Kerberos Authentication errors: Cannot find KDC for realm while getting initial credentials | kinit: configuration file does not specify default realm

Here are some errors i ran into while trying to setup Ansible for the second time in my test laboratory

Solution: These two errors here are based on a lot of factors

- My domain setup was not resolving correctly, when this is done,
- Ensure to include the realm with the principal you are logging in as
And remember to set the default realm also in the krb5.config file.

Concept: Authentication Services relies on DNS (Domain Naming Service) to locate the Key Distributions Center (KDC) which in AD is a domain controller, so if your DNS is not properly configured for your domain it will fail. To test the name resolution for your domain controller, simply use nslookup commands, see https://techdirectarchive.com/2020/03/21/cant-find-domain-non-existent-domain/

– Ensure the Ansible server is pointing to the DNS server Address or else this will not work correctly

Create an A record to ensure the domain name is resolvable via DNS.

nslookup techdirectarchive.local

Note: Alternatively, the name resolution requires the Ansible server to locate the domain controllers and can be manually added to the host file by including an entry in the /etc/hosts file for each domain controller, e.g

192.168.xxx.45 techdirectarchive.local

Next, After the krb5 configuration file has been updated correctly, you should be able to successfully authenticate and get a valid token. The following steps show how to authenticate and get a token:
– You may get the error “kinit: configuration file does not specify default realm” if you try to use the “kinit” command only, use the command below

kinit username@TECHDIRECT.LOCAL

As you can see above, the kinit command did not work correctly initially.

default_realm Identifies the default Kerberos realm for the client. Set its value to your Kerberos realm. If this value is not set, then a realm must be specified with every Kerberos principal when invoking programs such as kinit.

Other Possible tips to note.
– Ensure that the “krb5.conf” is correctly configured.
– The kinit command is fail for user authentication because kerberos is case sensitive.Here is the right syntax “kinit username@TECHDIRECT.LOCAL“. Ensure the domain name is in all CAPS, or else you will get an error.

What to note when settings up Ansible to work with Kerberos

Kerberos is reliant on a properly-configured environment to work. To troubleshoot Kerberos issues, ensure that the hostname set for the Windows host is the FQDN and not an IP address.

– The forward and reverse DNS lookups are working properly in the domain. To test this, ping the windows host by name and then use the ip address returned with nslookup. The same name should be returned when using nslookup on the IP address.

– The Ansible host’s clock is synchronized with the domain controller. Kerberos is time-sensitive, and a little clock drift can cause the ticket generation process to fail.

– Ensure that the fully qualified domain name for the domain is configured in the krb5.conf file. To check this, run:

kinit -C username@MY.DOMAIN.COM

And then klist to view the list all your active Kerberos tickets and their expiration dates.

klist

– If the domain name returned by klist is different from the one requested, an alias is being used. The krb5.conf file needs to be updated so that the fully qualified domain name is used and not an alias.

Ensure the Realms are written in CAPS because Kerberos is case sensitive, see the link below for more details.
https://techdirectarchive.com/2020/03/14/configuring-kerberos-for-ansible-authentication/

Various Ansible Authentication Options

It is worth knowing that proper configuration of the authentication is paramount to communicating properly with the target hosts. There are various authentication methods that are possible when authenticating with an account via a local or domain-based account.

Basic: 
Certifcate:
Keberous
NTLM
CredSSP

User authentication via Active Directory (AD), also referred to as authentication through Kerberos is my chosen option.

Why employ Kerberos?

Domain based authentication uses Kerberos authentication and this is supported by Windows Active Directory Services. Kerberos authentication with Windows Server environments uses the central user authentication that Active Directory supplies to configure and manage your Windows Server resources. This permits also greater level of trust for the WinRM connections to the remote Servers that are built in when using Active Directory credentials.

Note: Kerberos is the recommended authentication option to use when running in a domain environment. Kerberos supports features like credential delegation and message encryption over HTTP and is one of the more secure options that are available through WinRM.

Domain Authentication: Employs Kerberos authentication which is supported with Microsoft Active Directory Services and this allows the creation of a more reliable WinRM communication between the Ansible Server and Remote machines.

For steps needed to how to install Kerberos packages in Windows via Cygwin, see https://techdirectarchive.com/2020/03/14/kerberos-setup-in-windows-cygwin/

For steps to configuring Kerberos for Ansible Authentication, see https://techdirectarchive.com/2020/03/14/configuring-kerberos-for-ansible-authentication/

For more on Kerberos, see https://docs.ansible.com/ansible/latest/user_guide/windows_winrm.html#kerberos

For basic Ansible authentication, see the following link
http://bit.ly/2Qfx9Zn
https://www.virtualizationhowto.com/2018/04/basic-windows-server-automation-with-ansible/

Configuring Kerberos for Ansible Authentication

A Key Distribution Center (abbreviated KDC) is also known as the Trust Center in the Kerberos system, Kerberos server, issues an on-demand ID file(TGT) for logged-in users on request, which the user can use as an ID to protect their traffic.

The Ticket Granting Ticket (TGT) is a small file that provides access to a data exchange, similar to a password but more secure.

The TGT is considered more secure because it contains, in encrypted form, the client’s IP address, the lifetime of the TGT, and the previously generated session key, preventing a man-in-the-middle attack. The TGT is an essential part of the Kerberos system for data path backup.

The TGT is issued by the Key Distribution Center (KDC) for registered and designated (authenticated) users

This step is required for Kerberos to communicate with the domain effectively and this is achieved via the following path in my environment as shown below.

  • Modify the configuration files, krb5.conf to reflect the correct information, (such as domain-realm mappings to Kerberos servers’ names) for your realm.

Edit the file using any of your desired editors and populate it as follow C:\cygwin64\etc\crypto-policies\back-ends\krb5.config

Note: We will populate the file later, but for the initial test, this is absolutely ok.

Before making your first connection in any Cygwin session, you need to authenticate to the Kerberos service. In a Cygwin bash shell, type.

kinit <yourusername>

Here, you will be prompted to enter your password. After you must have successfully authenticated, you will have acquired a Kerberos ticket-granting ticket

Now, we have tested and it works, let’s configure the Host Kerberos in details as shown below. This is necessary because Kerberos is reliant on a properly-configured environment to work.

Note:
– Ensure to enter the realm name in capital letters and pay specific attention to how the file is written here https://docs.ansible.com/ansible-tower/latest/html/administration/kerberos_auth.html       

– The [realm] should include the FQDNs of your DCs’.
– The [domain_realm] Help map server hostname to Kerberos realm (This should include each domain that Ansible needs access to).
– The [libdefaults] should contain various settings used by Kerberos V5 library

Also see this link for more information https://web.mit.edu/kerberos/krb5-1.12/doc/admin/conf_files/krb5_conf.html

Ensure to save before quitting, by hitting the Esc key on your keyboard, followed by “:wq” in the test editor and then hit enter on your keyboard.

Below is how the file on the screenshot is layout (written).

[logging]
 default = FILE:/var/log/krb5libs.log
 kdc = FILE:/var/log/krb5kdc.log
 admin_server = FILE:/var/log/kadmind.log

[libdefaults]
 default_realm = TECHDIRECT.LOCAL
 dns_lookup_realm = false
 dns_lookup_kdc = false
 ticket_lifetime = 24h
 renew_lifetime = 7d
 forwardable = true

[realms]
 TECHDIRECT.LOCAL = {
  kdc = techdarchive.techdirect.local
  admin_server = techdarchive.techdirect.local
 }

[domain_realm]
 .techdirect.com = TECHDIRECT.LOCAL
 techdirect.com = TECHDIRECT.LOCAL 

Note (Information Only):  Most setup has it in this location /etc/krb5.conf.
– For seamless operation, both Kerberos and SSH can be configured. For me there was no need to configure Ansible to work with SSH, so this was ignored.

See the following links and the image above if you would like to perform SSH too
http://computing.help.inf.ed.ac.uk/kerberos-cygwin
http://nynim.org/blog/2012/08/25/using-kerberos-gssapi-auth-with-openssh-in-cygwin-on-windows/

Note: There are two types of Kerberos ticket management for Ansible. We will be using the manual Kerberos ticket management
– Automatic Kerberos Ticket management and
– Manual Kerberos Ticket Management.

Testing: Before making your first connection to a remote device in any Cygwin session, you need to authenticate to the Kerberos service by using your Kerberized credentials In a Cygwin bash shell, type: simply run kinit binary to acquire a new Kerberos ticket as shown below.

– kinit <yourusername@DOMAIN.COM>

Test using kinit, it will work correctly.
$ kinit user@TECHDIRECT.LOCAL
Password for user@TECHDIRECT.LOCAL:
$
- You can run “klist” to list all your active Kerberos tickets and their expiration dates.)

Note: To destroy all the tickets that have been acquired, use the following command:

$ kdestroy

For how to to setup Kerberos in Windows (Cygwin), see https://techdirectarchive.com/2020/03/14/kerberos-setup-in-windows-cygwin/

For similar kerberos errors, see https://techdirectarchive.com/2020/03/21/cannot-find-kdc-for-realm-while-getting-initial-credentials-kinit-configuration-file-does-not-specify-default-realm/

Error –“ kinit: Malformed representation of principal when parsing name”

This error is prompted when there is a mistake in the username when you wish to manually obtain a Kerberos ticket.

Solution: Run the command as shown below with your username and domain correctly.

kinit user@TEST.COM

The error will no longer be prompted. In my case I entered the @symbol twice 😉

Error – kinit: KDC reply did not match expectations while getting initial credentials

The following error is prompted when trying to initialize krb5 with AD as shown below. The issue was I had my realm in lower case and not all parameters were fully entered.

$ kinit user@test.com
Password for user@test.com:
kinit: KDC reply did not match expectations while getting initial credentials

Solution: Ensure your krb5 file is structured this way.
– The realm is in capital letters
– Access the krb5.config file via C:\cygwin64\etc\crypto-policies\back-ends.

[logging]
 default = FILE:/var/log/krb5libs.log
 kdc = FILE:/var/log/krb5kdc.log
 admin_server = FILE:/var/log/kadmind.log

[libdefaults]
 default_realm = TEST.COM
 dns_lookup_realm = false
 dns_lookup_kdc = false
 ticket_lifetime = 24h
 renew_lifetime = 7d
 forwardable = true

[realms]
 WEBSITE.COM = {
  kdc = myserver.test.com
  admin_server = myserver.test.com
 }

[domain_realm]
 .test.com = TEST.COM
 Test.com = TEST.COM 

Next, run the kinit command again with the domain name in upper case as shown below, the error will not be prompted and the user will be authenticated via Kerberos with AD.

For more information, see the following link.
https://docs.ansible.com/ansible/latest/user_guide/windows_winrm.html#kerberos

How to install and configure Ansible on Ubuntu

Ansible is a tool for managing a large number of servers centrally. With Ansible playbooks, you can create automated tasks to run against your servers remotely from a central control node. Installing Ansible on Ubuntu requires the following prerequisites and installation commands to be followed promptly.

1. Root privilege.

ansible@ansible-VirtualBox:~$ sudo -i
[sudo] password for ansible:
root@ansible-VirtualBox:~#

2. Install update packages and upgrades

root@ansible-VirtualBox:/# apt-get update && apt-get upgrade

3. One of those many ways to get Ansible running for Ubuntu is to install the project’s PPA (personal package archive), i.e the software-properties-common to your server.

root@ansible-VirtualBox:~# apt-get install software-properties-common
Reading package lists… Done
Building dependency tree
Reading state information… Done
software-properties-common is already the newest version (0.96.20.1).
software-properties-common set to manually installed.
0 upgraded, 0 newly installed, 0 to remove and 3 not upgraded.

4. After installing this package, add the Ansible PPA from the repository using this command.

root@ansible-VirtualBox:~# apt-add-repository ppa:ansible/ansible
Ansible is a radically simple IT automation platform that makes your applications and systems easier to deploy. Avoid writing scripts or custom code to deploy and update your applications‚ automate in a language that approaches plain English, using SSH, with no agents to install on remote systems.
http://ansible.com/
More info: https://launchpad.net/~ansible/+archive/ubuntu/ansible
Press [ENTER] to continue or ctrl-c to cancel adding it
gpg: keyring `/tmp/tmpooow5ue_/secring.gpg' created
gpg: keyring `/tmp/tmpooow5ue_/pubring.gpg' created
gpg: requesting key 7BB9C367 from hkp server keyserver.ubuntu.com
gpg: /tmp/tmpooow5ue_/trustdb.gpg: trustdb created
gpg: key 7BB9C367: public key "Launchpad PPA for Ansible, Inc." imported
gpg: Total number processed: 1
gpg: imported: 1  (RSA: 1)                                  
OK

Note: where you have to press enter to accept the PPA added process to continue.

5. Once again, run the apt-get update utility to have the system package refreshed to be aware of the newly added PPA.

root@ansible-VirtualBox:/# apt-get update

6. Now install Ansible software.

root@ansible-VirtualBox:/# apt-get install ansi
ansible ansible-fireball ansible-node-fireball ansiweather
root@ansible-VirtualBox:/# apt-get install ansible
Reading package lists… Done
Building dependency tree
Reading state information… Done
The following additional packages will be installed:
python-crypto python-ecdsa python-httplib2 python-jinja2 python-markupsafe python-paramiko
python-pkg-resources python-setuptools python-six python-yaml sshpass
Suggested packages:
python-crypto-dbg python-crypto-doc python-jinja2-doc python-setuptools-doc
The following NEW packages will be installed:
ansible python-crypto python-ecdsa python-httplib2 python-jinja2 python-markupsafe
python-paramiko python-pkg-resources python-setuptools python-six python-yaml sshpass
0 upgraded, 12 newly installed, 0 to remove and 3 not upgraded.
Need to get 2.182 kB of archives.
After this operation, 13,8 MB of additional disk space will be used.
Do you want to continue? [Y/n]

Note: Just hit enter of “y” key to continue with the installation.

7. SSH Key Setup: SSH keys allow communication (auth) between two hosts without the need for a password. SSH key authentication uses two keys, a private key, and a public key.

Ansible communicates with servers via SSH on Linux. While it certainly has the ability to handle password-based, SSH authentication keys help keep things simple. Simplay enter the following command to generate for you the keys using RSA algorithm

root@ansible-VirtualBox:~# ssh
Note: I used the tab key to display all available options
ssh               ssh-argv0         ssh-import-id     ssh-keygen        
ssh-add           ssh-copy-id       ssh-import-id-gh  ssh-keyscan       
ssh-agent         sshd              ssh-import-id-lp  sshpass   

Note: The step starts from here. During the process you will be prompted for a password, keep hitting the enter button to create the key.

root@ansible-VirtualBox:~# ssh-keygen -t rsa -K4096
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:3DLxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxE8yGA root@ansible-VirtualBox
The key's randomart image is:
+---[RSA 2048]----+
|    .oo..        |
|    .. o.o       |
|   . ..+*.       |
|    + .+==       |
|   . o .S.+ ..   |
|.Eo . =  +.o...  |
|.+ + = .  o=oB . |
|  o =      o@.=  |
|     .       =.  |
+----[SHA256]-----+

Note the following

  • Now you have the RSA key generated.
  • If you already have an SSH key pair, simply copy this to the specified path of the machine running Ansible (~/.ssh/id_rsa.pub)
  • By default, the public key is saved in the file ~/.ssh/id_rsa.pub,
  • while the private key is saved in ~/.ssh/id_rsa
  • Not important for this task but vital to add more users.

    Now copy the id_rsa.pub file to the remote host and append it to ~/.ssh/authorized_keys

    This step is necessary if you wish to have needed other users to administer ansible to connect to these machines.

    Now copy the id_rsa.pub file to the remote host and append it to ~/.ssh/authorized_keys using this command:
root@ansible-VirtualBox:~# ssh-copy-id username@remotehost

where remotehost is the IP address or FQDN of the server and username is the “root” or whatsoever name you have configured for the other machine.

Finally, double-check the permissions on the authorized_keys file, only the authenticated user should have read and write permissions. If the permissions are not correct to change them by.

Now assign permission: You should now be able to SSH to the host without being prompted for a password.

root@ansible-VirtualBox:~#chmod 600 .ssh/authorized_keys
To get (view) the public key run this command
root@ansible-VirtualBox:~# cat ~/.ssh/id_rsa.pub
ssh-rsa AAAAxxxxxxxxxxxxxxxxxxxxNxfz5NxxxxxxxxxxxRUVxxxxxxxx root@ansible-VirtualBox
root@ansible-VirtualBox:~#
To view where your hidden files are kept you use
root@ansible-VirtualBox:/etc# cd ~/.ssh
root@ansible-VirtualBox:~/.ssh# ls
id_rsa id_rsa.pub

To view the private keys, use any of the editor

root@ansible-VirtualBox:~/.ssh# vi id_rsa
-----BEGIN RSA PRIVATE KEY-----
MIIEpgIBAAKCAQEAyUZdGaqwHoxPb9XVXDN142xvrJN9gvKp6VSqZoIpEYw8LTlExxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx6w860wHVDdx/n1tDJqVjM0DJxGJwMJoC9QHejiSjJnz/SJkPB22/BDbuY4/l9tFYWSs+Ih9aekcRwg9KNtRelC8r5j0Vt1Qt3X4GB0q9qZ+hZC5EK219
-----END RSA PRIVATE KEY-----

Note: Make sure that you include the first and last marker lines. They are required in order for the key file to be valid and change permission if necessary.

To view the public key, run the same command but with the .pub

root@ansible-VirtualBox:~/.ssh# vi id_rsa.pub

8. Configuring Ansible Hosts:
As we all know, Ansible keeps track of all the servers it knows via a file called “Hosts” file. This needs to be configured (entered) before it will be able to communicate with other computers.
this is how it looks.

root@ansible-VirtualBox:/etc/ansible# ls
ansible.cfg hosts roles
root@ansible-VirtualBox:/etc/ansible# vi hosts
# This is the default ansible 'hosts' file.
#
# It should live in /etc/ansible/hosts
#
#   - Comments begin with the '#' character
#   - Blank lines are ignored
#   - Groups of hosts are delimited by [header] elements
#   - You can enter hostnames or ip addresses
#   - A hostname/ip can be a member of multiple groups
# Ex 1: Ungrouped hosts, specify before any group headers.
## green.example.com
## blue.example.com
## 192.168.100.1
## 192.168.100.10
# Ex 2: A collection of hosts belonging to the 'webservers' group
## [webservers]
## alpha.example.org
## beta.example.org
## 192.168.1.100
## 192.168.1.110
# If you have multiple hosts following a pattern you can specify
# them like this:
## www[001:006].example.com
# Ex 3: A collection of database servers in the 'dbservers' group
## [dbservers]
##
## db01.intranet.mydomain.net
## db02.intranet.mydomain.net
## 10.25.1.56
## 10.25.1.57
# Here's another example of host ranges, this time there are no
# leading 0s:
## db-[99:101]-node.example.co

Note: These are just configuration examples. None will work except they are commented out, i.e, by deleting the # symbol and adding the right devices (IP address) are added.

To do this, you have to uncomment them out and enter the idea ip address(es).

[monitoringservers]
alpha.example.org
beta.example.org
192.168.1.100
192.168.1.110
192.168.177.13

Note the following steps below.
Method 1: On how to copy the SSH public key to remote servers
Step 1: Always remember to copy the ssh public key from the ansible machine to the server you wish to manage.

root@ansible-VirtualBox:/etc# cd ~/.ssh
root@ansible-VirtualBox:~/.ssh# ls
id_rsa id_rsa.pub known_hosts
root@ansible-VirtualBox:~/.ssh# vim id_rsa.pub

And copy the key!!!

Step 2: Create a file called authorized_keys on the target machine you wish to use ansible on. the in the authorized_keys create copy and paste the public of the Ansible server in it.

Note: This has to be in the home directory of the ssh user which is without the bracket (~/.ssh). it should look this way.

Webserver@nagios:~$ vi .ssh/authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJRl0Zq

Note 2: http://docs.ansible.com/ansible/apt_module.html
Also, install the python-apt library on the target machine you wish to administer Ansible on, e.g., my Nagios server

Webserver@nagios:~$ sudo apt-get install python-apt
Webserver@nagios:~$ sudo apt-get install aptitude

or use a single command (any of these)

- Webserver@nagios:~$ sudo apt-get install python-apt; apt-get install aptitude
- Webserver@nagios:~$ sudo apt-get install python-apt && apt-get install aptitude
- Webserver@nagios:~$ sudo apt-get install python-apt || apt-get install aptitude

Note: Without this, it will fail if we want to do some package installations.

Method 2: This method seems to be the easiest way to copy the ssh public key from the ansible machine to the server you wish to manage.

root@ansible-VirtualBox:~/.ssh# ssh
ssh               ssh-argv0         ssh-import-id     ssh-keygen        
ssh-add           ssh-copy-id       ssh-import-id-gh  ssh-keyscan       
ssh-agent         sshd              ssh-import-id-lp  sshpass    
root@ansible-VirtualBox:~/.ssh# ssh-copy-id webserver@192.168.xxx.1x
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
webserver@192.168.xxx.xx's password:
Number of key(s) added: 1

Now try logging into the machine, with: “ssh ‘webserver@192.168.xxx.1x'”
and check to make sure that only the key(s) you wanted were added.

Note: This method has copied the public key to the server you wish to administer. now to see the key on the other server use this command

webserver@nagios:~$ cat .ssh/authorized_keys
ssh-rsa yc2EAAxxxxxxxxxxxxxxxxxxxxxxxDJRl0Zqrxxxxxxxxxxxxxxxxxxxxxxxx
root@ansible-VirtualBox:~/.ssh#

9. After defining your host file in step 8 above

root@ansible-VirtualBox:/etc/ansible# vi hosts
[monitoringservers]
## alpha.example.org
## beta.example.org
## 192.168.1.100
## 192.168.1.110
192.168.177.13 ansible_user=webserver

Let’s do some ping testing

Example 1:
root@ansible-VirtualBox:/etc/ansible# ansible monitoringservers -m ping

This will display the number of servers with IP address it successfully pinged.
192.168.xxx.1x | SUCCESS => {
"changed": false,
"ping": "pong"
}

Note: It shows it is using the module called ping, signified by -m to ping all the available servers in the group of monitoring servers.

root@ansible-VirtualBox:/etc/ansible# ansible -i hosts monitoringservers -m ping
192.168.xxx.1x | SUCCESS => {
"changed": false,
"ping": "pong

Here, a hosts is added and it delivers the same output.

Useful link
https://www.linode.com/docs/applications/ansible/getting-started-with-ansible