Kerberos Deep Dive

How Kerberos authentication works, the ticket lifecycle, kinit/klist/kdestroy, and fixing common Kerberos errors.

What Kerberos is

Kerberos is a network authentication protocol that uses time-limited cryptographic tickets instead of passwords. Once you have a ticket, you can prove your identity to any service in the same realm without re-entering your password.

FreeIPA uses Kerberos as its authentication backend. When you log in to a FreeIPA-enrolled system with your IPA credentials, you are authenticating via Kerberos. When you run ipa CLI commands, they use your Kerberos ticket.

Key concepts: realm, principal, KDC, TGT, ticket

Realm

The Kerberos domain — typically the uppercase DNS domain name. For example.com the realm is EXAMPLE.COM. FreeIPA creates a realm matching the domain.

Principal

An identity in the Kerberos system. Users: alice@EXAMPLE.COM. Services: host/web01.example.com@EXAMPLE.COM, HTTP/web01.example.com@EXAMPLE.COM.

KDC (Key Distribution Center)

The trusted server that issues tickets. In FreeIPA, the IPA server is the KDC.

TGT (Ticket Granting Ticket)

The first ticket you get when you authenticate. It proves your identity to the KDC and lets you request service tickets without re-entering your password.

Service ticket

A ticket for a specific service (SSH, LDAP, HTTP). The KDC issues these in exchange for a TGT.

The authentication flow

What happens when Alice runs ssh web01.example.com:

1. Alice runs kinit alice@EXAMPLE.COM (usually happens automatically at login)
2. Her workstation sends a request to the KDC: "I am alice, I want a TGT"
3. The KDC verifies alice's password (without sending it over the network) and issues a TGT
4. The TGT is stored in Alice's credential cache (/tmp/krb5cc_1000)
5. Alice runs ssh web01 — ssh sees she has a TGT and requests a service ticket for host/web01.example.com
6. The KDC issues the service ticket, encrypted with web01's key
7. SSH presents the service ticket to web01 — web01 decrypts it using its keytab
8. web01 verifies the ticket is valid and Alice is allowed access (via HBAC)
9. SSH session starts — no password was entered or transmitted

kinit — get a ticket

# Get a TGT for your user
kinit username@EXAMPLE.COM

# With explicit realm (if /etc/krb5.conf has the default realm set, you can omit it)
kinit username

# Get a ticket from a keytab (for automated services)
kinit -kt /etc/myapp.keytab myapp/host.example.com@EXAMPLE.COM

# Renew a ticket before it expires
kinit -R

klist — view tickets

klist

# Example output:
Ticket cache: KEYRING:persistent:1000:1000
Default principal: alice@EXAMPLE.COM

Valid starting       Expires              Service principal
04/10/2024 09:00:00  04/10/2024 19:00:00  krbtgt/EXAMPLE.COM@EXAMPLE.COM
  renew until 04/17/2024 09:00:00
04/10/2024 09:05:00  04/10/2024 19:00:00  host/web01.example.com@EXAMPLE.COM

• The first entry is the TGT (krbtgt/EXAMPLE.COM)
• Subsequent entries are service tickets acquired during the session
• Valid starting / Expires — the ticket is only usable in this window (typically 10 hours)
• renew until — you can renew without re-authenticating up to this time (typically 7 days)

# List keytab principals
klist -kt /etc/krb5.keytab

kdestroy — delete tickets

# Delete all tickets in the default cache (logout)
kdestroy

# Delete a specific credential cache
kdestroy -c /tmp/krb5cc_1000

After kdestroy, SSO stops working — you need to kinit again. Tickets are also automatically invalidated after their expiry time.

Keytabs — passwordless authentication for services

A keytab is a file containing a pre-shared secret for a service principal. It allows services (like SSH, Apache, or custom apps) to authenticate to the KDC without a human-entered password — used for automated and service-to-service authentication.

# Create a service principal in FreeIPA and retrieve its keytab
ipa service-add HTTP/web01.example.com
# IMPORTANT: use the FQDN — the principal must exactly match the hostname
# "web01" and "web01.example.com" are different principals
ipa-getkeytab -s ipa01.example.com -p HTTP/web01.example.com -k /etc/httpd/conf/httpd.keytab

# Verify the keytab works
kinit -kt /etc/httpd/conf/httpd.keytab HTTP/web01.example.com@EXAMPLE.COM
klist

# The host keytab at /etc/krb5.keytab is created during ipa-client-install
# and is used by SSH and SSSD for host authentication
klist -kt /etc/krb5.keytab

/etc/krb5.conf

The Kerberos client configuration. FreeIPA enrollment sets this up automatically.

[libdefaults]
default_realm = EXAMPLE.COM
dns_lookup_realm = false
dns_lookup_kdc = true
rdns = false
ticket_lifetime = 24h
forwardable = true

[realms]
EXAMPLE.COM = {
    kdc = ipa01.example.com
    master_kdc = ipa01.example.com
    admin_server = ipa01.example.com
}

[domain_realm]
.example.com = EXAMPLE.COM
example.com = EXAMPLE.COM

The clock requirement

Kerberos is extremely sensitive to clock skew. If the difference between client and KDC clocks is more than 5 minutes, authentication fails with:

kinit: Clock skew too great while getting initial credentials

This is why Chrony (NTP) is considered a dependency for any host joining a Kerberos realm. If you see clock skew errors:

# Check chrony sync status
chronyc tracking

# Force an immediate time sync
chronyc makestep

# Check the IPA server's time
ssh admin@ipa01.example.com 'date'

Common errors and fixes

kinit: Cannot contact any KDC for realm

# Check DNS resolution of the KDC
dig +short ipa01.example.com

# Check connectivity to the KDC
nc -zv ipa01.example.com 88    # Kerberos port

# Check /etc/krb5.conf has the right KDC
cat /etc/krb5.conf | grep kdc

kinit: Clients credentials have been revoked

The account is locked or disabled in FreeIPA. Check: ipa user-show username or the IPA web UI.

SSH: Server's host principal not found in keytab

The host's keytab is stale or missing. Re-enroll or re-retrieve the keytab:

# On the host
ipa-getkeytab -s ipa01.example.com -p host/$(hostname) -k /etc/krb5.keytab

# Then restart SSSD and SSH
systemctl restart sssd sshd

Ticket expired during a long operation

# Renew before it expires
kinit -R

# If already expired, re-authenticate
kinit username@EXAMPLE.COM

Credential cache types

The credential cache is where your TGT is stored after kinit. Different environments use different storage backends.

# Check which cache type is in use
klist -l             # list all caches (KCM / file)
echo $KRB5CCNAME     # override variable if set

# Check the configured default
grep default_ccache_name /etc/krb5.conf

# Force a specific cache type (useful when container can't reach KCM daemon)
KRB5CCNAME=FILE:/tmp/my-cache kinit username@EXAMPLE.COM

In containers you may see errors like "Can't contact KDC" or "No credential cache" when the KCM socket is not mounted. Setting KRB5CCNAME=FILE:/tmp/krb5cc forces file-based caching which works without the SSSD daemon.

KVNO and keytab rotation

Every keytab entry carries a Key Version Number (KVNO). When a service principal's password/key is rotated in the KDC, the KVNO increments — any keytab still holding the old KVNO will stop decrypting service tickets and you'll see KRB_AP_ERR_BADKEYVER or Decrypt integrity check failed on the server.

# Show the current KVNO for a principal (queries the KDC)
kvno HTTP/web01.example.com@EXAMPLE.COM

# Rotate the key and append the new KVNO to the existing keytab
ipa-getkeytab -r -s ipa01.example.com \
  -p HTTP/web01.example.com \
  -k /etc/httpd/conf/httpd.keytab

# Confirm both the old and new KVNO are present (grace period)
klist -kt /etc/httpd/conf/httpd.keytab

-r ("retrieve existing key") is the safe mode — it pulls the currently-active key without triggering a new random key, so other replicas and existing tickets stay valid. Without -r, ipa-getkeytab generates a brand-new key, invalidating every other keytab holding that principal.

For the full rotation procedure (schedules, rollback, multi-host SPNs), see Keytab Rotation →.

SPN duplication pitfalls

# FreeIPA: look for duplicate or stale service principals
ipa service-find HTTP/web01.example.com
ipa host-find web01.example.com

# AD: search for duplicate SPNs across the forest
setspn -X                                  # list all duplicates forest-wide
setspn -Q HTTP/web01.example.com           # who owns this exact SPN?

# Clean up: remove the stale registration, then re-issue the keytab
ipa service-del HTTP/web01.example.com     # only if confirmed stale
ipa service-add HTTP/web01.example.com
ipa-getkeytab -s ipa01.example.com -p HTTP/web01.example.com -k /etc/httpd/conf/httpd.keytab

Always delete the old host/service entry before re-enrolling a reused hostname. Re-running ipa-client-install on a host that already has a live host/ principal without --force-join will quietly leave both entries present — a classic cause of intermittent SSO failures weeks later.

Debugging with KRB5_TRACE

When kinit fails with a cryptic error, set KRB5_TRACE to see every step of the Kerberos exchange — which KDC was contacted, what was sent and received, and exactly where it failed.

# Print trace to stderr
KRB5_TRACE=/dev/stderr kinit username@EXAMPLE.COM

# Save trace to a file
KRB5_TRACE=/tmp/krb5_trace.log kinit username@EXAMPLE.COM 2>&1
cat /tmp/krb5_trace.log

# Trace SSSD Kerberos operations (restart after log level change)
# In /etc/sssd/sssd.conf:
# [domain/example.com]
# debug_level = 7    # 7-10 for Kerberos detail
systemctl restart sssd
journalctl -u sssd --since "1 min ago" -f

KRB5_TRACE output shows the exact error from the KDC (e.g. "KDC_ERR_PREAUTH_FAILED" = wrong password, "KDC_ERR_C_PRINCIPAL_UNKNOWN" = user does not exist, "KRB_AP_ERR_SKEW" = clock skew too large). Reading the trace code name is usually enough to identify the fix.