Kerberos Keytab Rotation

How to rotate or re-retrieve service keytabs safely, what KVNO means, and how to avoid breaking every host that shares a principal.

The dangerous bit
  • Plain ipa-getkeytab changes the service key and bumps the KVNO. Anything still using the old keytab will start failing.
  • ipa-getkeytab -r retrieves the currently active key without creating a new one. Use it when rebuilding a host or syncing peers to an already-rotated key.
  • Back up the current keytab, inventory every consumer of the principal, and decide the rollback point before you touch the KDC.
  • After a change, verify with klist -k, a real kinit -kt, and a service restart. "File copied" is not proof.
On this page
  1. When to rotate vs when to re-retrieve
  2. KVNO and what actually changes
  3. Safe rotation workflow
  4. Using ipa-getkeytab -r correctly
  5. Verification after the change
  6. Service restarts and rollout order
  7. Failure modes and troubleshooting

When to rotate vs when to re-retrieve

Two similar operations get confused constantly:

If multiple servers share one service principal, a true rotation is a coordinated change, not a host-local housekeeping task. That is why operators often prefer one principal per host or per VIP where possible; see Kerberos for the identity model and Apache GSSAPI SSO for a common service-principal use case.

KVNO and what actually changes

Every service key has a Key Version Number (KVNO). The KDC encrypts service tickets to the latest key version. The service must have a matching keytab entry or it cannot decrypt the ticket.

KDC current key:      HTTP/app01.example.com  KVNO 7
Service keytab file: HTTP/app01.example.com  KVNO 6
Result: service ticket decrypt fails

The usual symptoms are:

Useful checks:

# What KVNO is in the local keytab?
klist -k /etc/httpd/conf/httpd.keytab

# What KVNO does the KDC hand out for this principal?
kvno HTTP/app01.example.com@EXAMPLE.COM
Interpretation: if the KDC reports KVNO 7 and the service keytab only has KVNO 6, the file is stale. If the file has KVNO 7 but the service still fails, the process probably has not reloaded the keytab or is using the wrong path.

Safe rotation workflow

Use a change window if the principal is shared or fronted by a load balancer. True rotation invalidates the old key for new tickets.

1. Inventory the consumers and back up the current file

# Which service principal are we rotating?
ipa service-show HTTP/app01.example.com

# What is in the current keytab?
klist -k /etc/httpd/conf/httpd.keytab

# Take a local backup before replacing anything
cp -a /etc/httpd/conf/httpd.keytab /root/httpd.keytab.pre-rotation

2. Rotate the key once on the authority host

This step creates a new key and bumps the KVNO. Use a temporary file first so you can test before replacing the live one.

ipa-getkeytab -s ipa01.example.com \
  -p HTTP/app01.example.com \
  -k /etc/httpd/conf/httpd.keytab.new

# Test the new file directly
kinit -kt /etc/httpd/conf/httpd.keytab.new HTTP/app01.example.com@EXAMPLE.COM
kdestroy

# Replace the live file only after the test passes
mv /etc/httpd/conf/httpd.keytab.new /etc/httpd/conf/httpd.keytab
chown root:apache /etc/httpd/conf/httpd.keytab
chmod 0640 /etc/httpd/conf/httpd.keytab

3. Sync every additional consumer to the new active key

Once the KDC has the new key, every peer that shares the same principal must retrieve that active key with -r.

ipa-getkeytab -r -s ipa01.example.com \
  -p HTTP/app01.example.com \
  -k /etc/httpd/conf/httpd.keytab

klist -k /etc/httpd/conf/httpd.keytab

4. Restart the services in a controlled order

Do not assume the daemon rereads the keytab automatically. Restart or reload the service explicitly and validate each node before moving on.

Shared-principal warning: if five nodes share HTTP/app01.example.com and only three have been updated, traffic pinned to the stale two can still fail intermittently. Treat this like a rolling deploy, not a file copy.

Using ipa-getkeytab -r correctly

-r means "retrieve existing key". It is the safe option when you want the current KDC key but do not want to change it again.

# Good: re-create a missing keytab on a rebuilt host
ipa-getkeytab -r -s ipa01.example.com \
  -p host/web01.example.com \
  -k /etc/krb5.keytab

# Good: sync another node after a planned rotation already happened
ipa-getkeytab -r -s ipa01.example.com \
  -p HTTP/app01.example.com \
  -k /etc/httpd/conf/httpd.keytab

# Dangerous: running this casually rotates the service key again
ipa-getkeytab -s ipa01.example.com \
  -p HTTP/app01.example.com \
  -k /etc/httpd/conf/httpd.keytab

Operational rule:

This is especially important when a service principal is used by web nodes, proxies, or batch jobs in more than one place. For IPA replica health during a large rollout, check FreeIPA Replication.

Verification after the change

Verify in three layers: file contents, Kerberos auth, and real service behavior.

# 1. Confirm the principal and KVNO are present
klist -k /etc/httpd/conf/httpd.keytab

# 2. Confirm the keytab can obtain credentials
kinit -kt /etc/httpd/conf/httpd.keytab HTTP/app01.example.com@EXAMPLE.COM
klist
kdestroy

# 3. Ask the KDC for a service ticket and compare KVNO
kinit alice@EXAMPLE.COM
kvno HTTP/app01.example.com@EXAMPLE.COM
kdestroy

If the principal is a host keytab, the same pattern applies:

klist -k /etc/krb5.keytab
kinit -kt /etc/krb5.keytab host/web01.example.com@EXAMPLE.COM
kdestroy
Remember: a successful kinit -kt proves the file is valid. It does not prove the service is reading that same file or that the principal name matches the hostname clients actually request.

Service restarts and rollout order

Typical restarts after keytab work:

# Apache with GSSAPI
systemctl restart httpd

# SSSD and SSH after host keytab work
systemctl restart sssd
systemctl restart sshd

# Java app using JAAS / krb5 login modules
systemctl restart myapp

A sane rollout order:

  1. Drain one node from traffic.
  2. Update keytab and restart the service.
  3. Run kinit -kt and one real application test.
  4. Return the node to traffic.
  5. Repeat for the next node.

If authentication failures include clock skew errors, stop and check Chrony before blaming the rotation.

Failure modes and troubleshooting

SymptomLikely causeWhat to do
KRB_AP_ERR_BADKEYVER or decrypt failures Service still has an old KVNO. Retrieve the active keytab, verify with klist -k, then restart the service.
One node works, another fails Shared principal was not updated on every node. Inventory all consumers, sync them with ipa-getkeytab -r, and roll restarts cleanly.
Key table entry not found Wrong principal name or wrong file path. Check the exact FQDN-based principal and the service's configured keytab path.
Permission denied reading the keytab Ownership, mode, or SELinux context is wrong. Fix owner and permissions; if needed run restorecon and re-test.
Clock skew too great Time drift between client, service host, and KDC. Check chronyc tracking on both sides and fix NTP first.
SSO still fails after file replacement Daemon has not reloaded the keytab, or the app caches credentials. Restart the service and confirm the process is using the expected file path.
Rotation succeeded, but clients still fail intermittently Load balancer is sending traffic to stale nodes. Drain and validate each backend one by one instead of replacing files in parallel and hoping.

Related pages: Kerberos, FreeIPA HBAC & Sudo, FreeIPA Replication, Apache GSSAPI SSO, Chrony.