gNSI Credentialz — SSH Credential Rotation on Cisco IOS XR

The Problem

Certz handles TLS for the gRPC Management Plane, but most network devices also have SSH access — console, VTY, interactive troubleshooting. Rotating SSH Credentials (Passwords, Authorized Keys, Host Keys, Certificates) traditionally means manual file edits, config pushes, or platform-specific CLIs. There is no standard API to rotate SSH material across vendors, no built-in Rollback if something goes wrong, and no way to programmatically restrict which Authentication methods the SSH server allows. If a Host Key rotation fails, operators risk locking themselves out.

What is Credentialz

gNSI Credentialz solves this with two streaming RPCs that cover account-level and host-level SSH Credentials:

• RotateAccountCredentials — rotate Passwords, Authorized Keys, and Authorized Principals for user accounts
• RotateHostParameters — rotate Host Keys, Host Certificates, CA Public Keys, allowed Authentication methods, and AuthorizedPrincipalsCommand settings

Plus two unary RPCs for preflight:

• CanGenerateKey — check if the device can generate a Key Pair
• GetPublicKeys — retrieve the current Public Keys from the host

Both streaming RPCs follow the same stage-validate-finalize pattern as Certz Rotate: send the new material, validate it works (e.g., try SSH with the new Key), then finalize. Cancel the stream before finalize to Rollback.

Cisco IOS XR supports Credentialz from 24.2.1 onward. In this article, every command was validated against NCS-540 25.3.1. The OpenConfig contract is credentialz.proto.

Within AAA, Credentialz is the Authentication plane for SSH — it manages the host keys, authorized keys, and account passwords that gate SSH login. (Its sibling Certz plays the equivalent role for the gRPC plane.) Only one Credentialz RPC can be in progress at a time.

---

The four RPCs

RPC	Type	What it does
RotateAccountCredentials	Bidirectional stream	Rotate Passwords, Authorized Keys, Authorized Principals for user accounts. Finalize to commit; cancel to Rollback.
RotateHostParameters	Bidirectional stream	Rotate Host Keys (external or generated), Host Certificates, CA Public Keys, allowed Authentication types, AuthorizedPrincipalsCommand. Finalize to commit; cancel to Rollback.
CanGenerateKey	Unary	Check whether the device can generate a Key Pair for a given algorithm (RSA 2048/4096, ECDSA P-256/P-521, Ed25519).
GetPublicKeys	Unary	Retrieve the current Public Keys configured on the host.

Supported Key algorithms (Cisco documentation)

Algorithm	Key types
RSA	2048, 4096
ECDSA	P-256, P-521
Ed25519	—

The OpenConfig KeyGen enum also defines KEY_GEN_SSH_KEY_TYPE_RSA_3072 and KEY_GEN_SSH_KEY_TYPE_ECDSA_P_384. On IOS XR 25.3.1 (NCS-540) the device rejects both with Code: InvalidArgument, message Can Generate Key:Invaid KeyGen input (typo on the device). Pinning to the five algorithms above is required.

---

IOS XR Support

Release	What changed
24.2.1	Credentialz introduced — RotateAccountCredentials, RotateHostParameters, CanGenerateKey, GetPublicKeys. Account and Host SSH Credential Rotation.
24.2.11	Enhanced real-time Rotation (without reboot), account- and host-specific updates.

Prerequisites:

• Configure a user account on the router
• Configure SSH version 2

---

IOS XR CLI Reference

Configuration command	What it does
ssh server v2	Enable SSH version 2 (required for Credentialz)
username <name> group <group>	Create a user account for SSH access
crypto key generate ed25519	Generate Ed25519 Host Keys (alternative to Credentialz GenerateKeysRequest)
crypto key generate rsa <bits>	Generate RSA Host Keys
crypto key generate ecdsa nistp256	Generate ECDSA P-256 Host Keys

Show / exec command	What it shows
show ssh	Active SSH sessions
show ssh server	SSH server configuration and status
show crypto key mypubkey ed25519	Ed25519 Host Public Key
show crypto key mypubkey rsa	RSA Host Public Key
show crypto key mypubkey ecdsa	ECDSA Host Public Key
show gnsi state	gNSI subsystem status (Credentialz, RDSFS state)
show tech-support gnsi	Full gNSI diagnostic bundle (all services)

---

RotateAccountCredentials — what it can rotate

Message	What it does
PasswordRequest	Change the Password for a user account (plaintext or hashed). Version and created_on timestamp included.
AuthorizedKeysRequest	Replace the Authorized Keys list for password-less SSH. Supported key types: RSA 2048/4096, ECDSA P-256/P-521, Ed25519.
AuthorizedUsersRequest	Set Authorized Principals for Certificate-based SSH Authentication. Uses AuthorizedPrincipalsFile semantics (pairs of account names and principal names).
FinalizeRequest	Commit the rotation. Old Credentials are invalidated.

The stream follows: send one or more credential updates → validate externally (try SSH with the new material) → finalize or cancel.

RotateHostParameters — what it can rotate

Message	What it does
CaPublicKeyRequest	Replace the CA Public Key used to verify client SSH Certificates.
ServerKeysRequest (external)	Upload externally generated Host Keys and/or Host Certificates. Private Key and Certificate are base64-encoded.
GenerateKeysRequest	Ask the device to generate new Host Keys. Public Key is returned in the response for Certificate signing.
AllowedAuthenticationRequest	Restrict which SSH Authentication methods are globally allowed (password, pubkey, keyboard-interactive, certificate). Default: all methods.
AuthorizedPrincipalCheckRequest	Set AuthorizedPrincipalsCommand — TOOL_HIBA_DEFAULT selects HIBA binary for dynamic principal Authorization.
FinalizeRequest	Commit. If validation fails before finalize, cancel the stream to revert to old Host parameters.

---

Hands-on: End-to-End Credentialz

Prerequisites

• IOS XR router with gRPC and SSH v2 enabled (24.2.1 or later for Credentialz)
• grpcurl and ssh-keygen installed
• OpenConfig gNSI protos cloned: git clone https://github.com/openconfig/gnsi.git
• A user account configured on the router

Environment (workstation)

# Stage the proto so grpcurl's -import-path / -proto resolves cleanly
git clone https://github.com/openconfig/gnsi.git /tmp/gnsi
mkdir -p./credz-import/credentialz
cp /tmp/gnsi/credentialz/credentialz.proto./credz-import/credentialz/
export GNSI_IMPORT="${GNSI_IMPORT:-$PWD/credz-import}"
export XR="${XR:?set to <router-ip>:57400 — required}"
export XR_USER="${XR_USER:-cisco}"
export XR_PASS="${XR_PASS:?set XR_PASS — required (Credentialz needs valid AAA on every call)}"
GRPC_MD=(-H "username:${XR_USER}" -H "password:${XR_PASS}")
PROTO_CREDZ="credentialz/credentialz.proto"

credentialz.proto has no import statements, so the -import-path only needs the one credentialz/credentialz.proto file under it. There is no need to clone openconfig/gnmi.

Lab only: All examples below use -plaintext (no TLS). In production, replace -plaintext with -cacert, -cert, and -key flags. See the Certz article for TLS setup.

Router setup (IOS XR CLI)

Ensure SSH v2 and a user account exist:

configure
 ssh server v2
 username lab
 group root-lr
 group cisco-support
 commit
end

List Services (Reflection)

grpcurl -plaintext \
 -import-path "$GNSI_IMPORT" \
 -proto "$PROTO_CREDZ" \
 "${GRPC_MD[@]}" \
 "${XR}" \
 list gnsi.credentialz.v1.Credentialz

Expected output:

gnsi.credentialz.v1.Credentialz.CanGenerateKey
gnsi.credentialz.v1.Credentialz.GetPublicKeys
gnsi.credentialz.v1.Credentialz.RotateAccountCredentials
gnsi.credentialz.v1.Credentialz.RotateHostParameters

CanGenerateKey (preflight)

Check if the device can generate Key Pairs for each algorithm:

grpcurl -plaintext \
 -import-path "$GNSI_IMPORT" \
 -proto "$PROTO_CREDZ" \
 "${GRPC_MD[@]}" \
 -d '{"keyParams": "KEY_GEN_SSH_KEY_TYPE_EDDSA_ED25519"}' \
 "${XR}" \
 gnsi.credentialz.v1.Credentialz.CanGenerateKey

Router response:

{
 "canGenerate": true
}

Repeat for other algorithms — all return canGenerate: true on Cisco 8000:

# RSA 4096
grpcurl -plaintext \
 -import-path "$GNSI_IMPORT" -proto "$PROTO_CREDZ" \
 "${GRPC_MD[@]}" \
 -d '{"keyParams": "KEY_GEN_SSH_KEY_TYPE_RSA_4096"}' \
 "${XR}" gnsi.credentialz.v1.Credentialz.CanGenerateKey
# RSA 2048
grpcurl -plaintext \
 -import-path "$GNSI_IMPORT" -proto "$PROTO_CREDZ" \
 "${GRPC_MD[@]}" \
 -d '{"keyParams": "KEY_GEN_SSH_KEY_TYPE_RSA_2048"}' \
 "${XR}" gnsi.credentialz.v1.Credentialz.CanGenerateKey
# ECDSA P-256
grpcurl -plaintext \
 -import-path "$GNSI_IMPORT" -proto "$PROTO_CREDZ" \
 "${GRPC_MD[@]}" \
 -d '{"keyParams": "KEY_GEN_SSH_KEY_TYPE_ECDSA_P_256"}' \
 "${XR}" gnsi.credentialz.v1.Credentialz.CanGenerateKey
# ECDSA P-521
grpcurl -plaintext \
 -import-path "$GNSI_IMPORT" -proto "$PROTO_CREDZ" \
 "${GRPC_MD[@]}" \
 -d '{"keyParams": "KEY_GEN_SSH_KEY_TYPE_ECDSA_P_521"}' \
 "${XR}" gnsi.credentialz.v1.Credentialz.CanGenerateKey

GetPublicKeys

Retrieve the current Host Public Keys:

grpcurl -plaintext \
 -import-path "$GNSI_IMPORT" \
 -proto "$PROTO_CREDZ" \
 "${GRPC_MD[@]}" \
 -d '{}' \
 "${XR}" \
 gnsi.credentialz.v1.Credentialz.GetPublicKeys

Response (trimmed — one entry per key type, as observed on a default NCS-540 25.3.1):

{
 "publicKeys": [
 { "publicKey": "c3NoLXJzYSBBQUFBQjNOemFD...", "keyType": "KEY_TYPE_RSA_2048" },
 { "publicKey": "c3NoLWRzcyBBQUFBQjNOemFD..." },
 { "publicKey": "ZWNkc2Etc2hhMi1uaXN0cDUy...", "keyType": "KEY_TYPE_ECDSA_P_521" },
 { "publicKey": "ZWNkc2Etc2hhMi1uaXN0cDM4..." },
 { "publicKey": "ZWNkc2Etc2hhMi1uaXN0cDI1...", "keyType": "KEY_TYPE_ECDSA_P_256" },
 { "publicKey": "c3NoLWVkMjU1MTkgQUFBQUMz...", "keyType": "KEY_TYPE_ED25519" }
 ]
}

Each publicKey is base64-encoded OpenSSH format. Decode with echo "<base64>" | base64 -d to see the raw SSH Public Key.

On 25.3.1: the default response includes a DSA (ssh-dss) and an ECDSA P-384 host key alongside the usual four. Both come back with keyType omitted — DSA isn’t modeled by OpenConfig Credentialz, and P-384 is defined in the proto but not populated by the device. If your security baseline disallows ssh-dss, remove it via a multi-algo RotateHostParameters: generateKeys call (see GenerateKeys is destructive).

RotateAccountCredentials: rotate Authorized Keys

Generate a test SSH Key Pair and push the Public Key as an Authorized Key for a non-admin user account (using the same cisco user you authenticate gRPC with risks locking yourself out of every flow that follows). The authorizedKey field must be base64-encoded:

ssh-keygen -t ed25519 -f /tmp/credz-test-key -N "" -C "credz-test@lab"
PUBKEY_B64=$(base64 < /tmp/credz-test-key.pub | tr -d '\n')
CREATED_ON=$(date +%s)
grpcurl -plaintext \
 -import-path "$GNSI_IMPORT" \
 -proto "$PROTO_CREDZ" \
 "${GRPC_MD[@]}" \
 -d '@' \
 "${XR}" \
 gnsi.credentialz.v1.Credentialz.RotateAccountCredentials <<EOF
{"credential":{"credentials":[{"account":"mike","authorizedKeys":[{"authorizedKey":"${PUBKEY_B64}","keyType":"KEY_TYPE_ED25519"}],"version":"keys-mike-v1","createdOn":"${CREATED_ON}"}]}}
{"finalize":{}}
EOF

Router response:

{
 "credential": {}
}
{}

The first response acknowledges the Credential update; the second acknowledges the finalize. The old Authorized Keys for this account are replaced.

Validation step: In a production workflow, open a second terminal between the credential update and finalize to test SSH with the new Key. Cancel the stream (Ctrl+C) to Rollback if validation fails.

Verify Account Credentials on IOS XR

After RotateAccountCredentials, verify the Authorized Keys were applied:

show ssh server
show gnsi state

Or verify via gRPC with GetPublicKeys (see Verify on IOS XR for the full verification block).

RotateAccountCredentials: rotate Password

Change the Password for a user account. Use a non-admin user (e.g. mike, not the gRPC rotate-authority cisco) so a bad password does not lock you out of the management plane:

CREATED_ON=$(date +%s)
grpcurl -plaintext \
 -import-path "$GNSI_IMPORT" \
 -proto "$PROTO_CREDZ" \
 "${GRPC_MD[@]}" \
 -d '@' \
 "${XR}" \
 gnsi.credentialz.v1.Credentialz.RotateAccountCredentials <<EOF
{"password":{"accounts":[{"account":"mike","password":{"plaintext":"NewSecurePass123"},"version":"pwd-mike-v1","createdOn":"${CREATED_ON}"}]}}
{"finalize":{}}
EOF

Router response on IOS XR 25.3.1 (NCS-540):

{ "password": {} }
{}

The new password is immediately usable for gRPC AAA:

grpcurl -plaintext -import-path "$GNSI_IMPORT" -proto "$PROTO_CREDZ" \
 -H "username:mike" -H "password:NewSecurePass123" \
 -d '{}' "${XR}" gnsi.credentialz.v1.Credentialz.GetPublicKeys
# returns publicKeys[…]
grpcurl -plaintext -import-path "$GNSI_IMPORT" -proto "$PROTO_CREDZ" \
 -H "username:mike" -H "password:OLD-PASSWORD" \
 -d '{}' "${XR}" gnsi.credentialz.v1.Credentialz.GetPublicKeys
# ERROR: Code: Unauthenticated, Message: authentication failed

RotateHostParameters: generate new Host Keys

GenerateKeys is destructive. A single-algorithm generateKeys call wipes every other algorithm’s host key, including ones the proto cannot regenerate (DSA, ECDSA-P-384). Always pass every algorithm you want to keep in one keyParams list.

Recommended baseline call — regenerate the four algorithms IOS XR 25.3.1 actually supports through Credentialz, in one transaction:

grpcurl -plaintext \
 -import-path "$GNSI_IMPORT" \
 -proto "$PROTO_CREDZ" \
 "${GRPC_MD[@]}" \
 -d '@' \
 "${XR}" \
 gnsi.credentialz.v1.Credentialz.RotateHostParameters <<'EOF'
{"generateKeys":{"keyParams":[
 "KEY_GEN_SSH_KEY_TYPE_RSA_2048",
 "KEY_GEN_SSH_KEY_TYPE_ECDSA_P_256",
 "KEY_GEN_SSH_KEY_TYPE_ECDSA_P_521",
 "KEY_GEN_SSH_KEY_TYPE_EDDSA_ED25519"]}}
{"finalize":{}}
EOF

Router response:

{
 "generateKeys": {
 "publicKeys": [
 { "publicKey": "c3NoLXJzYS...", "keyType": "KEY_TYPE_RSA_2048" },
 { "publicKey": "ZWNkc2Etc2hhMi1uaXN0cDI1...","keyType": "KEY_TYPE_ECDSA_P_256" },
 { "publicKey": "ZWNkc2Etc2hhMi1uaXN0cDUy...","keyType": "KEY_TYPE_ECDSA_P_521" },
 { "publicKey": "c3NoLWVkMjU1MTkg...", "keyType": "KEY_TYPE_ED25519" }
 ]
 }
}
{}

Each entry in publicKeys contains the freshly generated Public Key (base64-encoded). In a production workflow, extract these Keys, sign them with your CA, then send a ServerKeysRequest with the Host Certificate before finalizing.

Single-algorithm calls only for greenfield. A generateKeys call with one algorithm wipes every other host key — and DSA / ECDSA-P-384 are not recoverable through Credentialz because the device’s CanGenerateKey rejects both. Use the multi-algo form above unless you really intend to flatten the host-key store.

RotateHostParameters: upload external Host Key

Destructive partial commit on 25.3.1. The serverKeys finalize returns Code: Unknown / "Write Metadata to file failed … No such file or directory", but the host-key store is still mutated — every existing host key is wiped and the uploaded private key becomes the only active one. Have a recovery generateKeys call ready before you run this (see GenerateKeys is destructive).

Upload an externally generated SSH Host Key. The privateKey field is base64-encoded:

ssh-keygen -t ed25519 -f /tmp/credz-ext-hostkey -N "" -C "ext-host@lab"
PRIVKEY_B64=$(base64 < /tmp/credz-ext-hostkey | tr -d '\n')
CREATED_ON=$(date +%s)
grpcurl -plaintext \
 -import-path "$GNSI_IMPORT" \
 -proto "$PROTO_CREDZ" \
 "${GRPC_MD[@]}" \
 -d '@' \
 "${XR}" \
 gnsi.credentialz.v1.Credentialz.RotateHostParameters <<EOF
{"serverKeys":{"authArtifacts":[{"privateKey":"${PRIVKEY_B64}"}],"version":"hostkey-ext-v1","createdOn":"${CREATED_ON}"}}
{"finalize":{}}
EOF

Documented (spec) response, on a healthy implementation:

{ "serverKeys": {} }
{}

Observed response on NCS-540 25.3.1:

{ "serverKeys": {} }
ERROR:
 Code: Unknown
 Message: Write Metadata to file failed: 'SSH' detected the 'fatal' condition 'Failed to open file': No such file or directory

After this error the device silently retains the uploaded Ed25519 as the only host key. Verify with GetPublicKeys and decode the returned key:

grpcurl ... -d '{}' "${XR}" gnsi.credentialz.v1.Credentialz.GetPublicKeys \
 | jq -r '.publicKeys[]|select(.keyType=="KEY_TYPE_ED25519").publicKey' \
 | base64 -d | ssh-keygen -lf /dev/stdin
# fingerprint matches /tmp/credz-ext-hostkey.pub
# Recover the missing algorithms in one call:
grpcurl ... -d @ "${XR}" gnsi.credentialz.v1.Credentialz.RotateHostParameters <<'EOF'
{"generateKeys":{"keyParams":[
 "KEY_GEN_SSH_KEY_TYPE_RSA_2048",
 "KEY_GEN_SSH_KEY_TYPE_ECDSA_P_256",
 "KEY_GEN_SSH_KEY_TYPE_ECDSA_P_521",
 "KEY_GEN_SSH_KEY_TYPE_EDDSA_ED25519"]}}
{"finalize":{}}
EOF

RotateHostParameters: upload CA Public Key

Upload a CA Public Key for verifying client SSH Certificates:

ssh-keygen -t ed25519 -f /tmp/credz-test-ca -N "" -C "test-ca"
CA_B64=$(base64 < /tmp/credz-test-ca.pub | tr -d '\n')
CREATED_ON=$(date +%s)
grpcurl -plaintext \
 -import-path "$GNSI_IMPORT" \
 -proto "$PROTO_CREDZ" \
 "${GRPC_MD[@]}" \
 -d '@' \
 "${XR}" \
 gnsi.credentialz.v1.Credentialz.RotateHostParameters <<EOF
{"sshCaPublicKey":{"sshCaPublicKeys":[{"publicKey":"${CA_B64}","keyType":"KEY_TYPE_ED25519"}],"version":"ca-v1","createdOn":"${CREATED_ON}"}}
{"finalize":{}}
EOF

Router response:

{
 "sshCaPublicKey": {}
}
{}

RotateHostParameters: restrict Authentication methods

Restrict which SSH Authentication methods are globally allowed. Available types: AUTHENTICATION_TYPE_PASSWORD, AUTHENTICATION_TYPE_PUBKEY, AUTHENTICATION_TYPE_KBDINTERACTIVE.

Allow all three methods (safe default):

grpcurl -plaintext \
 -import-path "$GNSI_IMPORT" \
 -proto "$PROTO_CREDZ" \
 "${GRPC_MD[@]}" \
 -d '@' \
 "${XR}" \
 gnsi.credentialz.v1.Credentialz.RotateHostParameters <<'EOF'
{"authenticationAllowed":{"authenticationTypes":["AUTHENTICATION_TYPE_PASSWORD","AUTHENTICATION_TYPE_PUBKEY","AUTHENTICATION_TYPE_KBDINTERACTIVE"]}}
{"finalize":{}}
EOF

Router response:

{
 "authenticationAllowed": {}
}
{}

Caution: To restrict to pubkey-only, pass ["AUTHENTICATION_TYPE_PUBKEY"]. This disables password and keyboard-interactive — validate SSH access before finalizing to avoid lockout.

RotateHostParameters: set AuthorizedPrincipalsCommand (HIBA)

Enable dynamic principal Authorization using the HIBA binary:

grpcurl -plaintext \
 -import-path "$GNSI_IMPORT" \
 -proto "$PROTO_CREDZ" \
 "${GRPC_MD[@]}" \
 -d '@' \
 "${XR}" \
 gnsi.credentialz.v1.Credentialz.RotateHostParameters <<'EOF'
{"authorizedPrincipalCheck":{"tool":"TOOL_HIBA_DEFAULT"}}
{"finalize":{}}
EOF

Router response:

{
 "authorizedPrincipalCheck": {}
}
{}

To clear HIBA and revert to static Authorization, use "tool":"TOOL_UNSPECIFIED".

Verify on IOS XR

After any Rotation, verify with CLI:

show ssh
show ssh server
show crypto key mypubkey ed25519
show gnsi state

Verify via gRPC — confirm the Host Keys reflect the latest Rotation:

grpcurl -plaintext \
 -import-path "$GNSI_IMPORT" \
 -proto "$PROTO_CREDZ" \
 "${GRPC_MD[@]}" \
 -d '{}' \
 "${XR}" \
 gnsi.credentialz.v1.Credentialz.GetPublicKeys

Response after an external key upload:

{
 "publicKeys": [
 {
 "publicKey": "c3NoLWVkMjU1MTkgQUFBQUMzTnph...",
 "keyType": "KEY_TYPE_ED25519"
 }
 ]
}

Decode the publicKey to confirm it matches the uploaded material:

echo "c3NoLWVkMjU1MTkgQUFBQUMzTnph..." | base64 -d
# ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAI... ext-host@lab

grpcurl Debugging Aids

Use describe and -msg-template to explore the message structure:

grpcurl -import-path "$GNSI_IMPORT" -proto "$PROTO_CREDZ" \
 describe gnsi.credentialz.v1.Credentialz.RotateAccountCredentials
grpcurl -import-path "$GNSI_IMPORT" -proto "$PROTO_CREDZ" \
 -msg-template describe gnsi.credentialz.v1.RotateAccountCredentialsRequest
grpcurl -import-path "$GNSI_IMPORT" -proto "$PROTO_CREDZ" \
 -msg-template describe gnsi.credentialz.v1.RotateHostParametersRequest
grpcurl -import-path "$GNSI_IMPORT" -proto "$PROTO_CREDZ" \
 -msg-template describe gnsi.credentialz.v1.CanGenerateKeyRequest

---

SPIFFE Mapping Path

The SPIFFE ID-based Authentication and Authorization workflow uses a mapping file at:

/misc/config/grpc/gnsi/credentialz/spiffe-user-map.json

If the file is missing or invalid, mapping is empty. When present, SPIFFE IDs are resolved to usernames for gRPC Authentication. See the SPIFFE ID-based Authentication section in the Cisco guide.

---

Monitoring with YANG Telemetry

The OpenConfig YANG model openconfig-gnsi-credentialz is intended to expose Credential metadata and access Counters.

On 25.3.1: gnmic capabilities advertises openconfig-gnsi-credentialz, 0.6.0, but gnmi Get returns NotFound on every documented Credentialz leaf (active-host-key-version, …/counters/access-{accepts,rejects}, the per-user password-version, etc.). The only Credentialz signal currently exposed is the per-RPC counter view under openconfig-gnsi-authz:authz-policy-counters (rows for gnsi.credentialz.v1.Credentialz/{CanGenerateKey,GetPublicKeys,RotateAccountCredentials,RotateHostParameters}). The leaves below are documented for reference — treat them as forward-looking on this build.

SSH server Credential metadata (base: /system/ssh-server/state):

YANG path	What it tells
/system/ssh-server/state/active-trusted-user-ca-keys-version	Version of the trusted CA Keys for client Certificate validation
/system/ssh-server/state/active-trusted-user-ca-keys-created-on	When the CA Keys were installed
/system/ssh-server/state/active-host-certificate-version	Version of the Host Certificate
/system/ssh-server/state/active-host-certificate-created-on	When the Host Certificate was installed
/system/ssh-server/state/active-host-key-version	Version of the Host Public Key
/system/ssh-server/state/active-host-key-created-on	When the Host Key was installed

SSH server access Counters (base: /system/ssh-server/state/counters):

YANG path	What it tells
/system/ssh-server/state/counters/access-accepts	Total successful SSH connections
/system/ssh-server/state/counters/access-rejects	Total denied SSH connections
/system/ssh-server/state/counters/last-access-accept	Timestamp of the most recent successful SSH connection
/system/ssh-server/state/counters/last-access-reject	Timestamp of the most recent SSH rejection

Per-user Credential metadata (base: /system/aaa/authentication/users/user[username=<user>]/state):

YANG path (relative to base)	What it tells
.../authorized-keys-list-version	Version of the Authorized Keys for this user
.../authorized-keys-list-created-on	When the Authorized Keys were installed
.../authorized-principals-list-version	Version of the Authorized Principals for this user
.../authorized-principals-list-created-on	When the Authorized Principals were installed
.../password-version	Version of the Password for this user
.../password-created-on	When the Password was set

Subscribe to access-rejects and last-access-reject to detect SSH Authentication failures after a Credential Rotation. A spike in rejects after a RotateHostParameters usually means the new Host Key or CA Key is not matching — verify with GetPublicKeys.

The *-version and *-created-on leaves mirror the exact strings passed during the Rotate RPCs, so Telemetry pipelines can correlate SSH access patterns with specific Credential deployments.

Note: When Credentialz is enabled, the standard YANG leaves for SSH keys and passwords under /system/aaa/authentication/users/user/config/ are treated as invalid. Credentialz owns those Credentials.

Quick Telemetry check with gnmic

Confirm what the device actually advertises and what it returns for the documented paths:

# 1) What model + revision does the device claim to support?
gnmic -a "${XR}" --insecure -u "${XR_USER}" -p "${XR_PASS}" capabilities \
 | grep -i credentialz
# openconfig-gnsi-credentialz, 0.6.0 (gnmi)
# cisco-xr-openconfig-gnsi-credentialz-deviations, 2024-08-29
# 2) Try the documented leaves — these will all be NotFound on 25.3.1:
gnmic -a "${XR}" --insecure -u "${XR_USER}" -p "${XR_PASS}" get \
 --path "/system/ssh-server/state/active-host-key-version" \
 --path "/system/ssh-server/state/counters/access-rejects" || true
# 3) Working fallback — gRPC-server RPC counters via openconfig-gnsi-authz:
gnmic -a "${XR}" --insecure -u "${XR_USER}" -p "${XR_PASS}" get \
 --path "/system/grpc-servers/grpc-server/network-instance/state/authz-policy-counters/rpcs/rpc[name=/gnsi.credentialz.v1.Credentialz/RotateHostParameters]/state"

The third call is the practical signal for “did anyone hit Credentialz?” today; the first two are placeholders until the YANG implementation lands.

---

Things to watch out for

• One RPC at a time. Overlapping RotateAccountCredentials or RotateHostParameters calls are rejected. Serialize across both RPCs.
• Validate before finalize. Always test SSH (or gRPC, for the password rotate) with the new Credentials in a second terminal before sending FinalizeRequest. Cancel the stream to Rollback if validation fails.
• Authorized Keys are replaced, not merged. AuthorizedKeysRequest replaces the entire Authorized Keys list for the account. Include all Keys that should remain valid. Sending an empty list is the documented “remove” path.
• Restricting Authentication methods can lock out users who rely on the disabled method. Always keep at least one working method.
• HIBA support. Setting TOOL_HIBA_DEFAULT in AuthorizedPrincipalCheckRequest enables dynamic principal Authorization via the HIBA binary. Clearing it (TOOL_UNSPECIFIED) reverts to static Authorization.
• PasswordRequest works on 25.3.1. Earlier images returned Unimplemented; the live build we tested re-binds the password and the new password is immediately usable for gRPC AAA. Always rotate against a non-admin user (e.g. mike), not the gRPC rotate-authority you are currently authenticated as.
• GenerateKeys is destructive across all algorithms. Even a single-algo keyParams wipes RSA, ECDSA-P-256/P-384/P-521, DSA and Ed25519. DSA and ECDSA-P-384 are irreversible through gRPC (the device’s CanGenerateKey rejects both). Always pass every algorithm you want to keep in the same keyParams list. See GenerateKeys is destructive.
• ServerKeysRequest finalize bug (25.3.1). Finalize returns Code: Unknown with an internal Failed to open file message, but the host-key store is still mutated — the uploaded private key replaces every existing host key. Have a multi-algo generateKeys recovery call ready.
• CanGenerateKey rejects two values that are in the proto enum. KEY_GEN_SSH_KEY_TYPE_RSA_3072 and KEY_GEN_SSH_KEY_TYPE_ECDSA_P_384 return InvalidArgument: "Can Generate Key:Invaid KeyGen input" (typo on the device). Stick to RSA-2048/4096, ECDSA-P-256/P-521, Ed25519.
• Default host-key inventory includes DSA and ECDSA-P-384. GetPublicKeys on a default NCS-540 lists 6 keys, two of which (ssh-dss, ECDSA-P-384) come back with keyType field omitted. DSA in particular is deprecated by OpenSSH; remove it via a clean multi-algo generateKeys call.
• Mid-stream cancel works but leaks subsystem name. Closing the bidi stream before sending finalize aborts the candidate, but the device returns Code: Canceled, Message: "Credz stream failure: EOF". Treat this as expected; a follow-up call with the same payload + finalize succeeds cleanly.
• YANG telemetry is advertised but not implemented (25.3.1). Every documented Credentialz leaf returns NotFound. Use openconfig-gnsi-authz:authz-policy-counters for RPC-level visibility until the YANG implementation lands. Application-layer errors (InvalidArgument, Unknown) do not increment access-rejects on those counters.
• Certz vs Credentialz. Certz handles gRPC TLS (Certificates, Trust Bundles, CRLs for the gRPC server). Credentialz handles SSH (Host Keys, user Keys, Passwords for interactive access). They are separate planes.

Conclusion

Credentialz closes the loop between programmatic gRPC Management and interactive SSH access. Keys, Passwords, and Host parameters rotate through a standard gRPC API with built-in Rollback — no file edits, no reloads, no vendor-specific CLIs. Together with Certz for TLS Authentication, Credentialz covers both Authentication planes. Adding Authz for Service-level Authorization, Pathz for Path-level Authorization, and Acctz for Accounting completes the gNSI Security Lifecycle on IOS XR.

• Credentialz rotates SSH Credentials — Passwords, Authorized Keys, Host Keys, Certificates, CA Keys, Authentication method restrictions — without a reload.
• Two streaming RPCs: RotateAccountCredentials (per-user) and RotateHostParameters (per-host). Both use the same stage-validate-finalize Transaction model as Certz.
• Two unary RPCs: CanGenerateKey (preflight) and GetPublicKeys (inspect current Host Keys).
• Only one Credentialz RPC at a time. Overlapping calls fail.
• Authorized Keys are replaced entirely — include all valid Keys in the request.
• Host-key rotates are destructive. Both generateKeys (single-algo) and serverKeys (failed finalize on 25.3.1) wipe every other algorithm. Always rotate host keys with a multi-algo generateKeys that lists every algorithm you want to keep.
• PasswordRequest works on 25.3.1 NCS-540 and the new password is immediately accepted by gRPC AAA — but always rotate non-admin users, not the gRPC rotate-authority.
• openconfig-gnsi-credentialz YANG telemetry is advertised but not implemented on 25.3.1. Use openconfig-gnsi-authz:authz-policy-counters for RPC-level visibility today.
• Certz = gRPC TLS. Credentialz = SSH. Different planes, same Transaction pattern.

What to do next: With Authentication covered (Certz + Credentialz), continue to Authorization with Authz for gRPC Service-level control.

---

Reverting / Undo

Situation	How to revert
Cancel a Rotate mid-stream	Close the grpcurl stream (Ctrl+C) before sending finalize. The Candidate is discarded and the previous Credentials stay active. The device may return Code: Canceled, Message: "Credz stream failure: EOF" — this is the expected rollback signature on 25.3.1.
Remove Authorized Keys	Send a RotateAccountCredentials with an empty authorizedKeys list for the account. This reverts to password-only Authentication for that user.
Restore a rotated password	Send another RotateAccountCredentials with password.accounts[*].password.plaintext set to the original value. New password becomes effective on finalize.
Restore all Authentication methods	Send RotateHostParameters with all three types: ["AUTHENTICATION_TYPE_PASSWORD","AUTHENTICATION_TYPE_PUBKEY","AUTHENTICATION_TYPE_KBDINTERACTIVE"].
Clear HIBA	Send RotateHostParameters with "tool":"TOOL_UNSPECIFIED" to revert to static principal Authorization.
Recover from a destructive generateKeys or a failed serverKeys	Re-run RotateHostParameters with a multi-algo generateKeys: ["KEY_GEN_SSH_KEY_TYPE_RSA_2048","KEY_GEN_SSH_KEY_TYPE_ECDSA_P_256","KEY_GEN_SSH_KEY_TYPE_ECDSA_P_521","KEY_GEN_SSH_KEY_TYPE_EDDSA_ED25519"] followed by finalize. DSA and ECDSA-P-384 cannot be recovered through Credentialz — fall back to crypto key generate dsa / crypto key generate ecdsa nistp384 from IOS XR CLI if you actually need them.
Replace bad Host Key	Upload corrected material through RotateHostParameters with serverKeys. Be aware of the 25.3.1 Write Metadata to file failed finalize bug — verify with GetPublicKeys and run a multi-algo generateKeys if other algorithms were silently wiped.

References

1. Cisco IOS XR — Programmability Configuration Guide: gRPC Authentication (gNSI Credentialz updates, Rotate Credentialz RPC, Rotate account credentials, Rotate host parameters)
2. OpenConfig — credentialz.proto, credentialz/README.md
3. OpenConfig — openconfig-gnsi-credentialz YANG model v0.7.0
4. Series — gNSI on IOS XR overview

---