A new feature in the cert-manager plugin google-cas-issuer streamlines rotation of the organization root Certificate Authorities. This article explains when it can help and how to use it.
If you are managing your workload TLS certificates using the ubiquitous cert-manager service for Kubernetes and the Google Cloud Certificate Authority Service (CAS), you have likely encountered the open-source project google-cas-issuer. This cert-manager plugin allows it to call CAS in Google Cloud to fetch certificates for Kubernetes workloads. This applies whether you are using Google GKE managed service, Google Distributed Cloud, or any arbitrary Kubernetes cluster with access to the Google Cloud API. While this is less critical for workloads managed by Google Cloud Service Mesh (which integrates with CAS and handles lifecycle management automatically), there are many other use cases where a TLS certificate is required within a Kubernetes cluster.
To obtain a TLS certificate from the Certificate Authority Service (CAS), you would typically install cert-manager in the cluster, define a Google Certificate Authority Issuer (or ClusterIssuer at the cluster level), and configure it with a reference to the CAS CA Pool resource in Google Cloud.
apiVersion: cas-issuer.jetstack.io/v1beta1
kind: GoogleCASClusterIssuer
metadata:
name: googlecasissuer
spec:
project: my-project-id
location: europe-west3
caPoolId: server-pool
Next, you must describe the required TLS certificate using the cert-manager API Certificate resource and reference a prepared Kubernetes Secret object that holds the keystore password. For example:
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: gateway-certificate
namespace: istio-system
spec:
secretName: gateway-cert-tls
commonName: crilb.hello.zone
dnsNames:
- crilb.hello.zone
duration: 720h
renewBefore: 8h
issuerRef:
group: cas-issuer.jetstack.io
kind: GoogleCASClusterIssuer
name: googlecasissuer
keystores:
jks:
create: true
passwordSecretRef:
name: gateway-cert-keystore-password
key: password
pkcs12:
create: true
passwordSecretRef:
name: gateway-cert-keystore-password
key: password
---
apiVersion: v1
kind: Secret
metadata:
name: gateway-cert-keystore-password
namespace: istio-system
data:
password: Y2hhbmdlaXQ=
Assuming everything is configured correctly (refer to the public documentation for a detailed overview), you will receive a Kubernetes Secret object containing three key components.
apiVersion: v1
kind: Secret
data:
ca.crt: LS0tLS1C.....
keystore.p12: MIIM4wIBAzCCD...
tls.crt: LS0tLS1CRUdJ...
tls.key: LS0tLS1CRUdJTiB...
truststore.jks: /u3+7QAAAAIA...
truststore.p12: MIIEHgIBAz...
metadata:
annotations:
cert-manager.io/alt-names: crilb.hello.zone
cert-manager.io/certificate-name: gateway-certificate
cert-manager.io/common-name: crilb.hello.zone
cert-manager.io/ip-sans: ""
cert-manager.io/issuer-group: cas-issuer.jetstack.io
cert-manager.io/issuer-kind: GoogleCASClusterIssuer
cert-manager.io/issuer-name: googlecasissuer
cert-manager.io/uri-sans: ""
labels:
controller.cert-manager.io/fao: "true"
name: gateway-cert-tls
namespace: istio-system
- tls.key — The private key of the TLS certificate, which the server uses to provide a TLS-encrypted endpoint.
- tls.crt — The public certificate that the server process presents to clients to prove its authenticity.
- ca.crt —The root CA certificate of the hierarchy from which the issuing CA originates.
The ca.crt value requires a bit more explanation.
In many cases (and following recommended best practices), especially in large organizations, issuing certificate authorities (CAs) are organized into tree-like hierarchies. In these setups, the CAs issuing the actual TLS certificates (known as “leaf” certificates) are not the only entities that clients must trust.
There are several reasons for such a setup. One is the ability to independently replace the CAs in the lower levels of the hierarchy if they expire or are (hopefully not) compromised. This alone allows for shorter issuing CA validity times, which improves the overall organizational security posture and follows modern recommended security practices.
Clients do not need to be reconfigured each time the intermediate CAs are replaced. They can be configured to trust the single parent CA of the hierarchy and, therefore, transitively trust any certificate issued by any intermediate-level CA. The parent or root CAs in this hierarchy typically have longer validity (lifetime) configured.
So far, so good. But eventually (unless you have configured very long intervals against security best practices), the root CA will expire as well. Application clients must then be reconfigured with a new trust bundle that includes the new root CA certificate.
By default, cert-manager places this root CA certificate into the ca.crt field of the generated Kubernetes Secret. This is conveniently available to all workloads in the cluster — both application servers providing TLS endpoints and application clients establishing encrypted, trusted connections to those endpoints.
Relying on the ca.crt field for a trust bundle is tempting, but it presents a significant hurdle for root CA rotation in downtime-sensitive workloads. Since application clients trust a single root CA certificate, rotating it requires the simultaneous replacement of the server TLS certificate and the client trust bundle to maintain service availability. This is often very difficult, if not impossible, to achieve seamlessly.
The best practice is to configure application clients to fetch their trust bundle from a central, well-defined location, such as a Google Cloud Storage bucket (see the KubeTrust reference architecture explained in this IEEE CSCLOUD2025 article). In such a location, several valid root CA certificates can coexist during the rotation period. This allows the application client configuration to be updated to the new trust bundle version independently — the most critical factor — without incurring application downtime.
However, because cert-manager generates a convenient Secret accessible by application clients, following that best practice can feel like a burden. Consequently, applications often accumulate “technical debt” by relying on the ca.crt field (containing only the single root CA) as their trust bundle source.
Since root CAs must eventually be rotated, how can we minimize application downtime during that process?
Google Cloud Certificate Authority Service uses the concept of a CA Pool. A CA Pool is a container for multiple CAs. When you request a certificate from the CAS service, you target the CA Pool rather than a specific CAS CA resource.
This is highly convenient for rotation: when a CA in the pool nears expiration, a new CA with a longer validity period is created and placed in the pool (initially in a Disabled state). Once the new CA certificate is distributed to all application client trust bundles, the new CA can be activated in the pool to issue TLS certificates alongside the old CA. The old CA can then be disabled from issuing new certificates (while remaining trusted), and after application servers have updated their certificates, it can be fully deactivated and removed. This is the intended rotation workflow in CAS.
For application clients that rely solely on the cert-manager Secret’s ca.crt field, this rotation procedure would typically cause downtime.
However, starting with the v0.11.0 release, the google-cas-issuer plugin includes a new feature to simplify this for those clients.
By optionally specifying the caFetchMode: PoolCAs attribute, the ca.crt field in the resulting Kubernetes Secret will be populated with the root CAs of all CAs currently present in the specified CAS CA Pool:
apiVersion: cas-issuer.jetstack.io/v1beta1
kind: GoogleCASClusterIssuer
metadata:
name: googlecasissuer
spec:
project: my-project-id
location: europe-west3
caPoolId: server-pool
caFetchMode: PoolCAs
If an application client now uses the ca.crt field as its trust bundle to verify application servers when establishing TLS connections, it will receive both the old, expiring root CA and the new, extended one. This allows the client to bridge the root CA rotation and avoid downtime by updating its configuration independently of the application server’s TLS update.
Here is what the resulting Kubernetes Secret might look like (the Base64-encoded ca.crt value has been decoded for illustration purposes):
apiVersion: v1
kind: Secret
data:
ca.crt: |-
-----BEGIN CERTIFICATE-----
MIID/TCCAuWgAwIBAgIUAOvVafxuYaRGgdctZqTWQ64ZJoowDQYJKoZIhvcNAQEL
BQAwgYUxCzAJBgNVBAYTAkRFMRwwGgYDVQQIExNOb3JkcmhlaW4td2VzdGZhbGVu
...
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIICbzCCAhagAwIBAgITfkwGizNFOiLKdSLvyTwKW7ekozAKBggqhkjOPQQDAjCB
hTELMAkGA1UEBhMCR2UxHDAaBgNVBAgTE05vcmRyaGVpbi13ZXN0ZmFsZW4xEDAO
...
-----END CERTIFICATE-----
keystore.p12: MIIM4wIBAzCCD...
tls.crt: LS0tLS1CRUdJ...
tls.key: LS0tLS1CRUdJTiB...
truststore.jks: /u3+7QAAAAIA...
truststore.p12: MIIEHgIBAz...
metadata:
annotations:
cert-manager.io/alt-names: crilb.hello.zone
cert-manager.io/certificate-name: gateway-certificate
cert-manager.io/common-name: crilb.hello.zone
cert-manager.io/ip-sans: ""
cert-manager.io/issuer-group: cas-issuer.jetstack.io
cert-manager.io/issuer-kind: GoogleCASClusterIssuer
cert-manager.io/issuer-name: googlecasissuer
cert-manager.io/uri-sans: ""
labels:
controller.cert-manager.io/fao: "true"
name: gateway-cert-tls
namespace: istio-system
The great thing here is that all CAS CA Pool root CAs can also be requested in P12 and JKS formats within the same Secret. Cert-manager will generate all of them in the desired format, making them convenient for direct mounting to applications that support trust bundles in those formats.
Hopefully, this new google-cas-issuer feature will help save some gray hair during future root CA rotations.
Please let me know what you think!
Simplified Google Cloud Certificate Authority Service CA Rotation for Kubernetes Workloads was originally published in Google Cloud – Community on Medium, where people are continuing the conversation by highlighting and responding to this story.
Source Credit: https://medium.com/google-cloud/simplified-google-cloud-certificate-authority-service-ca-rotation-for-kubernetes-workloads-8b0c98e0a17b?source=rss—-e52cf94d98af—4
About the Author
