GKE: service to service zonal affinity when crossing the cluster boundary. | by Mark Chilvers | Google Cloud – Community | Sep, 2025

Let’s walk through an example of this:

Proposed architecture

The proposed architecture for this use case (depicted below) uses the following GCP services:

• GKE. In this case we’ll create GKE Autopilot clusters. The advantage here is that it will enable both Cloud DNS and GKE Gateway automatically, as well as managing and provisioning appropriately sized worker nodes for us.
• Cloud DNS. A fully managed DNS service.
• GKE Gateway. To provision the external L7 Application Load Balancer we use GKE Gateway which is an implementation of the Gateway API kubernetes-sig.
• Private Service Connect (using a GKE ServiceAttachment CRD). To provide connectivity to our target service across distinct Google VPCs.

Press enter or click to view image in full size

We will create two clusters here. In one we’ll deploy a front end service and expose it externally.

In the other we’ll deploy a backend service, and expose it via a service of type LoadBalancer, preferring a close traffic distribution.

Each of these clusters will be in different VPCs, so we’ll use Private Services Connect (PSC) to connect the Internal Loadbalancer in one VPC to the target VPC where it will be consumed by the frontend service. If the clusters are in the same VPC then there is no need to use PSC.

Deployment Steps

You will need 2 GCP projects. We’ll begin by creating a GKE Autopilot cluster in one of these projects. Into this project & cluster we will deploy our “frontend” service.

1. We set up some variables and create a working directory:

export PROJECT_1=[your-source-project-id]
export PROJECT_2=[your-target-project-id]
gcloud config set project ${PROJECT_1}
mkdir -p ${HOME}/affinity-source
cd ${HOME}/affinity-source
export WORKDIR=`pwd`

2. We create an empty kubeconfig. In this case we deploy this cluster to us-east4, but you can pick any valid GCP region. We enable the GKE API.

touch affinity_kubeconfig
export KUBECONFIG=${WORKDIR}/affinity_kubeconfig
export CLUSTER_1_NAME=source-cluster
export CLUSTER_1_LOCATION=us-east4
gcloud services enable container.googleapis.com

3. We create an Autopilot cluster in the rapid release channel. Note: we assume a VPC named default exists.

gcloud container --project ${PROJECT_1} clusters create-auto \
${CLUSTER_1_NAME} --region ${CLUSTER_1_LOCATION} --release-channel rapid --enable-fleet

4. We export the kubecontext of our newly created cluster.

export CONTEXT_1=$(kubectl config current-context)

5. We create a namespace for the frontend service & deploy it.

Note: we use TopologySpreadConstraints on the deployment to ensure our application pods are spread over all three zones.

kubectl --context=${CONTEXT_1} create ns frontend
mkdir -p ${WORKDIR}/whereami-frontend/base
cat < ${WORKDIR}/whereami-frontend/base/kustomization.yaml 
resources:
- github.com/GoogleCloudPlatform/kubernetes-engine-samples/quickstarts/whereami/k8s
EOF
mkdir whereami-frontend/variant
cat < ${WORKDIR}/whereami-frontend/variant/cm-flag.yaml 
apiVersion: v1
kind: ConfigMap
metadata:
name: whereami
data:
BACKEND_ENABLED: "True"
BACKEND_SERVICE: "http://backend.${PROJECT_2}.mycompany.private/"
EOF
cat < ${WORKDIR}/whereami-frontend/variant/service-type.yaml 
apiVersion: "v1"
kind: "Service"
metadata:
name: "whereami"
spec:
ports:
- port: 8080
type: ClusterIP
EOF
cat < ${WORKDIR}/whereami-frontend/variant/deploy-topology.yaml 
apiVersion: "apps/v1"
kind: "Deployment"
metadata:
name: "whereami"
spec:
template:
spec:
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
minDomains: 3
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
app: whereami-frontend
EOF
cat < ${WORKDIR}/whereami-frontend/variant/kustomization.yaml 
nameSuffix: "-frontend"
namespace: frontend
commonLabels:
app: whereami-frontend
resources:
- ../base
patches:
- path: cm-flag.yaml
target:
kind: ConfigMap
- path: service-type.yaml
target:
kind: Service
- path: deploy-topology.yaml
target:
kind: Deployment
EOF
kubectl --context=${CONTEXT_1} apply -k ${WORKDIR}/whereami-frontend/variant

6. We reserve an external IP address for our frontend, and use Cloud Endpoints as a mechanism to create an external DNS entry. Then we create an SSL certificate for our frontend.

gcloud compute addresses create affinity-gclb-ip --global
export GCLB_IP=$(gcloud compute addresses describe affinity-gclb-ip \
--global --format "value(address)")
echo ${GCLB_IP}
#This creates a DNS entry
cat < ${WORKDIR}/dns-spec.yaml
swagger: "2.0"
info:
description: "Cloud Endpoints DNS"
title: "Cloud Endpoints DNS"
version: "1.0.0"
paths: {}
host: "frontend.endpoints.${PROJECT_1}.cloud.goog"
x-google-endpoints:
- name: "frontend.endpoints.${PROJECT_1}.cloud.goog"
target: "${GCLB_IP}"
EOF
gcloud endpoints services deploy ${WORKDIR}/dns-spec.yaml
gcloud services enable certificatemanager.googleapis.com --project=${PROJECT_1}
gcloud --project=${PROJECT_1} certificate-manager certificates create affinity-cert \
--domains="frontend.endpoints.${PROJECT_1}.cloud.goog"
gcloud --project=${PROJECT_1} certificate-manager maps create affinity-cert-map
gcloud --project=${PROJECT_1} certificate-manager maps entries create affinity-cert-map-entry \
--map="affinity-cert-map" \
--certificates="affinity-cert" \
--hostname="frontend.endpoints.${PROJECT_1}.cloud.goog"
cat < ${WORKDIR}/gke-gateway.yaml
kind: Gateway
apiVersion: gateway.networking.k8s.io/v1beta1
metadata:
name: external-http
namespace: frontend
annotations:
networking.gke.io/certmap: affinity-cert-map
spec:
gatewayClassName: gke-l7-global-external-managed # gke-l7-gxlb
listeners:
- name: http # list the port only so we can redirect any incoming http requests to https
protocol: HTTP
port: 80
- name: https
protocol: HTTPS
port: 443
addresses:
- type: NamedAddress
value: affinity-gclb-ip # reference the static IP created earlier
EOF
kubectl --context=${CONTEXT_1} apply -f ${WORKDIR}/gke-gateway.yaml
cat << EOF > ${WORKDIR}/default-httproute.yaml
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
name: default-httproute
namespace: frontend
spec:
parentRefs:
- name: external-http
namespace: frontend
sectionName: https
rules:
- matches:
- path:
value: /
backendRefs:
- name: whereami-frontend
port: 8080
EOF
kubectl --context=${CONTEXT_1} apply -f ${WORKDIR}/default-httproute.yaml
cat << EOF > ${WORKDIR}/default-httproute-redirect.yaml
kind: HTTPRoute
apiVersion: gateway.networking.k8s.io/v1beta1
metadata:
name: http-to-https-redirect-httproute
namespace: frontend
spec:
parentRefs:
- name: external-http
namespace: frontend
sectionName: http
rules:
- filters:
- type: RequestRedirect
requestRedirect:
scheme: https
statusCode: 301
EOF
kubectl --context=${CONTEXT_1} apply -f ${WORKDIR}/default-httproute-redirect.yaml

7. Next we’ll create a second Autopilot cluster as the target cluster for our remote service call. We put this in a different project with no connectivity to the first. Note: we assume a VPC named default exists.

Note: after creating our target cluster we must enable ILB subsetting on the cluster. This is a pre-requisite to creating a L4 ILB (a Service of type LoadBalancer) with zonal affinity.

gcloud config set project ${PROJECT_2}
mkdir -p ${HOME}/affinity-target
cd ${HOME}/affinity-target
export WORKDIR=`pwd`
export CLUSTER_2_NAME=target-cluster
export CLUSTER_2_LOCATION=us-east4
gcloud services enable container.googleapis.com
gcloud container --project ${PROJECT_2} clusters create-auto \
${CLUSTER_2_NAME} --region ${CLUSTER_2_LOCATION} --release-channel rapid
export CONTEXT_2=$(kubectl config current-context)
gcloud container clusters update ${CLUSTER_2_NAME} --project ${PROJECT_2} --region ${CLUSTER_2_LOCATION} --enable-l4-ilb-subsetting 

8. Now we’ll create a namespace and deploy our backend service. This is where we specify spec.trafficDistribution: PreferCloseon the Service of type LoadBalancer.

Note: again, we use TopologySpreadConstraints on the deployment to ensure our application pods are spread over all three zones.

kubectl --context=${CONTEXT_2} create ns backend
mkdir -p ${WORKDIR}/whereami-backend/base
cat < ${WORKDIR}/whereami-backend/base/kustomization.yaml 
resources:
- github.com/GoogleCloudPlatform/kubernetes-engine-samples/quickstarts/whereami/k8s
EOF
mkdir ${WORKDIR}/whereami-backend/variant
cat < ${WORKDIR}/whereami-backend/variant/cm-flag.yaml 
apiVersion: v1
kind: ConfigMap
metadata:
name: whereami
data:
BACKEND_ENABLED: "False" # assuming you don't want a chain of backend calls
METADATA:        "backend"
EOF
cat < ${WORKDIR}/whereami-backend/variant/service-type.yaml 
apiVersion: "v1"
kind: "Service"
metadata:
name: "whereami"
annotations:
networking.gke.io/load-balancer-type: "Internal"
spec:
type: LoadBalancer
trafficDistribution: PreferClose
externalTrafficPolicy: Local  
EOF
cat < ${WORKDIR}/whereami-backend/variant/deploy-topology.yaml 
apiVersion: "apps/v1"
kind: "Deployment"
metadata:
name: "whereami"
spec:
template:
spec:
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
minDomains: 3
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
app: whereami-backend
EOF
cat < ${WORKDIR}/whereami-backend/variant/kustomization.yaml 
nameSuffix: "-backend"
namespace: backend
commonLabels:
app: whereami-backend
resources:
- ../base
patches:
- path: cm-flag.yaml
target:
kind: ConfigMap
- path: service-type.yaml
target:
kind: Service
- path: deploy-topology.yaml
target:
kind: Deployment
EOF
kubectl --context=${CONTEXT_2} apply -k ${WORKDIR}/whereami-backend/variant

7. Now create a subnet for Private Service Connect (PSC). We’ll use PSC to expose our ILB into another GCP VPC (the VPC where the source cluster is deployed)

Note: producer and consumer region must be the same in this use case since we want to achieve zonal affinity (which assumes same region).

gcloud beta compute networks subnets create gke-nat-subnet \
--project ${PROJECT_2} \
--network default \
--region ${CLUSTER_2_LOCATION} \
--range 100.100.10.0/24 \
--purpose PRIVATE_SERVICE_CONNECT

8. Now we create a PSC service attachment based upon our target IG Service.

kubectl apply --context=${CONTEXT_2} -f - <   apiVersion: networking.gke.io/v1
kind: ServiceAttachment
metadata:
name: whereami-backend-svcatt
namespace: backend
spec:
connectionPreference: ACCEPT_AUTOMATIC
natSubnets:
- gke-nat-subnet
proxyProtocol: false
resourceRef:
kind: Service
name: whereami-backend
EOF

9. In the source project we’ll reserve an internal IP for the consumer endpoint, and create a Cloud DNS entry that points to that IP address.

Note: producer and consumer endpoint must be in same region in this example. We referenced this DNS name when we deployed the frontend service in step 5.

gcloud compute addresses create vpc-consumer-psc --project=${PROJECT_1} --region=${CLUSTER_1_LOCATION} --subnet=default
export PSC_CON_IP=$(gcloud compute addresses describe vpc-consumer-psc \
--region=${CLUSTER_1_LOCATION} --project=${PROJECT_1} --format "value(address)")
echo ${PSC_CON_IP}
gcloud dns managed-zones create test-private-zone \
--project ${PROJECT_1} \
--dns-name=mycompany.private \
--networks=default \
--description="Zone for external service" \
--visibility=private
gcloud dns record-sets transaction start \
--zone=test-private-zone --project=${PROJECT_1}
gcloud dns record-sets transaction add ${PSC_CON_IP} \
--name=backend.${PROJECT_2}.mycompany.private \
--ttl=300 \
--type=A \
--project=${PROJECT_1} \
--zone=test-private-zone
gcloud dns record-sets transaction execute \
--zone=test-private-zone --project=${PROJECT_1}

10. Finally, we attach the PSC producer (the L4 ILB in the target project) to the consumer IP address in the source project.

export SA_URL=$(kubectl get serviceattachment whereami-backend-svcatt --context=${CONTEXT_2} -n backend -o "jsonpath={.status['serviceAttachmentURL']}")
gcloud compute forwarding-rules create vpc-consumer-psc-fr --project=${PROJECT_1} \
--region=${CLUSTER_1_LOCATION} --network=default \
--address=vpc-consumer-psc --target-service-attachment=${SA_URL}

11. Now we can curl our frontend multiple times to validate that frontend and backend service zones match!

curl -s https://frontend.endpoints.${PROJECT_1}.cloud.goog | jq .

You should observe that the zone for the backend and frontend part of the results should be the same:

Conclusion

We tried out the new feature of zonal affinity feature of GKE Service of type LoadBalancer across 2 distinct GKE clusters, and shows that it will prefer to serve a target service from the same zone as the client service!

It’s worth pointing out that using zonal affinity can result in your upstream service calls to “hotspot” if the pods of the client service are not deployed evenly across zones (or more generally if client traffic is not evenly spread) . To put it simply, as you chain together services where zonal affinity is configured the likelihood of these zonal “hotspots” can increase if care isn’t taken to ensure an even deployment & split of traffic across all zones.

It should also be noted — deploying a small number of replicas will increase the likelihood that the “PreferClose” distribution cannot be fulfilled, and the load balancer would then fallback to serving from a healthy endpoint in another zone.

With these caveats, this feature (still in preview) can be used to reduce cost and latency for service to service calls that cross the GKE cluster boundary.