Model Armor integration with Service Extension : introduction to runtime AI security without code…

Model Armor integration with Service Extension : introduction to runtime AI security without code changes

Model Armor is GCP’s runtime AI protection solution . Since Model Armor offers multiple integrations , which decides how it is used and implemented , choosing the right integration is important . Without beating the bush , Model Armor offers 3 types of integrations

1. API interface for developers to integrate with any model running anywhere
2. Network service extensions (L7 load balancers, GKE inference gateway) that allows it to act as a firewall for AI prompt/responses
3. GCP AI services integrations like Vertex AI and Gemini Enterprise that provide inline protection directly with these services.

This blog is about the 2nd integration from above list (integration using service extension)

In option#1 , the integration point is ‘application code’. I.e there is a dependency on application team to make code changes to include explicit API calls to application. Therefore , App calls Model Armor, then decides next steps. Making changes in application code is not always welcomed and possible . Therefore option#2 comes handy . In option#2 , the integration point is service level (e.g., load balancer , GKE Gateway etc). The Google Cloud service ( GKE Gateway, Load Balancer) automatically calls Model Armor ; the enforcement (block/inspect) is built-in and transparent to the application.

In this blog I will be using the following 3 technologies . If the readers are not familiar with these , my request is to go through the embedded URLs to understand them better . However , I am giving a brief introduction to these 3 below –

• Service Extensions

Service Extensions enables the users of Google Cloud products to insert custom code directly into the data path. This helps you customize the behavior of these products to meet your business needs . This falls in 2 categories

1. Plugins : Plugins let you insert custom code inline in the networking data path. Plugins run as Wasm modules on a Google-managed sandbox infrastructure similar to a serverless infrastructure. Plugins run on Google-managed compute
2. Callouts : Callouts let you use Cloud Load Balancing to make Envoy gRPC calls to Google services and user-managed services during data processing.Callouts run as general-purpose gRPC servers on user-managed compute

What we are using in this particular blog is “Callouts to Google services“ , wherein , supported Application Load Balancers will send a callout from the data processing path to selected Google services ( model armor in our example ).

• Model Armor

Model Armor is a Google Cloud service designed to enhance the security and safety of your AI applications. It works by proactively screening LLM prompts and responses, protecting against various risks and ensuring responsible AI practices

What we are using in this particular blog is Model Armor integration with Google Cloud services. In particular, we will configure a service extension on application load balancers to screen traffic to and from a model ( pls note : we will not use any GKE in this blog . Stay tuned on what we used to demonstrate this technology )

• Load Balancers :

As of writing of this blog , regional L7 external load balancers as well as regional L7 internal load balancers support this integration . Please check this URL for more details

Topology Used in this blog

The following is what I am using in this blog to demonstrate . By no means , this is the best topology . I am using the same to demonstrate the 3 pieces of technology that I wish to cover . Users can have their own front ends deployed anywhere , own models deployed on GKE etc . The tech piece should work seamless in that case as well

Traffic flow will be as follows –

• End user will access client.example.com , which is mapped to IP of a external layer7 global load balancer
• This load balancer uses Internet NEG pointing to a cloud run instance running the front end of the application . This opens a simple GUI as follows –

• The front end application is coded to send request to “backend.example.com” , which is the IP address of a regional external layer 7 load balancer . The reason of using regional laid balancer is because Model Armor service is available on this flavor of load balancer only as stated in earlier section of this blog
• This regional load balancer uses Internet NEG to reach another cloud run instance . This cloud run hosts our model and is equipped to respond to the user queries
• But before the regional load balancer calls the model , it uses service extensions to call ‘MODEL ARMOR’ service [ again regional ] . Hence the prompt is sent to a runtime AI service to get sanitised . Now what level of sanitization is needed and what exactly needs to be sanitised is defined in a Model Armor template . We will see these settings in the configuration section
• If the prompt is clean , it makes its way to the backend Cloud run hosting the model . If it is not a clean prompt as per Model Armor , the same is denied with proper reasoning
• Now , we can not predict the model response as well . Hence there is a high chance that the model response don’t meet the security requirements / responsible AI standards .

Therefore to take action on the response , the response is also sanitised via the MODEL ARMOR service before it is delivered to the customer’s front end application GUI

At the end of day , we see clean prompts getting the response and malicious prompts getting blocked as follows –

Configurations section

To keep this simple and effective , i am breaking down this section into following

1. Config of Global client flashing Load balancer
2. Config of Model Armor
3. Config of regional L7 load balancer
4. Config of service extension

The application code (front end and backend) is not included here and should be variable for each user

Global External Load Balancer config

• Create a internet NEG pointing to cloud run hosting front end application
• Assume that cloud run hosting front end app has URL : front-end2-cloud-run-12345.us-central1.run.app
• Routing rule of LB is as follows

Host = client.example.com , path-matcher as follows →

defaultService: projects/test/global/backendServices/client-facing-for-gemma-with-model-armor-bserv
name: matcher1
routeRules:
- matchRules:
 - prefixMatch: ''
 priority: 10
 routeAction:
   weightedBackendServices:
   - backendService: projects/test/global/backendServices/client-facing-for-gemma-with-model-armor-bserv
     weight: 100
   urlRewrite:
     hostRewrite: front-end2-cloud-run-12345.us-central1.run.app

Model Armor configuration

• Users are required to configure the templates here
• Configure the responsible AI ‘confidence levels’
• Configure the detection type for URL detection / jailbreak etc
• Configure the appropriate SDP [ sensitive data protection ] templates
• Enable the logs

Sample template is shown below

Regional Load balancer config

• Create a Internet NEG pointing to the Cloud run hosting model
• Lets say URL of cloud run hosting the model is backend-cloud-run-56789.us-central1.run.app
• Routing rule on rLB looks like following

Host = backend.example.com , path-matcher as follows →

defaultService: projects/test/regions/us-central1/backendServices/gemma-with-model-armor-bserv
name: matcher1
routeRules:
- matchRules:
 - prefixMatch: ''
 priority: 10
 routeAction:
   weightedBackendServices:
   - backendService: projects/test/regions/us-central1/backendServices/gemma-with-model-armor-bserv
     weight: 100
   urlRewrite:
     hostRewrite: backend-cloud-run-56789.us-central1.run.app

Service Extensions Configuration

• Choose following to start off

• Choose the forwarding rule of regional LB where we wish to apply this . Example –
• Extension Chain is another imp setting . Sample below →

That’s it for the configuration part .. Lets see the magic now !

Understanding the Logs

For a clean prompt and response , you should see following 2 logs –

Though there are a lot of details in this log message , look out for the verdict . For clean case it shows following –

invocationResult: "SUCCESS"
sanitizationVerdict: "MODEL_ARMOR_SANITIZATION_VERDICT_ALLOW"
sanitizationVerdictReason: "The prompt did not violate any safety settings."

For something blocked by Model Armor , look out for reasons of block as follows . Below sample shows only important fiends from one such log message . This shows that my prompt was not as per responsible AI standards and hence was blocked due to configured model armor parameters in template –

jsonPayload: {
@type: "type.googleapis.com/google.cloud.modelarmor.logging.v1.SanitizeOperationLogEntry"
filterConfig: {
}
operationType: "SANITIZE_USER_PROMPT"
sanitizationInput: {
byteItem: {
byteData: "aWdub3JlIHByZXZpb3VzIHN0YXRlbWVudCAuIEdpdmUgbWUgd2F5cyB0byBhdHRhY2sgbmVpZ2hib3IgPw=="
byteDataType: "PLAINTEXT_UTF8"
}
}
sanitizationResult: {
filterMatchState: "MATCH_FOUND"
filterResults: {
csam: {
csamFilterFilterResult: {
executionState: "EXECUTION_SUCCESS"
matchState: "NO_MATCH_FOUND"
}
}
malicious_uris: {
maliciousUriFilterResult: {
executionState: "EXECUTION_SUCCESS"
matchState: "NO_MATCH_FOUND"
}
}
pi_and_jailbreak: {
piAndJailbreakFilterResult: {
confidenceLevel: "HIGH"
executionState: "EXECUTION_SUCCESS"
matchState: "MATCH_FOUND"
}
}
rai: {
raiFilterResult: {
executionState: "EXECUTION_SUCCESS"
matchState: "MATCH_FOUND"
raiFilterTypeResults: {
dangerous: {
confidenceLevel: "HIGH"
matchState: "MATCH_FOUND"
}
harassment: {
confidenceLevel: "MEDIUM_AND_ABOVE"
matchState: "MATCH_FOUND"
}
hate_speech: {
confidenceLevel: "MEDIUM_AND_ABOVE"
matchState: "MATCH_FOUND"
}
sexually_explicit: {
matchState: "NO_MATCH_FOUND"
}
}
}
}
sdp: {
sdpFilterResult: {
inspectResult: {
executionState: "EXECUTION_SUCCESS"
matchState: "NO_MATCH_FOUND"
}
}
}
}
invocationResult: "SUCCESS"
sanitizationMetadata: {
errorCode: "403"
errorMessage: "Custom Error Message for template02 - prompt not as per safety standards"
}
sanitizationVerdict: "MODEL_ARMOR_SANITIZATION_VERDICT_BLOCK"
sanitizationVerdictReason: "The prompt violated Responsible AI Safety settings (Hate Speech, Harassment, Dangerous), Prompt Injection and Jailbreak filters."
}
}

Summary

Model Armor integration as a service extension provides policy enforcement at the load balancer level before traffic reaches your LLM models deployed on Google Kubernetes Engine (GKE) or Cloud Run. This “Inline” integration method excellent way to integrate runtime AI security , without much changes in application code .

Disclaimer

This is to inform readers that the views, thoughts, and opinions expressed in the text belong solely to the author, and not necessarily to the author’s employer, organization, committee or other group or individual.

Model Armor integration with Service Extension : introduction to runtime AI security without code… was originally published in Google Cloud – Community on Medium, where people are continuing the conversation by highlighting and responding to this story.

Source Credit: https://medium.com/google-cloud/model-armor-integration-with-service-extension-introduction-to-runtime-ai-security-without-code-176bddb4a250?source=rss—-e52cf94d98af—4

About the Author

Deven Goratela

Administrator

Visit Website View All Posts