Skip to content
SunnyWriteUps
Go back

how to build production grade multi-agent systems on Google Cloud

Edit page

Building Enterprise AI Agent Teams with Google’s Antigravity SDK


Angular Agentic Engineering

Building a production-grade multi-agent system with the Google Antigravity SDK (pip install google-antigravity) separates agent reasoning (system instructions and tools) from governance (declarative policies and observability hooks).

Based on the reference architecture for the Financial Audit Agent Team on Google Cloud (BigQuery, Cloud Storage, Cloud Logging, Cloud Trace, and Cloud Run), the following blueprint outlines how to structure, secure, and deploy the system, along with how the pattern adapts to Insurance Claims Adjudication and HR Onboarding Compliance.


1. Multi-Agent Architecture & Least-Privilege Delegation

Instead of giving a single monolithic agent access to all databases, storage buckets, and write APIs, the architecture enforces strict separation of concerns across a 4-phase workflow (trigger $\rightarrow$ gather $\rightarrow$ reconcile $\rightarrow$ report):

Delegation Tool Implementation (tools/delegation_tools.py): In the Antigravity SDK, subagents are spawned inside standard Python functions with type hints and docstrings, keeping subagent context windows isolated so raw PDF data doesn’t bloat the Orchestrator’s context:

from google.antigravity import Agent
from tools.bigquery_tools import query_vendor_transactions
from tools.gcs_tools import list_invoices_in_gcs, read_invoice_from_gcs
from policies.audit_policies import get_readonly_policy

def delegate_to_researcher(quarter: str, project_id: str) -> str:
    """Spawns a read-only Data Researcher subagent to fetch pending transactions for a quarter."""
    researcher = Agent(
        name="data_researcher",
        model="gemini-3.5-flash-lite",
        system_instruction="You are a financial data researcher. Query BigQuery for pending vendor transactions and list matching GCS invoice paths.",
        tools=[query_vendor_transactions, list_invoices_in_gcs],
        policies=get_readonly_policy(),
    )
    response = researcher.run(f"Fetch all pending transactions and available invoice files for {quarter} in project {project_id}.")
    return response.text

2. Model Tiering Strategy

Right-sizing models to each agent’s cognitive load minimizes latency and token spend across multi-turn agent loops:


3. Safety Policies (dev / staging / prod Modes) & Guardrails

The Antigravity SDK uses Decide hooks (blocking, read-only interceptors) to enforce runtime policies before any tool executes. Combined with Least-Privilege GCP IAM Service Accounts, this creates defense-in-depth: both the SDK policy layer and GCP IAM must permit an action.

from google.antigravity.hooks import pre_tool_call, Decision

@pre_tool_call
def human_compliance_gate(context):
    """Pauses execution and escalates discrepancies > $1,000 for human review."""
    if context.tool_name == "write_audit_result":
        discrepancy = abs(float(context.tool_args.get("discrepancy_amount", 0.0)))
        if discrepancy > 1000.0:
            return Decision.require_approval(
                reason=f"Discrepancy of ${discrepancy:,.2f} exceeds $1,000 threshold. Escalating to Compliance Officer."
            )
    return Decision.allow()

4. Observability Hooks (Cloud Logging & Cloud Trace)

Using Inspect hooks (non-blocking, read-only), you can attach compliance telemetry across the agent lifecycle (pre_turn, post_turn, post_tool_call, on_tool_error) without cluttering business logic:

from google.antigravity.hooks import post_tool_call, post_turn
import google.cloud.logging

@post_tool_call
def audit_trail_logger(context):
    """Streams an immutable compliance record of every tool invocation to Cloud Logging."""
    log_payload = {
        "agent_name": context.agent_name,
        "tool_name": context.tool_name,
        "tool_args": context.tool_args,
        "execution_ms": context.duration_ms,
        "status": "SUCCESS" if not context.error else "ERROR",
    }
    cloud_logger.log_struct(log_payload, severity="INFO")

@post_turn
def track_token_consumption(context):
    """Captures prompt, candidate, cached, and thinking tokens per turn via usage_metadata."""
    usage = context.response.usage_metadata
    trace_span.set_attribute("antigravity.tokens.total", usage.total_token_count)
    trace_span.set_attribute("antigravity.tokens.thinking", usage.thinking_token_count)

5. Provisioning, Evaluation & Cloud Run Deployment

Step 1: Provision GCP Infrastructure & Seed Discrepancies

export PROJECT_ID="your-gcp-project-id"
gcloud config set project $PROJECT_ID

gcloud services enable bigquery.googleapis.com storage.googleapis.com \
  logging.googleapis.com cloudtrace.googleapis.com aiplatform.googleapis.com run.googleapis.com

# Create BigQuery dataset & tables
bq mk --dataset $PROJECT_ID:financial_audit
bq mk --table $PROJECT_ID:financial_audit.vendor_transactions \
  vendor_id:STRING,vendor_name:STRING,invoice_num:STRING,amount:FLOAT64,currency:STRING,tax_rate:FLOAT64,status:STRING,quarter:STRING,transaction_date:DATE

# Create GCS Bucket & upload generated sample PDFs
gsutil mb -l us-central1 gs://$PROJECT_ID-audit-invoices
python scripts/generate_sample_invoices.py
gsutil -m cp data/invoices/*.pdf gs://$PROJECT_ID-audit-invoices/Q3/

Step 2: Run the Evaluation Suite (eval/run_eval.py)

Before deploying, run deterministic evaluations against planted test discrepancies (e.g., overbilled line items, mismatched tax rates, missing invoices, and $> $1,000$ escalation triggers) to verify precision and policy adherence:

python eval/run_eval.py --project-id=$PROJECT_ID

Step 3: Deploy to Cloud Run

Package the agent behind a lightweight Flask/Gunicorn server (server.py) using a dedicated least-privilege service account (audit-agent-sa):

gcloud run deploy financial-audit-agent \
  --source . \
  --region us-central1 \
  --set-env-vars PROJECT_ID=$PROJECT_ID \
  --timeout 300 \
  --memory 1Gi \
  --service-account audit-agent-sa@$PROJECT_ID.iam.gserviceaccount.com \
  --no-allow-unauthenticated

Invoke the production service with an IAM identity token:

SERVICE_URL=$(gcloud run services describe financial-audit-agent --region us-central1 --format 'value(status.url)')
curl -X POST "$SERVICE_URL/audit" \
  -H "Authorization: Bearer $(gcloud auth print-identity-token)" \
  -H "Content-Type: application/json" \
  -d '{"quarter": "Q3"}'

6. Adapting the Blueprint: Bonus Sample Applications

The exact same 4-agent pattern (Orchestrator + DB Researcher + Document Analyzer + Adjudication/Compliance Engine + Policy Gate) ports directly to the two companion domain architectures in the samples/ directory:

  1. Insurance Claims Adjudication (samples/insurance-claims-agent/):
  1. HR Onboarding Compliance (samples/hr-onboarding-agent/):

Edit page
Share this post on:

Previous Post
Design MCP Tools
Next Post
The AI Factory How Organizations Move Beyond Science Projects to Scale AI

Comments