Agentic Orchestration Model

Overview

Digital media operations decompose into specialized agents supervised by a central orchestrator. Agents call domain services and platform connectors; they do not hold long-lived credentials directly — services inject scoped tokens per request.

Deployment: agents run on Vertex AI Agent Engine; models are served from Vertex AI Model Garden (see GCP deployment topology — AI & agent runtime). Cloud Run hosts the orchestrator control plane, connectors, and HITL API.

Agent hierarchy

Three layers — each with a defined model tier (see Model catalog):

Agent roster

State machine (client lifecycle)

Orchestration patterns

1. Plan–execute separation

• Agents never execute spend against an unapproved plan version.
• Plan document is immutable once approved; changes create plan_vN+1.

2. Guardrails (hard limits)

Examples (exact values configured per tenant):

• Max daily budget delta without approval: e.g. 20%
• Blocked actions: delete conversion actions, change billing account
• Required checks before launch: tracking health = green, feed errors = 0 critical

3. Tool access

Each agent has an allowlist of tools (service APIs):

Onboarding Agent → onboarding.createAccount, verification.request, bm.linkAsset
Plan Agent → planning.draftMaster, planning.draftTrack, planning.draftEvent, planning.validate
Plan Revise Agent → planning.revise, planning.diffManifest, planning.recommendRevise (per `track_id`)
Execution Agent → execution.applyPlan, execution.pauseCampaign, execution.syncManifest
Optimization Agent → optimization.propose, optimization.apply, optimization.computeDrift
Reporting Agent → reporting.changelog, reporting.planDrift, reporting.recommendRevise

4. Observability

Every agent run produces:

• run_id, agent, tenant_id, track_id, plan_version
• Input context hash (no PII in logs)
• Tool calls and outcomes
• Human escalation reason if blocked

QC correction loops are logged on every pass/fail and correction attempt — not only when A8 fires. See QC loop telemetry.

5. Per-task QC gates and correction loops

Every mutable or client-visible main task runs through a checker sub-agent (Q/A gate) before the orchestrator marks it complete or applies platform mutations. Read-only tasks (e.g. daily KPI digest) skip QC unless anomaly is detected.

Main task → QC checker pairing

Compliance QC may chain to qc.compliance.escalate (Sonnet) only on low-confidence or high-risk vertical — not on every run.

Loop limits (hard caps — no infinite loops)

Policy: agents never loop indefinitely. Every loop type has a hard ceiling enforced by the orchestrator in code — not by prompt instruction alone. When a ceiling is hit, the run stops immediately, state is frozen, and a red flag is raised for Kobi ops (see below). No silent retries, no "try one more time" without a new run_id and human acknowledgment.

Configurable per tenant in loop_policy (implementation phase); defaults above are maximums — tenants may be stricter, never looser without admin override.

Red flag on loop exhaustion

When any loop limit is exceeded or the global run_id ceiling is hit:

1. Stop — no further agent or tool calls for that run_id; partial platform mutations rolled back or marked needs_review
2. Emit agent.loop.exhausted with tenant_id, track_id, agent, loop_type, attempt_count, last_qc_failure, run_id
3. HITL ticket (A8) — appears in Human Touch inbox with red priority, SLA timer, structured summary, and recommended actions; full step trace in System Ops
4. Ops alert — notify on-call / ops channel (email, Slack, or PagerDuty — TBD); include tenant, task, and link to ticket
5. Client impact guard — block spend increases, launches, and optimization applies tied to that run_id until A8 is resolved or explicitly overridden (A6) by admin
6. No auto-restart — same logical task requires human to approve a new run_id or take manual action; prevents loop-until-lucky behavior

QC checker receives: main agent output JSON, approved plan_version, relevant playbook slice, and only the fields needed to verify — not the full conversation history.

QC loop telemetry

Deterministic orchestrator logging (not LLM-generated summaries) records every QC gate and correction loop so ops can answer: which agents fail most, on which models, in which tasks, with which inputs/docs?

Emit on every QC invocation (agent.qc.result):

Emit on each correction iteration (agent.qc.loop):

On A8 exhaustion, agent.loop.exhausted includes last_qc_failure plus qc_loop_trace_id → full ordered list of agent.qc.result / agent.qc.loop rows for that run_id.

BigQuery tables (partitioned by event_date, clustered by tenant_id, main_task_id):

Example ops queries (illustrative):

• QC fail rate by main_agent + model_main (rolling 7d)
• Top failure_codes for qc.compliance on vertical.health
• Runs where playbook_versions.qc changed and fail rate spiked same day
• opt.cycle loops where context_refs include stale manifest_slice_id

Dashboard (System Ops) — QC health and statistics: leaderboard, model breakdown, drill-down to input_slice_uri / qc_feedback_uri. Human Touch shows breach alerts only, not the statistics console.

Feeds automatic model promotion — rolling 24h fail rate per main_task_id from agent_qc_results drives model promotion / demotion; no manual spreadsheet.

QC success threshold alerts (80% floor)

A deterministic alert job (not an agent) evaluates rollups from agent_qc_results / agent_qc_failures_rollup and fires when a task or subtask cannot maintain the 80% success floor. Below that floor means the pairing is under-performing — prompts, models, playbooks, or context must be optimized, not ignored.

Scope grains (evaluated independently):

Metrics (both computed per grain, rolling window default 24h):

Alert rule (default):

if sample_count >= min_sample
   AND (first_pass_rate < 0.80 OR eventual_success_rate < 0.80):
     emit agent.qc.threshold.breached
     open optimization ticket (engineering — not client HITL)
     run auto-optimization actions (below)

On breach (agent.qc.threshold.breached) — payload includes grain, main_task_id, qc_task_id, main_agent, first_pass_rate, eventual_success_rate, sample_count, top_failure_codes[], model_breakdown, playbook_versions_mode, sample_run_ids[].

Recovery: alert clears when the same grain stays ≥80% for a full 24h cooldown window (hysteresis — avoids flap).

Optimize what? Use breach payload to pick the lever:

# Illustrative — implementation phase
qc_telemetry_policy:
  log_every_qc_invocation: true
  store_full_input_in_gcs: true      # BQ = refs + hashes only
  retention_days_bq: 400
  retention_days_gcs: 90             # extend on A8 / compliance hold
  rollup_views: [daily, weekly]

qc_threshold_policy:
  success_floor: 0.80                # alert below 80%
  window_hours: 24
  min_sample_global: 20
  min_sample_tenant: 10
  recovery_cooldown_hours: 24
  auto_promote_on_breach: true
  auto_bump_thinking_on_breach: true

Cost Guard — deterministic spend circuit breaker (not AI)

A separate deterministic service — not an agent, not LLM-mediated — sits in front of every Vertex call and enforces hard spend limits per run_id. It uses the per-task cost catalog and a versioned pricing table (model → $/1M input/output) loaded from config. No model chooses whether to stop; the math does.

Why: loop caps limit iterations but not token blowups within a step (thinking tokens, tool-loop context re-send). Cost Guard catches runaway spend even when loop counts are still legal.

Single cost formula (same for catalog planning, run budget, and live metering):

step_cost = (input_tokens / 1e6 × price_in[model])
          + (output_tokens / 1e6 × price_out[model])   # output includes thinking tokens

Only the token source differs between estimate and actual:

Estimate at run start (when router dispatches a gated task):

# Per planned step — same formula as catalog "Cost / run" column
step_estimate = (catalog[task].in_tok / 1e6 × price_in[model])
              + (catalog[task].out_tok / 1e6 × price_out[model])

estimated_run_cost = Σ step_estimate over planned steps   # e.g. opt.cycle + qc.spend
                   × loop_buffer                          # default 1.08 for gated tasks

catalog[task].expected_cost in the table is exactly this math pre-computed; Cost Guard may read either the USD column or recompute from In/Out + pricing_table_version — they must match.

Stored on run_id before the first LLM call. Estimate is immutable for that run unless admin resets (A6).

Actual after each LLM response — Vertex returns token counts on every generate call:

# Map usage_metadata fields (names vary slightly by SDK; normalize in Cost Guard)
input_tokens  = prompt_token_count
              + cached_content_token_count   # billed at cached input rate if applicable
output_tokens = candidates_token_count
              + thoughts_token_count         # 3.5 Flash thinking — billed at output rate

step_cost = (input_tokens / 1e6 × price_in[model])
          + (output_tokens / 1e6 × price_out[model])
actual_run_cost += step_cost

Cost Guard logs per step: run_id, step_index, model, input_tokens, output_tokens, step_cost_usd, actual_run_cost_usd, estimated_run_cost_usd, ratio. Ledger and BigQuery use API-reported tokens only — never prompt guesses or agent claims.

Optional pre-call check (still deterministic): before forwarding a request, Cost Guard may count input tokens in the outbound payload (Vertex tokenizer or count_tokens API) to warn if a single call's prompt alone exceeds 2× catalog.in_tok for that step — does not replace the 3× run-level trip; catches context blowups early.

Trip rule (default):

if actual_run_cost >= trip_multiplier × estimated_run_cost:
    TERMINATE  # default trip_multiplier = 3.0

On terminate (cost.guard.tripped):

1. Hard stop — orchestrator cancels in-flight agent work; Cost Guard rejects subsequent Vertex requests for that run_id with 403 cost_guard_tripped
2. Rollback policy — same as A8: rollback or mark needs_review for any partial platform mutations
3. HITL ticket (A9) — distinct from A8 (loop exhaustion): shows estimated vs actual USD, per-step token breakdown, model used, trip multiplier
4. Ops alert — red notification to on-call with tenant, run_id, task_id, actual / estimated ratio
5. No auto-restart — new run_id requires human acknowledgment; optional admin may raise estimate ceiling (A6) with audit log
6. Emit cost.guard.tripped on event bus

Cost Guard cannot be bypassed by agents, prompts, or orchestrator retries. Only admin override (A6) with logged reason may raise the multiplier or authorize a new run with a higher estimate.

# Illustrative — implementation phase
cost_guard_policy:
  trip_multiplier: 3.0          # stop at 3× estimate
  loop_buffer_in_estimate: 1.08 # baked into estimated_run_cost
  pricing_table_version: "2026-06"  # must match catalog revision
  tenant_daily_cap_usd: null    # optional; e.g. 50.00
  block_on_trip: true           # always true in prod

What monthly costs include

Per-task prices in the cost catalog are single successful runs. Monthly totals explicitly count main + QC as separate line items (e.g. opt.cycle + qc.spend).

Correction loops are budgeted separately:

Composite unit examples (main + QC + expected loop overhead):

Monthly profile totals use separate main/QC counts; add ~6–8% for correction-loop overhead on gated tasks (lower with 3.5 QC), or use composite units above when estimating.

Event bus (conceptual)

Async events (illustrative names):

Implementation may use Pub/Sub, Kafka, or equivalent — TBD with stack.

Model strategy and routing

Principles

1. Capable, never cheap-at-all-costs — sub-agents use the cheapest tier that still passes QC for that task class. If QC failure rate rises, auto-promote tier (see playbook §4).
2. Gemini-first on Vertex — native tool calling, context caching, Agent Engine integration, single billing/IAM surface.
3. gemini-3.5-flash as the primary agentic surface — GA model built for agentic execution, tool loops, and sub-agent deployment. Prefer one capable model + thinking_level per task over mixing many smaller models for agent work. See 3.5 Flash thinking matrix.
4. Compliance QC on Gemini 3.5 Flash — default compliance gate uses gemini-3.5-flash at high thinking: GA, strong reasoning, structured policy checks, ~50% cheaper than Sonnet on Vertex. Cross-model is optional escalation, not the default (see below).
5. Cross-model escalation (optional) — claude-sonnet-4-6 only when compliance QC is ambiguous (low confidence), vertical risk flag is health / education, or a human requests a second opinion. Reduces cost while keeping an independent check for edge cases.
6. Opus / Pro escalation only — claude-opus-4-8 or gemini-3.1-pro-preview with extended thinking reserved for human-triggered or router-scored "hard" tasks.
7. No autonomous loops on Lite-only models — gemini-2.5-flash-lite is allowed only for deterministic structured I/O (JSON transform, field mapping) with schema validation; not for multi-step tool chains.

Model tiers (summary)

3.5 Flash thinking matrix (recommended default)

Gemini 3.5 Flash is more capable than 3.1 Flash-Lite and 3 Flash Preview for agentic work: GA, stronger tool use, thought preservation across turns, and near-Pro quality at Flash-tier cost. Google positions it explicitly for sub-agent deployment and rapid agentic loops.

Does using more 3.5 with different thinking_level per task make sense? Yes — for any task that calls tools, blocks mutations, or runs a QC correction loop. Use one endpoint, many depths instead of jumping to Pro or mixing preview models. Reserve Flash-Lite for high-volume non-agentic routing and batch digests only.

Quality vs cost tradeoff: Putting gated QC on 3.5 Flash (vs a cheaper Flash-Lite QC mix) raises LLM spend by roughly $4–6 / client / month on Standard, but buys fewer correction loops (8% buffer vs ~12%) and a higher first-pass QC rate — usually net-positive once human-review time is counted. Do not run the router or batch digests on 3.5; that would 3–6× those rows for no quality gain. Expected Standard total is **$25.60 / client / month** (see cost catalog and cost band).

Thinking levels (gemini-3.1-flash-lite only)

For T0 tasks on Flash-Lite per Google's guidance:

Model catalog (Vertex AI, June 2026)

Source: Vertex AI Generative AI pricing — Global endpoint, standard (non-batch) rates per 1M tokens, prompts ≤200K. Cached input shown where available. Verify before implementation; preview models may change.

Primary — Gemini (Google)

Secondary — Anthropic on Vertex (QC & cross-check)

Optional — Meta Llama on Vertex (batch / cost experiments)

Models we do not default to

Per-task LLM cost catalog

Vertex global standard pricing. Cost formula:

cost = (input_tokens / 1M × input_rate) + (output_tokens / 1M × output_rate)

Token assumptions are planning defaults for a single successful run of that step. 3.5 Flash output column includes thinking tokens billed at output rate. QC checker runs are separate line items — gated main tasks always incur main + QC (see pairing table). Correction loops add ~6–8% on gated tasks with 3.5 QC (see loop limits). Context caching on system prompts typically reduces input cost 30–50% in steady state — figures below are without cache (conservative).

† For gemini-3.5-flash, Out tok = visible output + thinking tokens (billed at output rate per Vertex pricing).

Batch API for scheduled / non-urgent tasks

Vertex Batch API = 50% discount on eligible Gemini models, in exchange for async completion (up to ~24h). Use it only for scheduled, non-blocking work — never for real-time routing, gated QC, platform mutations, interactive planning, or anomaly alerts (<1h SLA).

Reality check: batch trims the reporting + feed-sweep slice only (~2–4% of total). The real cost drivers — optimization cycles, execution mutations, QC gates — are real-time and stay at standard rates. Batch is worthwhile (free 50% on eligible tasks) but is not where the savings concentrate; context caching matters far more.

Monthly task volume & LLM cost by client profile

Illustrative steady-state volumes (after onboarding month). Costs assume multiple concurrent plan tracks per client (always-on, branding, engagement, special-day events) — not a single monolithic plan. See plan tracks.

QC pairing: each plan.* row includes + qc.plan ($0.077) unless noted.

Profile A — Starter (Google + Meta, low activity)

~$15–25K/mo media spend, 2 active tracks (always-on + seasonal/event), optimization 2–3×/week.

• With context caching (40% input savings): **$6.40–6.90 / client / month**
• With caching + batch on scheduled reports (saves $0.19): **$6.25–6.70 / client / month**

Profile B — Standard (Google + Meta + GA4 + CRM, active optimization)

~$50–150K/mo media spend, 3–4 active tracks (always-on, branding, engagement, seasonal), daily optimization.

• With context caching (40% input savings): **$19–22 / client / month**
• With caching + batch on scheduled reports (saves $0.34): **$18.70–21.70 / client / month**

Profile C — Ecommerce (feeds, catalog, high optimization)

~$150K+/mo media spend, 4–6 active tracks (+ flash sales, catalog pushes), hourly pacing.

• With context caching (40% input savings): **$27–31 / client / month**
• With caching + batch on scheduled reports + feed sweeps (saves $0.93): **$26.50–30 / client / month**

Onboarding month (one-time, any profile)

Portfolio-level estimate

Assumes multi-track planning volumes above; actuals scale with count of active event/sale plans per client. Batch column reflects scheduled reporting + feed sweeps only — the marginal step beyond caching is small because cost concentrates in real-time optimization, execution, and QC.

Note: These are Vertex LLM inference only — not media spend, Kobi management fees, Cloud Run, BigQuery, or platform API costs. Log actual run_id token usage to BigQuery per tenant_id for invoice-grade allocation if pass-through is contracted. See Billing & invoicing for client-facing monthly invoices (media spend + fees).

Realistic cost band (treat point estimates as the expected case)

The per-task tokens above are single-shot, expected-case assumptions. Two factors can push real cost up materially; size budgets with a band, not a point:

Even the high case stays a tiny fraction of media spend and Kobi fees — but design for the high band, then measure real run_id usage in a pilot to replace these assumptions.

Actual cost is dominated by optimization frequency and multi-turn tool loops — the playbook below targets both.

Token-efficiency playbook

Goal: maximum quality per token — structured outputs, cached context, minimal re-prompting.

1. Prompt architecture

2. Context budget

Trim platform API responses to fields the schema requires. Store full responses in GCS/BQ; pass URI to reporting agents only.

3. Tool-call discipline

• Plan tools explicitly — max 3 tools per sub-agent invocation unless router scores complex.
• Idempotent tools — safe to retry; orchestrator dedupes by run_id.
• No LLM in hot path for math — budget splits, bid deltas, % thresholds computed in code; LLM proposes intent, code validates numbers.
• Batch non-urgent work — route all scheduled, non-blocking tasks (daily/weekly/monthly digests, per-track drift reports, scheduled feed sweeps) through Vertex Batch API for a 50% discount on eligible Gemini models. Never batch real-time routing, gated QC, mutations, or anomaly alerts. See Batch API for scheduled tasks.

4. Model promotion / demotion (automatic)

5. Playbook registry (per tenant / vertical)

Versioned artifacts cached in Vertex context cache:

Router selects playbook IDs; agents never inline full vertical rules in every prompt.

6. Output quality without extra tokens

• Two-pass by design: every gated main task → paired QC checker (see §5). Max 2 correction loops, then human. No third QC pass unless compliance escalates to Sonnet.
• Confidence gate: if router confidence < 0.85 on classification, escalate to medium thinking or T1 model — cheaper than a failed tool loop + retry.
• Log token usage per run_id, agent, model, thinking_level → BigQuery for cost attribution per tenant.
• Log every QC loop — agent.qc.result + agent.qc.loop with agent, model, task, playbook versions, and input/doc refs (see QC loop telemetry).

Failure handling

Related documents

• GCP deployment topology — AI & agent runtime
• System overview
• Human control plane
• System Ops Dashboard
• Lifecycle: optimization
• Vertex AI pricing (verify at implementation)