AI Harness Design for Game LiveOps

Anthropic recently published "Harness Design for Long-Running Application Development", documenting how multi-agent architectures enable Claude to autonomously build complete applications over multi-hour sessions. The results are striking: a solo agent spent 20 minutes and $9 to produce a broken game editor, while a full harness spent 6 hours and $200 to produce a polished, functional one with 16 features across 10 sprints.

This isn't just an AI engineering story. The patterns Anthropic describes -- separating generation from evaluation, using sprint contracts, managing context decay -- map remarkably well to the two hardest problems in live-service game operations: scaling content production and optimizing monetization.

Let's break down how.

The Harness Pattern, Briefly

Anthropic's approach borrows from GANs: a Generator produces work, and a separate Evaluator judges it. Add a Planner that expands requirements into detailed specs, and you get a three-agent system:

• Planner Agent: Turns a brief prompt into a comprehensive product specification
• Generator Agent: Implements features in sprints, self-evaluates, then hands off to QA
• Evaluator Agent: Tests the running application through Playwright, verifying against pre-agreed "Sprint Contracts"

Four principles make this work:

1. Self-Evaluation Bias: When agents evaluate their own work, they "respond by confidently praising the work -- even when quality is obviously mediocre." Generation and evaluation must be separated.
2. Context Decay: As context windows fill, models lose coherence and prematurely wrap up. Structured handoffs with clean context resets outperform compaction.
3. Specialized Agents > Generalists: Role separation enables targeted tuning for each function.
4. Assumptions Decay: Every harness component encodes an assumption about what the model can't do alone. As models improve, scaffolding must be re-examined.

"Every component in a harness encodes an assumption about what the model can't do on its own."

The Two Problems That Define Game LiveOps

If you run a live-service game, two challenges tower above everything else:

Content volume + Revenue optimization

Content volume is typically bottlenecked by art production. Revenue optimization requires both data-driven economy tuning and gameplay sophistication that creates organic purchase conversion points.

These aren't independent problems -- they're tightly coupled. More content means more engagement surface, which means more monetization opportunities. But only if the content is balanced, the difficulty curve is right, and the economy doesn't leak value in the wrong places.

The Anthropic harness pattern offers a framework for tackling both systematically.

1. Content Pipeline: Generator-Evaluator for Sustainable Production

The Bottleneck

Every live-service team knows this pain: sustaining LiveOps requires a continuous stream of events, assets, and content variations. Art is almost always the bottleneck. You can hire more artists, outsource more work, maximize every production channel -- but the fundamental constraint remains: content takes time to produce, and LiveOps doesn't wait.

The standard playbook: secure baseline content volume (especially events), then explore variations on existing content to stretch the production budget.

Mapping the Three-Agent System

Planner Agent = Content Strategist

[Event Calendar + Player Data + Revenue Targets] → Planner
→ Auto-generate content specs for the next 4-week cycle
→ Prioritized art asset requirements
→ Outsource vs. in-house allocation optimization
→ Variation opportunities flagged for existing content

When AI generates the first draft of an event specification, designers shift from blank-canvas creation to validation and refinement. This is exactly what Anthropic's Planner does: "expands brief prompts into comprehensive product specs, emphasizing high-level design over granular implementation details."

Generator Agent = Asset Production Pipeline

[Content Spec] → Generator
→ AI-generated asset drafts (backgrounds, UI elements, item variants)
→ Art team focuses on polish and brand consistency
→ Batch production in sprint cadence

When art is the bottleneck, having an AI Generator produce the first 70% of an asset while artists focus on the remaining 30% can dramatically increase throughput. Content variations -- color palettes, seasonal themes, rarity tiers -- are precisely where generative AI excels.

This mirrors how studios like King and Supercell already approach data-driven content: small teams with heavy automation. The harness pattern formalizes what their best teams do intuitively.

Evaluator Agent = Quality Gate

[Generated Content] → Evaluator
→ Visual consistency check against existing asset library
→ In-engine rendering test (Playwright equivalent for game clients)
→ Player segment response prediction
→ Economy impact assessment

Anthropic's key insight about Self-Evaluation Bias is critical here: when the team that creates content also evaluates it, quality assessments skew optimistic. A structurally separated Evaluator -- driven by data, not ownership -- catches what creators miss.

Translating Grading Criteria

Anthropic used four weighted criteria for frontend evaluation. Here's the game content equivalent:

The Evaluator's job isn't subjective approval -- it's threshold-based verification against these criteria. As Anthropic puts it: "If any one fell below it, the sprint failed."

2. Revenue Optimization: Multi-Agent Economy Balancing

A. Currency Balance: The Biggest Lever

In most live-service games, the premium currency economy is the single biggest revenue driver. Getting the earn-spend balance right is everything. Too generous, and players never need to purchase. Too restrictive, and players churn before converting.

The challenge: teams often default to conservative adjustments because the downside risk of breaking the economy feels larger than the upside of optimizing it. This leads to a slow drift toward suboptimal equilibria -- the "boiling frog" problem of game economics.

Game economy management has evolved through distinct phases:

• Spreadsheet era (pre-2015): Manual tuning, playtest intuition, post-launch hotfixes
• Analytics-driven (2015-2022): Telemetry dashboards, A/B testing, dedicated data science teams
• ML-augmented (2022-present): Simulation environments, reinforcement learning, causal inference models, digital twins

The harness pattern pushes this evolution further by structuring how these ML tools interact.

Generator-Evaluator for Economy Tuning:

Generator (Balance Simulator)
├── Premium currency earn/spend simulation
├── Secondary reward system optimization scenarios
├── Event economy distribution modeling
└── A/B test scenario auto-generation

Evaluator (Revenue Validator)
├── Revenue impact prediction (primary currency → secondary rewards → events)
├── Churn rate simulation across player segments
├── LTV impact analysis by cohort
└── Competitive benchmark validation

The key concept is Anthropic's Sprint Contract applied to economy changes. Before any balance change ships, define the contract:

interface BalanceSprintContract {
  target: {
    arpdau_change: ">= +5%";
    d7_retention_change: ">= -0.5%";  // acceptable churn tolerance
    payer_conversion_rate: ">= +2%";
  };
  scope: "primary_currency" | "secondary_rewards" | "event_economy";
  rollback_trigger: "d1_retention < 35% OR revenue_drop > 15%";
}

This structurally solves the "conservative action" problem. When the Evaluator judges against a contract rather than gut feeling, bold decisions become defensible. The data says go or no-go -- not a committee's risk aversion.

CCP Games (EVE Online) employs a full-time economist for exactly this kind of rigor. The harness pattern democratizes that approach: you don't need a PhD in econometrics if you have an AI Evaluator stress-testing every proposed change against explicit success criteria.

B. Difficulty Sophistication: Context-Aware Challenge Systems

The second revenue problem is subtler: when gameplay is too uniform, players plateau instead of converting. If the challenge distribution is flat -- every task roughly the same difficulty -- there are no natural pressure points where spending feels valuable.

This problem maps directly to Anthropic's Context Decay concept.

Reframing Through Agent Design:

As a player's game state accumulates (longer playtime, more resources, more completed content), the experience converges toward "comfortable mediocrity" -- the gameplay equivalent of an AI losing coherence as its context window fills. The solution is the same:

Structured difficulty resets -- just as Anthropic uses context resets to restore agent coherence, gameplay needs strategic reset points that re-engage players with fresh challenge.

Player State Monitoring (Planner)
├── Consecutive success streak tracking
├── Resource surplus detection
├── Session pattern analysis (frequency, duration, time-of-day)
└── Purchase history and conversion point analysis

Challenge Generation (Generator)
├── High-difficulty task injection (rare resource requirements, compound conditions)
├── Resource demand spikes that stress current holdings
├── Multi-constraint challenges (several scarce resources simultaneously)
└── Time-limited challenges that create urgency

Hurdle Calibration (Evaluator)
├── Helper item / hint timing optimization
├── Random bonus appearance probability tuning
├── Per-player frustration threshold estimation
└── Purchase conversion probability optimization

The Key Insight: Build the Inverse of Easy-Mode Systems

Most games already have systems that serve returning or struggling players easier content to reduce churn. The inverse is equally important -- and equally systematic:

IF player.consecutive_successes >= N AND player.resource_surplus > threshold:
    → Increase high-difficulty challenge probability
    → This is a "purchase conversion point"

IF player.high_difficulty_failures >= M:
    → Increase helper item / bonus appearance rate
    → Provide contextual hints
    → This is a "churn prevention point"

EA's patented Dynamic Difficulty Adjustment system drew controversy precisely because it connected difficulty to monetization without transparency. The harness approach is different: the Evaluator enforces explicit constraints (fairness bounds, maximum frustration thresholds, designer-defined guardrails) that prevent the Generator from optimizing a single metric at the expense of player experience.

This is Anthropic's "Specialized Agents beat Generalists" principle applied to games. Instead of one universal difficulty curve, specialized agents orchestrate player experience situationally -- one for challenge generation, one for calibration, one for monitoring, each tunable independently.

3. The Full Architecture: Game LiveOps Harness

Combining content production, economy balancing, and difficulty tuning into a unified harness:

┌──────────────────────────────────────────────────────┐
│               GAME LIVEOPS HARNESS                   │
├──────────────────────────────────────────────────────┤
│                                                       │
│  ┌───────────┐   Sprint     ┌───────────────┐        │
│  │  PLANNER  │   Contract   │  GENERATOR    │        │
│  │           │─────────────▶│               │        │
│  │ Content   │              │ Content prod. │        │
│  │ Strategy  │              │ Balance sim.  │        │
│  │ Roadmap   │              │ Difficulty    │        │
│  └───────────┘              └───────┬───────┘        │
│                                     │                 │
│                             ┌───────▼───────┐        │
│                             │   EVALUATOR   │        │
│                             │               │        │
│                             │ Revenue check │        │
│                             │ Retention     │        │
│                             │ Quality gate  │        │
│                             │ Fairness      │        │
│                             └───────┬───────┘        │
│                                     │                 │
│                             Pass? ──┤── Fail?         │
│                             ▼       │    ▼            │
│                          DEPLOY   ITERATE             │
│                                                       │
└──────────────────────────────────────────────────────┘

The Natural Mapping

The harness roles map to game development functions more cleanly than you might expect:

Why the Cost Is Justified

In software, shipping a broken feature means a hotfix. In live-service games, shipping a broken economy means player exodus, revenue collapse, and community trust damage that takes months to repair. The harness investment isn't just justified -- it's cheap insurance.

Execution Priority

Start with what directly impacts revenue, then expand:

1. Immediate: Economy balance Evaluator -- simulate premium currency flows, automate A/B test design and analysis
2. Week 2-3: Difficulty Generator-Evaluator -- prototype context-aware dynamic challenge system
3. Month 1: Content production pipeline -- AI asset generation with quality gate integration
4. Ongoing: Re-examine scaffolding as models improve (Assumptions Decay principle)

Final Thought

The biggest message from Anthropic's harness design:

"Increased model capability expands interesting harness combinations rather than eliminating them."

The same applies to game operations. As AI capabilities grow, the range of automatable LiveOps functions -- content generation, economy simulation, difficulty calibration, QA testing -- expands with it. Unity's ML-Agents, Ubisoft's La Forge lab, and EA's SEED research group are all pushing in this direction.

The key is building the structure first: separate Generator from Evaluator. Define Sprint Contracts with explicit thresholds. Let data drive decisions, not committee risk-aversion.

If conservative action has been the problem, the answer isn't reckless action. It's structured boldness. The harness provides that structure.