Federated from workspace · PRD-001 · Dental Clinic Revenue Operating System/docs/technical/ZAIXOS_AGENT_BENCHMARK_PLATFORM.md Do not edit canonical truth here — update the source repo, then re-run npm run docs:sync.

---

ZAIXOS Agent Benchmark & Evaluation Platform

Status: Implemented (2026-07-05)
Authority: App\Modules\AI\Application\Benchmark
Product: PRD-001 (prd-001)

Purpose

Enterprise AI Quality Engineering infrastructure for Clinic Agent. This is an evaluation asset layer — not runtime routing, not a new orchestrator, not a platform redesign.

Frozen: PL-003, PL-004, PL-005, PL-006 — benchmark integrates with existing Intent → Signals → CapabilityMatcher → PlatformGoalPlanner pipeline only.

Architecture

Benchmark Dataset (JSON + DB)
        ↓
ClinicAgentBenchmarkRunner
        ↓
Conversation Behavior Engine (social exit detection)
        ↓
ClinicIntentUnderstandingEngine (LLM or fallback per config)
        ↓
ClinicIntentSignalMapper → signals
        ↓
CapabilityMatcher (PL-003)
        ↓
PlatformGoalPlanner (PL-003)
        ↓
ClinicAgentBenchmarkMockExecutor (no business tool side effects)
        ↓
ClinicAgentBenchmarkEvaluator
        ↓
ClinicAgentBenchmarkMetricsCalculator → Report
        ↓
ai_benchmark_runs + ai_benchmark_results

Dataset format

Each entry contains:

Field	Description
id	Unique benchmark ID
language	en, ar, tr, …
category	Benchmark taxonomy category
capability	Related capability key (optional)
semantic_intent	Expected semantic intent
user_message	User utterance
expected_capability	Expected matched capability
expected_intent	Expected semantic intent key
expected_planner	platform_goal_planner or null
expected_tool	Expected mock tool name
expected_action	Expected action key
expected_response_mode	social / guidance / clarification / tool_execution
expected_clarification	bool
expected_confidence_min/max	Optional confidence range
tags	Free-form tags
product	prd-001
version	Dataset version
conversation_history	Optional prior turns (follow-up cases)

Storage:

• File: database/benchmark/{product}/{version}/entries.json
• Database: ai_benchmark_entries (includes human-approved imports)

Categories

ClinicAgentBenchmarkCategory enum covers: Greeting, Social, Help, Navigation, Instruction, Appointments, Revenue, Invoices, Payments, Patients, Leads, Marketing, Doctors, Scheduling, Website, Settings, Finance, Reports, Analytics, Permissions, Admin, Unknown, Clarification, Follow-up, Cancellation, Conversation continuation, Meta questions, Capability discovery, System questions.

CLI

# Sync canonical dataset + run benchmark (LLM off by default)
php artisan ai:benchmark --sync

# Run with Groq intent engine (nightly eval)
php artisan ai:benchmark --use-llm

# Enterprise report
php artisan ai:benchmark-report --latest

# Export dataset JSON
php artisan ai:benchmark-export --sync

# Import production conversations → review queue
php artisan ai:benchmark-import --limit=20

# Approve candidate → dataset (human required)
php artisan ai:benchmark-import --approve={uuid} --reviewer=moomen

# List failures
php artisan ai:benchmark-failures --run={uuid}
php artisan ai:benchmark-failures --pending

Reports

Enterprise metrics per run:

• Coverage (capability coverage ratio)
• Accuracy, Precision, Recall
• Intent Accuracy
• Capability Accuracy
• Clarification Accuracy
• Routing Accuracy
• Language Accuracy
• Regression Detection (vs baseline run)
• False Positives / False Negatives
• Per-language, per-category, per-capability scores
• Trend history (ai_benchmark_runs)

Reports saved to: storage/benchmark/reports/{run_id}.json

Production failure import pipeline

Production conversation
        ↓
ai:benchmark-import
        ↓
Failure detection (re-evaluate proposed entry)
        ↓
ai_benchmark_import_candidates (pending_review)
        ↓
Human review (--approve / --reject)
        ↓
ai_benchmark_entries (source=import)
        ↓
Dataset export

No automatic dataset modification. Human approval required.

Evaluation methodology

1. Deterministic CI (default): AI_BENCHMARK_USE_LLM=false — validates signal mapper + matcher + planner wiring.
2. LLM eval (nightly): AI_BENCHMARK_USE_LLM=true — validates Groq intent quality on same dataset.
3. Regression gate: --baseline={previous_run_uuid} — fails if previously passing cases regress.

CI integration

- name: Agent Benchmark
  run: |
    php artisan migrate --force
    php artisan ai:benchmark --sync
    php artisan ai:benchmark-report --latest

Threshold gate: AI_BENCHMARK_ACCURACY_THRESHOLD (default 0.85).

Runtime isolation (Part 6)

Benchmark phrases are never wired into:

• ClinicIntentCatalog (runtime LLM prompt)
• ClinicIntentRegexFallbackAnalyzer
• ClinicAssistantIntentCatalog
• Any capability activation regex

Runtime remains: LLM → Intent → Signals → Planner (regex emergency fallback only).

Tests

php artisan test app/Modules/AI/Tests/Unit/ClinicAgentBenchmarkRunnerTest.php

Related docs

• CLINIC_INTENT_UNDERSTANDING_ENGINE.md
• CLINIC_CAPABILITY_COVERAGE_TESTING.md
• ZAIXOS_AGENT_BENCHMARK_CERTIFICATION_REPORT.md