Introduction
Truvyx User Guide
Truvyx is an evaluation platform for multi-agent AI systems. You submit your agent's outputs against structured test scenarios, and Truvyx tells you whether it satisfied its constraints, why it failed, and how it compares to other systems over time.
Start with Quick start if you're new, or jump straight to any module in the sidebar.
Key concepts
Scenario — a structured test case: task description and a set of constraints the agent must satisfy.
Run — one execution of your agent against a scenario.
Constraint — a rule (HARD = must pass; SOFT = scored) the agent output must satisfy.
Violation — a constraint the agent broke, with a severity level.
More concepts
RCA report — Root Cause Analysis. After a failing run, classifies which category of fault caused it.
Verifier — a generated script that encodes constraint logic for reproducible evaluation.
HITL — Human-in-the-Loop. The agent escalates uncertain decisions to a human reviewer.
Registry — a community catalogue of benchmark results across agent systems.
Getting started
Quick start — first evaluation in five steps
1
Create a scenario
Go to Scenario Studio (/scenarios). Click New scenario, choose a domain, fill in the task description. Save it.
2
Add constraints
Open the Constraint Engine (/constraint-engine). Add at least one HARD constraint — the rule that must pass — and any SOFT constraints for scoring.
3
Submit a run
From /runs, click Submit evaluation. Choose your scenario, paste your agent's output JSON, name the agent system, and submit. The verifier runs automatically.
4
Review results
Click the run to open its detail view. The Results tab shows constraint scores and violations. The Audit Trail tab shows the verifier signature and attribution map.
5
Set up monitoring (optional)
Go to /monitoring and create a config watching your scenario suite. It will alert you via Slack, email, or PagerDuty if future runs regress.
🔑
Connecting your system programmatically? Generate an API key under Settings → API Keys and use the SDK to submit runs from CI. See
API & SDK.
Architecture
How all the modules connect
Every evaluation flows through the same backbone. You build scenarios and constraints first, submit runs against them, then use the analysis tools to understand and improve results.
Scenario Studio
→
Constraint Engine
→
Verifier Factory
Runs ↓
RCA Engine
←
Fault Events
→
HITL Analytics
Cost Benchmarks
→
Registry / Leaderboard
→
Monitoring
Slack alerts
Email alerts
PagerDuty
PDF exports
SDK / API
The core loop: Author a scenario → attach constraints → submit a run → review violations → trace root causes in RCA Engine → escalate decisions via HITL → benchmark cost efficiency → publish to Registry → have Monitoring watch for regressions automatically.
Module 01
Scenario Studio
The starting point for all evaluations. A scenario describes what task the agent should perform, in what domain, at what difficulty, and with what expected output shape. Scenarios are reusable — run many agent systems through the same scenario to compare them.
1
Browse existing scenarios
The main list shows all scenarios for your organisation. Filter by domain or difficulty. Click any row to open it.
2
Create a new scenario
Click New scenario. Choose between From Template (pick a pre-built domain template) or From Natural Language. In the Natural Language tab, use the guided intake — answer what your product does, what decisions it makes, and what could go wrong — and Truvyx assembles the problem statement for you.
3
Open a scenario's detail page
Click a scenario to reach /scenarios/[id]. Here you see run history, constraints, verifier status, and can trigger decomposition or contract inference.
4
Decompose the scenario
Click Decompose to open the visual task-decomposition canvas at /scenarios/[id]/decomposition. This breaks the scenario into sub-agent steps.
feeds intoConstraint EngineVerifier FactoryDecomposition DesignerRegistryMonitoring
💡
New to prompt writing? Use the guided intake on the Natural Language tab. Answer four plain-language questions about your product and Truvyx assembles a well-formed problem statement automatically — no prompt-writing experience needed.
Module 02
Constraint Engine
Constraints are the rules that define what “correct” means for a scenario. HARD constraints must pass (the run fails if violated). SOFT constraints contribute to a score.
1
Add a constraint
Click Add constraint. Choose the scenario, Name, Type (HARD/SOFT), Category, and a natural language description — the plain-English rule the LLM verifier evaluates against.
2
Use a template
Switch to the Templates tab to browse pre-built constraints grouped by category. Click a template to load it, then customise the natural language.
3
Preview CDL
The CDL Preview tab shows the machine-readable Constraint Definition Language representation — useful for debugging or exporting.
4
Restrict sensitive fields
Under Restricted Fields, add output fields that should be Redacted, Blocked, or Audited. Enforced across all runs for your organisation.
| Category | What it covers |
|---|
| Temporal | Deadlines, sequencing, time windows |
| Resource | Token budgets, memory, compute limits |
| Dependency | Agent A must complete before agent B |
| Geographic | Data locality, regional restrictions |
| Regulatory | GDPR, HIPAA, financial compliance rules |
| Safety | Content filters, harm avoidance, access limits |
| Data Governance | Lineage, retention, classification |
| Fairness | Bias, demographic parity |
| Optimization | Cost efficiency, redundancy |
receives fromScenario Studiofeeds intoVerifier FactoryRCA EngineHITL Analytics
Module 03
Verifier Factory
The evaluation engine. Submit a run and the platform automatically generates a verifier script, checks each constraint, records violations, and produces a signed audit record. The run detail page is the central view most teams use day-to-day.
1
Submit a run
Click Submit evaluation. Select a scenario, enter your agent system name, paste or upload the agent output as JSON. The verifier runs and you are taken to the run detail page.
2
Read the Results tab
Shows overall pass/fail status, a score breakdown by constraint, and a list of violations with severity (CRITICAL → HIGH → MEDIUM → LOW). Each violation shows the constraint broken and the agent output that broke it.
3
Inspect the Audit Trail tab
Shows the cryptographic verifier signature, submitter, environment metadata, and a rule-attribution map. Click Generate Examiner Narrative to produce a regulator-ready PDF report.
4
Trigger RCA on failures
If the run has HARD constraint violations, a Run RCA button appears. Clicking it opens the RCA Engine with pre-loaded fault events for this run.
5
Export
Download results as JSON, or the Examiner Narrative as PDF/DOCX. Choose the audience type (Central Bank, Healthcare Auditor, Tax Authority, Tribunal, or General Regulator) — the language and regulatory references adapt automatically.
receives fromScenario StudioConstraint Enginefeeds intoRCA EngineHITL AnalyticsCost BenchmarksRegistryMonitoring
Module 04
Decomposition Designer
A visual canvas for mapping task splits across specialised sub-agents — which handles which step, what data flows between them, and where human handoffs happen.
1
Open the canvas
Reach it via /decomposition-designer for a blank canvas, or from a scenario detail page via Decompose which pre-loads the scenario context.
2
Add agents and connect them
Drag agents onto the canvas, name them, assign roles. Connect agents with edges representing data flow or control flow.
3
Mark escalation points
Where an agent should escalate to a human, mark the edge as a HITL handoff. This feeds the HITL Analytics module with baseline expectations.
4
Infer contracts
Once saved, navigate to the scenario's contracts view and click Infer contracts to automatically derive the output schema each producer agent hands to its consumer. These become inter-agent contract tests.
receives fromScenario Studiofeeds intoHITL AnalyticsVerifier Factory
Module 05
Registry
A shared catalogue of benchmark results. When you publish a run, it enters the Registry. The Leaderboard ranks agent systems by performance within a domain. Use it to compare against others, find existing scenarios, or contribute results to the community.
1
Browse entries
Filter by domain, difficulty, or agent system name. Click any entry for details. Each published entry has a permanent URL at /registry/[slug] you can share externally.
2
View the Leaderboard
Navigate to /registry/leaderboard. Sort by overall score, HARD constraint pass rate, cost-per-correct-decision, or efficiency percentile. Filter by scenario or domain for a like-for-like comparison.
3
Contribute a result
Go to /registry/contribute. Select a run and confirm publication. Published runs include agent system name, scenario, scores, violation counts, and verifier checksum — no raw agent output is shared.
receives fromVerifier FactoryCost Benchmarksfeeds intoMonitoring (baselines)
Module 06
RCA Engine
When runs fail, Root Cause Analysis classifies why. The engine collects fault events, groups them by fault type, identifies systemic weaknesses across multiple runs, and surfaces recurring patterns.
| Fault type | What it means |
|---|
| Decomposition Fault | The task split between agents was incorrect or incomplete |
| Constraint Blindness | The agent acted without checking a constraint it should have |
| Coordination Breakdown | Agents disagreed or handed off data in an incompatible format |
| Optimization Failure | The agent chose a locally optimal path that violates a global constraint |
| Hallucinated Dependency | The agent assumed a dependency that does not exist |
| Information Boundary Violation | Agent accessed or leaked data outside its authorised scope |
| Temporal Misalignment | Actions taken out of sequence or outside time windows |
| Unknown | Pattern not yet classified — review manually |
1
View cross-run patterns
The RCA Engine home shows fault-type frequency across all your runs. High-frequency fault types indicate a systemic issue with your agent architecture.
2
Open an RCA report from a run
From any failing run's detail page, click Run RCA. This creates an RCA report at /runs/[id]/rca showing fault events, evidence, and root-cause flags.
receives fromVerifier FactoryHITL Analyticsfeeds intoMonitoring
Module 07
HITL Analytics
Tracks escalations — when an agent passes a decision to a human reviewer. Helps you tune escalation thresholds to reduce unnecessary interruptions while keeping humans in the loop where they genuinely add value.
1
View the escalation dashboard
Shows total escalations over time, escalation type breakdown (FALSE_AUTONOMY, UNCERTAINTY, POLICY_CONFLICT, RISK_AVERSION, COORDINATION_BREAKDOWN), approval vs rejection rate, average response time, and semantic drift score.
2
Inspect individual escalations
Each escalation has a decision record: which agent escalated, what action it was considering, who reviewed it, what the human decided (approve / reject / modify), and any corrections.
3
Generate an escalation policy
From a scenario's detail page, click Generate escalation policy. The platform drafts a policy from your constraint definitions and historical escalation patterns. Review and save it.
4
Record an outcome via API
POST to /api/escalations/[decisionId]/outcome to record the human's decision. This feeds friction analytics and can automatically create RCA fault events if the agent was wrong to escalate.
receives fromVerifier FactoryDecomposition Designerfeeds intoRCA Engine
Module 08
Cost Benchmarks
Profiles each run for token spend, API call count, redundant calls, and cost per correct decision. Generates plain-language procurement recommendations at a specified volume.
1
View cost profiles
Each submitted run with LLM call traces automatically generates a cost profile: estimated USD cost, total tokens, API call count, redundant call count, efficiency percentile, and cost per correct decision.
2
Generate an efficiency report
Select a set of runs, set a projected monthly volume, click Generate efficiency report. The report compares agents on cost vs accuracy with annual cost projections. Download as PDF or JSON.
3
Redundancy detection
The engine automatically flags duplicate calls (same input/output within a session), verification loops (same verifier tool called more than twice), and hallucination recovery (rapid corrective calls).
receives fromVerifier Factory (run traces)feeds intoRegistry / Leaderboard
Module 09
Monitoring
Watches your agent system continuously. Configure thresholds and scenario suites; when a new run's scores fall below them — or a new fault type appears, or a regulatory violation is detected — Monitoring fires an alert.
1
Create a monitoring config
Click New config. Set: Scenario suite (which scenarios to watch), Feasibility threshold (minimum overall score 0–1), Completeness threshold, Fail on regulatory violation toggle, Auto-trigger RCA toggle, and notification channels.
2
Configure notification channels
Each config can notify via Slack webhook, email, or PagerDuty. Configure credentials first under Settings → Integrations, then add the channel to the config.
3
Enable / disable a config
Toggle the config active or inactive from the monitoring list. Inactive configs are preserved but do not trigger alerts.
4
Review regression alerts
The alerts panel shows: SCORE_REGRESSION, NEW_FAULT_TYPE, REGULATORY_VIOLATION, THRESHOLD_BREACH, and IMPROVEMENT. Each links to the run that triggered it. Mark alerts resolved when actioned.
receives fromVerifier FactoryRCA EnginefiresSlack / Email / PagerDuty
Configuration
Settings
Settings are at /settings. Six sub-pages: General, API Keys, Integrations, Members, Compliance, Exports, and Billing.
| Scope | Allows |
|---|
| runs:read | Read evaluation runs and results |
| runs:write | Submit new evaluation runs |
| registry:read | Read the public registry |
| rca:read | Read RCA reports |
| monitoring:write | Trigger monitoring evaluation |
🔒
Least privilege. For a CI pipeline that only submits runs, grant only runs:write. For a read-only dashboard, grant only runs:read. Never use an all-scopes key in automated pipelines.
Slack
Enter your Slack incoming webhook URL. Once saved, select Slack as a notification channel in any Monitoring config.
Email (SMTP)
Enter SMTP host, port (typically 587), username, password, and From address. Works with Gmail, SendGrid, Postmark, etc.
PagerDuty
Enter your Events API routing key. Critical regulatory violations can be set to page on-call immediately when Fail on regulatory is enabled.
Compliance Mode
Toggle on to enforce stricter audit logging — all runs recorded with immutable verifier signatures and restricted-field enforcement is mandatory.
Enter a team member's email and select their role. Owners can manage billing. Admins can manage members and settings. Members can submit runs and view results. Invitees receive an email link to join.
Programmatic access
API & SDK
Every Truvyx feature is accessible via REST API. All requests use Bearer authentication with your trk_-prefixed API key. The SDK Docs page at /docs/sdk provides live, copyable code examples.
| Operation | Method | Endpoint | Scope |
|---|
| Submit a run | POST | /api/runs | runs:write |
| Get run results | GET | /api/runs/[id] | runs:read |
| List runs | GET | /api/runs | runs:read |
| Trigger RCA | POST | /api/runs/[id]/rca | runs:write |
| Get RCA report | GET | /api/runs/[id]/rca | rca:read |
| Record HITL outcome | POST | /api/escalations/[id]/outcome | runs:write |
| List registry entries | GET | /api/registry | registry:read |
| Trigger monitoring | POST | /api/monitoring/[id]/trigger | monitoring:write |
Authentication: Authorization: Bearer trk_your_key_here
Content type: All request bodies are application/json.
Integration guide
Connecting your system
The most common pattern is a CI step that submits a run after every deployment, then fails the build if critical constraints are violated.
1
Create an API key with runs:write scope
Go to Settings → API Keys. Create a key named after your pipeline (e.g. 'GitHub Actions – prod'). Save the trk_ key as a secret in your CI environment.
2
Run your system against the scenario
Your agent runs its task and produces output. Capture the output as a JSON object, plus any LLM call traces if you want cost profiling.
3
Submit the run via API
POST to /api/runs with body: { scenarioId, agentSystemName, agentOutput, environment }. Use your API key in the Authorization header. The response contains the run ID and immediate status.
4
Poll for completion and check results
GET /api/runs/[id] until status is PASSED, FAILED, or PARTIAL. Check severityCounts.CRITICAL — if above zero, fail the build.
5
Configure Monitoring to alert on regressions
After the integration is working, set up a Monitoring config watching the scenarios your CI covers. Connect it to your Slack channel for alerts on runs submitted from any source.
📋
Full SDK examples including TypeScript, Python, and cURL are on the SDK Docs page at /docs/sdk — pre-filled with your organisation's scenario IDs once you log in.