Documentation
Learn Truvyx
Guides and references for every part of the platform: from running your first evaluation to integrating Truvyx into a CI pipeline.
Quick Start
From zero to your first evaluation result in about ten minutes. No programming experience is required for the dashboard route.
Create your free account
Create an organisation
Write your first Scenario
Run your system and capture its output
{ "decisions": {}, "reasoning": "...", "actions_taken": [] }.Submit the output and read the report
Connect your system
Truvyx evaluates outputs — it never calls your AI system directly. You run your pipeline externally and submit the result as JSON. The Connect Wizard on every scenario page makes setup instant.
Open a Scenario and click Connect
The teal Connect button (plug icon) appears in the scenario header. It opens a three-step wizard.
Choose your integration
Pick REST/cURL, TypeScript SDK, Python SDK, or GitHub Actions. The wizard generates a pre-filled code snippet that posts your system's JSON output to POST /api/runs with your scenario ID already embedded.
Generate an API key and copy the snippet
Create a runs:write-scoped API key inline (or pick an existing one), copy the snippet into your codebase or CI YAML, and fire a test run from the wizard to confirm everything reaches Truvyx.
What does my output need to look like? Any JSON object works. At minimum, include a decisions object, a reasoning string, and an actions_taken array. Toggle Show advanced options in the wizard to add token-cost fields for cost profiling, compliance mode, or monitoring config IDs.
Core Concepts
These are the building blocks of Truvyx. Read each one once and you will understand how the whole system fits together.
Scenarios
Login requiredA Scenario is a single, self-contained test case for an AI agent. You describe an input (the message or task the agent receives) and a set of expected behaviours (what it should or should not do in response). Think of it as a unit test for AI behaviour rather than code logic. Scenarios can be written in plain language with no programming required.
Tip: Start with the 3 to 5 situations where a wrong AI response would cause the most damage to your users or your business. Those become your first Scenarios.
Templates
Login requiredA Template is a pre-built Scenario with placeholder fields. Truvyx ships a library of templates covering finance, healthcare, customer service, legal, and software development. Fill in your agent-specific values and run the template immediately. Templates are peer-reviewed and updated as regulations change.
Tip: Browse templates before writing from scratch. Adapting an existing template is faster and catches edge cases you might not have considered.
Constraint Engine
Login requiredA Constraint is a rule your AI agent must never break. Examples: never share a user email with a third party; always refuse requests to generate malware; response must be under 400 words. The Constraint Engine checks every agent response against your active constraints and flags any violation. It can also prove mathematically that certain combinations of constraints cannot be satisfied at the same time, so you catch contradictory rules before deploying.
Tip: Define constraints in plain language first. Truvyx will suggest formal representations. Start with 5 to 10 high-priority constraints.
Root-Cause Analysis
Login requiredWhen an agent breaks a rule or produces a wrong answer, Root-Cause Analysis identifies exactly where the reasoning went wrong. Truvyx classifies failures into eight fault types: Misunderstanding, Hallucination, Instruction Conflict, Context Overflow, Tool Failure, Guardrail Bypass, Drift, and Contract Violation. For each failure the system constructs a causal chain, a step-by-step trace from input to bad output. It can also run a counterfactual replay: change one variable and see whether the failure disappears.
Tip: Use the counterfactual replay to test prompt fixes before deploying them. It is cheaper than re-running live evaluations.
Evaluation Runs
Login requiredAn Evaluation Run is what happens when you execute one or more Scenarios against your AI agent. Truvyx records every input, every output, every constraint check, and every score. Runs are immutable after creation. Each run receives a unique ID and a cryptographic signature so you can prove to regulators that the record has not been tampered with.
Tip: Run evaluations on every model version before you deploy. Store the run IDs in your deployment notes so you always know which evaluation certified each release.
Regression Monitoring
Login requiredTruvyx catches two kinds of AI model drift. Deployment-triggered monitoring fires evaluations on every release via a CI/CD webhook, blocking the deploy if scores fall below your threshold. Continuous production sampling runs in the background at all times, sampling a configurable fraction of live traffic and comparing aggregate scores to a stored baseline — so silent provider-side model updates (no deployment on your end) are caught by statistical observation rather than by waiting for a signal that may never arrive. Both send alerts to Slack, email, or PagerDuty, labelled differently so on-call teams know whether to roll back a deploy or investigate an upstream model change.
Tip: Set deployment-triggered monitoring on your CI pipeline and continuous sampling on your 10 most critical scenarios. Together they cover both known changes and silent drift.
Scenario Studio
Login requiredSome AI agent failures only appear after a sequence of turns: the agent says the right thing on message 1 but contradicts itself by message 4. Scenario Studio is a visual editor for building multi-turn conversations, branching dialogue trees, and adversarial prompt chains. You can model an entire user session, not just a single message.
Tip: Map your most common real-world user conversation paths. A 5-turn adversarial test often reveals more than 20 single-turn tests.
Verifier Factory
Login requiredSome rules cannot be expressed as simple text constraints because they require business logic or domain-specific checks. The Verifier Factory lets engineers write custom verifier functions in Python that Truvyx calls during every evaluation. For example: a HIPAA verifier that cross-checks patient identifiers, or a financial verifier that confirms transaction amounts are within approved limits. Non-programmers link existing verifiers to Scenarios without writing code.
Tip: Have your engineers write one verifier for your most domain-specific requirement first. Your compliance team can then use it across all relevant Scenarios.
Decomposition Designer
Login requiredA complex AI task such as research this company and write an investment memo is actually dozens of sub-tasks stitched together. If the final output is wrong, you need to know which sub-task failed. The Decomposition Designer lets you model your agent task as a tree of sub-tasks, assign constraints to each node, and evaluate them independently. This is especially valuable for regulated industries where you must demonstrate which step of a process failed and why.
Tip: Map your agent workflow as a flowchart first. Then reproduce it in the Decomposition Designer and add one constraint per node.
Audit Records
Login requiredEvery evaluation run generates an Audit Record: a structured document containing the inputs, outputs, constraint results, scores, timestamps, and the model version used. Each record is signed with an RS256 cryptographic key that you control. The signature means anyone, including a regulator or an auditor, can verify the record was created by your organisation at the stated time and has not been modified since. Records can be exported as PDFs, JSON, or submitted directly to compliance portals.
Tip: Keep your signing key in a secrets manager and rotate it annually. Never store it in code.
Inter-Agent Contract Testing
Login requiredWhen multiple AI agents collaborate, each agent implicitly promises to send data in a certain shape. If one agent changes its output format, a downstream agent silently receives garbage. Contract Testing makes these promises explicit: Truvyx infers JSON Schema contracts for every agent-to-agent edge from your run history, then validates each new run against them. Breaking changes surface as CONTRACT_VIOLATION faults in the RCA engine before they reach production. The Decomposition Designer canvas colour-codes each edge by its last contract test result.
Tip: Run contract tests in CI after any prompt change. The most common source of silent multi-agent failures is an upstream agent changing its output keys.
HITL & Escalation
Login requiredSome agent decisions are too high-stakes to run without human review. The Human-in-the-Loop (HITL) module lets you define Escalation Policies: tiered rules specifying which decisions require human approval, what context the reviewer must receive, and how long they have to respond. The Escalation Evaluator then classifies each decision as CORRECT_ESCALATION, CORRECT_AUTONOMY, FALSE_ESCALATION, or FALSE_AUTONOMY. FALSE_AUTONOMY events (agent acted when it should have waited) propagate directly to the RCA engine as EU AI Act Article 14 violations. Friction Analytics tracks how reviewers actually respond, surfacing recurring correction patterns as actionable signals.
Tip: Start with your highest-stakes decision type: financial commitments, clinical recommendations, or customer data changes. Add escalation tiers from there.
Cost & Efficiency
Login requiredAccuracy alone does not tell you whether a multi-agent system is economically viable. The Cost & Efficiency module records token usage, API call counts, and estimated cost for every run. A redundancy detector identifies four classes of wasted calls: duplicate prompts, unnecessary re-fetches, verification loops, and hallucination-recovery spirals — each propagated to the RCA engine as OPTIMIZATION_FAILURE faults. Efficiency Score (0–100, inverse of cost) and Combined Score (70% correctness, 30% efficiency) appear on the global leaderboard alongside a cost-at-scale projection for your projected monthly volume.
Tip: Run the Cost & Efficiency tab after your first 10 evaluations. One redundant verification loop caught early can save hundreds of dollars at production volume.
Compliance Reporting
Produce regulator-ready documentation from any evaluation run: audience-tuned narratives, decision attribution maps, and restricted-field access reports.
Examiner Narrative
Generate a formal regulatory narrative PDF directly from a run. Choose your audience (central bank supervisory authority, healthcare compliance auditor, tax authority, administrative tribunal, or general regulator) and Truvyx writes a five-section document — System Description, Evaluation Methodology, Findings, Remediation, and Ongoing Controls — in the formal third-person register that examiners expect. An optional constraint appendix lists every hard and soft rule in scope.
Rule-Attribution Map
For every agent decision in a run, Truvyx traces which constraint governed it and whether the agent respected it. Decisions are classified as DETERMINING, CONTRIBUTING, or CHECKED_NOT_TRIGGERED. Decisions that cannot be traced to any constraint are flagged as UNATTRIBUTABLE and automatically block a PASS certificate in Compliance Mode. The Attribution tab in the Run detail page shows coverage as a percentage of attributed decisions.
Restricted-Field Permissions
Register fields that your agents must never access — patient identifiers, social security numbers, salary data — along with the regulatory basis (GDPR Art. 9, HIPAA § 164.502, etc.) and the restriction type: PROHIBITED, ROLE_RESTRICTED, PURPOSE_RESTRICTED, or AGGREGATION_ONLY. Truvyx scans every agent output for these fields and their aliases. PROHIBITED and ROLE_RESTRICTED violations create CRITICAL-severity INFORMATION_BOUNDARY_VIOLATION faults in the RCA engine.
Citation & Grounding Fidelity
Agents in regulated industries regularly cite statutes, clinical guidelines, and policy documents to justify their decisions. Truvyx verifies those citations are real, current, and accurately represented.
Source Corpus
Upload the authoritative documents your agents are supposed to cite: regulatory texts, clinical protocols, internal policies, or contract templates. Documents are organised into named corpora (Regulatory, Clinical, Internal Policy, Contractual, or Custom) and stored per organisation. Use the Source Corpus tab under Settings → Compliance to add documents one by one or bulk-import up to 500 at a time. Each document carries an identifier, version, effective date, and an optional supersededBy pointer so citations to outdated versions are caught automatically.
Citation Fidelity Check
After a run completes, open the Citation Fidelity tab and click Run Citation Check. Truvyx extracts every citation from the agent's reasoning — using pattern matching for known regulatory formats (USC, CFR, IRC, Article/Section references) plus an LLM extraction pass for less-structured text — then looks each one up in your source corpus. Each citation receives one of five verdicts:
VERIFIEDThe document says exactly what the agent claimed.PARTIAL_MATCHThe document was found but the agent's characterisation was incomplete or imprecise.MISREPRESENTEDThe document was found but the agent's claim contradicts or overstates it.FABRICATEDThe citation has a plausible regulatory format but the document is not in your corpus — the agent invented it.SOURCE_NOT_FOUNDThe reference cannot be checked because no matching document exists in your corpus.
Compliance integration: FABRICATED citations create a CITATION_FABRICATION fault in the RCA engine (CRITICAL severity). MISREPRESENTED citations create a CONSTRAINT_BLINDNESS fault. Both block a PASS certificate in Compliance Mode and appear in the Examiner Narrative under Findings.
Continuous Production Sampling
Model providers can update the model behind a hosted API endpoint without notice, with no version bump and no signal on your side. Continuous sampling detects this by observing live traffic rather than waiting for a deployment event that may never come.
Probabilistic sampling
Set a sampling rate (default 5%) and a strategy — Random, Stratified across decision types, or High-Stakes Weighted (oversample decisions above a configured stakes threshold). Truvyx selects runs automatically as they are submitted, with no changes to your submission flow.
Rolling windows
Sampled runs are grouped into rolling time windows (default 24 hours). At the end of each window, Truvyx aggregates scores across all sampled runs in that window and compares them to a stored baseline. Windows with fewer than the minimum sample count are marked INSUFFICIENT DATA to prevent misleading signals from statistically small populations.
Silent drift alerts
If the window score drops more than your configured threshold below the baseline — or if the fault rate increases by more than the fault-rate threshold — an alert fires through your existing Slack, email, or PagerDuty channels. The alert explicitly states no deployment event was recorded, so on-call teams know to investigate an upstream provider change rather than look for a local rollback.
Baseline management
A baseline is the score distribution your system is expected to achieve. Establish it from a set of known-good run IDs (for example, the runs from your last formal pre-deployment evaluation gate) or let Truvyx self-establish it from the first computed window once you have sufficient traffic. After an intentional, approved model upgrade, click Re-establish in the Monitoring tab to reset the comparison point without losing historical window data.
How does this differ from deployment-triggered monitoring? Deployment monitoring (P10) fires evaluations when you deploy a new version. Continuous sampling fires on live traffic continuously, independent of deployment events. Both run simultaneously. A deployment regression points to a specific code or prompt change you can revert. A silent drift alert points to something that changed upstream — investigate whether your model provider announced an update, or whether your input data distribution shifted.
Scenario Registry
A searchable library of evaluation Scenarios published by the Truvyx community, accessible from the dashboard at /registry.
Browse and search
Full-text search across all published Scenarios by title, domain, or constraint type. Filter by industry (healthcare, finance, legal, customer service, and more). Each listing shows the author, version history, download count, and community upvote score.
Download and fork
Download any published Scenario as a structured YAML or JSON spec and import it directly into your organisation. Forked Scenarios are private to your org until you choose to publish them back. Version history is preserved.
Contribute
Publish your own Scenarios to the registry from the Contribute page. Published Scenarios are versioned and attributed to your organisation. The community can upvote, comment, and propose amendments. Peer-reviewed Scenarios are promoted to the featured list.
Deployment
Use Truvyx as a hosted service, plug it into your CI pipeline, or run it entirely on your own servers.
Cloud (hosted)
The default option. Truvyx hosts everything: database, workers, email delivery, and the dashboard. Suitable for teams that do not have strict data-residency requirements. The free plan includes 500 evaluation runs per month with no credit card required.
Tip: Start here. You can migrate to on-premise later if your data requirements change.
CI/CD Integration
Connect Truvyx to your deployment pipeline so a failed evaluation automatically blocks the release. Truvyx provides official GitHub Actions, a GitLab CI template, and a generic CLI command that returns a non-zero exit code on failure. Your pipeline runs tests, then Truvyx runs evaluations, and the deploy only proceeds if evaluations pass.
Tip: Gate your production deploy, not just staging. A failed evaluation in production is more expensive than a delayed release.
On-Premise
For organisations that cannot send data to a third-party service (healthcare, government, finance). The on-premise edition uses a Docker Compose stack that runs inside your firewall. All data stays on your infrastructure with no telemetry leaving your network. A paid plan or enterprise licence is required.
Tip: You will need Docker, at least 4 GB of RAM, and SMTP credentials for email. Setup takes about 30 minutes.
API and SDK
Everything you can do in the dashboard you can also do programmatically. The Python SDK wraps the REST API for the most common workflows.
Python SDK
Install with pip and authenticate with your API key (Settings → API Keys). Run your system externally, then submit its output. Truvyx evaluates the output — it never calls your system directly.
pip install truvyx
from truvyx import evaluate
# Run your system externally, then submit its JSON output
result = evaluate(
scenario_id="scn_abc123",
agent_system_name="My AI System",
agent_output={
"decisions": {},
"reasoning": "...",
"actions_taken": [],
},
api_key="trk_...",
)
print(result.passed, result.score)API keys start with trk_. Generate them under Settings → API Keys, or use the Connect button on any scenario page for a pre-filled snippet in your preferred stack.
REST API
All endpoints accept and return JSON. Authenticate by passing your API key in the Authorization header as a Bearer token.
/api/runsSubmit agent output for evaluation
/api/runs/:id/statusPoll until the run completes (PENDING → PASS/FAIL)
/api/runs/:idRetrieve full results, scores, and constraint breakdown
/api/scenariosList all Scenarios in your org
/api/audit/:idDownload a signed Audit Record
Full OpenAPI spec coming soon. Contact support for early access.
OperationsBench
The public leaderboard for multi-agent AI systems. Submit your agent, run the standardised evaluation suite, and receive a verified score you can display publicly. The leaderboard ranks systems on three dimensions: Most Correct (constraint satisfaction rate), Most Efficient (lowest cost per correct decision), and Best Value (Combined Score: 70% correctness, 30% efficiency). Filter by estimated cost per run to find the best system within your budget.
Glossary
Plain-English definitions for every term used in Truvyx.
Agent
An AI system that takes actions on its own: it reads input, decides what to do, and produces an output or performs a task without a human approving every step. Examples include a customer-service chatbot, a code reviewer, or an automated data analyst.
Evaluation
The process of systematically testing an AI agent to measure how well it follows rules, handles edge cases, and produces correct outputs. Think of it as a quality-assurance inspection for AI behaviour.
Scenario
A single test case. It defines an input, the context the agent is operating in, and the expected behaviour. Example: given this customer complaint, the agent must not offer a refund above 50 pounds without manager approval.
Constraint
A hard rule the agent must never violate, regardless of the situation. Constraints are non-negotiable. Example: never reveal another user's personal data.
Run
A single execution of one or more Scenarios. Each run produces a score, a pass/fail result, and a detailed breakdown of every constraint check.
Score
A number between 0 and 1 representing how well the agent performed in a run. A score of 1.0 means every constraint was satisfied. Anything below your configured threshold triggers an alert.
Audit Record
A tamper-proof document proving that a specific evaluation happened at a specific time and produced a specific result. It is signed with a cryptographic key, so a regulator or auditor can verify it has not been altered.
Root-Cause Analysis (RCA)
The process of identifying the exact reason an agent failed. Truvyx classifies failures into seven types and traces the causal chain from input to bad output so you know precisely what to fix.
Hallucination
When an AI model invents facts, citations, names, or data that do not exist. A hallucination verifier checks agent outputs against a knowledge base or search result to detect invented information.
Regression
When a model update or prompt change causes a previously passing test to fail. Regressions are often silent: the agent looks fine until you run your Scenarios again. Regression Monitoring catches them automatically.
Prompt
The instructions you give to an AI model: usually a system prompt that sets the agent role, plus the user message it receives. Your constraints are checked against the agent's response to the prompt.
Model
The AI brain powering your agent, for example GPT-4o, Claude, or Gemini. Truvyx is model-agnostic and can evaluate any model that exposes an API.
API Key
A secret string starting with trk_ that identifies your account when you call the Truvyx API programmatically. Treat it like a password: never share it or commit it to version control.
Organisation
Your team's shared workspace inside Truvyx. All Scenarios, Runs, and Audit Records belong to an Organisation. Members can have different roles: Owner, Admin, Editor, or Viewer.
Causal Chain
A step-by-step trace showing which input, which model decision, and which piece of context led to a failure. Reading the causal chain is like rewinding a recording of the agent's reasoning to see exactly where it went wrong.
Counterfactual Replay
Re-running the agent with one variable changed (a different prompt word, a different model temperature) to see whether the failure disappears. It confirms that your proposed fix actually solves the problem before you deploy it.
Fault Type
Truvyx classifies every failure into one of eight categories: Misunderstanding, Hallucination, Instruction Conflict, Context Overflow, Tool Failure, Guardrail Bypass, Drift, and Contract Violation. Each type points to a different remediation strategy.
CI/CD Pipeline
The automated process that builds, tests, and deploys your software or AI agent whenever code changes. Truvyx integrates into this pipeline so evaluations run automatically before every release.
Verifier
A custom function written in Python that checks a specific rule during evaluation. Used when a rule is too complex for a plain-language constraint, for example checking that a drug dosage does not exceed a patient's maximum based on their weight.
SLA (Service-Level Agreement)
A formal commitment about the quality or availability of a service. The enterprise plan includes an SLA covering platform uptime and support response times. Contact us to discuss terms.
Cost Profile
A per-run record of token usage, API call count, tool call count, estimated cost in USD, and a breakdown by agent. Used to compute the Efficiency Score and to detect redundant calls before they inflate your production API bill.
Efficiency Score
A 0–100 metric derived from a run's estimated cost relative to other runs on the same scenario. A score of 100 means the cheapest run on record; lower scores indicate higher relative cost. Combined Score blends correctness (70%) and efficiency (30%) into a single ranking number on the leaderboard.
HITL (Human-in-the-Loop)
A design pattern where an AI agent pauses and requests human approval before taking a high-stakes action. Truvyx's Escalation Evaluator checks whether your agent escalated decisions correctly according to your Escalation Policy, and flags violations as EU AI Act Article 14 non-compliance.
Escalation Policy
A set of tiered rules specifying which agent decisions require human review, what context the reviewer needs, and the deadline for response. Created in the Constraint Engine under the Escalation Policy tab. FALSE_AUTONOMY events (agent acted without required approval) create COORDINATION_BREAKDOWN faults in the RCA engine.
Friction Signal
A root-cause classification produced when a human reviewer corrects or rejects an agent decision. Signals include DECOMPOSITION_FAULT (the agent misunderstood its task), CONSTRAINT_GAP (a rule was missing), FEW_SHOT_NEEDED (the agent lacked examples), PROMPT_AMBIGUITY, and DATA_QUALITY_ISSUE. Recurring signals trigger pattern alerts in the HITL Analytics dashboard.
Attribution Map
A per-decision record showing which constraint governed each agent action and how: DETERMINING (the constraint was the primary reason), CONTRIBUTING (it played a secondary role), or CHECKED_NOT_TRIGGERED (it was evaluated but not activated). Decisions with no matching constraint are flagged UNATTRIBUTABLE and block certificate sign-off in Compliance Mode.
Restricted Field
A data field registered in the Restricted-Field Permissions registry that agents are not permitted to access. Each field carries a regulatory basis (e.g. GDPR Art. 9) and a restriction type: PROHIBITED (never), ROLE_RESTRICTED (only named agent roles), PURPOSE_RESTRICTED (usage warning), or AGGREGATION_ONLY (no individual-level access).
Examiner Narrative
A formal PDF document generated from an evaluation run for a specific regulatory audience. Structured in five sections (System Description, Evaluation Methodology, Findings, Remediation, Ongoing Controls) in the third-person register expected by regulators. An optional constraint appendix lists every rule in scope.
Agent Contract
A machine-verifiable JSON Schema promise that one agent makes to another about the shape of data it will send. Inferred automatically from run history by Truvyx's contract-inference engine. Violations are classified as BREAKING (type or field presence changed), DEGRADED (enum or range narrowed), or WARNING (new optional field added).
Source Corpus
The collection of authoritative documents your agents are expected to cite: regulations, clinical guidelines, internal policies, or contract templates. Uploaded per organisation under Settings → Compliance → Source Corpus. The citation fidelity engine looks up every agent citation against this corpus to determine whether it is accurate, misrepresented, or fabricated.
Citation Fidelity
A measure of whether agent-cited sources actually say what the agent claims they say. Each citation is classified as VERIFIED, PARTIAL_MATCH, MISREPRESENTED, FABRICATED, or SOURCE_NOT_FOUND. FABRICATED and MISREPRESENTED citations automatically create CRITICAL fault events in the RCA engine and block compliance certificate sign-off.
Fabricated Citation
A citation that uses a structurally plausible regulatory format (for example 42 U.S.C. § 1395dd) but does not correspond to any document in the organisation's source corpus. Distinct from SOURCE_NOT_FOUND, which means the corpus simply does not cover that domain. A fabricated citation is a hallucination of a real-looking authority reference.
Silent Drift
A change in an AI model's behaviour that occurs without any deployment event on the customer's side — typically because the model provider updated the model behind the same API endpoint. Silent drift cannot be detected by deployment-triggered evaluation alone because there is no webhook signal to trigger on. Continuous production sampling catches it by comparing rolling window scores against a stored baseline.
Sampling Window
A fixed time period (default 24 hours) during which a configured fraction of production runs is probabilistically selected and evaluated. At window close, Truvyx aggregates scores across all sampled runs, computes drift magnitude and direction versus the baseline, and fires alerts if thresholds are exceeded. Windows with too few samples are marked INSUFFICIENT DATA rather than producing a potentially misleading aggregate.
Baseline (sampling)
The reference score distribution that continuous sampling windows are compared against to detect drift. Established from a set of known-good historical run IDs, or self-established from the first computed window with sufficient data. Should be re-established after any intentional, approved model upgrade to reset the drift comparison point.
Ready to start evaluating?
Free plan: 500 runs per month, unlimited Scenarios, community templates included.