> ## Documentation Index
> Fetch the complete documentation index at: https://braintrust.dev/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# TypeSafe
> Trace TypeSafe Jev calls, inspect structured decisions, and use Jev as an LLM judge to score or label AI outputs against your criteria.
[TypeSafe](https://typesafe.ai/) provides Jev, a model that makes structured decisions instead of generating text. You define the question and its allowed answers before the call, and Jev returns one of them, so nothing downstream has to find a decision inside prose or malformed JSON. You can use Braintrust with Jev in two ways:
* **[Trace Jev in your application](#tracing)**: If your application calls Jev to route support requests, assess urgency, or make other decisions, instrument those calls to inspect their inputs, answers, and probabilities in Braintrust traces.
* **[Use Jev for evaluations](#llm-as-a-judge)**: Configure Jev as the LLM judge to check AI outputs against your criteria, such as whether a support response follows a refund policy. Braintrust runs the evaluator and turns Jev's decisions into scores or labels in experiments or online scoring.
## Tracing
Instrument TypeSafe calls in your application to capture their inputs, results, timing, and errors in Braintrust question spans.
Setup
Requires Braintrust v3.34.0+ and `@typesafe-ai/sdk` v0.6.0 or later within v0.x. You need a TypeSafe account and API key.
```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
pnpm add braintrust@^3.34.0 @typesafe-ai/sdk@^0.6.0
```
Set your [Braintrust API key](/docs/admin/authentication#api-authentication) and TypeSafe API key in your shell:
```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
export BRAINTRUST_API_KEY=""
export TYPESAFE_API_KEY=""
```
For the EU data plane, also set `BRAINTRUST_API_URL` to `https://api-eu.braintrust.dev`. For a self-hosted deployment, use your data plane URL.
Auto-instrumentation
Use Braintrust's import hook to trace TypeSafe calls throughout your application.
Save this example as `trace-typesafe-auto.js`. It asks Jev to route a support request, assess its urgency, and identify whether it mentions a duplicate charge.
```javascript title="trace-typesafe-auto.js" theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
import { initLogger } from "braintrust";
import { TypeSafeClient, choice, noul, score } from "@typesafe-ai/sdk";
const logger = initLogger({ projectName: "typesafe-example" });
const client = new TypeSafeClient();
const response = await client.systemOne({
state: "I was charged twice. Please refund the duplicate charge today.",
questions: {
category: choice("Which team should handle this request?", {
billing: "Payments and refunds",
technical: "Software problems",
other: "Other requests",
}),
urgency: score("How urgent is this request?", [
"routine",
"soon",
"urgent",
]),
duplicate_charge: noul("Does the customer report a duplicate charge?"),
},
});
console.log(response.answers);
await logger.flush();
```
```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
node --import braintrust/hook.mjs trace-typesafe-auto.js
```
The auto-instrumentation example uses plain JavaScript so `node --import` can run the file directly. The Braintrust APIs work the same in TypeScript projects — compile your TypeScript to JavaScript, then run the compiled file with the import hook.
If you're using a bundler, see [Trace LLM calls](/docs/instrument/trace-llm-calls#auto-instrumentation) for plugin and loader setup.
Go to your project's [** Logs**](https://www.braintrust.dev/app/~/logs) and select the `typesafe.systemOne` span to [inspect the decisions](#inspect-jev-decisions).
Manual instrumentation
Wrap individual clients with `wrapTypeSafe()` to choose which TypeSafe calls to trace.
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
import { initLogger, wrapTypeSafe } from "braintrust";
import { TypeSafeClient, choice } from "@typesafe-ai/sdk";
const logger = initLogger({ projectName: "typesafe-example" });
const client = wrapTypeSafe(new TypeSafeClient());
const response = await client.systemOne({
state: "I was charged twice. Please refund the duplicate charge today.",
questions: {
category: choice("Which team should handle this request?", {
billing: "Payments and refunds",
technical: "Software problems",
other: "Other requests",
}),
},
});
console.log(response.answers.category.choice);
await logger.flush();
```
**AI SDK** — If your application calls Jev through AI SDK's `experimental_evaluate()`, wrap the `ai` module with `wrapAISDK()` instead. This requires AI SDK v7.0.103 or later within v7. In your existing AI SDK application, replace the direct import of `experimental_evaluate()` with the wrapped export:
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
import * as ai from "ai";
import { initLogger, wrapAISDK } from "braintrust";
initLogger({ projectName: "typesafe-example" });
const { experimental_evaluate } = wrapAISDK(ai);
```
Call the wrapped `experimental_evaluate()` with your existing evaluation model, state, and questions. Braintrust records an `evaluate` span with type `question`. The import hook also instruments this function. For evaluation calls, use the wrapper or import hook rather than relying only on AI SDK telemetry callbacks.
What Braintrust traces
For each `TypeSafeClient.systemOne()` call, Braintrust records a `typesafe.systemOne` span with type `question`:
* Input state and questions, including question identifiers, instructions, and criteria
* Structured answers, including choices, scores, Noul values, and returned confidence and probabilities
* Model and provider metadata
* Token usage reported by TypeSafe and request duration
* Errors raised by the call
Setup
Requires Braintrust v0.41.0+ and `typesafe-sdk` v0.6.0+. You need a TypeSafe account and API key.
```bash uv theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
uv add "braintrust>=0.41.0" "typesafe-sdk>=0.6.0"
```
```bash pip theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
pip install "braintrust>=0.41.0" "typesafe-sdk>=0.6.0"
```
Set your [Braintrust API key](/docs/admin/authentication#api-authentication) and TypeSafe API key in your shell:
```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
export BRAINTRUST_API_KEY=""
export TYPESAFE_API_KEY=""
```
For the EU data plane, also set `BRAINTRUST_API_URL` to `https://api-eu.braintrust.dev`. For a self-hosted deployment, use your data plane URL.
Auto-instrumentation
Call `braintrust.auto_instrument()` before creating your TypeSafe clients to trace synchronous and asynchronous calls.
Save this example as `trace_typesafe.py`. It asks three questions about the same support request.
```python title="trace_typesafe.py" theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
import braintrust
from typesafe_sdk import Choice, Noul, Score, TypeSafeClient
logger = braintrust.init_logger(project="typesafe-example")
braintrust.auto_instrument()
with TypeSafeClient() as client:
response = client.system_one(
state="I was charged twice. Please refund the duplicate charge today.",
questions={
"category": Choice(
instructions="Which team should handle this request?",
criteria={
"billing": "Payments and refunds",
"technical": "Software problems",
"other": "Other requests",
},
),
"urgency": Score(
instructions="How urgent is this request?",
criteria=["routine", "soon", "urgent"],
),
"duplicate_charge": Noul(
instructions="Does the customer report a duplicate charge?",
),
},
)
print(response.answers)
logger.flush()
```
```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
python trace_typesafe.py
```
Go to your project's [** Logs**](https://www.braintrust.dev/app/~/logs) and select the `typesafe.systemOne` span to [inspect the decisions](#inspect-jev-decisions).
Manual instrumentation
Wrap individual clients with `wrap_typesafe()`. This example uses the asynchronous client. The same wrapper supports `TypeSafeClient` for synchronous calls.
```python Python theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
import asyncio
import braintrust
from braintrust.integrations.typesafe import wrap_typesafe
from typesafe_sdk import AsyncTypeSafeClient, Choice
logger = braintrust.init_logger(project="typesafe-example")
async def main():
async with wrap_typesafe(AsyncTypeSafeClient()) as client:
response = await client.system_one(
state="I was charged twice. Please refund the duplicate charge today.",
questions={
"category": Choice(
instructions="Which team should handle this request?",
criteria={
"billing": "Payments and refunds",
"technical": "Software problems",
"other": "Other requests",
},
),
},
)
print(response.answers["category"].choice)
asyncio.run(main())
logger.flush()
```
What Braintrust traces
For each synchronous or asynchronous `system_one()` call, Braintrust records a `typesafe.systemOne` span with type `question`:
* Input state and questions, including question identifiers, instructions, and criteria
* Structured answers, including choices, scores, Noul values, and returned confidence and probabilities
* Model and provider metadata
* Token usage reported by TypeSafe and request duration
* Errors raised by the call
### Inspect Jev decisions
Select a question span in a trace and use the **Pretty** view for its input and output. The input shows the state sent to Jev and each question's instructions and criteria. The output matches answers to questions by identifier, taken from the keys in your request, so you can inspect the decision alongside the question that produced it.
What each answer type shows, keeping in mind that model responses can vary:
* **Noul**: Probability of yes, displayed as a percentage on a scale from no to yes.
* **Choice**: The selected option, per-option probabilities, and confidence when returned.
* **Score**: The numeric result on the rubric's scale, level descriptions, per-level probabilities, and confidence when returned.
Per-option probabilities show how Jev distributes probability across the possible answers. [TypeSafe confidence](https://docs.typesafe.ai/confidence), a value between `0` and `1`, summarizes how concentrated that distribution is. It is separate from the selected option's probability and from an evaluation score. High confidence does not guarantee a correct decision.
## LLM-as-a-judge
Use Jev as the model in an [LLM-as-a-judge](/docs/evaluate/llm-as-a-judge) evaluator. It reads AI output, plus any context you pass it, and returns one of the choices you define rather than text. Braintrust turns that choice into either a numeric score, such as `pass` → `1` and `fail` → `0`, or a classifying label, such as `billing` or `technical`.
Jev works best for frequent, focused checks with a clear answer, such as whether a support reply follows your refund policy. For broad or nuanced criteria like tone or helpfulness, where the useful output is a judgment rather than a selection, a general-purpose LLM is the better judge. See [Choose a judge model](/docs/evaluate/llm-as-a-judge#choose-a-judge-model).
### Configure access
Braintrust runs the evaluator, so you give it access to Jev in your settings rather than in your application. There are two ways:
* **Braintrust's built-in Jev**, which needs no TypeSafe account of your own. Braintrust's built-in Jev is free to use for scorers and classifiers. It is not available through the AI gateway. Go to ** Settings** > [** AI providers**](https://www.braintrust.dev/app/~/configuration/org/secrets), click **Enable Jev**, and agree to send data submitted to Jev to TypeSafe for processing.
This requires [built-in models](/docs/admin/ai-providers#manage-built-in-models) to be allowed for your organization, and only members of the **Owners** [permission group](/docs/admin/access-control), or a custom permission group with the **Manage settings** organization permission, can change the setting. Disabling Jev can take up to a minute to take effect.
* **A key from your own TypeSafe account**, if you already have one. [Add TypeSafe as an AI provider](/docs/admin/ai-providers#organization-providers) at organization or project scope.
### Create an evaluator
Go to [** Scorers**](https://www.braintrust.dev/app/~/scorers) and click **Scorer**. Enter a name and slug, select **LLM judge**, and choose **Jev** under the provider you configured above. Jev appears in the model picker for scorers and classifiers, not in the picker for generating outputs in prompts or playgrounds.
Selecting Jev hides the model parameter and tool controls, because it takes no sampling parameters and selects from your choices without a tool schema.
Write a prompt that includes the context needed to make the decision and explains what each choice means. Jev judges text, so media and tool message content are rejected. To judge a trace that contains tool calls, pass the relevant context as text.
For example:
```text theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
Evaluate whether the support response follows this refund policy:
Unused items can be refunded within 30 days of purchase.
Used items and purchases older than 30 days are not eligible.
Customer request: {{input}}
Support response: {{output}}
Choose pass if the response applies the policy correctly to the request.
Choose fail if it contradicts the policy or promises an ineligible refund.
```
The prompt defines the criteria. The output configuration in the next step defines what Jev's decision becomes.
Both output types require at least one non-empty choice. Jev returns its decision directly rather than reasoning first, so the **Use chain of thought (CoT)** setting doesn't appear.
Set **Output type** to **Score**. Under **Choice scores**, add each choice and the number it maps to, such as `pass` with score `1` and `fail` with score `0`. Choices and scores must be unique.
If your criteria do not apply to every input, enable **Allow skip** and explain when to skip in the prompt. Jev gains a reserved `Skip` choice, and selecting it returns a `null` score so the case is excluded from score aggregates rather than counted as `0`.
Set **Output type** to **Classification**. Under **Classifications**, add each label Jev can choose. Labels must be unique, and Jev selects exactly one. The selected label becomes both the `id` and the `label` of the resulting classification.
Reach for a classifier when the interesting answer is which failure occurred, not how good the output was. Grading a support reply for whether it can be sent, for example, distinguishes cases a single number would flatten together:
* `Incorrect, unsafe, or invents information`
* `Accurate, but does not answer the customer's request`
* `Answers the request, but misses a necessary detail or next step`
* `Accurate, answers the request, and clearly explains the next step`
To let Jev return no label when none of yours fit, enable **Allow "No match"**. The result is then `no_match`.
In the **Run** section, enter test values for `input` and `output` in the editor, then click **Test**.
For the scorer above, set `input` to `I bought this item 10 days ago and have not used it. Can I get a refund?` and `output` to `Yes, your unused item is eligible for a refund within 30 days.` The intended result is `pass`, with score `1`.
Repeat with `input` set to `I bought this item 45 days ago and have used it. Can I get a refund?` and the same output. The intended result is `fail`, with score `0`. Compare Jev's actual decisions with these judgments and refine the prompt if they disagree.
Click **Save as custom scorer**. Select the saved evaluator when you [run an experiment](/docs/evaluate/run-evaluations) or [score production traces](/docs/evaluate/score-online) to apply the same criteria to new outputs.
### Interpret the result
Jev returns one of the choices you define, which Braintrust maps to a number for a scorer or keeps as a label for a classifier. The mapping ignores confidence, so a `pass` choice scored `1` returns `1` even when Jev is unsure. Both scorers and classifiers carry the same metadata:
| Field | Meaning |
| --------------------------------- | ------------------------------------------------- |
| `metadata.choice` | The selected choice. |
| `metadata.typesafe.model` | The resolved Jev model. |
| `metadata.typesafe.confidence` | How concentrated the probability distribution is. |
| `metadata.typesafe.probabilities` | The probability assigned to each choice. |
Use confidence and probabilities to investigate uncertain judgments. Compare confidence with correctness on human-reviewed examples before choosing a threshold for further review. Routing uncertain results to a person or another judge requires your own logic.
An evaluator's model call appears as a `TypeSafe Jev` LLM span.
## Resources
**Tracing**
* [TypeSafe quickstart](https://docs.typesafe.ai/introduction/quickstart)
* [TypeSafe client SDKs](https://docs.typesafe.ai/sdk)
* [Customize traces](/docs/instrument/trace-application-logic) beyond what the integration records
**LLM-as-a-judge**
* [Run experiments](/docs/evaluate/run-evaluations) with your saved evaluator
* [Configure online scoring](/docs/evaluate/score-online) to evaluate production traces
* [Develop LLM-as-a-judge evaluators](/docs/evaluate/llm-as-a-judge) for span, trace, or group scoring