TypeSafe

TypeSafe provides Jev, a model that makes structured decisions instead of generating text. You define the question and its allowed answers before the call, and Jev returns one of them, so nothing downstream has to find a decision inside prose or malformed JSON. You can use Braintrust with Jev in two ways:

Trace Jev in your application: If your application calls Jev to route support requests, assess urgency, or make other decisions, instrument those calls to inspect their inputs, answers, and probabilities in Braintrust traces.
Use Jev for evaluations: Configure Jev as the LLM judge to check AI outputs against your criteria, such as whether a support response follows a refund policy. Braintrust runs the evaluator and turns Jev’s decisions into scores or labels in experiments or online scoring.

Tracing

Instrument TypeSafe calls in your application to capture their inputs, results, timing, and errors in Braintrust question spans.

Setup

Requires Braintrust v3.34.0+ and @typesafe-ai/sdk v0.6.0 or later within v0.x. You need a TypeSafe account and API key.

Install packages

pnpm add braintrust@^3.34.0 @typesafe-ai/sdk@^0.6.0

Set environment variables

Set your Braintrust API key and TypeSafe API key in your shell:

export BRAINTRUST_API_KEY="<your-braintrust-api-key>"
export TYPESAFE_API_KEY="<your-typesafe-api-key>"

For the EU data plane, also set BRAINTRUST_API_URL to https://api-eu.braintrust.dev. For a self-hosted deployment, use your data plane URL.

Auto-instrumentation

Use Braintrust’s import hook to trace TypeSafe calls throughout your application.

Initialize Braintrust and call Jev

Save this example as trace-typesafe-auto.js. It asks Jev to route a support request, assess its urgency, and identify whether it mentions a duplicate charge.

import { initLogger } from "braintrust";
import { TypeSafeClient, choice, noul, score } from "@typesafe-ai/sdk";

const logger = initLogger({ projectName: "typesafe-example" });
const client = new TypeSafeClient();

const response = await client.systemOne({
  state: "I was charged twice. Please refund the duplicate charge today.",
  questions: {
    category: choice("Which team should handle this request?", {
      billing: "Payments and refunds",
      technical: "Software problems",
      other: "Other requests",
    }),
    urgency: score("How urgent is this request?", [
      "routine",
      "soon",
      "urgent",
    ]),
    duplicate_charge: noul("Does the customer report a duplicate charge?"),
  },
});

console.log(response.answers);
await logger.flush();

Run with the import hook

node --import braintrust/hook.mjs trace-typesafe-auto.js

The auto-instrumentation example uses plain JavaScript so node --import can run the file directly. The Braintrust APIs work the same in TypeScript projects — compile your TypeScript to JavaScript, then run the compiled file with the import hook.

If you’re using a bundler, see Trace LLM calls for plugin and loader setup.

Go to your project’s Logs and select the typesafe.systemOne span to inspect the decisions.

Manual instrumentation

Wrap individual clients with wrapTypeSafe() to choose which TypeSafe calls to trace.

import { initLogger, wrapTypeSafe } from "braintrust";
import { TypeSafeClient, choice } from "@typesafe-ai/sdk";

const logger = initLogger({ projectName: "typesafe-example" });
const client = wrapTypeSafe(new TypeSafeClient());

const response = await client.systemOne({
  state: "I was charged twice. Please refund the duplicate charge today.",
  questions: {
    category: choice("Which team should handle this request?", {
      billing: "Payments and refunds",
      technical: "Software problems",
      other: "Other requests",
    }),
  },
});

console.log(response.answers.category.choice);
await logger.flush();

AI SDK — If your application calls Jev through AI SDK’s experimental_evaluate(), wrap the ai module with wrapAISDK() instead. This requires AI SDK v7.0.103 or later within v7. In your existing AI SDK application, replace the direct import of experimental_evaluate() with the wrapped export:

import * as ai from "ai";
import { initLogger, wrapAISDK } from "braintrust";

initLogger({ projectName: "typesafe-example" });
const { experimental_evaluate } = wrapAISDK(ai);

Call the wrapped experimental_evaluate() with your existing evaluation model, state, and questions. Braintrust records an evaluate span with type question. The import hook also instruments this function. For evaluation calls, use the wrapper or import hook rather than relying only on AI SDK telemetry callbacks.

What Braintrust traces

For each TypeSafeClient.systemOne() call, Braintrust records a typesafe.systemOne span with type question:

Input state and questions, including question identifiers, instructions, and criteria
Structured answers, including choices, scores, Noul values, and returned confidence and probabilities
Model and provider metadata
Token usage reported by TypeSafe and request duration
Errors raised by the call

Inspect Jev decisions

Select a question span in a trace and use the Pretty view for its input and output. The input shows the state sent to Jev and each question’s instructions and criteria. The output matches answers to questions by identifier, taken from the keys in your request, so you can inspect the decision alongside the question that produced it.

A typesafe.systemOne span showing the question criteria sent to Jev and three rendered answers: a Choice of billing at 100 percent, a Score of 1.74 out of 2 across routine, soon, and urgent levels, and a Noul of 99 percent true.

What each answer type shows, keeping in mind that model responses can vary:

Noul: Probability of yes, displayed as a percentage on a scale from no to yes.
Choice: The selected option, per-option probabilities, and confidence when returned.
Score: The numeric result on the rubric’s scale, level descriptions, per-level probabilities, and confidence when returned.

Per-option probabilities show how Jev distributes probability across the possible answers. TypeSafe confidence, a value between 0 and 1, summarizes how concentrated that distribution is. It is separate from the selected option’s probability and from an evaluation score. High confidence does not guarantee a correct decision.

LLM-as-a-judge

Use Jev as the model in an LLM-as-a-judge evaluator. It reads AI output, plus any context you pass it, and returns one of the choices you define rather than text. Braintrust turns that choice into either a numeric score, such as pass → 1 and fail → 0, or a classifying label, such as billing or technical. Jev works best for frequent, focused checks with a clear answer, such as whether a support reply follows your refund policy. For broad or nuanced criteria like tone or helpfulness, where the useful output is a judgment rather than a selection, a general-purpose LLM is the better judge. See Choose a judge model.

Configure access

Braintrust runs the evaluator, so you give it access to Jev in your settings rather than in your application. There are two ways:

Braintrust’s built-in Jev, which needs no TypeSafe account of your own. Braintrust’s built-in Jev is free to use for scorers and classifiers. It is not available through the AI gateway. Go to Settings > AI providers, click Enable Jev, and agree to send data submitted to Jev to TypeSafe for processing.
This requires built-in models to be allowed for your organization, and only members of the Owners permission group, or a custom permission group with the Manage settings organization permission, can change the setting. Disabling Jev can take up to a minute to take effect.
A key from your own TypeSafe account, if you already have one. Add TypeSafe as an AI provider at organization or project scope.

Create an evaluator

Create the evaluator

Go to Scorers and click Scorer. Enter a name and slug, select LLM judge, and choose Jev under the provider you configured above. Jev appears in the model picker for scorers and classifiers, not in the picker for generating outputs in prompts or playgrounds.Selecting Jev hides the model parameter and tool controls, because it takes no sampling parameters and selects from your choices without a tool schema.

Write the prompt

Write a prompt that includes the context needed to make the decision and explains what each choice means. Jev judges text, so media and tool message content are rejected. To judge a trace that contains tool calls, pass the relevant context as text.For example:

Evaluate whether the support response follows this refund policy:
Unused items can be refunded within 30 days of purchase.
Used items and purchases older than 30 days are not eligible.

Customer request: {{input}}
Support response: {{output}}

Choose pass if the response applies the policy correctly to the request.
Choose fail if it contradicts the policy or promises an ineligible refund.

The prompt defines the criteria. The output configuration in the next step defines what Jev’s decision becomes.

Configure the output

Both output types require at least one non-empty choice. Jev returns its decision directly rather than reasoning first, so the Use chain of thought (CoT) setting doesn’t appear.

Scorer
Classifier

Set Output type to Score. Under Choice scores, add each choice and the number it maps to, such as pass with score 1 and fail with score 0. Choices and scores must be unique.If your criteria do not apply to every input, enable Allow skip and explain when to skip in the prompt. Jev gains a reserved Skip choice, and selecting it returns a null score so the case is excluded from score aggregates rather than counted as 0.

Set Output type to Classification. Under Classifications, add each label Jev can choose. Labels must be unique, and Jev selects exactly one. The selected label becomes both the id and the label of the resulting classification.Reach for a classifier when the interesting answer is which failure occurred, not how good the output was. Grading a support reply for whether it can be sent, for example, distinguishes cases a single number would flatten together:

Incorrect, unsafe, or invents information
Accurate, but does not answer the customer's request
Answers the request, but misses a necessary detail or next step
Accurate, answers the request, and clearly explains the next step

To let Jev return no label when none of yours fit, enable Allow “No match”. The result is then no_match.

Test known cases

In the Run section, enter test values for input and output in the editor, then click Test.For the scorer above, set input to I bought this item 10 days ago and have not used it. Can I get a refund? and output to Yes, your unused item is eligible for a refund within 30 days. The intended result is pass, with score 1.Repeat with input set to I bought this item 45 days ago and have used it. Can I get a refund? and the same output. The intended result is fail, with score 0. Compare Jev’s actual decisions with these judgments and refine the prompt if they disagree.

Save the evaluator

Click Save as custom scorer. Select the saved evaluator when you run an experiment or score production traces to apply the same criteria to new outputs.

Interpret the result

Jev returns one of the choices you define, which Braintrust maps to a number for a scorer or keeps as a label for a classifier. The mapping ignores confidence, so a pass choice scored 1 returns 1 even when Jev is unsure. Both scorers and classifiers carry the same metadata:

Field	Meaning
`metadata.choice`	The selected choice.
`metadata.typesafe.model`	The resolved Jev model.
`metadata.typesafe.confidence`	How concentrated the probability distribution is.
`metadata.typesafe.probabilities`	The probability assigned to each choice.

Use confidence and probabilities to investigate uncertain judgments. Compare confidence with correctness on human-reviewed examples before choosing a threshold for further review. Routing uncertain results to a person or another judge requires your own logic. An evaluator’s model call appears as a TypeSafe Jev LLM span.

Resources

Tracing

TypeSafe quickstart
TypeSafe client SDKs
Customize traces beyond what the integration records

LLM-as-a-judge

Run experiments with your saved evaluator
Configure online scoring to evaluate production traces
Develop LLM-as-a-judge evaluators for span, trace, or group scoring

AI providers

SDKs

Developer tools

Runtime environments

Tracing

Setup

Auto-instrumentation

Manual instrumentation

What Braintrust traces

Inspect Jev decisions

LLM-as-a-judge

Configure access

Create an evaluator

Interpret the result

Resources

​Tracing

​Setup

​Auto-instrumentation

​Manual instrumentation

​What Braintrust traces

​Inspect Jev decisions

​LLM-as-a-judge

​Configure access

​Create an evaluator

​Interpret the result

​Resources

Tracing

Setup

Auto-instrumentation

Manual instrumentation

What Braintrust traces

Inspect Jev decisions

LLM-as-a-judge

Configure access

Create an evaluator

Interpret the result

Resources