Evaluation

DeepEval: Pytest-Native Evaluation for LLM Applications and Agents

By Omer DemirAug 28, 2026⏱ 13 min read👁 — reads⧉ — copies

AI disclosure: This article was moderated by AI.

DeepEval treats an LLM evaluation as a software test. The test case contains the inputs and observed behavior. A metric decides whether that behavior is acceptable, and a failed threshold returns a non-zero result that CI can block.

The familiar Pytest syntax is only part of the appeal. DeepEval can score a final answer, an agent’s full trajectory, or a single span within that trajectory, and its metric catalog is much larger than those of the other tools in this series. Those capabilities come with a substantial Python dependency graph and frequent LLM-judge calls. The product runs locally first, but its path to collaboration and production monitoring leads to the commercial Confident AI platform.

This assessment covers DeepEval 4.2.0 using public information available on August 28, 2026. The Python SDK is the production reference here. A TypeScript package exists, but its registry and dependency metadata were inconsistent during this review.

Evaluation model

DeepEval has four core data structures:

Keep goldens separate from test cases. A golden should not contain last week’s actual_output. The evaluation loop needs to invoke the current application and capture its current output and trace before constructing the test case. Otherwise, the suite grades a fixture instead of the code under review.

There are three evaluation scopes:

DeepEval has the broadest application scope in this comparison. Vally and Waza are better suited to filesystem-changing coding agents and agent skills. DeepEval works best when the unit under test can be represented as an LLM test case or an instrumented trace.

Installation and dependencies

The Python package supports Python 3.9 through 3.14 in its published metadata:

python -m venv .venv
source .venv/bin/activate
pip install 'deepeval==4.2.0'
export DEEPEVAL_TELEMETRY_OPT_OUT=1

DeepEval is considerably larger than an assertion library. Its mandatory dependencies include Pytest, parallel and retry plugins, Pydantic 2, OpenAI, Rich, Typer, OpenTelemetry, PostHog, aiohttp, requests, gRPC, and supporting libraries. In return, you get an integrated CLI, concurrent execution, tracing, telemetry, and cloud connectivity. You also get more lockfile churn, longer installs, and more transitive packages for security teams to review.

The optional trace inspector adds more dependencies:

pip install 'deepeval[inspect]==4.2.0'
deepeval inspect

DeepEval also publishes a TypeScript SDK:

npm install --save-dev deepeval

At the review snapshot, npm’s website displayed 0.9.13, while the registry API exposed 0.9.11 as latest. Requesting 0.9.13 returned ETARGET, and a clean 0.9.11 CLI invocation failed because @sentry/node was missing. Revalidate the TypeScript API before adoption rather than assuming the documentation tabs imply feature parity.

First test: use a deterministic gate

Most built-in DeepEval metrics call an LLM. When a requirement can be expressed as a fact, start the release gate there. This custom metric checks an exact business outcome without spending tokens or introducing judge variance:

from deepeval import assert_test
from deepeval.metrics import BaseMetric
from deepeval.test_case import LLMTestCase


class ExactMatchMetric(BaseMetric):
    def __init__(self, threshold: float = 1.0):
        self.threshold = threshold
        self.score = None
        self.reason = None
        self.success = None
        self.error = None

    def measure(self, test_case: LLMTestCase) -> float:
        actual = (test_case.actual_output or '').strip()
        expected = (test_case.expected_output or '').strip()
        self.score = float(actual == expected)
        self.reason = f'Normalized exact match: {actual!r} == {expected!r}'
        self.success = self.score >= self.threshold
        return self.score

    async def a_measure(self, test_case: LLMTestCase) -> float:
        return self.measure(test_case)

    def is_successful(self) -> bool:
        if self.error is not None:
            self.success = False
        else:
            self.success = self.score >= self.threshold
        return self.success

    @property
    def __name__(self) -> str:
        return 'Exact Match'


def test_refund_policy() -> None:
    test_case = LLMTestCase(
        input='Can I return an unopened item after 14 days?',
        actual_output='Yes. Unopened items can be returned within 30 days.',
        expected_output='Yes. Unopened items can be returned within 30 days.',
    )
    assert_test(test_case, [ExactMatchMetric()], run_async=False)

Run the test through DeepEval rather than invoking Pytest directly:

DEEPEVAL_TELEMETRY_OPT_OUT=1 deepeval test run test_refund.py -n 1

The CLI prints each metric’s score, threshold, reason, and pass/fail status, then the Pytest summary. A successful run has this information shape:

Metrics:
    Exact Match (score: 1.0, threshold: 1.0, strict: False)
  Reason: Normalized exact match: 'Yes. ... 30 days.' == 'Yes. ... 30 days.'

1 passed

This output is a shortened representation of DeepEval’s documented result fields, not a captured 4.2.0 terminal transcript. Two installation attempts for this article reached the 4.2.0 package metadata but timed out while downloading the wheel, so I did not execute a local test. The code uses the current custom-metric contract, though terminal colors, labels, and spacing may differ. Only the output structure was verified; the result was not measured locally.

Add a semantic judge with G-Eval

Deterministic checks cannot tell whether an explanation is complete, grounded, or appropriately cautious. GEval converts natural-language criteria into evaluation steps, then asks a judge model for a normalized verdict:

from deepeval import assert_test
from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCase, SingleTurnParams


correctness = GEval(
    name='Policy correctness',
    criteria=(
        'The answer must state the 30-day return window and must not invent '
        'restocking fees, receipt requirements, or product exclusions.'
    ),
    evaluation_params=[
        SingleTurnParams.ACTUAL_OUTPUT,
        SingleTurnParams.EXPECTED_OUTPUT,
    ],
    threshold=0.8,
    model='gpt-4o-mini',
)


def test_semantic_refund_policy() -> None:
    assert_test(
        LLMTestCase(
            input='Can I return an unopened item after 14 days?',
            actual_output='Yes. Unopened items can be returned within 30 days.',
            expected_output='Unopened items may be returned within 30 days.',
        ),
        [correctness],
    )

This path requires a provider key unless the judge runs locally. DeepEval can route judges to OpenAI, Azure OpenAI, Anthropic, Gemini, Bedrock, Ollama, OpenRouter, LiteLLM, and OpenAI-compatible endpoints. Custom providers implement DeepEvalBaseLLM, including synchronous and asynchronous generation methods and structured output behavior.

The resulting score is a model judgment, not a physical measurement. It depends on the prompt, model version, sampling configuration, and framework template. Calibrate it against human-labeled examples, and rerun borderline cases before allowing a difference such as 0.79 versus 0.81 to decide a release.

RAG evaluation

DeepEval splits RAG quality into retriever and generator questions:

from deepeval import evaluate
from deepeval.metrics import AnswerRelevancyMetric, FaithfulnessMetric
from deepeval.test_case import LLMTestCase

test_case = LLMTestCase(
    input='How long can an unopened item be returned?',
    actual_output='An unopened item can be returned within 30 days.',
    retrieval_context=[
        'Return policy: unopened merchandise is eligible for return for 30 days.'
    ],
)

evaluate(
    test_cases=[test_case],
    metrics=[
        AnswerRelevancyMetric(threshold=0.8),
        FaithfulnessMetric(threshold=0.9),
    ],
)

Avoid averaging these metrics too early. A relevant but unsupported answer points to generation, while an irrelevant chunk set points to retrieval. Keeping that boundary intact is what makes the metrics useful for diagnosis.

Agent trajectories and component metrics

For agents, @observe creates a trace tree. update_current_trace supplies the observable input and output, while integrations can capture nested model, retriever, and tool spans automatically:

from deepeval.dataset import EvaluationDataset, Golden
from deepeval.evaluate import AsyncConfig
from deepeval.metrics import TaskCompletionMetric
from deepeval.tracing import observe, update_current_trace


@observe()
def support_agent(query: str) -> str:
    policy = lookup_policy(query)
    answer = compose_answer(query, policy)
    update_current_trace(input=query, output=answer)
    return answer


dataset = EvaluationDataset(
    goldens=[Golden(input='Can I return an unopened item after 14 days?')]
)

for golden in dataset.evals_iterator(
    metrics=[TaskCompletionMetric(threshold=0.8)],
    async_config=AsyncConfig(run_async=False),
):
    support_agent(golden.input)

Trajectory metrics include task completion, step efficiency, plan adherence, and plan quality. Tool correctness and argument correctness operate at the component level for individual tool-calling decisions. They can be combined, but the suite still needs to answer three separate questions:

1. Did the agent achieve the outcome?
2. Was its path acceptable and efficient?
3. Which component caused the failure?

DeepEval has integrations for LangChain, LangGraph, OpenAI, OpenAI Agents, Anthropic, Pydantic AI, CrewAI, LlamaIndex, Google ADK, AWS AgentCore, Strands, Mastra, Vercel AI SDK, and OpenInference. Integration depth varies across languages and frameworks. Verify that the integration captures the spans you need before selecting a metric that depends on them.

CI and result persistence

assert_test converts metric thresholds into test failures. A minimal GitHub Actions gate is:

name: llm-evals
on: [pull_request]

jobs:
  deepeval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.12'
          cache: pip
      - run: pip install 'deepeval==4.2.0'
      - run: deepeval test run tests/evals -n 2
        env:
          DEEPEVAL_TELEMETRY_OPT_OUT: '1'
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

Use one worker while debugging rate limits, then increase concurrency with care. DeepEval runs metrics concurrently by default, and one test case may trigger several internal judge requests. A simple request count is therefore only a lower bound.

Results can stay local as JSON by setting DEEPEVAL_RESULTS_FOLDER. Logging in or setting a CONFIDENT_API_KEY enables upload to Confident AI for shared reports, regression views, dataset management, trace storage, and online evaluation.

Cost model

The open-source framework uses the Apache-2.0 license and has no license fee. Its costs fall into three independent categories:

For $S$ test cases, $R$ repeated runs, and $M$ judge-backed metrics, the floor is approximately $S \times R \times M$ metric evaluations. It is not necessarily the number of billable LLM calls: faithfulness, relevancy, and similar algorithms can use multiple calls to extract claims, generate questions, or produce reasons.

Confident AI’s public plans at the review date were:

Hosted pricing does not include model-provider charges. Prices and included usage change, so capture the current pricing page in an architecture decision instead of treating this table as a contract.

Privacy and data boundaries

“Runs locally” does not mean that all data stays on the machine. Three controls determine egress:

1. Anonymous PostHog telemetry is enabled by default. Disable it with DEEPEVAL_TELEMETRY_OPT_OUT=1.
2. LLM-backed metrics send test-case or trace content to the configured judge provider. Use an approved endpoint or a local judge for restricted data.
3. Confident AI credentials enable result and trace upload. Omit them when artifacts must remain local.

A zero-egress design requires all three controls. Local JSON and trace snapshots are sensitive as well because they can contain prompts, retrieved documents, tool arguments, model responses, and application metadata.

Ownership, support, and community

DeepEval is maintained by Confident AI rather than a cloud provider or standards body. The Python repository is Apache-2.0, and Confident AI sells the hosted platform and enterprise support around it.

The public GitHub snapshot showed approximately 17.9k stars, 1.9k forks, 308 contributors, 62 releases, 251 open issues, and 236 open pull requests. Commits had landed within hours of the review. Those numbers indicate substantial adoption and active development. The backlog also makes clear that community support is not an SLA. Hosted Team and Enterprise plans advertise progressively stronger support arrangements.

The repository CI tested Python and TypeScript paths, but Python’s core workflow concentrated on Python 3.11 even though package metadata declared a wider interpreter range. Test your selected interpreter rather than inferring equivalent coverage from the metadata classifier.

Pros and cons

Good fit and poor fit

Choose DeepEval when:

• The team is Python-first and already uses Pytest.
• One suite must cover RAG, chat, structured output, and agent behavior.
• Trace-level diagnosis matters alongside black-box quality.
• The team wants a choice of providers for LLM judges.
• Hosted collaboration may be added later without replacing the test API.

Look elsewhere when:

• The primary artifact is a SKILL.md file whose trigger precision is the main risk.
• The team needs hermetic filesystem and shell-command graders for coding agents.
• Every release gate must be deterministic and model-free.
• The production stack is TypeScript-only and cannot tolerate packaging churn.
• An air-gapped environment cannot host an approved local judge.

Verdict

DeepEval is the strongest general-purpose starting point in this series. Its metrics cover output quality, while its trace support helps diagnose agent failures. The practical approach is to start with deterministic outcomes, then add one or two calibrated semantic metrics for requirements that code cannot express. Do not add five LLM judges and average their scores. Reserve hard CI gates for signals that remain stable across repeated runs.

The next article examines Microsoft Vally, whose declarative YAML, trajectory graders, and offline re-grading trade DeepEval’s Pytest-centric model for a Node.js agent-evaluation pipeline.

References

• DeepEval introduction
• DeepEval quickstart
• Metric architecture and catalog
• Custom metric contract
• End-to-end evaluation
• Data privacy and telemetry
• DeepEval GitHub repository
• Confident AI pricing

#evaluation#deepeval#ai-agents#llm-judge#pytest