Systematic Evaluation of LLMs and Agents – From Heuristics to Measurability

Since the launch of ChatGPT in 2022, general-purpose large language models (LLMs) have become widely available. Natural language processing is thus no longer the exclusive domain of data scientists and ML specialists. Thanks to their ability to solve a wide range of tasks right “out of the box,” the time-consuming training of specialized models is often no longer necessary.

Since these generative AI (GenAI) models require enormous computing power, they often exceed the limits of traditional infrastructure. For this reason, they are primarily run in the cloud and integrated into existing software applications via REST interfaces.

However, this leads to a common misconception when integrating GenAI: content generation is far more unpredictable than with traditional software. As a result, the verifiability of the entire application—or at least the subprocesses controlled by LLMs—is inherently non-deterministic.

In practice, this means that initial manual test runs may yield excellent results. However, with a larger number of user queries, an agent may unexpectedly go astray, get caught in endless reasoning loops, or cause the quality of a RAG system to plummet drastically following even a minor prompt adjustment.

Data science already provides proven ML methods for statistical evaluation. In the following, we’ll demonstrate how these approaches can be successfully adapted to the age of GenAI.

Anatomy of an AI App: Input, Context, and Parameters

Whether the AI-based application we want to develop is an autonomous agent, a Retrieval-Augmented Generation (RAG) system, or a complex prompt chain is, for the purposes of this discussion, of secondary importance. At its core, each of these applications can be reduced to a fundamental paradigm: a system defined by specific control variables (parameters) that transforms inputs into outputs.

The key difference from traditional software lies in the nature and processing of this data:

1. Input Distribution: The inputs do not come from a manageable set of test cases, but rather from a broad, highly unpredictable distribution of real-world user queries that vary greatly in linguistic form.
2. Dynamic Context: This primary input is almost always accompanied by a dynamic context, such as documents retrieved from a vector database or the chat history to date.
3. Parameters: How and which context is used is dictated by the application’s parameters. These include architectural decisions (chunk size, retrieval methods), hyperparameters (temperature), the base model, and, of course, the system prompt.

When this interplay meets probabilistic language models, the downright chaotic nature of generative AI becomes apparent: These systems behave in a highly nonlinear manner and are extremely sensitive to their initial conditions. Even minimal deviations in the input or in the dynamically loaded context can drastically alter the output.

This high sensitivity makes it clear that spot-checking—also known as “vibe checking,” analogous to vibe coding—is no longer sufficient. A future-proof development workflow must incorporate continuous, statistical evaluation as an integral component from day one.

To illustrate this process visually, we’ll use the analogy of building a rail line. The first PoC of our AI application represents the actual rail system. This is where daily application traffic flows: user inputs are comparable to the various trains rolling along our track.

Observability: The Signaling Mechanism of AI

To continuously evaluate a system, we must first make its runtime behavior transparent and measurable. The foundation for this is the principle of observability: How do the system’s outputs change in response to incoming user requests, the loaded context, and the selected parameters?

The operational lever for this is the interplay between logging and tracing. Traditional logging merely records isolated events—which falls short when dealing with multi-layered AI architectures. Tracing, on the other hand, makes the entire execution path of a request transparent. Here, complex processes are broken down into a hierarchical tree structure consisting of individual sub-steps (spans). Only tracing reveals the exact cause-and-effect relationships, latencies, and costs of every single decision.

In the analogy of our rail project, there is another component—besides the main track—that is essential for preventing disasters: signaling technology. It makes traffic measurable and prevents collisions on the track—or, more specifically, safety risks such as hallucinations, data leakage, or prompt injections, as defined in the industry-wide OWASP Top 10 for Large Language Model Applications.

Without signaling systems, our rail line will not receive approval from the Federal Railway Authority (EBA). This is not necessarily the case with AI applications. Only when an application falls under Level 3 (high-risk system) of the EU AI Act does the use of certain observability mechanisms become mandatory. However, while the EBA’s regulations contain strict requirements, the EU AI Act is kept rather open-ended. So how do we implement these requirements in practice?

Tracing with OpenTelemetry

To establish the required “signaling system” for our rail line and meet the requirements of regulatory authorities as well as our internal quality standards, we must install sensors along the route. These sensors track every train movement, log switch positions, and report anomalies in real time to a central signal office. Applied to our AI architecture, this means we must instrument the data processing path.

Technologically, this relies on the concept of tracing. The established de facto standard for this is OpenTelemetry (OTel) —an open-source, vendor-neutral framework from the Cloud Native Computing Foundation (CNCF). OTel defines a uniform specification for how telemetry data (traces, metrics, logs) is generated, collected, and exported in a standardized manner, regardless of the programming language used or the final analysis tool. A complete run of a user request—the entire journey of our train from the starting station to the destination station – is referred to as a trace in this standard. This trace is composed of individual spans, which represent the isolated sub-steps and nodes of information processing. Each span includes several attributes.

Let’s look at a concrete example: a Retrieval-Augmented Generation (RAG) pipeline. A user asks a question; the system searches a vector database for relevant context and passes it to the LLM, which generates the final answer.

To make this execution path transparent, we instrument the code using the OTel SDK:

Distributed Traces as a Data Structure

Modern LLM observability frameworks (such as Langfuse, Arize Phoenix, MLFlow, or specialized integrations from Datadog, to name just a few) increasingly rely on this OTel standard under the hood, while incorporating key simplifications that improve developer experience. Instead of manually opening traces and assigning attributes one by one, these frameworks often rely on simple decorators that reduce boilerplate code to a minimum and automatically capture model-specific details—such as token usage or the selected system prompt.

But how do these local logs make their way to the developer’s dashboard? This is where OpenTelemetry’s standardized architecture really shines: The metadata generated by code extensions (instrumentation) is not written directly to the main application but is sent (asynchronously) in the background by a so-called exporter. This exporter transmits the data to an OTel Collector—essentially the control center—which receives the data, filters it, and forwards it to the desired front end. This front end can be a specialized GenAI platform or a general-purpose tool like Grafana. Importantly, this asynchronous transmission process does not slow down the actual execution of the AI application.

Standardized Vocabulary: Semantic Conventions

To ensure that not only the format but also the content of the data is understood across different vendors, the CNCF additionally defines so-called Semantic Conventions. These specify a standardized vocabulary for telemetry data. This is essential in the GenAI environment: If Tool A calls the input tokens prompt_tokens and Tool B calls them input_length, centralized analysis becomes impossible. The Semantic Conventions therefore stipulate that this value be logged consistently across the industry, for example, under the key genai.usage.input_tokens.

From a Black Box to an Auditable Glass Box

What appears to be a mandatory technical task at the code level provides an immense level of detail through this standardization in the frontend. Instead of a simple “Status: OK,” a hierarchical trace tree is visualized. A detailed payload (usually in JSON format) is recorded for each span.

To illustrate this, let’s look at an example payload for the LLM call (the second span in our example), which follows CNCF conventions:

Based on this precisely structured data, we can determine:

• What dynamic context was loaded from the database during the retrieval span?
• Which system prompt, model, and hyperparameters were used?
• What were the latency and token consumption, accurate to the millisecond and cent?

This is precisely where the circle closes regarding auditability and regulations such as the EU AI Act: If an agent produces a hallucination, violates guidelines, or generates toxic content, these standardized payloads make it possible to retrospectively and completely trace whether the cause was an irrelevant entry in the vector database, a faulty system instruction, or a model artifact. The inherent “black box” nature of generative AI thus becomes—at least at the process level—an auditable “glass box.” This establishes the fundamental prerequisite for professional operation.

However, tracing is not yet an evaluation; rather, it is a necessary prerequisite for one. While a trace shows exactly how the AI arrived at a result, it does not reveal whether the final answer is correct. To stay with the metaphor: The signal box measures the train’s punctuality but does not know the passengers’ satisfaction.

The Test Track: From Production Traces to Structured Experiments

To distill real quality metrics from the collected raw data, the data now leaves the main track and is routed to our parallel measurement system. Real data is the foundation for robust experiments.

So we draw on our production traces collected via OpenTelemetry and extract hundreds of real user requests from them. To perform meaningful performance comparisons, this dataset is often enriched with so-called “golden answers”—that is, human-verified, ideal reference responses. This validated dataset now serves as our test bed.

We can use any experimentation framework as the “lab” for this test bed. Most established experimentation frameworks have adapted to today’s GenAI landscape. In our examples, we’ll use MLflow.

While OTel represents the sensor technology in production, MLflow is the orchestration platform for the development cycle. It allows us to systematically test a new configuration (such as a new base model, a changed chunk size, or an optimized system prompt) against our historical dataset.

In practice, such an experimental toolkit can be set up quite easily with the right framework. For example, MLFlow has an OTel HTTP endpoint (/v1/traces) to which production traces can be sent by common OTel exporters. If the traces are created using GenAI semantic conventions, MLFlow can recognize and display them directly. In the MLFlow UI, we can then filter and annotate the collected traces and compile them into evaluation datasets.

The End of the Vibe Check: Why We Need to Experiment

Anyone who develops LLM applications without such an experimentation toolkit will inevitably fall into the regression trap. A classic scenario: A user reports that the RAG agent answered a specific question incorrectly. The developer then adjusts the system prompt, tests the question manually, and is pleased with the now-correct result (the so-called “vibe check”). What they don’t see is that this minor adjustment has rendered the application unusable for 20 other queries that were previously working perfectly.

Systematic experiments eliminate this flying blind and offer three key advantages:

• Preventing Regressions: Every change must prove itself against the entire historical dataset. If overall performance declines, the update does not go into production.
• Data-Driven Architecture Decisions: Is it worth switching from GPT-4O-Mini to the more expensive GPT-4O? The experiment provides hard numbers: Does the error rate drop significantly, or are API costs the primary factor driving the increase?
• Complete reproducibility: MLflow versions not only the results but also the exact code, prompt, and hyperparameters of each run. Nothing is lost; every state of the application can be restored.

Why We Need to Trace in the Lab, Too

A common misconception is that tracing (as described in the previous section) is relevant only for monitoring live production. In fact, it is at least as critical in the experimental lab.

If, after a run, our test bench reports that Version B has a 15% higher error rate than Version A, we have a symptom but not yet a diagnosis. This is precisely where tracing in an experiment shows its strength: it enables comparability. In the UI, we can place the trace of a failed request from Version B right next to the successful trace from Version A.

This allows us to see exactly at which node in the graph the new pipeline took a wrong turn. Did the modified retrieval suddenly load irrelevant chunks from the vector database? Or was the context correct, but the LLM hallucinated due to the new prompt? Tracing in the experiment provides the answer before the rollout.

In addition, many evaluation methods today are “LLM-as-a-Judge” approaches. That is, the outputs of the AI application are verified using another model. However, this makes our test bench an AI application in its own right. Therefore, to ensure that our LLM-based judge actually classifies positive examples as positive and negative examples as negative, we should verify the experiments ourselves by tracing them against human-reviewed reference judgments.

Outlook: The TÜV for Our Models

To stick with our railroad analogy: We have now built a modern track system, complete with signaling technology and a state-of-the-art test bench, and equipped both with the most sophisticated sensors. We can measure exactly how fast the train is traveling and how much electricity it consumes. But who actually decides whether the ride was pleasant? Who verifies that the AI’s responses are factually correct, safe, and relevant?

The mere existence of infrastructure does not answer questions of quality. The specific evaluation methods—ranging from classic heuristic text metrics (such as ROUGE or BLEU) to RAG-specific frameworks (such as RAGAS) and cutting-edge approaches in which a powerful LLM evaluates the responses of another LLM (LLM-as-a-Judge)—constitute the actual mechanism for quality control. They are the “TÜV inspectors” and “passenger surveyors” of our system and will be examined in detail in the next article in this series.