{"id":71258,"date":"2026-10-29T13:51:59","date_gmt":"2026-10-29T12:51:59","guid":{"rendered":"https:\/\/www.inovex.de\/?p=71258"},"modified":"2026-11-03T09:32:58","modified_gmt":"2026-11-03T08:32:58","slug":"systematic-evaluation-of-llms-and-agents-from-heuristics-to-measurability","status":"publish","type":"post","link":"https:\/\/www.inovex.de\/en\/blog\/systematic-evaluation-of-llms-and-agents-from-heuristics-to-measurability\/","title":{"rendered":"Systematic Evaluation of LLMs and Agents \u2013 From Heuristics to Measurability"},"content":{"rendered":"<p>Since the launch of <a href=\"https:\/\/openai.com\/de-DE\/index\/chatgpt\/\">ChatGPT<\/a> in 2022, general-purpose large language models (LLMs) have become widely available. Natural language processing is thus no longer the exclusive domain of data scientists and ML specialists. Thanks to their ability to solve a wide range of tasks right \u201cout of the box,\u201d the time-consuming training of specialized models is often no longer necessary.<\/p>\n<p>Since these generative AI (GenAI) models require enormous computing power, they often exceed the limits of traditional infrastructure. For this reason, they are primarily run in the cloud and integrated into existing software applications via REST interfaces.<!--more--><\/p>\n<p>However, this leads to a common misconception when integrating GenAI: content generation is far more unpredictable than with traditional software. As a result, the verifiability of the entire application\u2014or at least the subprocesses controlled by LLMs\u2014is inherently non-deterministic.<\/p>\n<p>In practice, this means that initial manual test runs may yield excellent results. However, with a larger number of user queries, an agent may unexpectedly go astray, get caught in endless reasoning loops, or cause the quality of a RAG system to plummet drastically following even a minor prompt adjustment.<\/p>\n<p>Data science already provides proven ML methods for statistical evaluation. In the following, we\u2019ll demonstrate how these approaches can be successfully adapted to the age of GenAI.<\/p>\n<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_89 counter-hierarchy ez-toc-counter ez-toc-custom ez-toc-container-direction\">\n<div class=\"ez-toc-title-container\"><p class=\"ez-toc-title\" style=\"cursor:inherit\"><\/p>\n<\/div><nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/www.inovex.de\/en\/blog\/systematic-evaluation-of-llms-and-agents-from-heuristics-to-measurability\/#Anatomy-of-an-AI-App-Input-Context-and-Parameters\" >Anatomy of an AI App: Input, Context, and Parameters<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/www.inovex.de\/en\/blog\/systematic-evaluation-of-llms-and-agents-from-heuristics-to-measurability\/#Observability-The-Signaling-Mechanism-of-AI\" >Observability: The Signaling Mechanism of AI<\/a><ul class='ez-toc-list-level-3' ><li class='ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/www.inovex.de\/en\/blog\/systematic-evaluation-of-llms-and-agents-from-heuristics-to-measurability\/#Tracing-with-OpenTelemetry\" >Tracing with OpenTelemetry<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/www.inovex.de\/en\/blog\/systematic-evaluation-of-llms-and-agents-from-heuristics-to-measurability\/#Distributed-Traces-as-a-Data-Structure\" >Distributed Traces as a Data Structure<\/a><ul class='ez-toc-list-level-4' ><li class='ez-toc-heading-level-4'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/www.inovex.de\/en\/blog\/systematic-evaluation-of-llms-and-agents-from-heuristics-to-measurability\/#Standardized-Vocabulary-Semantic-Conventions\" >Standardized Vocabulary: Semantic Conventions<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-4'><a class=\"ez-toc-link ez-toc-heading-6\" href=\"https:\/\/www.inovex.de\/en\/blog\/systematic-evaluation-of-llms-and-agents-from-heuristics-to-measurability\/#From-a-Black-Box-to-an-Auditable-Glass-Box\" >From a Black Box to an Auditable Glass Box<\/a><\/li><\/ul><\/li><\/ul><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-7\" href=\"https:\/\/www.inovex.de\/en\/blog\/systematic-evaluation-of-llms-and-agents-from-heuristics-to-measurability\/#The-Test-Track-From-Production-Traces-to-Structured-Experiments\" >The Test Track: From Production Traces to Structured Experiments<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-8\" href=\"https:\/\/www.inovex.de\/en\/blog\/systematic-evaluation-of-llms-and-agents-from-heuristics-to-measurability\/#The-End-of-the-Vibe-Check-Why-We-Need-to-Experiment\" >The End of the Vibe Check: Why We Need to Experiment<\/a><ul class='ez-toc-list-level-3' ><li class='ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-9\" href=\"https:\/\/www.inovex.de\/en\/blog\/systematic-evaluation-of-llms-and-agents-from-heuristics-to-measurability\/#Why-We-Need-to-Trace-in-the-Lab-Too\" >Why We Need to Trace in the Lab, Too<\/a><\/li><\/ul><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-10\" href=\"https:\/\/www.inovex.de\/en\/blog\/systematic-evaluation-of-llms-and-agents-from-heuristics-to-measurability\/#Outlook-The-TUV-for-Our-Models\" >Outlook: The T\u00dcV for Our Models<\/a><\/li><\/ul><\/nav><\/div>\n<h2><span class=\"ez-toc-section\" id=\"Anatomy-of-an-AI-App-Input-Context-and-Parameters\"><\/span>Anatomy of an AI App: Input, Context, and Parameters<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Whether the AI-based application we want to develop is an autonomous agent, a Retrieval-Augmented Generation (RAG) system, or a complex prompt chain is, for the purposes of this discussion, of secondary importance. At its core, each of these applications can be reduced to a fundamental paradigm: a system defined by specific control variables (parameters) that transforms inputs into outputs.<\/p>\n<p>The key difference from traditional software lies in the nature and processing of this data:<\/p>\n<ol>\n<li>Input Distribution: The inputs do not come from a manageable set of test cases, but rather from a broad, highly unpredictable distribution of real-world user queries that vary greatly in linguistic form.<\/li>\n<li>Dynamic Context: This primary input is almost always accompanied by a dynamic context, such as documents retrieved from a vector database or the chat history to date.<\/li>\n<li>Parameters: How and which context is used is dictated by the application\u2019s parameters. These include architectural decisions (chunk size, retrieval methods), hyperparameters (temperature), the base model, and, of course, the system prompt.<\/li>\n<\/ol>\n<p>When this interplay meets probabilistic language models, the downright chaotic nature of generative AI becomes apparent: These systems behave in a highly nonlinear manner and are extremely sensitive to their initial conditions. Even minimal deviations in the input or in the dynamically loaded context can drastically alter the output.<\/p>\n<p>This high sensitivity makes it clear that spot-checking\u2014also known as \u201cvibe checking,\u201d analogous to vibe coding\u2014is no longer sufficient. A future-proof development workflow must incorporate continuous, statistical evaluation as an integral component from day one.<\/p>\n<p>To illustrate this process visually, we\u2019ll use the analogy of building a rail line. The first PoC of our AI application represents the actual rail system. This is where daily application traffic flows: user inputs are comparable to the various trains rolling along our track.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Observability-The-Signaling-Mechanism-of-AI\"><\/span>Observability: The Signaling Mechanism of AI<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>To continuously evaluate a system, we must first make its runtime behavior transparent and measurable. The foundation for this is the principle of observability: How do the system\u2019s outputs change in response to incoming user requests, the loaded context, and the selected parameters?<\/p>\n<p>The operational lever for this is the interplay between logging and tracing. Traditional logging merely records isolated events\u2014which falls short when dealing with multi-layered AI architectures. Tracing, on the other hand, makes the entire execution path of a request transparent. Here, complex processes are broken down into a hierarchical tree structure consisting of individual sub-steps (spans). Only tracing reveals the exact cause-and-effect relationships, latencies, and costs of every single decision.<\/p>\n<p>In the analogy of our rail project, there is another component\u2014besides the main track\u2014that is essential for preventing disasters: signaling technology. It makes traffic measurable and prevents collisions on the track\u2014or, more specifically, safety risks such as hallucinations, data leakage, or prompt injections, as defined in the industry-wide <a class=\"ng-star-inserted\" href=\"https:\/\/owasp.org\/www-project-top-10-for-large-language-model-applications\/\" target=\"_blank\" rel=\"noopener\">OWASP Top 10 for Large Language Model Applications<\/a>.<\/p>\n<p>Without signaling systems, our rail line will not receive approval from the Federal Railway Authority (EBA). This is not necessarily the case with AI applications. Only when an application falls under <a href=\"https:\/\/artificialintelligenceact.eu\/de\/high-level-summary\/\">Level 3 (high-risk system) of the EU AI Act<\/a> does the use of certain observability mechanisms become mandatory. However, while the EBA\u2019s regulations contain strict requirements, the EU AI Act is kept rather open-ended. So how do we implement these requirements in practice?<\/p>\n<h3><span class=\"ez-toc-section\" id=\"Tracing-with-OpenTelemetry\"><\/span>Tracing with OpenTelemetry<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>To establish the required \u201csignaling system\u201d for our rail line and meet the requirements of regulatory authorities as well as our internal quality standards, we must install sensors along the route. These sensors track every train movement, log switch positions, and report anomalies in real time to a central signal office. Applied to our AI architecture, this means we must instrument the data processing path.<\/p>\n<p>Technologically, this relies on the concept of tracing. The established de facto standard for this is <a href=\"https:\/\/opentelemetry.io\/\"><b><em>OpenTelemetry<\/em> (OTel)<\/b><\/a> \u2014an open-source, vendor-neutral framework from the <a href=\"https:\/\/www.cncf.io\/\">Cloud Native Computing Foundation (CNCF)<\/a>. OTel defines a uniform specification for how telemetry data (traces, metrics, logs) is generated, collected, and exported in a standardized manner, regardless of the programming language used or the final analysis tool. A complete run of a user request\u2014the entire journey of our train from the starting station to the destination station \u2013 is referred to as a trace in this standard. This trace is composed of individual spans, which represent the isolated sub-steps and nodes of information processing. Each span includes several attributes.<\/p>\n<p>Let\u2019s look at a concrete example: a Retrieval-Augmented Generation (RAG) pipeline. A user asks a question; the system searches a vector database for relevant context and passes it to the LLM, which generates the final answer.<\/p>\n<p>To make this execution path transparent, we instrument the code using the OTel SDK:<\/p>\n<pre class=\"lang:python decode:true\" title=\"Tracing RAG Beispiel\">from opentelemetry import trace\r\nfrom my_rag_functions import answer_query, get_embedding, retreive_docs\r\n\r\n# Initialize the TracerProvider\r\ntracer = trace.get_tracer(\"my_ai_app\")\r\n\r\n\r\n@tracer.start_as_current_span(\"retrieval H7STPQYOND\")\r\ndef retrieve_documents(query: str, n_docs: int = 10) -&gt; list[str]:\r\n    \"\"\"records a span for retrieval.\"\"\"\r\n    current_span = trace.get_current_span()\r\n\r\n    # set some span attributes\r\n    current_span.set_attribute(\"gen_ai.operation.name\", \"retieval\")\r\n    current_span.set_attribute(\"gen_ai.data_source.id\", \"H7STPQYOND\")\r\n    current_span.set_attribute(\"gen_ai.retreival.top_k\", n_docs)\r\n    current_span.set_attribute(\"gen_ai.retrieval.query.text\", query)\r\n\r\n    # perform the actual operation\r\n    query_embedding = get_embedding(query)\r\n    retrieved_docs = retreive_docs(query_embedding, n_docs=n_docs)\r\n\r\n    # Record the retreived docs\r\n    current_span.add_attribute(\"gen_ai.retrieval.documents\", retrieved_docs)\r\n    return retrieved_docs\r\n\r\n\r\n@tracer.start_as_current_span(\"text_completion gpt-6-astra\")\r\ndef generate_answer(query: str, context: list[str]) -&gt; str:\r\n    \"\"\"Record a span for generating a response based on the context docs.\"\"\"\r\n    current_span = trace.get_current_span()\r\n\r\n    # compile input prompt\r\n    input_message = f\"\"\"{query}\r\n    CONTEXT:\r\n    {\"\\n\\n\".join(context)}\r\n    \"\"\"\r\n\r\n    # set span attributes for the inputs\r\n    current_span.set_attributes(\r\n        {\r\n            \"gen_ai.operation.name\": \"text_completion\",\r\n            \"gen_ai.provider.name\": \"openai\",\r\n            \"gen_ai.request.model\": \"gpt-6-astra\",\r\n            \"gen_ai.request.temperature\": 0.7,\r\n            \"gen_ai.input.messages\": [input_message],\r\n        }\r\n    )\r\n\r\n    answer = answer_query(input_message)\r\n\r\n    # 2. set span attributes for outputs\r\n    current_span.set_attributes(\r\n        {\r\n            \"gen_ai.conversation.id\": answer.session_id,\r\n            \"gen_ai.usage\": {\r\n                \"input_tokens\": answer.input_tokens,\r\n                \"output_tokens\": answer.output_tokens,\r\n            },\r\n            \"gen_ai.output.messages\": answer.output_messages,\r\n        }\r\n    )\r\n    return answer\r\n\r\n\r\ndef rag_pipeline(query: str):\r\n    \"\"\"E2E Rag Pipeline.\"\"\"\r\n    with tracer.start_as_current_span(\"rag_pipeline\"):\r\n        # Step 1: Retrieval\r\n        docs = retrieve_documents(query)\r\n\r\n        # Step 2: Generation\r\n        answer = generate_answer(query, docs)\r\n\r\n        return answer\r\n<\/pre>\n<h3><span class=\"ez-toc-section\" id=\"Distributed-Traces-as-a-Data-Structure\"><\/span>Distributed Traces as a Data Structure<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>Modern LLM observability frameworks (such as <a href=\"https:\/\/langfuse.com\/\"><em>Langfuse<\/em><\/a>,<a href=\"https:\/\/arize.com\/phoenix\/\"><em> Arize Phoenix<\/em><\/a>, <a href=\"https:\/\/mlflow.org\/\"><em>MLFlow<\/em><\/a>, or specialized integrations from <a href=\"https:\/\/www.datadoghq.com\/\"><em>Datadog<\/em><\/a>, to name just a few) increasingly rely on this OTel standard under the hood, while incorporating key simplifications that improve developer experience. Instead of manually opening traces and assigning attributes one by one, these frameworks often rely on simple decorators that reduce boilerplate code to a minimum and automatically capture model-specific details\u2014such as token usage or the selected system prompt.<\/p>\n<p>But how do these local logs make their way to the developer\u2019s dashboard? This is where OpenTelemetry\u2019s standardized architecture really shines: The metadata generated by code extensions (instrumentation) is not written directly to the main application but is sent (asynchronously) in the background by a so-called exporter. This exporter transmits the data to an OTel Collector\u2014essentially the control center\u2014which receives the data, filters it, and forwards it to the desired front end. This front end can be a specialized GenAI platform or a general-purpose tool like <a href=\"https:\/\/grafana.com\/\"><em>Grafana<\/em><\/a>. Importantly, this asynchronous transmission process does not slow down the actual execution of the AI application.<\/p>\n<h4><span class=\"ez-toc-section\" id=\"Standardized-Vocabulary-Semantic-Conventions\"><\/span>Standardized Vocabulary: Semantic Conventions<span class=\"ez-toc-section-end\"><\/span><\/h4>\n<p>To ensure that not only the format but also the content of the data is understood across different vendors, the CNCF additionally defines so-called <a href=\"https:\/\/github.com\/open-telemetry\/semantic-conventions-genai\"><b>Semantic Conventions<\/b><\/a>. These specify a standardized vocabulary for telemetry data. This is essential in the GenAI environment: If Tool A calls the input tokens <span class=\"lang:default decode:true crayon-inline \">prompt_tokens<\/span> and Tool B calls them <span class=\"lang:default decode:true crayon-inline\">input_length<\/span>, centralized analysis becomes impossible. The Semantic Conventions therefore stipulate that this value be logged consistently across the industry, for example, under the key <span class=\"lang:default decode:true crayon-inline \">genai.usage.input_tokens<\/span>.<\/p>\n<h4><span class=\"ez-toc-section\" id=\"From-a-Black-Box-to-an-Auditable-Glass-Box\"><\/span>From a Black Box to an Auditable Glass Box<span class=\"ez-toc-section-end\"><\/span><\/h4>\n<p>What appears to be a mandatory technical task at the code level provides an immense level of detail through this standardization in the frontend. Instead of a simple \u201cStatus: OK,\u201d a hierarchical trace tree is visualized. A detailed payload (usually in JSON format) is recorded for each span.<\/p>\n<p>To illustrate this, let\u2019s look at an example payload for the LLM call (the second span in our example), which follows CNCF conventions:<\/p>\n<pre class=\"lang:default decode:true \" title=\"Spans JSON Beispiel\">[\r\n    {\r\n        \"name\": \"rag_pipeline\",\r\n        \"context\": {\r\n            \"trace_id\": \"0x5b8aa5a2d2c872e8321cf37308d69df2\",\r\n            \"span_id\": \"0xf6c109f48195b451\",\r\n            \/\/ ...\r\n        },\r\n        \"parent_id\": null,\r\n        \"start_time\": \"2026-10-20T09:14:02.100876Z\",\r\n        \"end_time\": \"2026-10-20T09:14:03.913102Z\",\r\n        \/\/ ...\r\n    },\r\n    {\r\n        \"name\": \"retrieval H7STPQYOND\",\r\n        \"context\": {\r\n            \"trace_id\": \"0x5b8aa5a2d2c872e8321cf37308d69df2\",\r\n            \"span_id\": \"0x051581bf3cb55c13\",\r\n            \/\/ ...\r\n        },\r\n        \"parent_id\": \"0xf6c109f48195b451\",\r\n        \"start_time\": \"2026-10-20T09:14:02.101234Z\",\r\n        \"end_time\": \"2026-10-20T09:14:02.347981Z\",\r\n        \"attributes\": {\r\n            \"gen_ai.operation.name\": \"retrieval\",\r\n            \"gen_ai.data_source.id\": \"H7STPQYOND\",\r\n            \"gen_ai.retrieval.top_k\": 3,\r\n            \"gen_ai.retrieval.query.text\": \"How many vacation days do new employees get?\",\r\n            \"gen_ai.retrieval.documents\": [\r\n                \"New employees are entitled to 42 vacation days per calendar year ...\",\r\n                \/\/ ...\r\n            ]\r\n        },\r\n        \/\/ ...\r\n    },\r\n    {\r\n        \"name\": \"text_completion gpt-6-astra\",\r\n        \"context\": {\r\n            \"trace_id\": \"0x5b8aa5a2d2c872e8321cf37308d69df2\",\r\n            \"span_id\": \"0x3a1f9c7e5d2b8046\",\r\n            \/\/ ...\r\n        },\r\n        \"parent_id\": \"0xf6c109f48195b451\",\r\n        \"start_time\": \"2026-10-20T09:14:02.348502Z\",\r\n        \"end_time\": \"2026-10-20T09:14:03.912775Z\",\r\n        \"attributes\": {\r\n            \"gen_ai.operation.name\": \"text_completion\",\r\n            \"gen_ai.provider.name\": \"openai\",\r\n            \"gen_ai.request.model\": \"gpt-6-astra\",\r\n            \"gen_ai.request.temperature\": 0.7,\r\n            \"gen_ai.input.messages\": [\r\n                \/\/ ...\r\n            ],\r\n            \"gen_ai.conversation.id\": \"sess_9f3c2a71\",\r\n            \"gen_ai.usage.input_tokens\": 142,\r\n            \"gen_ai.usage.output_tokens\": 38,\r\n            \"gen_ai.output.messages\": [\r\n                \/\/ ...\r\n            ]\r\n        },\r\n        \/\/ ...\r\n    }\r\n]<\/pre>\n<p>&nbsp;<\/p>\n<pre class=\"lang:default decode:true\">[ { <span class=\"hljs-attr\">\"traceId\"<\/span>: <span class=\"hljs-string\">\"71380d3221a71e21b8c1605ad1322ab7\"<\/span>, <span class=\"hljs-attr\">\"spanId\"<\/span>: <span class=\"hljs-string\">\"341829abc12837ff\"<\/span>, <span class=\"hljs-attr\">\"parentSpanId\"<\/span>: <span class=\"hljs-string\">\"\"<\/span>, <span class=\"hljs-attr\">\"name\"<\/span>: <span class=\"hljs-string\">\"rag_pipeline\"<\/span>, <span class=\"hljs-attr\">\"kind\"<\/span>: <span class=\"hljs-string\">\"SPAN_KIND_INTERNAL\"<\/span>, <span class=\"hljs-attr\">\"startTimeUnixNano\"<\/span>: <span class=\"hljs-string\">\"1726066228000000000\"<\/span>, <span class=\"hljs-attr\">\"endTimeUnixNano\"<\/span>: <span class=\"hljs-string\">\"1726066230500000000\"<\/span>, <span class=\"hljs-attr\">\"status\"<\/span>: { <span class=\"hljs-attr\">\"code\"<\/span>: <span class=\"hljs-string\">\"STATUS_CODE_UNSET\"<\/span> }, <span class=\"hljs-attr\">\"attributes\"<\/span>: [] }, { <span class=\"hljs-attr\">\"traceId\"<\/span>: <span class=\"hljs-string\">\"71380d3221a71e21b8c1605ad1322ab7\"<\/span>, <span class=\"hljs-attr\">\"spanId\"<\/span>: <span class=\"hljs-string\">\"a987654321fedcba\"<\/span>, <span class=\"hljs-attr\">\"parentSpanId\"<\/span>: <span class=\"hljs-string\">\"341829abc12837ff\"<\/span>, <span class=\"hljs-attr\">\"name\"<\/span>: <span class=\"hljs-string\">\"retrieval H7STPQYOND\"<\/span>, <span class=\"hljs-attr\">\"kind\"<\/span>: <span class=\"hljs-string\">\"SPAN_KIND_INTERNAL\"<\/span>, <span class=\"hljs-attr\">\"startTimeUnixNano\"<\/span>: <span class=\"hljs-string\">\"1726066228050000000\"<\/span>, <span class=\"hljs-attr\">\"endTimeUnixNano\"<\/span>: <span class=\"hljs-string\">\"1726066228850000000\"<\/span>, <span class=\"hljs-attr\">\"status\"<\/span>: { <span class=\"hljs-attr\">\"code\"<\/span>: <span class=\"hljs-string\">\"STATUS_CODE_UNSET\"<\/span> }, <span class=\"hljs-attr\">\"attributes\"<\/span>: [ { <span class=\"hljs-attr\">\"key\"<\/span>: <span class=\"hljs-string\">\"gen_ai.operation.name\"<\/span>, <span class=\"hljs-attr\">\"value\"<\/span>: { <span class=\"hljs-attr\">\"stringValue\"<\/span>: <span class=\"hljs-string\">\"retrieval\"<\/span> } }, { <span class=\"hljs-attr\">\"key\"<\/span>: <span class=\"hljs-string\">\"gen_ai.data_source.id\"<\/span>, <span class=\"hljs-attr\">\"value\"<\/span>: { <span class=\"hljs-attr\">\"stringValue\"<\/span>: <span class=\"hljs-string\">\"H7STPQYOND\"<\/span> } }, { <span class=\"hljs-attr\">\"key\"<\/span>: <span class=\"hljs-string\">\"gen_ai.retrieval.top_k\"<\/span>, <span class=\"hljs-attr\">\"value\"<\/span>: { <span class=\"hljs-attr\">\"intValue\"<\/span>: <span class=\"hljs-string\">\"10\"<\/span> } }, { <span class=\"hljs-attr\">\"key\"<\/span>: <span class=\"hljs-string\">\"gen_ai.retrieval.query.text\"<\/span>, <span class=\"hljs-attr\">\"value\"<\/span>: { <span class=\"hljs-attr\">\"stringValue\"<\/span>: <span class=\"hljs-string\">\"How do I trace GenAI apps?\"<\/span> } }, { <span class=\"hljs-attr\">\"key\"<\/span>: <span class=\"hljs-string\">\"gen_ai.retrieval.documents\"<\/span>, <span class=\"hljs-attr\">\"value\"<\/span>: { <span class=\"hljs-attr\">\"arrayValue\"<\/span>: { <span class=\"hljs-attr\">\"values\"<\/span>: [ { <span class=\"hljs-attr\">\"stringValue\"<\/span>: <span class=\"hljs-string\">\"Doc1 content...\"<\/span> }, { <span class=\"hljs-attr\">\"stringValue\"<\/span>: <span class=\"hljs-string\">\"Doc2 content...\"<\/span> } ] } } } ] }, { <span class=\"hljs-attr\">\"traceId\"<\/span>: <span class=\"hljs-string\">\"71380d3221a71e21b8c1605ad1322ab7\"<\/span>, <span class=\"hljs-attr\">\"spanId\"<\/span>: <span class=\"hljs-string\">\"1a2b3c4d5e6f7a8b\"<\/span>, <span class=\"hljs-attr\">\"parentSpanId\"<\/span>: <span class=\"hljs-string\">\"341829abc12837ff\"<\/span>, <span class=\"hljs-attr\">\"name\"<\/span>: <span class=\"hljs-string\">\"generate_answer\"<\/span>, <span class=\"hljs-attr\">\"kind\"<\/span>: <span class=\"hljs-string\">\"SPAN_KIND_INTERNAL\"<\/span>, <span class=\"hljs-attr\">\"startTimeUnixNano\"<\/span>: <span class=\"hljs-string\">\"1726066228860000000\"<\/span>, <span class=\"hljs-attr\">\"endTimeUnixNano\"<\/span>: <span class=\"hljs-string\">\"1726066230490000000\"<\/span>, <span class=\"hljs-attr\">\"status\"<\/span>: { <span class=\"hljs-attr\">\"code\"<\/span>: <span class=\"hljs-string\">\"STATUS_CODE_UNSET\"<\/span> }, <span class=\"hljs-attr\">\"attributes\"<\/span>: [ { <span class=\"hljs-attr\">\"key\"<\/span>: <span class=\"hljs-string\">\"gen_ai.operation.name\"<\/span>, <span class=\"hljs-attr\">\"value\"<\/span>: { <span class=\"hljs-attr\">\"stringValue\"<\/span>: <span class=\"hljs-string\">\"chat\"<\/span> } }, { <span class=\"hljs-attr\">\"key\"<\/span>: <span class=\"hljs-string\">\"gen_ai.provider.name\"<\/span>, <span class=\"hljs-attr\">\"value\"<\/span>: { <span class=\"hljs-attr\">\"stringValue\"<\/span>: <span class=\"hljs-string\">\"openai\"<\/span> } }, { <span class=\"hljs-attr\">\"key\"<\/span>: <span class=\"hljs-string\">\"gen_ai.request.model\"<\/span>, <span class=\"hljs-attr\">\"value\"<\/span>: { <span class=\"hljs-attr\">\"stringValue\"<\/span>: <span class=\"hljs-string\">\"gpt-4o\"<\/span> } }, { <span class=\"hljs-attr\">\"key\"<\/span>: <span class=\"hljs-string\">\"gen_ai.request.temperature\"<\/span>, <span class=\"hljs-attr\">\"value\"<\/span>: { <span class=\"hljs-attr\">\"doubleValue\"<\/span>: <span class=\"hljs-number\">0.7<\/span> } }, { <span class=\"hljs-attr\">\"key\"<\/span>: <span class=\"hljs-string\">\"gen_ai.response.model\"<\/span>, <span class=\"hljs-attr\">\"value\"<\/span>: { <span class=\"hljs-attr\">\"stringValue\"<\/span>: <span class=\"hljs-string\">\"gpt-4o-2024-05-13\"<\/span> } }, { <span class=\"hljs-attr\">\"key\"<\/span>: <span class=\"hljs-string\">\"gen_ai.usage\"<\/span>, <span class=\"hljs-attr\">\"value\"<\/span>: { <span class=\"hljs-attr\">\"stringValue\"<\/span>: <span class=\"hljs-string\">\"{\\\"input_tokens\\\": 120, \\\"output_tokens\\\": 45}\"<\/span> } }, { <span class=\"hljs-attr\">\"key\"<\/span>: <span class=\"hljs-string\">\"gen_ai.output.messages\"<\/span>, <span class=\"hljs-attr\">\"value\"<\/span>: { <span class=\"hljs-attr\">\"stringValue\"<\/span>: <span class=\"hljs-string\">\"[{\\\"role\\\": \\\"assistant\\\", \\\"content\\\": \\\"OpenTelemetry makes tracing easy...\\\"}]\"<\/span> } } ] } ] } ]<\/pre>\n<p>Based on this precisely structured data, we can determine:<\/p>\n<ul>\n<li>What dynamic context was loaded from the database during the retrieval span?<\/li>\n<li>Which system prompt, model, and hyperparameters were used?<\/li>\n<li>What were the latency and token consumption, accurate to the millisecond and cent?<\/li>\n<\/ul>\n<p>This is precisely where the circle closes regarding auditability and regulations such as the EU AI Act: If an agent produces a hallucination, violates guidelines, or generates toxic content, these standardized payloads make it possible to retrospectively and completely trace whether the cause was an irrelevant entry in the vector database, a faulty system instruction, or a model artifact. The inherent \u201cblack box\u201d nature of generative AI thus becomes\u2014at least at the process level\u2014an auditable \u201cglass box.\u201d This establishes the fundamental prerequisite for professional operation.<\/p>\n<p>However, tracing is not yet an evaluation; rather, it is a necessary prerequisite for one. While a trace shows exactly how the AI arrived at a result, it does not reveal whether the final answer is correct. To stay with the metaphor: The signal box measures the train\u2019s punctuality but does not know the passengers\u2019 satisfaction.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"The-Test-Track-From-Production-Traces-to-Structured-Experiments\"><\/span>The Test Track: From Production Traces to Structured Experiments<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>To distill real quality metrics from the collected raw data, the data now leaves the main track and is routed to our parallel measurement system. Real data is the foundation for robust experiments.<\/p>\n<p>So we draw on our production traces collected via OpenTelemetry and extract hundreds of real user requests from them. To perform meaningful performance comparisons, this dataset is often enriched with so-called \u201cgolden answers\u201d\u2014that is, human-verified, ideal reference responses. This validated dataset now serves as our test bed.<\/p>\n<p>We can use any experimentation framework as the \u201clab\u201d for this test bed. Most established experimentation frameworks have adapted to today\u2019s GenAI landscape. In our examples, we\u2019ll use MLflow.<\/p>\n<p>While OTel represents the sensor technology in production, MLflow is the orchestration platform for the development cycle. It allows us to systematically test a new configuration (such as a new base model, a changed chunk size, or an optimized system prompt) against our historical dataset.<\/p>\n<p>In practice, such an experimental toolkit can be set up quite easily with the right framework. For example, MLFlow has an OTel HTTP endpoint (\/v1\/traces) to which production traces can be sent by common OTel exporters. If the traces are created using GenAI semantic conventions, MLFlow can recognize and display them directly. In the MLFlow UI, we can then filter and annotate the collected traces and compile them into evaluation datasets.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"The-End-of-the-Vibe-Check-Why-We-Need-to-Experiment\"><\/span>The End of the Vibe Check: Why We Need to Experiment<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Anyone who develops LLM applications without such an experimentation toolkit will inevitably fall into the regression trap. A classic scenario: A user reports that the RAG agent answered a specific question incorrectly. The developer then adjusts the system prompt, tests the question manually, and is pleased with the now-correct result (the so-called \u201cvibe check\u201d). What they don\u2019t see is that this minor adjustment has rendered the application unusable for 20 other queries that were previously working perfectly.<\/p>\n<p>Systematic experiments eliminate this flying blind and offer three key advantages:<\/p>\n<ul>\n<li><strong>Preventing Regressions<\/strong>: Every change must prove itself against the entire historical dataset. If overall performance declines, the update does not go into production.<\/li>\n<li><strong>Data-Driven Architecture Decisions<\/strong>: Is it worth switching from GPT-4O-Mini to the more expensive GPT-4O? The experiment provides hard numbers: Does the error rate drop significantly, or are API costs the primary factor driving the increase?<\/li>\n<li><strong>Complete reproducibility<\/strong>: MLflow versions not only the results but also the exact code, prompt, and hyperparameters of each run. Nothing is lost; every state of the application can be restored.<\/li>\n<\/ul>\n<h3><span class=\"ez-toc-section\" id=\"Why-We-Need-to-Trace-in-the-Lab-Too\"><\/span>Why We Need to Trace in the Lab, Too<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>A common misconception is that tracing (as described in the previous section) is relevant only for monitoring live production. In fact, it is at least as critical in the experimental lab.<\/p>\n<p>If, after a run, our test bench reports that Version B has a 15% higher error rate than Version A, we have a symptom but not yet a diagnosis. This is precisely where tracing in an experiment shows its strength: it enables comparability. In the UI, we can place the trace of a failed request from Version B right next to the successful trace from Version A.<\/p>\n<p>This allows us to see exactly at which node in the graph the new pipeline took a wrong turn. Did the modified retrieval suddenly load irrelevant chunks from the vector database? Or was the context correct, but the LLM hallucinated due to the new prompt? Tracing in the experiment provides the answer before the rollout.<\/p>\n<p>In addition, many evaluation methods today are \u201cLLM-as-a-Judge\u201d approaches. That is, the outputs of the AI application are verified using another model. However, this makes our test bench an AI application in its own right. Therefore, to ensure that our LLM-based judge actually classifies positive examples as positive and negative examples as negative, we should verify the experiments ourselves by tracing them against human-reviewed reference judgments.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Outlook-The-TUV-for-Our-Models\"><\/span>Outlook: The T\u00dcV for Our Models<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>To stick with our railroad analogy: We have now built a modern track system, complete with signaling technology and a state-of-the-art test bench, and equipped both with the most sophisticated sensors. We can measure exactly how fast the train is traveling and how much electricity it consumes. But who actually decides whether the ride was pleasant? Who verifies that the AI\u2019s responses are factually correct, safe, and relevant?<\/p>\n<p>The mere existence of infrastructure does not answer questions of quality. The specific evaluation methods\u2014ranging from classic heuristic text metrics (such as ROUGE or BLEU) to RAG-specific frameworks (such as RAGAS) and cutting-edge approaches in which a powerful LLM evaluates the responses of another LLM (LLM-as-a-Judge)\u2014constitute the actual mechanism for quality control. They are the \u201cT\u00dcV inspectors\u201d and \u201cpassenger surveyors\u201d of our system and will be examined in detail in the next article in this series.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Since the launch of ChatGPT in 2022, general-purpose large language models (LLMs) have become widely available. Natural language processing is thus no longer the exclusive domain of data scientists and ML specialists. Thanks to their ability to solve a wide range of tasks right \u201cout of the box,\u201d the time-consuming training of specialized models is [&hellip;]<\/p>\n","protected":false},"author":460,"featured_media":60394,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"inline_featured_image":false,"ep_exclude_from_search":false,"footnotes":""},"tags":[],"service":[473,455,477,471],"level":[1281],"coauthors":[{"id":460,"display_name":"Max Vieweg","user_nicename":"mvieweg"},{"id":436,"display_name":"Denise Hartmann","user_nicename":"dhartmann"}],"class_list":["post-71258","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","service-artificial-intelligence-en","service-data-engineering-en","service-data-science-en","service-nlp-en"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v28.8 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Systematic Evaluation of LLMs and Agents \u2013 From Heuristics to Measurability - inovex GmbH %<\/title>\n<meta name=\"description\" content=\"Why manual testing isn&#039;t enough for LLMs, and why a measurable and auditable evaluation infrastructure is essential for GenAI apps.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.inovex.de\/en\/blog\/systematic-evaluation-of-llms-and-agents-from-heuristics-to-measurability\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Systematic Evaluation of LLMs and Agents \u2013 From Heuristics to Measurability - inovex GmbH %\" \/>\n<meta property=\"og:description\" content=\"Why manual testing isn&#039;t enough for LLMs, and why a measurable and auditable evaluation infrastructure is essential for GenAI apps.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.inovex.de\/en\/blog\/systematic-evaluation-of-llms-and-agents-from-heuristics-to-measurability\/\" \/>\n<meta property=\"og:site_name\" content=\"inovex GmbH\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/inovexde\" \/>\n<meta property=\"article:published_time\" content=\"2026-10-29T12:51:59+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2026-11-03T08:32:58+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.inovex.de\/wp-content\/uploads\/How-to-Use-Mimesis-and-dbt-to-Test-Data-Pipelines.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1500\" \/>\n\t<meta property=\"og:image:height\" content=\"880\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Max Vieweg, Denise Hartmann\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:image\" content=\"https:\/\/www.inovex.de\/wp-content\/uploads\/How-to-Use-Mimesis-and-dbt-to-Test-Data-Pipelines-1024x601.png\" \/>\n<meta name=\"twitter:creator\" content=\"@inovexgmbh\" \/>\n<meta name=\"twitter:site\" content=\"@inovexgmbh\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Max Vieweg, Denise Hartmann\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"15 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/en\\\/blog\\\/systematic-evaluation-of-llms-and-agents-from-heuristics-to-measurability\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/en\\\/blog\\\/systematic-evaluation-of-llms-and-agents-from-heuristics-to-measurability\\\/\"},\"author\":{\"name\":\"Max Vieweg\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/en\\\/#\\\/schema\\\/person\\\/0f159b433119271e258ab68daa8e4fb8\"},\"headline\":\"Systematic Evaluation of LLMs and Agents \u2013 From Heuristics to Measurability\",\"datePublished\":\"2026-10-29T12:51:59+00:00\",\"dateModified\":\"2026-11-03T08:32:58+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/en\\\/blog\\\/systematic-evaluation-of-llms-and-agents-from-heuristics-to-measurability\\\/\"},\"wordCount\":2369,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/en\\\/#organization\"},\"image\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/en\\\/blog\\\/systematic-evaluation-of-llms-and-agents-from-heuristics-to-measurability\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.inovex.de\\\/wp-content\\\/uploads\\\/How-to-Use-Mimesis-and-dbt-to-Test-Data-Pipelines.png\",\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/www.inovex.de\\\/en\\\/blog\\\/systematic-evaluation-of-llms-and-agents-from-heuristics-to-measurability\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/en\\\/blog\\\/systematic-evaluation-of-llms-and-agents-from-heuristics-to-measurability\\\/\",\"url\":\"https:\\\/\\\/www.inovex.de\\\/en\\\/blog\\\/systematic-evaluation-of-llms-and-agents-from-heuristics-to-measurability\\\/\",\"name\":\"Systematic Evaluation of LLMs and Agents \u2013 From Heuristics to Measurability - inovex GmbH %\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/en\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/en\\\/blog\\\/systematic-evaluation-of-llms-and-agents-from-heuristics-to-measurability\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/en\\\/blog\\\/systematic-evaluation-of-llms-and-agents-from-heuristics-to-measurability\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.inovex.de\\\/wp-content\\\/uploads\\\/How-to-Use-Mimesis-and-dbt-to-Test-Data-Pipelines.png\",\"datePublished\":\"2026-10-29T12:51:59+00:00\",\"dateModified\":\"2026-11-03T08:32:58+00:00\",\"description\":\"Why manual testing isn't enough for LLMs, and why a measurable and auditable evaluation infrastructure is essential for GenAI apps.\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/en\\\/blog\\\/systematic-evaluation-of-llms-and-agents-from-heuristics-to-measurability\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/www.inovex.de\\\/en\\\/blog\\\/systematic-evaluation-of-llms-and-agents-from-heuristics-to-measurability\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/en\\\/blog\\\/systematic-evaluation-of-llms-and-agents-from-heuristics-to-measurability\\\/#primaryimage\",\"url\":\"https:\\\/\\\/www.inovex.de\\\/wp-content\\\/uploads\\\/How-to-Use-Mimesis-and-dbt-to-Test-Data-Pipelines.png\",\"contentUrl\":\"https:\\\/\\\/www.inovex.de\\\/wp-content\\\/uploads\\\/How-to-Use-Mimesis-and-dbt-to-Test-Data-Pipelines.png\",\"width\":1500,\"height\":880,\"caption\":\"IT-Security-Assessment\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/en\\\/blog\\\/systematic-evaluation-of-llms-and-agents-from-heuristics-to-measurability\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/www.inovex.de\\\/en\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Systematic Evaluation of LLMs and Agents \u2013 From Heuristics to Measurability\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/en\\\/#website\",\"url\":\"https:\\\/\\\/www.inovex.de\\\/en\\\/\",\"name\":\"inovex GmbH\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/en\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/www.inovex.de\\\/en\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/en\\\/#organization\",\"name\":\"inovex GmbH\",\"url\":\"https:\\\/\\\/www.inovex.de\\\/en\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/en\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/www.inovex.de\\\/wp-content\\\/uploads\\\/2021\\\/03\\\/inovex-logo-16-9-1.png\",\"contentUrl\":\"https:\\\/\\\/www.inovex.de\\\/wp-content\\\/uploads\\\/2021\\\/03\\\/inovex-logo-16-9-1.png\",\"width\":1921,\"height\":1081,\"caption\":\"inovex GmbH\"},\"image\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/en\\\/#\\\/schema\\\/logo\\\/image\\\/\"},\"sameAs\":[\"https:\\\/\\\/www.facebook.com\\\/inovexde\",\"https:\\\/\\\/x.com\\\/inovexgmbh\",\"https:\\\/\\\/www.instagram.com\\\/inovexlife\\\/\",\"https:\\\/\\\/www.linkedin.com\\\/company\\\/inovex\",\"https:\\\/\\\/www.youtube.com\\\/channel\\\/UC7r66GT14hROB_RQsQBAQUQ\"]},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/en\\\/#\\\/schema\\\/person\\\/0f159b433119271e258ab68daa8e4fb8\",\"name\":\"Max Vieweg\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/e97d39105606e6cc4f7732ef88141ae9b5d55442966766986a311b9fa7c0a7ca?s=96&d=retro&r=g092fdda8feef24b121ad617c8d704e03\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/e97d39105606e6cc4f7732ef88141ae9b5d55442966766986a311b9fa7c0a7ca?s=96&d=retro&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/e97d39105606e6cc4f7732ef88141ae9b5d55442966766986a311b9fa7c0a7ca?s=96&d=retro&r=g\",\"caption\":\"Max Vieweg\"},\"url\":\"https:\\\/\\\/www.inovex.de\\\/en\\\/blog\\\/author\\\/mvieweg\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Systematic Evaluation of LLMs and Agents \u2013 From Heuristics to Measurability - inovex GmbH %","description":"Why manual testing isn't enough for LLMs, and why a measurable and auditable evaluation infrastructure is essential for GenAI apps.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.inovex.de\/en\/blog\/systematic-evaluation-of-llms-and-agents-from-heuristics-to-measurability\/","og_locale":"en_US","og_type":"article","og_title":"Systematic Evaluation of LLMs and Agents \u2013 From Heuristics to Measurability - inovex GmbH %","og_description":"Why manual testing isn't enough for LLMs, and why a measurable and auditable evaluation infrastructure is essential for GenAI apps.","og_url":"https:\/\/www.inovex.de\/en\/blog\/systematic-evaluation-of-llms-and-agents-from-heuristics-to-measurability\/","og_site_name":"inovex GmbH","article_publisher":"https:\/\/www.facebook.com\/inovexde","article_published_time":"2026-10-29T12:51:59+00:00","article_modified_time":"2026-11-03T08:32:58+00:00","og_image":[{"width":1500,"height":880,"url":"https:\/\/www.inovex.de\/wp-content\/uploads\/How-to-Use-Mimesis-and-dbt-to-Test-Data-Pipelines.png","type":"image\/png"}],"author":"Max Vieweg, Denise Hartmann","twitter_card":"summary_large_image","twitter_image":"https:\/\/www.inovex.de\/wp-content\/uploads\/How-to-Use-Mimesis-and-dbt-to-Test-Data-Pipelines-1024x601.png","twitter_creator":"@inovexgmbh","twitter_site":"@inovexgmbh","twitter_misc":{"Written by":"Max Vieweg, Denise Hartmann","Est. reading time":"15 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.inovex.de\/en\/blog\/systematic-evaluation-of-llms-and-agents-from-heuristics-to-measurability\/#article","isPartOf":{"@id":"https:\/\/www.inovex.de\/en\/blog\/systematic-evaluation-of-llms-and-agents-from-heuristics-to-measurability\/"},"author":{"name":"Max Vieweg","@id":"https:\/\/www.inovex.de\/en\/#\/schema\/person\/0f159b433119271e258ab68daa8e4fb8"},"headline":"Systematic Evaluation of LLMs and Agents \u2013 From Heuristics to Measurability","datePublished":"2026-10-29T12:51:59+00:00","dateModified":"2026-11-03T08:32:58+00:00","mainEntityOfPage":{"@id":"https:\/\/www.inovex.de\/en\/blog\/systematic-evaluation-of-llms-and-agents-from-heuristics-to-measurability\/"},"wordCount":2369,"commentCount":0,"publisher":{"@id":"https:\/\/www.inovex.de\/en\/#organization"},"image":{"@id":"https:\/\/www.inovex.de\/en\/blog\/systematic-evaluation-of-llms-and-agents-from-heuristics-to-measurability\/#primaryimage"},"thumbnailUrl":"https:\/\/www.inovex.de\/wp-content\/uploads\/How-to-Use-Mimesis-and-dbt-to-Test-Data-Pipelines.png","inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.inovex.de\/en\/blog\/systematic-evaluation-of-llms-and-agents-from-heuristics-to-measurability\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.inovex.de\/en\/blog\/systematic-evaluation-of-llms-and-agents-from-heuristics-to-measurability\/","url":"https:\/\/www.inovex.de\/en\/blog\/systematic-evaluation-of-llms-and-agents-from-heuristics-to-measurability\/","name":"Systematic Evaluation of LLMs and Agents \u2013 From Heuristics to Measurability - inovex GmbH %","isPartOf":{"@id":"https:\/\/www.inovex.de\/en\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.inovex.de\/en\/blog\/systematic-evaluation-of-llms-and-agents-from-heuristics-to-measurability\/#primaryimage"},"image":{"@id":"https:\/\/www.inovex.de\/en\/blog\/systematic-evaluation-of-llms-and-agents-from-heuristics-to-measurability\/#primaryimage"},"thumbnailUrl":"https:\/\/www.inovex.de\/wp-content\/uploads\/How-to-Use-Mimesis-and-dbt-to-Test-Data-Pipelines.png","datePublished":"2026-10-29T12:51:59+00:00","dateModified":"2026-11-03T08:32:58+00:00","description":"Why manual testing isn't enough for LLMs, and why a measurable and auditable evaluation infrastructure is essential for GenAI apps.","breadcrumb":{"@id":"https:\/\/www.inovex.de\/en\/blog\/systematic-evaluation-of-llms-and-agents-from-heuristics-to-measurability\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.inovex.de\/en\/blog\/systematic-evaluation-of-llms-and-agents-from-heuristics-to-measurability\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.inovex.de\/en\/blog\/systematic-evaluation-of-llms-and-agents-from-heuristics-to-measurability\/#primaryimage","url":"https:\/\/www.inovex.de\/wp-content\/uploads\/How-to-Use-Mimesis-and-dbt-to-Test-Data-Pipelines.png","contentUrl":"https:\/\/www.inovex.de\/wp-content\/uploads\/How-to-Use-Mimesis-and-dbt-to-Test-Data-Pipelines.png","width":1500,"height":880,"caption":"IT-Security-Assessment"},{"@type":"BreadcrumbList","@id":"https:\/\/www.inovex.de\/en\/blog\/systematic-evaluation-of-llms-and-agents-from-heuristics-to-measurability\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.inovex.de\/en\/"},{"@type":"ListItem","position":2,"name":"Systematic Evaluation of LLMs and Agents \u2013 From Heuristics to Measurability"}]},{"@type":"WebSite","@id":"https:\/\/www.inovex.de\/en\/#website","url":"https:\/\/www.inovex.de\/en\/","name":"inovex GmbH","description":"","publisher":{"@id":"https:\/\/www.inovex.de\/en\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.inovex.de\/en\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.inovex.de\/en\/#organization","name":"inovex GmbH","url":"https:\/\/www.inovex.de\/en\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.inovex.de\/en\/#\/schema\/logo\/image\/","url":"https:\/\/www.inovex.de\/wp-content\/uploads\/2021\/03\/inovex-logo-16-9-1.png","contentUrl":"https:\/\/www.inovex.de\/wp-content\/uploads\/2021\/03\/inovex-logo-16-9-1.png","width":1921,"height":1081,"caption":"inovex GmbH"},"image":{"@id":"https:\/\/www.inovex.de\/en\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/inovexde","https:\/\/x.com\/inovexgmbh","https:\/\/www.instagram.com\/inovexlife\/","https:\/\/www.linkedin.com\/company\/inovex","https:\/\/www.youtube.com\/channel\/UC7r66GT14hROB_RQsQBAQUQ"]},{"@type":"Person","@id":"https:\/\/www.inovex.de\/en\/#\/schema\/person\/0f159b433119271e258ab68daa8e4fb8","name":"Max Vieweg","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/e97d39105606e6cc4f7732ef88141ae9b5d55442966766986a311b9fa7c0a7ca?s=96&d=retro&r=g092fdda8feef24b121ad617c8d704e03","url":"https:\/\/secure.gravatar.com\/avatar\/e97d39105606e6cc4f7732ef88141ae9b5d55442966766986a311b9fa7c0a7ca?s=96&d=retro&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/e97d39105606e6cc4f7732ef88141ae9b5d55442966766986a311b9fa7c0a7ca?s=96&d=retro&r=g","caption":"Max Vieweg"},"url":"https:\/\/www.inovex.de\/en\/blog\/author\/mvieweg\/"}]}},"_links":{"self":[{"href":"https:\/\/www.inovex.de\/en\/wp-json\/wp\/v2\/posts\/71258","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.inovex.de\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.inovex.de\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.inovex.de\/en\/wp-json\/wp\/v2\/users\/460"}],"replies":[{"embeddable":true,"href":"https:\/\/www.inovex.de\/en\/wp-json\/wp\/v2\/comments?post=71258"}],"version-history":[{"count":3,"href":"https:\/\/www.inovex.de\/en\/wp-json\/wp\/v2\/posts\/71258\/revisions"}],"predecessor-version":[{"id":71316,"href":"https:\/\/www.inovex.de\/en\/wp-json\/wp\/v2\/posts\/71258\/revisions\/71316"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.inovex.de\/en\/wp-json\/wp\/v2\/media\/60394"}],"wp:attachment":[{"href":"https:\/\/www.inovex.de\/en\/wp-json\/wp\/v2\/media?parent=71258"}],"wp:term":[{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.inovex.de\/en\/wp-json\/wp\/v2\/tags?post=71258"},{"taxonomy":"service","embeddable":true,"href":"https:\/\/www.inovex.de\/en\/wp-json\/wp\/v2\/service?post=71258"},{"taxonomy":"level","embeddable":true,"href":"https:\/\/www.inovex.de\/en\/wp-json\/wp\/v2\/level?post=71258"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.inovex.de\/en\/wp-json\/wp\/v2\/coauthors?post=71258"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}