The Observability Problem in AI Agents and Workflows + promptTrace

When we move from an experimental Python notebook to a production-grade AI application, we lose something fundamental: visibility. In traditional software, if a function fails, it throws a stack trace. In an LLM-powered application, a “failure” looks different.

The code might execute perfectly or the response might look normal, returning a 200 OK status, while the model itself provides a nonsensical answer, hallucinated data, or a JSON object that breaks your downstream logic.

This is the Observability Gap—the distance between knowing a system is running and knowing it is right.

An Example?

Most teams start by logging simple inputs and outputs. But as systems grow into multi-agent configurations or complex Retrieval-Augmented Generation (RAG) pipelines, a simple log is no longer enough.

Our AI assistant, Asyra AI leverages a context mapping process which involves ranking the likelihood that a given user query is related to a given conversation pair. This process helps Asyra respond more intuitively to vague or contextual queries.

To produce a response, Asyra might:

• Retrieve the 5 most recent conversation pairs from a vector database
• Rewrite the user query for better clarity.
• Rank the chunks based on relevance using a specialized prompt.
• Format the top three chunks into a specific JSON schema.

If the final output is incorrect, where did the error happen? Was the retrieval step flawed? Did the “Reasoning” agent misunderstand the ranking criteria? It is is not easy to decipher this from the final response.

What are we actually monitoring?

Effective LLM observability isn’t just about saving text; it’s about capturing the execution context. At Asycd, we believe “complete” monitoring must track four distinct pillars:

• The Inputs & Outputs: Not just the raw strings, but the prompt templates, system instructions, and the final structured data (e.g., JSON).
• The Reasoning Chain: Many models are now prompted to “think” before they act (Chain-of-Thought). Capturing this intermediate reasoning is vital for debugging why a model reached a specific conclusion.
• Performance Metadata: Real-time data including latency (time-to-first-token), token count (to manage context window limits), and inference costs.
• Model Lineage: Tracking which specific model (e.g., GPT-4o, Claude 3.5, Gemini 1.5 Flash) and version generated the response.

These pillars are inherent foundations for even deeper analysis in the future. For instance, by storing the model information alongside granular prompt responses, you can…

The Modern Observability Stack

We aren’t building in a vacuum. The industry has responded to this gap with a new generation of tools designed specifically for the non-deterministic nature of AI:

• LangSmith – A comprehensive suite for logging, debugging, and tracing LLM applications, natively integrated with the LangChain ecosystem.
• LangFuse – An open-source observability platform focused on production monitoring, lifecycle tracking, and calculating the ROI of individual prompts.
• Arize Phoenix – An open-source library specialized in tracing, evaluating RAG pipelines, and visualizing embedding clusters to detect "data drift."
• Weights & Biases Weave — A lightweight toolkit for tracking inputs, outputs, and prompt versions, designed to bridge the gap between experimentation and production.

Moving Toward “promptTrace”

At the Asycd Research Lab, we are exploring a solution we call promptTrace. The goal is simple: total visibility for every individual completion.

By treating every LLM call as a traceable event — complete with its own alerts for abnormal behaviour — we can turn “soft failures” into actionable data. Whether you are gathering data for future fine-tuning or setting alerts for when a model’s reasoning starts to drift, observability is the bridge between a “cool demo” and a “reliable product.”