Most AI observability content is written for solo developers or enterprise teams with dedicated ML ops staff. If your engineering team sits somewhere between five and fifty people, you’ve probably noticed the gap: tutorials are too thin, enterprise guides are too heavy, and none of them acknowledge the real constraint — you need this working without adding a full-time role.
So in this article we’re going to define the minimum viable observability stack (MVOS) for a small engineering team, compare five platforms — Langfuse, Arize Phoenix, MLflow, LangSmith, and Braintrust — and give you a decision tree for your context. The open-source vs. SaaS question is the first fork. Team capacity for self-hosting is the second.
If you want the conceptual foundation for what you are instrumenting before diving into tooling comparisons, that is worth reading first. For the broader platform context this builds on, the AI observability and guardrails platform guide covers the full landscape.
What does a minimum viable AI observability stack actually include?
A minimum viable observability stack for a 5–10 person engineering team has three components: LLM tracing, cost tracking, and basic output evaluation. Everything else can wait.
LLM tracing captures the full request-response lifecycle — prompt inputs, model outputs, intermediate chain steps, tool calls, and per-span latency. Think of it as the call stack for AI systems. Without it, debugging a production issue in a non-deterministic AI system is guesswork. You can observe that something went wrong, but you can’t reconstruct why.
Cost tracking monitors token consumption and API spend per model, per feature, and per user segment. Token spend can escalate faster than you expect — it’s often the first metric a small team actually cares about in production. Basic output evaluation uses automated metrics or LLM-as-judge techniques to detect quality regressions, hallucinations, and relevance failures. That’s the semantic quality layer that latency monitoring simply cannot provide.
What’s deferrable at this scale: advanced guardrails, shadow evaluation pipelines, canary deployment workflows, and sophisticated prompt management. Valuable. Not the starting point.
One distinction worth locking in before you pick a tool: monitoring and evaluation are different things. Monitoring is operational — latency, error rates, throughput, cost. Evaluation is semantic — whether outputs are correct, relevant, grounded, and safe. AI observability goes beyond traditional monitoring by requiring qualitative, semantic assessment of model outputs. Traditional APM tools assume deterministic software: working or broken. AI systems drift. Start with tracing and cost visibility. You can’t debug what you never instrumented.
How should a small team decide between open-source and SaaS observability tools?
The open-source vs. SaaS decision is the primary branching point. Every recommendation that follows depends on where you land here.
Open-source self-hosted tools — Langfuse, Arize Phoenix, MLflow — eliminate per-trace SaaS costs but introduce infrastructure overhead: provisioning, patching, scaling, and monitoring the observability system itself. SaaS and managed cloud tools — LangSmith, Braintrust, Langfuse Cloud — reduce operational burden but incur usage-based pricing that scales with trace volume.
For a 5–10 person team without dedicated DevOps capacity, SaaS or managed cloud is typically the right starting point. The engineering time cost of self-hosting usually exceeds the subscription cost in the first year. The “people TCO” for self-hosted stacks — operational personnel and on-call duties — often adds up to $1,600–$4,800 per month in engineering time, on top of zero licensing costs.
For teams with data residency requirements — EU/GDPR, regulated industries, customer contracts restricting data storage — self-hosted open source provides data sovereignty and eliminates per-trace cost at scale.
One clarification that trips people up: “open source” and “self-hosted” are not the same thing. Langfuse and Arize Phoenix are both open source, but self-hosting requires real infrastructure investment. Both also offer managed cloud tiers. Braintrust offers a third path — hybrid deployment, where your data stays in your AWS/GCP/Azure environment while you use the managed UI. Useful for teams with data residency requirements that aren’t ready for full self-hosting.
For platform-level decisions that precede tooling selection, there’s more context on how observability fits into the broader AI platform architecture.
What does Langfuse offer and when is it the right choice for a small team?
Langfuse is the most widely adopted open-source LLM observability platform. It covers tracing, prompt management, evaluation, and cost tracking in a single self-hostable package. The self-hosted version includes all features at no licensing cost. The managed Langfuse Cloud tier has a free Hobby plan with core features and paid plans starting at $29/month.
That free cloud tier is the practical starting point for most small teams — you get the observability without provisioning infrastructure, and you can graduate to self-hosting when cost or data residency makes it worthwhile.
Prompt versioning is a standout capability — teams can version, test, and deploy prompt templates alongside observability data. The @observe() decorator for Python provides function-level tracing without significant instrumentation effort. Langfuse integrates natively with LangChain, LlamaIndex, and Haystack, and supports any LLM via SDK. It also integrates with LLM security libraries including LLM Guard, Azure AI Content Safety, and Lakera, providing the evaluation layer on top.
Langfuse is the right default choice when data sovereignty matters, you want to avoid vendor lock-in, you have at least one infrastructure-capable engineer, or cost control at scale is a priority. If nobody wants to own infrastructure, start with Langfuse Cloud rather than skipping Langfuse entirely.
What does Arize Phoenix offer and how does it compare to Langfuse?
Arize Phoenix is an open-source LLM tracing and evaluation platform from Arize AI. It’s OTel-native, framework-agnostic, and purpose-built for evaluation depth.
Where Langfuse’s strongest differentiation is prompt management, Phoenix’s is evaluation. It ships with native LLM-as-judge metrics, structured evaluation workflows, and a plugin system for custom eval judges. Phoenix provides deeper support for agent evaluation compared with Langfuse, capturing complete multi-step agent traces that let you assess how agents make decisions over time. A prompt management module was added in April 2025, closing the main gap that previously separated it from Langfuse.
The distinctive capability is embedding drift detection. Phoenix monitors how the distribution of input embeddings changes over time, giving early warning of data distribution shifts before quality degrades. For teams building RAG pipelines where retrieval quality is the primary risk, this matters.
Phoenix is free and open-source for self-hosting. Managed cloud starts at $50/month. The commercial tier, Arize AX, is the enterprise upgrade path — but it’s designed for large-scale managed environments and is less suited for small single-server setups. Phoenix is the recommended path for small teams.
Compare it directly with Langfuse: Phoenix has better out-of-the-box evaluation and embedding drift detection; Langfuse has more mature prompt management and broader community adoption. Phoenix’s lightweight footprint means it can run locally during development, which lowers the initial barrier for self-hosting.
Phoenix is the right choice when evaluation depth is the priority, when you’re building RAG pipelines where retrieval quality matters, or when embedding-level observability is a requirement. For evaluation methodology in depth, the evaluation concepts article covers what you need before choosing an evaluation-first tool.
When do MLflow, LangSmith, and Braintrust each make sense for a small team?
These three occupy different niches. None is a general-purpose default, but each is the right tool for a specific context.
MLflow makes sense when your team is already in the Databricks ecosystem or running both classical ML pipelines and LLM workloads from the same infrastructure. It monitors both classical ML models and modern LLMs from a single platform, which matters when you have engineers crossing between both paradigms. The @mlflow.trace decorator and mlflow.openai.autolog() for automatic OpenAI tracing work well for teams that want minimal instrumentation overhead. The trade-off: MLflow’s LLM-specific observability is less mature than dedicated tools. Choose it when toolchain consolidation is worth more than best-of-breed LLM observability.
LangSmith is the choice for teams fully committed to LangChain or LangGraph. Its offline and online evaluation are tightly integrated with LangChain primitives. The limitation is framework lock-in — LangSmith’s tracing is designed around LangChain workflows and doesn’t translate smoothly to other orchestrators. Moving evaluation data into BigQuery or Snowflake requires bulk exports that can be slow. Self-hosting is Enterprise only. If you’re not deeply committed to LangChain, that lock-in is a real cost.
Braintrust makes sense when you want evaluation and monitoring unified in a single platform with framework flexibility and data residency options. It supports 13+ framework integrations — LangChain, LlamaIndex, Vercel AI SDK, OpenAI Agents SDK, and others — making it the most framework-agnostic commercial option. The hybrid deployment model keeps your data in your own cloud environment. The free tier is generous: 1M trace spans, 10K evaluation scores, and unlimited team members per month. The Pro tier is $249/month. Self-hosting is Enterprise only — if budget is tight, Langfuse or Phoenix are more accessible paths.
For broader platform selection context, see how to select an AI platform on observability and control-plane maturity for how observability tooling fits into the AI platform architecture.
How do you choose between these tools? A decision tree for small teams
Work through four questions in order. Your answers narrow the field quickly.
Question 1: What framework ecosystem is your team using?
If your team is committed to LangChain or LangGraph — it’s central to your stack and you have no plans to move — LangSmith is worth evaluating first. Its tight integration is a genuine feature for that context. If your team is framework-agnostic or multi-framework, eliminate LangSmith and look at the remaining four.
Question 2: Does your team have the capacity and willingness to self-host?
If yes — at least one engineer wants to own infrastructure and maintenance — open-source self-hosted options (Langfuse, Phoenix, MLflow) are viable. If no, go managed cloud or SaaS (Langfuse Cloud, Braintrust, or LangSmith if you’re LangChain-native). Self-hosting without internal capacity is an ongoing maintenance commitment, not a one-off setup task.
Question 3: Do you have data residency requirements?
If you’re subject to GDPR, industry regulations, or customer contracts restricting where LLM trace data can be stored, your options are: self-hosted Langfuse, self-hosted Phoenix, or Braintrust’s hybrid deployment. LangSmith’s self-hosting is Enterprise only — not viable for most small teams with this requirement.
Question 4: What is your evaluation maturity?
If you’re just starting, begin with tracing and cost visibility. Langfuse Cloud free tier is the default recommendation. If you have an established evaluation practice and need evaluation-first tooling, Phoenix or Braintrust are better fits.
The opinionated default: For most small teams — 5–10 engineers, no hard data residency requirements, no existing framework lock-in — start with Langfuse Cloud on the free tier. Add tracing and cost tracking first. Graduate to self-hosted Langfuse or add Phoenix for evaluation when production incidents demonstrate the need, not before.
Tool comparison:
Langfuse — open source, self-hostable (free, all features), free cloud Hobby tier, paid from $29/month. Best fit: teams wanting prompt management, cost analytics, and SQL access to trace data. Data residency: self-hosted or Langfuse Cloud.
Arize Phoenix — open source, self-hostable (free), managed cloud from $50/month. Best fit: teams prioritising evaluation depth and embedding drift detection. Data residency: self-hosted.
MLflow — open source (Apache 2.0), self-hostable (free). Best fit: teams in the Databricks ecosystem or running classical ML and LLM workloads together. Data residency: self-hosted.
LangSmith — not open source, Enterprise self-hosting only, free tier 5,000 traces/month on cloud SaaS. Best fit: LangChain/LangGraph-committed teams where evaluation-driven development is the priority.
Braintrust — not open source, Enterprise self-hosting only, free tier 1M spans and 10K scores per month. Best fit: framework-agnostic teams wanting evaluation and monitoring unified with data residency options. Data residency: hybrid deployment (AWS/GCP/Azure).
Upgrade triggers: Move from free tier to managed when trace volume exceeds free plan limits. Move from managed to self-hosted when per-trace cost consistently exceeds your infrastructure cost estimate. Add evaluation tooling after a production quality incident you couldn’t detect or debug from tracing alone.
The AI observability and guardrails platform guide has more on how these decisions connect to the broader platform architecture.
What does a minimum viable AI observability stack actually cost?
Every tool on this list has a free tier. For a small team early in production, $0 is a realistic starting budget for the tooling itself.
The real cost is engineering time, not licensing. Budget one to two engineering days for initial setup with a managed or SaaS tool. Self-hosted deployment adds one to two additional days for infrastructure provisioning. Ongoing maintenance runs 2–4 hours per month on a stable self-hosted deployment.
SaaS and managed cloud costs at SMB trace volumes are typically $0–$200/month in the first year. Most small teams start well within free tier limits and only hit paid tiers after significant production scale. The exception is Braintrust Pro at $249/month — a meaningful number for bootstrapped teams, though the free tier covers 1M spans per month, which handles most small production workloads.
For leadership justification, the investment case is simple: it pays for itself after the first production quality incident you catch early. Cost-per-feature analysis — attributing token spend to product features and user segments through trace tagging — often reveals budget overruns that would otherwise go undetected until the finance team’s review.
FAQ
Is Langfuse free to use?
Langfuse is open source and can be self-hosted at no licensing cost. Self-hosting includes all features. Langfuse Cloud also offers a free Hobby tier with core features. Paid cloud plans start at $29/month. Self-hosting is free to run but incurs infrastructure costs (compute, storage) and engineering maintenance time.
Can I use MLflow without Databricks?
Yes. MLflow is an independent open-source project under the Apache 2.0 licence. It can be self-hosted on any infrastructure without Databricks. Databricks offers a managed MLflow service, but the open-source version runs independently. MLflow Evaluation Datasets require a SQL backend (PostgreSQL, MySQL, SQLite, or MSSQL) — not available with FileStore.
What is the data residency argument for self-hosting AI observability?
Self-hosting keeps all LLM traces — including prompts, outputs, and user data — on your own infrastructure. This matters for teams subject to GDPR, industry regulations, or customer contracts that restrict where data can be stored. For teams that want data residency control without full self-hosting, Braintrust’s hybrid deployment keeps your data in your own cloud environment while using the managed platform.
Do I need AI observability before going to production?
Yes. At minimum you need tracing and cost tracking before production. Without tracing, you can’t debug production issues in non-deterministic AI systems — traditional metrics can’t detect model drift or quality degradation. Without cost tracking, token spend can escalate faster than billing cycles reveal. Add evaluation once you have production traffic to evaluate against.
What is the difference between LLM monitoring and LLM evaluation?
Monitoring is operational — latency, error rates, throughput, cost. Evaluation is semantic — whether outputs are correct, relevant, grounded, and safe. Both are necessary. Small teams should start with monitoring (tracing plus cost tracking) and add evaluation as a second step, once there’s enough production traffic to make quality review meaningful.
How long does it take to set up AI observability for a small team?
Initial setup with a managed or SaaS tool like Langfuse Cloud or LangSmith — SDK integration, basic dashboard configuration, cost tracking setup — typically takes one to two engineering days. You can get first traces flowing in under an hour. Self-hosted deployment adds one to two additional days for infrastructure provisioning and configuration.
What is LLM-as-judge evaluation and do I need it?
LLM-as-judge uses a language model to score your production model’s outputs on criteria like accuracy, relevance, and safety. It scales better than human review. You need it once you have enough production traffic to make manual quality review impractical — typically more than a few hundred traces per day. Before that, spot-checking traces manually is sufficient.
Can a 5–10 person engineering team realistically implement responsible AI practices?
Yes, with appropriate scope. A small team can implement tracing, cost monitoring, basic output evaluation, and PII redaction without dedicated ML ops staff. Advanced guardrails, shadow evaluation pipelines, and formal audit trails can be deferred until team size or compliance requirements justify the investment.
How do I add observability to an existing AI agent already in production?
Prioritise tracing first — it gives you the debugging foundation everything else depends on. Add cost tracking at the same time to establish a spend baseline before usage grows. Evaluation comes third, once you have production traces to evaluate against. The decision tree in this article applies whether you’re starting from scratch or retrofitting.
What happens if I pick the wrong observability tool?
The switching cost is moderate, not catastrophic. Most tools use similar instrumentation patterns — OpenTelemetry-compatible SDKs, decorator-based tracing. The main cost of switching is reconfiguring dashboards and evaluation workflows, not rewriting application code. Starting with any tool is better than waiting for the perfect choice.
Should I use the same tool for AI observability and traditional application monitoring?
Not usually. Traditional APM tools like Datadog and New Relic have added LLM features, but they are built on the assumption that software operates in deterministic states — working or broken — and lack the statistical frameworks and AI-specific insights that non-deterministic systems require. Use your existing APM for infrastructure monitoring and a dedicated tool (Langfuse, Phoenix, LangSmith, or Braintrust) for AI-specific observability.
