Best Open Source LLM Observability and Evaluation Tools (November 2026)
By Alston Antony, editor · Updated November 2026 · How we rank
As of November 2026, we list 13 tools in open source LLM observability and evaluation tools, 17% of the 75 LLM observability and evaluation tools we track. 83% of the 12 with known pricing offer a free plan. Weights & Biases, OpenLIT and Label Studio have the most complete listings.
- Key facts
- 13 open source LLM observability and evaluation tools listed as of November 2026.
- 83% of the 12 with known pricing offer a free plan or are free to use.
- Highest-ranked on our score: Weights & Biases, OpenLIT and Label Studio.
- Most common focus areas: experiment tracking (3) and guardrails (2).
- Ranked by a published score (listing completeness, pricing clarity, recency); sponsored placements never change the order.
- Tools
- 13
- 92% have a known pricing model
- Free plan
- 83%
- 10 of 12 where known
- Median entry price
- –
- too few prices to report
- Have an API
- –
- too few known to report
Top picks: open source LLM observability and evaluation tools
The most complete listings with clear pricing, ordered by our published score.
Top 5 side by side
The same facts for each of the leading listings. “Not published” means we found no answer, not that the answer is no.
| Tool | Best for | Free plan | Entry price | Runs on | API | Key limit |
|---|---|---|---|---|---|---|
| Weights & Biases | ML and AI teams logging experiments and evaluating models | Yes | $60/mo | Not published | Not published | Free tier caps seats, storage and ingestion; Pro limited to teams under 50 |
| OpenLIT | Teams needing OpenTelemetry-based tracing, evals and guardrails for agents | Yes | Not published | Self-hosted | Yes | Cloud not launched; self-hosting requires running the stack yourself |
| Label Studio | Teams labeling multimodal data and running AI evaluation workflows | Yes | Not published | Self-hosted | Not published | Paid plan pricing and limits are not published |
| MLflow | Teams tracking, evaluating and deploying LLM and ML models | Yes | Not published | Not published | Yes | Self-hosted server must be run and maintained by your team |
| Comet | ML and GenAI teams tracking experiments and debugging LLM agents | Yes | Not published | Not published | Not published | Paid pricing, usage limits and retention terms are not published |
Free to start
- Weights & BiasesFree plan
- OpenLITFree plan
- Label StudioFree plan
- MLflowFree plan
- bottest.aiFree plan
Free tools and tools with a free plan, best listing score first.
How to choose open source LLM observability and evaluation tools
- Budget: 10 of 12 tools with known pricing have a free plan. The lowest paid price we found is $60/mo (Weights & Biases); the middle is $60/mo.
- Match the job first: the most common focus areas here are Experiment Tracking (3) and Guardrails (2). Pick the one closest to what you need before comparing price.
- Who it is built for: most listings target developers (13) and enterprise (7).
Best free open source LLM observability and evaluation tools
10 of the 12 open source LLM observability and evaluation tools with known pricing offer a free plan or are free to use (83%). These are the highest-ranked ones that are free to start. Free plans usually limit usage, so check the limits on each tool page.
Compare all 13 open source LLM observability and evaluation tools
| Tool | Best for | Free plan | Starting price | Focus areas |
|---|---|---|---|---|
| Weights & Biases | ML and AI teams logging experiments and evaluating models | Yes | $60/mo | Experiment Tracking |
| OpenLIT | Teams needing OpenTelemetry-based tracing, evals and guardrails for agents | Yes | — | Guardrails |
| Label Studio | Teams labeling multimodal data and running AI evaluation workflows | Yes | — | — |
| MLflow | Teams tracking, evaluating and deploying LLM and ML models | Yes | — | Experiment Tracking, Model Deployment |
| bottest.ai | Teams automating regression, performance and security tests for chatbots | Yes | — | — |
| Comet | ML and GenAI teams tracking experiments and debugging LLM agents | Yes | — | Experiment Tracking |
| Fiddler AI | Enterprises governing agents across creation and production layers | Yes | — | AI Governance, Guardrails |
| Scale AI | AI labs and enterprises needing training data and model evaluations | No | — | Training Data |
| Giskard | An open-source framework for testing all ML models, from tabular to LLMs. | Yes | Free | — |
| Arthur | An AI performance platform that helps enterprises monitor, measure, and improve their models. | No | — | Agent Governance |
| WhyLabs | Teams studying open source AI observability and data logging | Unknown | — | — |
| Cleanlab | An open-source framework for data-centric AI that automatically finds and fixes errors in datasets. | Yes | — | — |
| LangSmith | A platform for debugging, testing, evaluating, and monitoring your LLM applications, built by the LangChain team. | Yes | — | — |
Open source LLM observability and evaluation tools: common questions
Are there free open source LLM observability and evaluation tools?
Yes. 10 of the 12 open source LLM observability and evaluation tools with known pricing have a free plan (83%). Examples: Weights & Biases, OpenLIT and Label Studio.
What is the cheapest of the open source LLM observability and evaluation tools?
Weights & Biases has the lowest starting price we found, $60/mo. Across 1 tools with a readable price the median is $60/mo.
How many open source LLM observability and evaluation tools are there?
We list 13 open source LLM observability and evaluation tools as of November 2026, 17% of the 75 LLM observability and evaluation tools we track.
How many open source LLM observability and evaluation tools are open source?
13 tools (100%) are open source or list a license: Weights & Biases, OpenLIT and Label Studio and more.
Who are open source LLM observability and evaluation tools built for?
Most listings target developers (13) and enterprise (7).
What do open source LLM observability and evaluation tools usually focus on?
The most common focus areas are Experiment Tracking (3) and Guardrails (2).
Open source LLM observability and evaluation tools: the numbers
- Tools listed
- 13
- 17% of 75 in LLM observability and evaluation tools
- Pricing models
- freemium 42% · free or open source 42% · paid only 17%
- based on 12 tools with known pricing
- With a free plan
- 83%
- 10 of 12 where we know
- Added in 90 days
- 13
- before 2026-11-18
Ready to try one? See Weights & Biases, our top listing for open source LLM observability and evaluation tools, or compare all 13.
Part of LLM Observability & Evaluation Tools. Data last updated November 2026 · Methodology · LaunchRounds
