LaunchRoundsSubmit Software

Best Open Source LLM Observability and Evaluation Tools (November 2026)

By Alston Antony, editor · Updated November 2026 · How we rank

As of November 2026, we list 13 tools in open source LLM observability and evaluation tools, 17% of the 75 LLM observability and evaluation tools we track. 83% of the 12 with known pricing offer a free plan. Weights & Biases, OpenLIT and Label Studio have the most complete listings.

Tools
13
92% have a known pricing model
Free plan
83%
10 of 12 where known
Median entry price
–
too few prices to report
Have an API
–
too few known to report

Top picks: open source LLM observability and evaluation tools

The most complete listings with clear pricing, ordered by our published score.

Top 5 side by side

The same facts for each of the leading listings. “Not published” means we found no answer, not that the answer is no.

Top open source LLM observability and evaluation tools compared on best use, free plan, entry price, platforms, API and main limit
ToolBest forFree planEntry priceRuns onAPIKey limit
Weights & BiasesML and AI teams logging experiments and evaluating modelsYes$60/moNot publishedNot publishedFree tier caps seats, storage and ingestion; Pro limited to teams under 50
OpenLITTeams needing OpenTelemetry-based tracing, evals and guardrails for agentsYesNot publishedSelf-hostedYesCloud not launched; self-hosting requires running the stack yourself
Label StudioTeams labeling multimodal data and running AI evaluation workflowsYesNot publishedSelf-hostedNot publishedPaid plan pricing and limits are not published
MLflowTeams tracking, evaluating and deploying LLM and ML modelsYesNot publishedNot publishedYesSelf-hosted server must be run and maintained by your team
CometML and GenAI teams tracking experiments and debugging LLM agentsYesNot publishedNot publishedNot publishedPaid pricing, usage limits and retention terms are not published

Free to start

Free tools and tools with a free plan, best listing score first.

How to choose open source LLM observability and evaluation tools

Best free open source LLM observability and evaluation tools

10 of the 12 open source LLM observability and evaluation tools with known pricing offer a free plan or are free to use (83%). These are the highest-ranked ones that are free to start. Free plans usually limit usage, so check the limits on each tool page.

Compare all 13 open source LLM observability and evaluation tools

Filter:
Comparison of open source LLM observability and evaluation tools: best for, free plan, starting price and focus areas
ToolBest forFree planStarting priceFocus areas
Weights & BiasesML and AI teams logging experiments and evaluating modelsYes$60/moExperiment Tracking
OpenLITTeams needing OpenTelemetry-based tracing, evals and guardrails for agentsYes—Guardrails
Label StudioTeams labeling multimodal data and running AI evaluation workflowsYes——
MLflowTeams tracking, evaluating and deploying LLM and ML modelsYes—Experiment Tracking, Model Deployment
bottest.aiTeams automating regression, performance and security tests for chatbotsYes——
CometML and GenAI teams tracking experiments and debugging LLM agentsYes—Experiment Tracking
Fiddler AIEnterprises governing agents across creation and production layersYes—AI Governance, Guardrails
Scale AIAI labs and enterprises needing training data and model evaluationsNo—Training Data
GiskardAn open-source framework for testing all ML models, from tabular to LLMs.YesFree—
ArthurAn AI performance platform that helps enterprises monitor, measure, and improve their models.No—Agent Governance
WhyLabsTeams studying open source AI observability and data loggingUnknown——
CleanlabAn open-source framework for data-centric AI that automatically finds and fixes errors in datasets.Yes——
LangSmithA platform for debugging, testing, evaluating, and monitoring your LLM applications, built by the LangChain team.Yes——

Open source LLM observability and evaluation tools: common questions

Are there free open source LLM observability and evaluation tools?

Yes. 10 of the 12 open source LLM observability and evaluation tools with known pricing have a free plan (83%). Examples: Weights & Biases, OpenLIT and Label Studio.

What is the cheapest of the open source LLM observability and evaluation tools?

Weights & Biases has the lowest starting price we found, $60/mo. Across 1 tools with a readable price the median is $60/mo.

How many open source LLM observability and evaluation tools are there?

We list 13 open source LLM observability and evaluation tools as of November 2026, 17% of the 75 LLM observability and evaluation tools we track.

How many open source LLM observability and evaluation tools are open source?

13 tools (100%) are open source or list a license: Weights & Biases, OpenLIT and Label Studio and more.

Who are open source LLM observability and evaluation tools built for?

Most listings target developers (13) and enterprise (7).

What do open source LLM observability and evaluation tools usually focus on?

The most common focus areas are Experiment Tracking (3) and Guardrails (2).

Open source LLM observability and evaluation tools: the numbers

Tools listed
13
17% of 75 in LLM observability and evaluation tools
Pricing models
freemium 42% · free or open source 42% · paid only 17%
based on 12 tools with known pricing
With a free plan
83%
10 of 12 where we know
Added in 90 days
13
before 2026-11-18

Ready to try one? See Weights & Biases, our top listing for open source LLM observability and evaluation tools, or compare all 13.

Part of LLM Observability & Evaluation Tools. Data last updated November 2026 · Methodology · LaunchRounds