LLM Observability and Evaluation Tools List (November 2026)
By Alston Antony, editor · Updated November 2026 · How we rank
This LLM observability and evaluation tools list for November 2026 compares 75 tools; the highest-ranked are Langfuse, Portkey and PromptLayer. Most offer a free plan, so shortlist two or three, test them on your own work, then compare entry prices before you pay.
- Key facts
- 75 LLM observability and evaluation tools listed as of November 2026.
- 86% of the 28 with known pricing offer a free plan or are free to use.
- Highest-ranked on our score: Langfuse, Portkey and PromptLayer.
- Most common focus areas: AI gateway (6), guardrails (6) and prompt management (6).
- Ranked by a published score (listing completeness, pricing clarity, recency); sponsored placements never change the order.
- Tools
- 75
- 37% have a known pricing model
- Free plan
- 86%
- 24 of 28 where known
- Median entry price
- –
- too few prices to report
- Have an API
- 100%
- 16 of 16 where known
Top picks: LLM observability and evaluation tools
The most complete listings with clear pricing, ordered by our published score.
Top 5 side by side
The same facts for each of the leading listings. “Not published” means we found no answer, not that the answer is no.
| Tool | Best for | Free plan | Entry price | Runs on | API | Key limit |
|---|---|---|---|---|---|---|
| Langfuse | Teams tracing, evaluating and improving LLM apps in production | Yes | $29/mo | Self-hosted | Yes | Free tier caps at 50k units/month and 30 days of data |
| Portkey | Teams taking LLM apps to production with monitoring and control | Yes | $49/mo | Not published | Yes | Free tier caps logs at 10k per month and is not for production |
| PromptLayer | Teams versioning, evaluating and monitoring prompts and agents | Yes | $49/mo | Web | Yes | Free tier caps requests, eval executions and dataset size |
| Weights & Biases | ML and AI teams logging experiments and evaluating models | Yes | $60/mo | Not published | Not published | Free tier caps seats, storage and ingestion; Pro limited to teams under 50 |
| Agenta | Teams building and operating AI agents they can self-host | Yes | $29/mo | Self-hosted | Not published | Model costs excluded; Hobby caps at 2 members and 5,000 runs monthly |
How to choose LLM observability and evaluation tools
- Budget: 24 of 28 tools with known pricing have a free plan. The lowest paid price we found is $25/mo (Vivgrid); the middle is $49/mo.
- Match the job first: the most common focus areas here are AI Gateway (6), Guardrails (6) and Prompt Management (6). Pick the one closest to what you need before comparing price.
- Who it is built for: most listings target developers (72), enterprise (43) and researchers (3).
- 13 tools here are open source or have a stated license, which matters if you want to self-host.
- Features to compare: guardrails (21%), cost tracking (17%), prompt management (17%) and ai gateway (14%).
- Check pricing on the vendor's site: we could not read it for 47 of these 75 tools.
Best free LLM observability and evaluation tools
24 of the 28 LLM observability and evaluation tools with known pricing offer a free plan or are free to use (86%). These are the highest-ranked ones that are free to start. Free plans usually limit usage, so check the limits on each tool page.
Compare all 75 LLM observability and evaluation tools
Showing 1 to 25 of 75, in score order. The rest are on the next 2 pages.
| Tool | Best for | Free plan | Starting price | Focus areas |
|---|---|---|---|---|
| Langfuse | Teams tracing, evaluating and improving LLM apps in production | Yes | $29/mo | Prompt Management |
| Portkey | Teams taking LLM apps to production with monitoring and control | Yes | $49/mo | AI Gateway, Guardrails, Prompt Management |
| PromptLayer | Teams versioning, evaluating and monitoring prompts and agents | Yes | $49/mo | Prompt Library |
| Weights & Biases | ML and AI teams logging experiments and evaluating models | Yes | $60/mo | Experiment Tracking |
| Agenta | Teams building and operating AI agents they can self-host | Yes | $29/mo | — |
| Langtail | Teams that need shared, testable prompt management for LLM apps | Yes | $99/mo | Prompt Management |
| OpenLIT | Teams needing OpenTelemetry-based tracing, evals and guardrails for agents | Yes | — | Guardrails |
| Vivgrid | Teams running agent tools in the cloud with tracing and evaluation | Yes | $25/mo | AI Agents |
| Label Studio | Teams labeling multimodal data and running AI evaluation workflows | Yes | — | — |
| MLflow | Teams tracking, evaluating and deploying LLM and ML models | Yes | — | Experiment Tracking, Model Deployment |
| bottest.ai | Teams automating regression, performance and security tests for chatbots | Yes | — | — |
| Promptfoo | Developers and security teams testing AI apps for vulnerabilities | Yes | — | Guardrails |
| Comet | ML and GenAI teams tracking experiments and debugging LLM agents | Yes | — | Experiment Tracking |
| Fiddler AI | Enterprises governing agents across creation and production layers | Yes | — | AI Governance, Guardrails |
| Hamming AI | QA and reliability for voice and chat agents before and after launch | No | — | — |
| Maxim AI | Teams centralising LLM access, cost control and observability | Yes | — | AI Gateway |
| Opik | Developers tracing, evaluating and monitoring LLM agents | Yes | — | Prompt Management |
| Scale AI | AI labs and enterprises needing training data and model evaluations | No | — | Training Data |
| Asymptote | Security teams governing coding agents | Yes | — | — |
| Giskard | An open-source framework for testing all ML models, from tabular to LLMs. | Yes | Free | — |
| Agentops | Engineers tracing, debugging and costing AI agent runs | Yes | — | — |
| Rhesis AI | Teams building AI agents with domain-expert collaboration | Yes | — | — |
| SaaSpec | Pay-as-you-go LLM software billed per user | No | — | — |
| Foundry | Teams training and evaluating browser agents at scale | Unknown | — | Training Data |
| Orq | Teams governing many AI agents, models and tool calls | Unknown | — | AI Gateway, Guardrails |
LLM observability and evaluation tools: common questions
Are there free LLM observability and evaluation tools?
Yes. 24 of the 28 LLM observability and evaluation tools with known pricing have a free plan (86%). Examples: Langfuse, Portkey and PromptLayer.
What is the cheapest of the LLM observability and evaluation tools?
Vivgrid has the lowest starting price we found, $25/mo. Across 7 tools with a readable price the median is $49/mo.
How many LLM observability and evaluation tools are there?
We list 75 LLM observability and evaluation tools as of November 2026.
How many LLM observability and evaluation tools are open source?
13 tools (17%) are open source or list a license: Weights & Biases, OpenLIT and Label Studio and more.
Who are LLM observability and evaluation tools built for?
Most listings target developers (72), enterprise (43) and researchers (3).
What do LLM observability and evaluation tools usually focus on?
The most common focus areas are AI Gateway (6), Guardrails (6) and Prompt Management (6).
LLM observability and evaluation tools: the numbers
- Tools listed
- 75
- in LLM observability and evaluation tools
- Pricing models
- freemium 68% · free or open source 18% · paid only 14%
- based on 28 tools with known pricing
- With a free plan
- 86%
- 24 of 28 where we know
- Median starting price
- $49/mo
- 7 tools with a readable price, from $25 to $99
- With an API
- 100%
- 16 of 16 where we know
- Platforms
- self hosted 13 · cli 2 · web 1
- 14 tools list a platform
- Added in 90 days
- 75
- before 2026-11-18
LLM observability and evaluation tools: in-depth data
How the 75 LLM observability and evaluation tools we list break down by pricing, openness, audience and features, counted from our own data as of November 2026.
- A free plan is 21 points more common here than across the whole directory (86% against 65%).
Show the full breakdown (7 charts)
How they are priced
Freemium is the most common model: 68% of the 28 LLM observability and evaluation tools with a known pricing model.
Based on 28 of 75 tools with a known pricing model.
Access and openness
86% offer a free plan, 17% open source or licensed, 100% have an API and 88% built around AI.
Based on each bar uses the tools where that fact is known; open source uses all tools in the set.
Who they are built for
Developers is the largest audience at 96% of the 75 tools that name one.
Based on 75 of 75 tools that name an audience (a tool can name several).
What they are used for
AI Gateway comes up most, on 24% of the 25 tools with tagged use cases.
Based on 25 of 75 tools with at least one tagged use case (a tool can carry several).
Most common features
Guardrails appears on 21% of the 29 tools that list features.
Based on 29 of 75 tools with a feature list.
Where they run
Self-hosted is the most common platform, listed by 93% of the 14 tools that name one.
Based on 14 of 75 tools that list a platform.
How many plans they offer
The typical tool publishes 3 plans; 33% publish four or more.
Based on 12 of 75 tools with published plans.
LLM observability and evaluation tools against the whole directory
| Measure | LLM observability and evaluation tools | All tools |
|---|---|---|
| Free plan | 86% | 65% |
| API available | 100% | 100% |
| Open source or licensed | 17% | 1% |
| Freemium pricing | 68% | 55% |
Ready to try one? See Langfuse, our top listing for LLM observability and evaluation tools, or compare all 75.
Part of AI Developer Tools. Data last updated November 2026 · Methodology · LaunchRounds
