If you have been trying to pick an AI model for a real production use case, you have probably noticed that the benchmark scores look impressive but the models themselves behave… less impressively. That is not a coincidence.
The structural failures in AI benchmarking — contamination, cherry-picking, saturation — have made the standard leaderboard ecosystem unreliable for anyone trying to make a real procurement decision. Hugging Face responded in February 2026 with Community Evals: an open, reproducible alternative to vendor-controlled benchmark reporting.
The interesting part is not just that it exists — it is the three-tier trust hierarchy it introduces. Community-submitted, author-submitted, and verified scores each carry different weight, and knowing how to read that distinction is the practical skill that turns a leaderboard page into a useful signal. This guide to benchmark governance covers how Community Evals works technically, how to interpret its scores, how it compares to the alternatives, and how to use it to make a model selection decision.
Let’s get into it.
What problem does Hugging Face Community Evals actually solve?
The short version: evaluation is broken, and everyone knows it.
The longer version involves benchmark saturation. MMLU — the go-to test for broad language model capability for years — is now saturated above 91%. GSM8K sits above 94%. HumanEval has essentially been conquered. And yet models that ace these benchmarks still cannot reliably browse the web, write production code, or handle multi-step tasks without hallucinating.
That gap is made worse by benchmark contamination. When test data leaks into training sets through web scraping — many benchmarks are publicly available online — a model learns the test rather than the skill. And then there is the selective reporting problem: studies released in 2025 found selective disclosure inflating proprietary model scores by as much as 112%. Labs report the scores that make their models look good, with no requirement to disclose where a model performed poorly.
The old model was simple: a vendor submits scores to a centralised leaderboard, no audit trail required. Community Evals changes that. It creates an auditable trail — you can trace who ran what evaluation, when, and with which configuration. That traceability is what the existing system has always lacked.
How does the Community Evals system work technically?
Community Evals runs on Hugging Face Hub using the same Git-based infrastructure the Hub is already built on.
A benchmark creator registers a dataset repository by adding an eval.yaml file that defines how the benchmark should be run. Once registered, that dataset repo automatically collects and displays evaluation results submitted across the Hub. Anyone who wants to evaluate a model runs the evaluation using LightEval — Hugging Face’s evaluation library — and submits the results via pull request to the benchmark’s dataset repository. Results then appear on the evaluated model’s card with a tier badge indicating whether the score was community-submitted, author-submitted, or verified.
The audit trail comes from the Git history. There is a record of when evaluations were added and by whom. A community member can link to sources and the discussion happens like any other pull request. That traceability is what distinguishes Community Evals from a vendor sending numbers to a centralised system with no verifiable chain of custody.
Initial benchmarks at launch include MMLU-Pro, GPQA, and HLE (Humanity’s Last Exam). The system launched in beta on 4 February 2026.
One important caveat: model authors retain the ability to close pull requests or hide results. The incentive problem is not fully eliminated for author-submitted scores — which is part of why the tier system matters.
What is eval.yaml and why does it matter for reproducibility?
The eval.yaml file is the technical mechanism that makes Community Evals more than just another submission form.
It lives in the benchmark’s dataset repository — not the model repository, which is where score files live. It contains a machine-readable specification of exactly how the benchmark should be run: which evaluation framework to use, where to find the dataset, how to score the outputs, and what configuration parameters apply. Anyone with access to the model and the eval.yaml can reproduce the benchmark result independently. That reproducibility is the mechanism behind the verified badge.
The format is based on Inspect AI, an open standard developed by the UK AI Safety Institute. That gives the specification institutional weight beyond any single company’s internal convention. LightEval now supports Inspect AI as a backend — you write the spec, LightEval executes it, and the result is in the format Community Evals expects.
One distinction to keep straight: eval.yaml in the dataset repository defines how to run the evaluation. The .eval_results/*.yaml files in the model repository record the results. They are separate artefacts serving different purposes, and confusing them is easy until you have worked with the system in practice.
What are the three tiers of Community Eval scores and how do I interpret them?
The three-tier system is the practical mechanism for deciding how much weight to put on any given score. Here is what each one means.
Community-submitted: Any user can submit evaluation results for any model via a pull request, appearing as “community” tier without waiting for the model author. These are unverified — no independent party has confirmed them — but they are traceable. Treat them as directional indicators that need corroboration.
Author-submitted: The model creator submits their own scores by publishing YAML files in .eval_results/ on their model repository. These carry reputational weight, but they are subject to the same self-reporting incentive problem as vendor benchmarks. Use them as a secondary signal.
Verified: The highest-confidence tier. Verified scores mean the result has been independently reproduced using the public eval.yaml specification — a third party ran the same evaluation with the same configuration and obtained a statistically equivalent result. This is where the auditability of Community Evals translates into actual confidence.
When community-submitted and author-submitted scores diverge significantly on the same benchmark, that is worth investigating. It may indicate different evaluation configurations, different model versions, or potential score inflation. Check whether the same eval.yaml spec was used and whether any verified scores exist as a reference.
Verified scores are relatively few while the system is in beta — you will often need to work with all three tiers simultaneously and triangulate. That is fine, but do not treat a community-submitted score as equivalent to a verified one.
How does Community Evals compare to the Open LLM Leaderboard and Chatbot Arena?
The Open LLM Leaderboard is a curated, centrally-managed project run by the Hugging Face team with a fixed set of benchmarks. Community Evals is its distributed, extensible counterpart — same platform, different governance model. The Open LLM Leaderboard tells you how models compare on the benchmarks Hugging Face chose; Community Evals tells you how models compare on whatever benchmarks the community has registered.
Chatbot Arena (now operating commercially as LMArena, following the LMSYS rebrand) uses an Elo rating system built on crowdsourced human preference votes from head-to-head model comparisons. It measures something genuinely different from benchmark performance: which model humans actually prefer in open-ended conversation. The fact that a model ranks differently on Chatbot Arena than on Community Evals is not a bug — it reflects the genuine difference between performing well on defined capability benchmarks and being preferred by humans in conversation.
Artificial Analysis publishes an Intelligence Index combining 10 evaluations across Agents (25%), Coding (25%), General (25%), and Scientific Reasoning (25%) with a 95% confidence interval of less than ±1%. It is independent and methodologically rigorous. Use it as a cross-reference when Community Evals coverage is incomplete.
Here is the short version of how they each fit together:
Community Evals — Open pull request submission with three trust tiers. Measures specific benchmark performance with auditable provenance. Best for initial model shortlisting with traceable scores.
Open LLM Leaderboard — Centrally managed with a fixed, curated benchmark set. Best for consistent cross-model comparison on Hugging Face-selected benchmarks.
Chatbot Arena / LMArena — Crowdsourced human preference Elo ratings. Best for conversational use cases where human preference matters.
Artificial Analysis — Independent third-party composite evaluation. Best as a cross-reference when Community Evals coverage is sparse.
The same model can rank first on one platform and fifth on another. That is expected and informative, not a flaw. Your job is to know which question each platform is actually answering.
What are live benchmarks and when do they provide better signal than community evals?
Community Evals makes existing benchmarks more trustworthy. It does not make them contamination-resistant. Those are two different things.
If a benchmark’s test data has leaked into training data, Community Evals can confirm the evaluation was reproducible — but it cannot confirm the score reflects real capability rather than memorisation. Live benchmarks take a different approach.
LiveBench refreshes its questions regularly, creating a moving target that prevents models from being optimised against a fixed test set. It uses verifiable, objective ground-truth answers rather than LLM judges.
PeerBench uses a proctored evaluation model with sealed test sets and cryptographic audit trails. The test data is not publicly accessible, making it structurally difficult to game through targeted optimisation.
The relationship is complementary. Community Evals provides breadth — many benchmarks, many models, auditable provenance. Live benchmarks provide contamination-resistant depth on specific capabilities. A model that scores consistently well on both is giving you more signal than either one alone.
How do I use community eval data in a real model selection decision?
Benchmarks are a filter, not a verdict. Here is the practical process.
Step 1: Define your capability requirements. A coding assistant needs models that excel on coding benchmarks (HumanEval, SWE-Bench, MBPP). A customer service application needs conversational quality. A data analysis tool needs mathematical reasoning. Start by identifying which benchmarks map to your actual use case.
Step 2: Filter Community Evals by relevant benchmarks. Pull the scores for those specific benchmarks across your candidate models. Note the tier of each score you are relying on.
Step 3: Prioritise verified scores. Where verified scores exist, use them as your baseline. Supplement with author-submitted scores as a secondary signal. Use community-submitted scores directionally where no verified score exists.
Step 4: Cross-reference. Check Chatbot Arena Elo rankings if conversational quality matters. Pull Artificial Analysis Intelligence Index scores as an independent composite reference. A model that ranks consistently well across multiple independent platforms gives you a stronger signal than one that looks good on a single platform.
Step 5: Flag and investigate divergences. If community-submitted scores differ substantially from author-submitted scores, or if Community Evals ranking diverges significantly from Chatbot Arena, investigate before committing. No single platform is reliable enough on its own.
Step 6: Validate with your own data. Production AI evaluation tools are where the shortlist from steps 1–5 gets tested against real inputs from your domain. The benchmark phase gets you to a shortlist of 3–5 candidates; your own evaluation phase gets you to a deployment decision.
Community Evals is in beta. Use it as one layer of a multi-signal approach, not as a sole source of truth.
How do I submit evaluation results and contribute to the community ecosystem?
Contributing requires technical familiarity — the eval.yaml format and LightEval are prerequisites, not a consumer-facing feature.
That said, contributing matters. Valuable benchmarks fade when maintainers lack resources to keep leaderboards running. Community Evals addresses this by decentralising hosting on the Hub — no separate infrastructure required. More participation means more coverage, and more pressure on model labs to submit consistent results.
One governance gap worth flagging: who controls the benchmark shortlist and who approves new registrations is not yet publicly documented. A community-based evaluation system that relies on centralised gatekeeping reintroduces some of the same problems it was designed to address. Worth watching as the system matures. The longer-term trajectory — from community benchmarks into AI benchmark standards and the regulatory frameworks coalescing around them — is a separate question, but one worth tracking now.
Frequently Asked Questions
Are community-submitted results more reliable than vendor-reported scores? More transparent and verifiable — not necessarily more accurate. Every community submission has a traceable pull request and the potential for independent reproduction using the public eval.yaml specification. Vendor-reported scores lack this audit trail. Reliability increases further when a score earns a verified badge.
Do I need technical expertise to submit an evaluation to Community Evals? Yes. Submitting requires familiarity with the eval.yaml format, LightEval, and the Hugging Face pull request workflow. It is not a one-click process.
What does the verified badge actually mean? A third party ran the same evaluation with the same configuration and got a statistically equivalent result. It is the highest-confidence score tier in the system.
Why do the same AI models rank differently on different leaderboards? Different platforms measure different things. Community Evals aggregates benchmark scores on specific capabilities. Chatbot Arena uses human preference Elo ratings. Artificial Analysis combines 10 evaluations into a composite index. Ranking divergence is expected and informative, not a flaw.
Can community members game Community Evals the way vendors game MMLU? Gaming is more visible — every submission is traceable and the eval.yaml allows anyone to reproduce the claimed result. But if the underlying benchmark’s test data has leaked into training data, Community Evals cannot prevent contamination-inflated scores. That is why live benchmarks like LiveBench complement it.
What is the difference between the Open LLM Leaderboard and Community Evals? The Open LLM Leaderboard is centrally managed with a fixed, curated benchmark set. Community Evals is distributed — any community member can submit results for any registered benchmark. Both live on Hugging Face Hub, but Community Evals is extensible while the Open LLM Leaderboard is curated.
How does Chatbot Arena’s Elo rating differ from Community Evals benchmark scores? Chatbot Arena uses crowdsourced human preference judgements, producing an Elo rating reflecting subjective quality in open-ended conversation. Community Evals aggregates automated scores measuring specific, defined capabilities. Chatbot Arena tells you which model humans prefer; Community Evals tells you which model performs better on specific tasks.
What is Inspect AI and why does Hugging Face use it for Community Evals? Inspect AI is an evaluation framework developed by the UK AI Safety Institute defining a standard format for describing and running LLM evaluations. Hugging Face adopted it as the specification format because it provides an institutionally-backed, open standard for machine-readable evaluation definitions.
What happens when community-submitted and author-submitted scores disagree significantly? Score divergence is worth investigating. Check whether the same eval.yaml spec was used, whether model versions match, and whether verified scores exist as a reference. Significant divergence may indicate different configurations, different model versions, or score inflation.
Is Hugging Face Community Evals ready for production use in model procurement decisions? It launched in beta on 4 February 2026. Not all models have verified scores, and not all benchmarks are registered. Useful now as one signal among several — alongside Chatbot Arena, Artificial Analysis, and your own production evaluations — but not a sole basis for procurement decisions yet.
Community Evals is one piece of a larger shift in how AI model quality gets measured and governed. For a complete overview of the benchmark governance landscape — including the regulatory drivers, internal operationalisation frameworks, and vendor procurement implications — see our guide to what AI benchmark governance is and why it matters now.
