hugging-face-community-evals
Run evaluations for Hugging Face Hub models using inspect-ai and lighteval on local hardware. Use for backend selection, local GPU evals, and choosing between vLLM / Transformers / accelerate.
Overview
When to Use
Use this skill when you need run evaluations for Hugging Face Hub models using inspect-ai and lighteval on local hardware. Use for backend selection, local GPU evals, and choosing between vLLM / Transformers / accelerate. Not for HF Jobs orchestration, model-card PRs, .eval_results publication, or community-evals...
This skill is for running evaluations against models on the Hugging Face Hub on local hardware.
It covers:
inspect-aiwith local inferencelightevalwith local inference- choosing between
vllm, Hugging Face Transformers, andaccelerate - smoke tests, task selection, and backend fallback strategy
It does not cover:
- Hugging Face Jobs orchestration
- model-card or
model-indexedits - README table extraction
- Artificial Analysis imports
.eval_resultsgeneration or publishing- PR creation or community-evals automation
If the user wants to run the same eval remotely on Hugging Face Jobs, hand off to the hugging-face-jobs skill and pass it one of the local scripts in this skill.
If the user wants to publish results into the community evals workflow, stop after generating the evaluation run and hand off that publishing step to ~/code/community-evals.
All paths below are relative to the directory containing this SKILL.md.
When To Use Which Script
| Use case | Script |
|---|---|
| Local `ins |
Subscribers only
The full skill, its 5 bundled files and every download is included with every paid Complete AI plan.
Details
| Source | huggingface/skills |
|---|---|
| License | Apache-2.0 |
| Risk label | critical ("critical" means the skill may run commands or touch files — read before use) |
| Files | SKILL.md, examples/USAGE_EXAMPLES.md, scripts/inspect_eval_uv.py, scripts/inspect_vllm_uv.py, scripts/lighteval_vllm_uv.py |
| Added | 2026-07-01 |
Related skills
advanced-evaluation
This skill should be used when the user asks to "implement LLM-as-judge", "compare model outputs", "create evaluation rubrics", "mitigate evaluation bias", or mentions direct scoring, pairwise comparison, position bias, evaluation pipelines, or automated quality assessment.
agent-creator
Create custom AI subagents with proper plugin structure, persona generation, and companion routing skills.
agent-framework-azure-ai-py
Build persistent agents on Azure AI Foundry using the Microsoft Agent Framework Python SDK.
agent-memory
A hybrid memory system that provides persistent, searchable knowledge management for AI agents.
agent-memory-mcp
A hybrid memory system that provides persistent, searchable knowledge management for AI agents (Architecture, Patterns, Decisions).
agent-orchestration-improve-agent
Systematic improvement of existing agents through performance analysis, prompt engineering, and continuous iteration.