BEIR Repo Skill

Use this skill for BEIR, the Python benchmark framework for information retrieval datasets, retrievers, rerankers, query generation, and SentenceTransformers-style training.

First Checks

• Install the public package with pip install beir, or use an editable install only when working on a checkout.
• Verify the core import with python -c "import beir; print(beir.__all__)".
• Run python scripts/check_beir_environment.py to inspect core and optional dependencies without contacting external services.
• Read references/repo-provenance.md before deciding whether this skill matches a current checkout or should be refreshed.
• Use references/troubleshooting.md for cross-cutting install, optional backend, data, service, credential, and scale failures.

Route by Task

• Use data-loading for BEIR corpus.jsonl, queries.jsonl, qrels/<split>.tsv, Hugging Face datasets, local validation, downloads, runfiles, and result JSON persistence.
• Use retrieval-evaluation for EvaluateRetrieval, dense exact search, FAISS, sparse retrieval, BM25/Elasticsearch, API embeddings, custom model protocols, metrics, embedding caches, and runfile export.
• Use reranking when first-stage retrieval results need a cross-encoder, MonoT5, SBERT, or custom pair-scoring second stage.
• Use generation for synthetic query generation, generated qrels, passage expansion, output prefixes, and fake-model smoke checks.
• Use training for TrainRetriever, pair/triplet construction, IR evaluators, BEIR losses, hard-negative workflows, and safe training-run planning.

Common Workflow Order

1. Start in data-loading to validate local files or understand remote dataset requirements.
2. Move to retrieval-evaluation to choose a backend and compute BEIR metrics.
3. Add reranking only after you have candidate results from a first-stage retriever.
4. Use generation when you need generated queries, qrels, or expanded corpora before retrieval or training.
5. Use training when preparing model fine-tuning data or planning a long SentenceTransformers run.

Bundled Smoke Tests

Run these from inside the skills/beir/ directory or pass the full script paths from elsewhere:

python scripts/check_beir_environment.py
python sub-skills/data-loading/scripts/make_tiny_beir_dataset.py /tmp/beir-tiny
python sub-skills/data-loading/scripts/validate_beir_dataset.py /tmp/beir-tiny --split test
python sub-skills/retrieval-evaluation/scripts/retrieval_smoke.py
python sub-skills/reranking/scripts/rerank_smoke.py
python sub-skills/generation/scripts/generation_smoke.py
python sub-skills/training/scripts/training_data_smoke.py --exercise-max-corpus-error

These checks are offline and deterministic. They do not download BEIR datasets, model weights, or contact Elasticsearch/API services.

Dependency Boundaries

• Core BEIR data loading and metric evaluation need the package runtime dependencies from pyproject.toml.
• FAISS workflows require faiss-cpu or a compatible GPU FAISS installation.
• BM25 workflows require the elasticsearch extra and a running Elasticsearch-compatible service; installing the Python package alone is not enough.
• Cohere and Voyage API embedding workflows require provider packages, environment variables, network access, and account credentials.
• vLLM, LoRA, NVEmbed, LLM2Vec, PEFT, TILDE, MonoT5, and full training workflows can require large model downloads, GPU memory, and careful PyTorch/Transformers compatibility.

Refresh Guidance

Refresh this skill if BEIR package metadata, public APIs, examples, optional dependency names, supported Python versions, retrieval model wrappers, generation wrappers, or training helpers change. The provenance file records the source commit and evidence paths used to generate this version.