8 skills · sorted by relevance
Filesystem RAG benchmarks: corpus/, train.json, evaluate_rag.py (RAGAS quality). Not for prod monitoring, latency/throughput benchmarking (use rag-perf), or evals outside this repo layout.
The English summary is being prepared. Showing the original description.
Implement comprehensive observability for LLM applications including tracing (Langfuse/Helicone), cost tracking, token optimization, RAG evaluation metrics (RAGAS), hallucination detection, and production monitoring. Essential for debugging, optimizing costs, and ensuring AI output quality. Use when ", llm-monitoring, tracing, langfuse, helicone, cost-tracking, ragas, evaluation, hallucination-detection, prompt-caching" mentioned.
The English summary is being prepared. Showing the original description.
Design and implement RAG systems, which retrieve documents to inform AI answers. Get document preparation strategies, vector database comparisons, combined keyword and meaning-based search, result reranking, and an evaluation plan for retrieval quality.
Performance benchmarking for a deployed NVIDIA RAG Blueprint server: profiling pass + aiperf load test driven by a single YAML config. Not for accuracy / RAGAS scoring (use rag-eval) or for deploying / repairing services (use rag-blueprint).
The English summary is being prepared. Showing the original description.
Design RAG pipelines: chunking, retrieval evaluation, and architecture. Use when building a RAG system, selecting a chunking strategy, choosing a vector database, optimizing retrieval quality, or evaluating with RAGAS metrics.
The English summary is being prepared. Showing the original description.
Decomposed, multi-criteria metric design for LLM pipelines. The metric IS the model — change the metric and the optimizer changes behavior. Decompose by default; bool during compile, float during eval; calibrate against human; mitigate judge bias. Search keywords: LLM-as-judge, llm as judge, eval metric, evaluation score, scoring function, rubric, RAGAS, G-Eval, judge bias, verbosity bias, how to evaluate LLM output.
The English summary is being prepared. Showing the original description.
Think and work like an expert Machine Learning Engineer. Use when a task calls for Machine Learning Engineer judgment. Reasons from decision-policy framing, Google's Rules of ML, point-in-time data, and prefill/decode inference physics through GBDT/PyTorch baselines, vLLM/SGLang/KServe serving with FP8/AWQ quantization, hybrid-retrieval RAG, RAGAS and human-validated LLM judges, OpenTelemetry GenAI tracing, and post-Omnibus EU AI Act obligations while treating train-serve skew, temporal leakage, prompt injection, LLM nondeterminism, and degenerate feedback loops as first-class failure modes.
The English summary is being prepared. Showing the original description.
模型评估全流程。覆盖分类/回归/排序/生成/RAG/LLM 评估、A/B 测试、 在线监控、数据漂移检测、评估集构建、基准测试。 触发关键词：评估、指标、Precision、Recall、F1、AUC、BLEU、ROUGE、 A/B 测试、基准测试、数据漂移、模型监控、RAGAS、LLM-as-Judge。
The English summary is being prepared. Showing the original description.