4.6K skills · sorted by relevance
Build repeatable evaluations for AI agent responses, tool choices, and multi-step tasks. Use reference datasets and scoring criteria to compare changes, detect quality regressions, and set automated checks that must pass before deployment.
Design criteria for using an LLM to judge AI responses through scoring or comparisons between two answers. Includes checks for order and length bias, plus validation against human judgments to assess evaluation reliability.
Design agent memory that preserves facts, preferences, and conversation history across sessions. Plan storage, retrieval, and retention separately for each organization or user, with checks for unauthorized access, outdated information, conflicting records, and retrieval quality.
Turn graded execution traces and production logs into model training examples and preferred-versus-rejected response pairs. Select useful records, remove sensitive information, exclude evaluation examples, and preserve links to the original runs.
Create shareable links that open specific Arize resources, including traces, datasets, and evaluators. Use IDs from logs or exported data, with suitable time ranges, to direct teammates to the relevant view.
Plan technology strategy and engineering team growth with frameworks for technical debt, hiring, architecture decisions, and vendor evaluation. Use Python tools and decision templates to prioritize improvements, structure teams, and track engineering performance.
Build document search and AI retrieval workflows with Chroma, a database for numerical representations of text. Cover document storage, meaning-based search, metadata filters, persistence, and connections to LangChain and LlamaIndex.
Design and improve RAG systems, which retrieve documents to support language model answers. Explore meaningful document splitting, vector storage, combined keyword and semantic search, and relevance ranking while checking common causes of poor retrieval.
Set up Langfuse to record LLM application activity, costs, and response times. Manage prompt versions, organize evaluation datasets, and compare results to investigate failures and guide improvements using recorded evidence.
Design memory systems that help AI agents retrieve relevant past information. Choose short-term and long-term memory structures, compare vector storage options, and review document splitting and retrieval strategies for different kinds of knowledge and experience.
Write tests that demonstrate a pull request’s user-visible changes through before-and-after steps, screenshots, and recorded observations. Extend an existing core journey or create a one-off proof that helps reviewers understand the outcome and access boundaries.
Evaluate whether Google SecOps rules detect threats described in articles or supplied text. Generate simulated attack logs, identify coverage gaps, summarize matching rules, and create rules for uncovered threats, with user approval before adding them.
Review BigQuery capacity, table layouts, storage retention, and SQL to improve cost efficiency. Compare billing options, plan partitioned tables, and evaluate search indexes, stored aggregation results, and BI Engine for suitable workloads.
Helps design semantic search and RAG, where an AI answers using retrieved documents. Covers embeddings, numerical representations of content, plus document splitting, search indexes, keyword search combinations, and evaluation of retrieval quality.
Design evaluation systems that measure agent results across accuracy, completeness, citations, and tool efficiency. Build repeatable tests, compare changes against a baseline, and define quality checks that catch regressions before deployment and monitor production performance.
Write and review launch prompts for agents working autonomously for hours or days. Define successful deliverables, excluded partial results, stopping rules, worker handoffs, and evidence-based review requirements so completion can be assessed against explicit criteria.
Find relevant passages in an Obsidian vault using a local word-based search index. Optionally reorder results by meaning with a multilingual Nomic model, returning note paths and snippets while leaving the original notes unchanged.
Set up a CARLA driving evaluation environment for Leaderboard 1.0, 2.0, or 2.1. Match the simulator and scenario_runner versions, configure Python paths, and verify that the required components load correctly.
Guides iterative improvement of AI agent prompts, skills, and memory using A-Evolve. It explains how to evaluate agents against benchmark tasks, apply changes, validate results, and track or roll back revisions with Git.
Build document search and AI retrieval workflows with Chroma, a database for numerical representations of text. Cover document storage, meaning-based search, metadata filters, persistence, and connections to LangChain and LlamaIndex.
Guides building AI applications that search documents before answering questions, an approach called RAG. Covers importing files and web data, creating searchable indexes, adding conversational interfaces, and evaluating whether answers match the retrieved material.
Design and improve prompts for language models, then test their accuracy and consistency. Create reusable templates, JSON output schemas that define response structure, evaluation criteria, and test cases while considering token use and model differences.
Turn an idea, existing code, or a previous Codex conversation into a runnable OpenAI Agents SDK app. Supports local verification, focused evaluations of agent behavior, and local deployment through the Deployment Manager.
Test whether an AI agent executes commands hidden in user input, evaluates untrusted code, or runs scripts from unverified sources. Record confirmed weaknesses with the test conversation, potential impact, and suggested safeguards.