무료GitHub에 공개

databricks-spark-structured-streaming

Comprehensive guide to Spark Structured Streaming for production workloads. Use when building streaming pipelines, working with Kafka ingestion, implementing Real-Time Mode (RTM), configuring triggers (processingTime, availableNow), handling stateful operations with watermarks, optimizing checkpoints, performing stream-stream or stream-static joins, writing to multiple sinks, or tuning streaming cost and performance.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

databricks/databricks-agent-skills

연동 도구PostgreSQL

설치 방법 보기

포함된 파일(15)

SKILL.md3.7 KB
agents/openai.yaml414 B
assets/databricks.png15.0 KB
assets/databricks.svg582 B
references/checkpoint-best-practices.md8.4 KB
references/kafka-streaming.md16.0 KB
references/lakebase-sink-python.md20.6 KB
references/merge-operations.md10.7 KB
references/multi-sink-writes.md12.6 KB
references/real-time-mode.md18.8 KB
references/stateful-operations.md11.7 KB
references/stream-static-joins.md13.9 KB
references/stream-stream-joins.md16.5 KB
references/streaming-best-practices.md9.4 KB
references/trigger-and-cost-optimization.md13.6 KB

SKILL.md(원문)

설치하기 전에 에이전트에게 전달되는 지시 내용을 확인할 수 있습니다.

Spark Structured Streaming

Production-ready streaming pipelines with Spark Structured Streaming. This skill provides navigation to detailed patterns and best practices.

Quick Start

from pyspark.sql.functions import col, from_json

# Basic Kafka to Delta streaming
df = (spark
    .readStream
    .format("kafka")
    .option("kafka.bootstrap.servers", "broker:9092")
    .option("subscribe", "topic")
    .load()
    .select(from_json(col("value").cast("string"), schema).alias("data"))
    .select("data.*")
)

df.writeStream \
    .format("delta") \
    .outputMode("append") \
    .option("checkpointLocation", "/Volumes/catalog/checkpoints/stream") \
    .trigger(processingTime="30 seconds") \
    .start("/delta/target_table")

Core Patterns

Pattern	Description	Reference
Kafka Streaming	Kafka to Delta, Kafka to Kafka, Real-Time Mode	See references/kafka-streaming.md
Real-Time Mode (RTM)	Sub-second E2E latency — cluster setup, slot math, supported ops (incl. stream-stream inner join on DBR 18+), `transformWithState`, observability, error classes, delivery semantics	See references/real-time-mode.md
Lakebase Sink	Write streaming records into Lakebase Postgres with transactional upserts. Native `format("postgresql")` sink (DBR 18.3+) and manual `foreach` sink as a fallback	See references/lakebase-sink-python.md
Stream Joins	Stream-stream joins, stream-static joins	See references/stream-stream-joins.md, references/stream-static-joins.md
Multi-Sink Writes	Write to multiple tables, parallel merges	See references/multi-sink-writes.md
Merge Operations	MERGE performance, parallel merges, optimizations	See references/merge-operations.md

Configuration

Topic	Description	Reference
Checkpoints	Checkpoint management and best practices	See references/checkpoint-best-practices.md
Stateful Operations	Watermarks, state stores, RocksDB configuration	See references/stateful-operations.md
Trigger & Cost	Trigger selection, cost optimization, RTM	See references/trigger-and-cost-optimization.md

Best Practices

Topic	Description	Reference
Production Checklist	Comprehensive best practices	See references/streaming-best-practices.md

Production Checklist

Checkpoint location is persistent (UC volumes, not DBFS)
Unique checkpoint per stream
Fixed-size cluster (no autoscaling for streaming)
Monitoring configured (input rate, lag, batch duration)
Exactly-once verified (txnVersion/txnAppId)
Watermark configured for stateful operations
Left joins for stream-static (not inner)

리뷰

아직 리뷰가 없습니다. 사용해 본 소감을 남겨 주세요.

로그인 없이 쓸 수 있습니다. 사용 후기, 질문, 작동하지 않은 환경 등을 편하게 남겨 주세요.

아직 댓글이 없습니다. 첫 댓글을 남겨 보세요.

같은 저장소의 스킬

databricks-agent-bricks

무료

Create Agent Bricks: Knowledge Assistants (KA) for document Q&A and Supervisor Agents for multi-agent orchestration (MAS).

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

databricks/databricks-agent-skills3522026년 10월 18일 업데이트

databricks-ai-functions

무료

Use Databricks built-in AI Functions (ai_classify, ai_extract, ai_summarize, ai_mask, ai_translate, ai_fix_grammar, ai_gen, ai_analyze_sentiment, ai_similarity, ai_parse_document, ai_prep_search, ai_query, ai_forecast) to add AI capabilities directly to SQL and PySpark pipelines without managing model endpoints. Also covers document parsing and building custom RAG pipelines (parse → prep_search → index → query).

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

databricks/databricks-agent-skills3522026년 10월 18일 업데이트

databricks-ai-runtime

무료

Databricks AI Runtime, the `databricks air` CLI commands for submitting and managing GPU training workloads on Databricks serverless compute. Use for: writing and submitting `databricks air` workload YAML, passing hyperparameters and secrets, checking run status, listing/cancelling runs, streaming a run's logs and watching its progress, custom Docker image setup, and environment configuration.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

databricks/databricks-agent-skills3522026년 10월 18일 업데이트

databricks-aibi-dashboards

무료

Create Databricks AI/BI dashboards. Must use when creating, updating, or deploying Lakeview dashboards as Databricks Dashboard have a unique json structure. CRITICAL: You MUST test ALL SQL queries via CLI BEFORE deploying. Follow guidelines strictly.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

databricks/databricks-agent-skills3522026년 10월 18일 업데이트

databricks-app-design

무료

Design the UX of custom-code Databricks Apps (AppKit/React) data screens — KPI/overview pages, reports, charts, tables, and Genie/chat data assistants — mapped to concrete AppKit components. Use when BUILDING or reviewing the UI of an AppKit/React app that displays data or answers data questions: choosing genre, layout, charts, KPIs, semantic color, required states (loading/empty/error), IBCS notation, and AI-result trust (showing generated SQL/sources for Genie/chat). A plain "create a dashboard" request means a managed AI/BI (Lakeview) dashboard → use databricks-aibi-dashboards, NOT this skill. Also NOT for non-data frontend (forms, settings, auth, marketing) or scaffolding/build/deploy (→ databricks-apps). Complements databricks-apps; use it alongside whenever a custom app has a chart, table, KPI, report, or Genie/chat/AI surface.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

databricks/databricks-agent-skills3522026년 10월 18일 업데이트

databricks-apps

무료

Build apps on Databricks Apps platform. Use when asked to create data apps, analytics tools, or custom interactive visualizations. A plain "create a dashboard" request means a managed AI/BI (Lakeview) dashboard → use databricks-aibi-dashboards, not this skill. Evaluates data access patterns (analytics vs Lakebase synced tables) before scaffolding. Invoke BEFORE starting implementation.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

databricks/databricks-agent-skills3522026년 10월 18일 업데이트

databricks의 스킬 모두 보기

이 스킬의 문제 신고

설치

명령으로 설치

npx skills add https://github.com/databricks/databricks-agent-skills --skill databricks-spark-structured-streaming

터미널에 붙여 넣고 실행합니다(Node.js 필요).

Claude에게 부탁하기

다음 Agent Skill을 설치해 주세요.
https://github.com/databricks/databricks-agent-skills/tree/main/skills/databricks-spark-structured-streaming

먼저 SKILL.md와 첨부 파일의 내용을 확인해 수상한 처리(외부 전송, 인증 정보 읽기 등)가 없는지 확인한 뒤 ~/.claude/skills/databricks-spark-structured-streaming/ 에 배치해 주세요.

이 문장을 그대로 Claude Code에 보내면 내용을 확인한 뒤 설치해 줍니다.

GitHub에서 소스 보기

저장소: databricks/databricks-agent-skills
스타: 352
포크: 93
카테고리: 기타
원문 언어: 영어
최종 업데이트: 2026년 10월 18일

스킬은 에이전트를 위한 지시와 스크립트의 묶음입니다. cccskills는 내용의 안전성을 보증하지 않습니다. 설치하기 전에 내용을 확인하세요.