Bright Data Snapshot Delivery Receiver

Overview

Treat snapshot delivery as an untrusted, repeatable data-transfer event. Configure delivery with the current POST /datasets/v3/deliver/SNAPSHOT_ID API, authenticate the managed destination, stream the body, validate it, and make downstream processing idempotent.

Prerequisites

• A completed approved snapshot and its expected schema
• A managed HTTPS endpoint with destination authentication
• Durable quarantine storage and a replay ledger

Instructions

Step 1: Establish the delivery contract

Read the receiver and Grep for body buffering, unauthenticated routes, hard-coded provider IPs, guessed retry schedules, and direct writes to trusted tables. Record the expected schema, format, destination, and ownership.

Step 2: Configure an approved destination

Write or Edit the integration so an authorized control-plane client requests POST /datasets/v3/deliver/SNAPSHOT_ID with type=webhook and managed destination authentication. Never place destination credentials in the URL or logs.

Step 3: Receive defensively

Stream the request into bounded quarantine storage. Authenticate before processing, cap bytes and duration, compute a digest, validate format and schema, and derive an idempotency key from trusted delivery context plus content digest.

Step 4: Acknowledge and replay safely

Acknowledge only after durable acceptance. Make duplicates no-ops, quarantine malformed or oversized content, and replay from retained evidence through the same validator. Do not invent provider retry or source-IP guarantees.

Tool Discipline

Use Read and Grep to inspect receiver, routing, and storage code. Use Write and Edit for receiver logic, validation, idempotency, fixtures, and the replay runbook. This skill does not invoke delivery or expose a public endpoint.

Output

• Authenticated streaming receiver contract
• Schema, idempotency, quarantine, and replay implementation
• Redaction-safe success and failure tests

Examples

The receiver authenticates a managed header, streams NDJSON to quarantine while hashing it, rejects an unexpected schema, and promotes a validated object exactly once even if the same payload arrives again.

Error Handling

Failure	Meaning	Response
Destination auth is missing	Anyone may submit data	Reject before reading the payload
Payload exceeds the configured ceiling	Transfer is outside the workload contract	Stop streaming and quarantine metadata
Digest already exists	Delivery is a duplicate	Return the documented safe acknowledgement without reprocessing

Resources

• Deliver snapshot
• Scraper data delivery
• Download snapshot
• REST API authentication