Bright Data Async Snapshot Pipeline

Overview

Implement the documented trigger, progress, and download lifecycle as a state machine. Keep dataset identifiers and approved inputs separate from the API key; cap polling, validate terminal states, and stream results rather than loading arbitrary payloads into memory.

Prerequisites

• A reviewed dataset ID and approved public input manifest
• A named-user Bright Data API key in the runtime secret manager
• A retention, schema, maximum-record, and maximum-byte policy

Instructions

Step 1: Validate the manifest

Read the requested dataset/inputs and Grep for disallowed target classes or fields. Hash the approved input manifest before submission.

Step 2: Trigger once

Use Bash(curl:*) only against the fixed Bright Data API origin with the API key as a Bearer header.

curl --fail-with-body --request POST \
  'https://api.brightdata.com/datasets/v3/trigger?dataset_id=DATASET_ID' \
  --header "Authorization: Bearer $BRIGHTDATA_API_KEY" \
  --header 'Content-Type: application/json' \
  --data-binary @approved-inputs.json

Step 3: Poll deliberately

Persist the returned snapshot_id; poll GET /datasets/v3/progress/SNAPSHOT_ID with bounded attempts and provider-directed delay. Handle ready, failed, empty, expired, and still-building states explicitly.

Step 4: Download and verify

Stream GET /datasets/v3/snapshot/SNAPSHOT_ID in the approved format. Use parts for large results, hold format/compression parameters constant, enforce byte/record ceilings, and validate the schema before promotion.

Tool Discipline

Use Read and Grep for policy and schema checks. Use Write and Edit only for the manifest, state machine, tests, and redacted receipt. Use Bash(curl:*) for fixed-origin Bright Data API calls after authorization; never print the Bearer value or raw result data.

Output

• Input-manifest hash and snapshot identifier
• Bounded state-transition log without target data or credentials
• Schema/size validation and a promoted-or-quarantined result

Examples

Submit a small approved URL batch, record the returned snapshot ID, poll until ready, and stream JSON to quarantine. Reject a response whose schema, record count, or byte count exceeds the manifest even if the provider marks it ready.

Error Handling

Failure	Meaning	Response
400 validation response	Dataset ID or input shape is invalid	Correct the manifest; do not retry unchanged input
429 or too many jobs	Tenant or dataset concurrency is exhausted	Pause new triggers and wait for owned jobs
Snapshot expired or empty	Result cannot be promoted	Trigger a newly approved run or investigate inputs

Resources

• Web Scraper API asynchronous requests
• Download snapshot
• Scraper async requests guide
• Authentication and API keys