Bright Data Governed Collection Architecture

Overview

Design collection as a policy-governed data pipeline, not a direct application call. Separate human approval and configuration from collection workers, then quarantine and validate all results before any trusted consumer or external destination receives them.

Prerequisites

• Approved purpose, targets, fields, products, retention, recipients, and owners
• Current proxy, Browser API, scraper, snapshot, and delivery requirements
• Platform identity, queue, storage, policy, and observability capabilities

Instructions

Step 1: Discover the existing planes

Read design and runtime files and Grep for Bright Data credentials, zones, API endpoints, browser sessions, queues, snapshot storage, and downstream destinations. Mark every trust transition and uncontrolled shortcut.

Step 2: Define the topology

Write or Edit an architecture with these responsibilities:

Approval + workload registry -> admission controller -> bounded collection workers
                                                   -> quarantine storage
Quarantine -> schema/policy validation -> approved internal consumer
                                    \-> approved snapshot delivery destination
Telemetry <- redacted events, error classes, usage units, and decision receipts

Keep proxy and Browser API credentials in worker-specific secret bindings and REST API keys in authorized control-plane identities.

Step 3: Apply cross-cutting controls

Specify target and destination allowlists, least privilege, environment isolation, job and byte ceilings, backpressure, idempotency, schema validation, retention, deletion, redaction, audit evidence, and independent emergency stop.

Step 4: Challenge the design

Test policy denial, revoked credentials, 429, provider 5xx, browser timeout, snapshot failure or expiry, schema drift, duplicate delivery, storage exhaustion, and downstream outage. Require an owner and recovery decision for each path.

Tool Discipline

Use Read and Grep for topology discovery and evidence gathering. Use Write and Edit for diagrams, threat models, contracts, tests, and architecture decisions. This skill does not provision Bright Data resources or initiate collection.

Output

• Control-plane and data-plane topology with trust boundaries
• Responsibility, secret, data, and destination matrix
• Failure analysis, rollback path, and decision record

Examples

An admission controller accepts only signed workload manifests. Product-specific workers emit raw results to quarantine, validators release an approved schema, and delivery resolves only named destinations while redacted telemetry preserves an audit trail.

Error Handling

Failure	Meaning	Response
Application code can choose arbitrary targets	Admission boundary is missing	Route requests through the workload registry
Raw collection reaches analytics directly	Validation boundary is bypassed	Require quarantine and schema promotion
Control and collection share one broad API key	Identity blast radius is excessive	Split identities and scopes

Resources

• REST API authentication
• Browser API introduction
• Asynchronous scraper requests
• Deliver snapshot