otel-collector-config-tester

A misconfigured Collector pipeline silently drops or misroutes spans - no error at deploy time, only missing data at query time. This skill tests two distinct failure modes:

1. Static config errors - invalid YAML, undefined component references, missing required fields. Caught by otelcol validate before the process starts.
2. Dynamic pipeline errors - a syntactically valid config that doesn't actually forward spans (wrong pipeline wiring, processor ordering, filter that drops everything). Caught only by sending real spans and asserting on the output.

When to use

• A new receiver, processor, or exporter is added to a Collector config and the team needs to confirm the pipeline is wired correctly before deploying.
• A processor chain (batch, filter, transform/OTTL) was reordered and span loss or attribute mutation must be ruled out.
• CI should gate on "collector config is valid and spans reach the exporter" rather than relying on post-deploy observation.

Step 1 - Validate config syntax with otelcol validate

Run:

otelcol validate --config=collector-config.yaml

This checks that all components referenced in service.pipelines are defined in their respective top-level sections, required fields are present, and the YAML parses cleanly. It does not start the collector process.

The config structure the validator checks:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: "0.0.0.0:4317"

processors:
  batch: {}

exporters:
  otlp/backend:
    endpoint: "https://backend.example.com:4317"

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/backend]

Components follow type[/name] naming (otlp/backend above), which allows multiple instances of the same type in one config. Every component referenced in service.pipelines must be declared in its top-level section - validate reports undefined references as errors.

In CI:

- name: Validate collector config
  run: otelcol validate --config=collector-config.yaml

Exit code is non-zero on any validation error, so a failing step blocks the pipeline.

Step 2 - Wire the debug exporter to observe span flow

Add the debug exporter to a test pipeline alongside (or instead of) the production exporter. This exporter writes span data to the collector process stdout without requiring a backend.

Three verbosity levels are available:

Level	Output per batch
basic (default)	Single-line count summary: "resource spans": 1, "spans": 2
normal	One line per span record
detailed	Full multi-line dump: TraceID, ParentID, timestamps, status, all attributes

Config to route a test pipeline through the debug exporter:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: "0.0.0.0:4317"

exporters:
  debug:
    verbosity: detailed

service:
  pipelines:
    traces/test:
      receivers: [otlp]
      processors: []
      exporters: [debug]

Multiple pipelines of the same signal type use type/name syntax (traces/test above), so the test pipeline does not conflict with the production traces pipeline in the same config.

Send a span to the collector and grep stdout for the trace ID or a known attribute to assert receipt:

# Send a test span via grpcurl or the OTel SDK
grpcurl -plaintext -d @ localhost:4317 \
  opentelemetry.proto.collector.trace.v1.TraceService/Export \
  < test-span.json

# Assert the debug exporter emitted the span
docker logs <container> 2>&1 | grep "my.attribute"

Step 3 - Wire the file exporter for machine-readable assertions

The debug exporter writes to stdout, which is inconvenient for assertion scripts. The file exporter writes each exported batch as a JSON object per line, making it grep- and jq-parseable:

exporters:
  file:
    path: /tmp/collector-spans.jsonl

service:
  pipelines:
    traces/test:
      receivers: [otlp]
      processors: []
      exporters: [file]

After sending spans, assert on the output file:

# Check at least one span was exported
[ $(wc -l < /tmp/collector-spans.jsonl) -gt 0 ] || { echo "No spans exported"; exit 1; }

# Assert a specific attribute value was preserved through processors
jq -e '
  .resourceSpans[].scopeSpans[].spans[]
  | select(.name == "order.create")
  | .attributes[]
  | select(.key == "order.item_count")
  | .value.intValue == 1
' /tmp/collector-spans.jsonl

Each line in the file is a JSON object, which matches the OTLP/JSON protobuf encoding. The default flush_interval is 1 second - wait at least 2 seconds after the last span before asserting on the file in a test script.

Step 4 - Test processor behavior end-to-end

Processors modify spans in transit. A common failure mode: a filter or transform processor was added but its OTTL condition is wrong, silently dropping all spans.

Test pattern using the file exporter as the oracle:

processors:
  # Filter keeps only spans with http.response.status_code >= 400
  filter/errors_only:
    error_mode: ignore
    traces:
      span:
        - 'attributes["http.response.status_code"] < 400'

service:
  pipelines:
    traces/test:
      receivers: [otlp]
      processors: [filter/errors_only]
      exporters: [file]

Send two spans - one with http.response.status_code = 200, one with http.response.status_code = 500 - then assert the file contains exactly one span with the 500 status code and zero spans with 200.

The Transform processor uses OTTL (OpenTelemetry Transformation Language) for advanced mutations. Test attribute mutations the same way: send a known input span, read the file exporter output, assert the mutated attribute value.

Step 5 - CI integration

Full pipeline: validate config, start the collector in Docker, send test spans, assert on the file exporter output, stop the container. Minimal shape:

- name: Validate config
  run: otelcol validate --config=collector-config-test.yaml
- name: Start collector, send spans, assert
  run: |
    docker run -d --name otel-test -p 4317:4317 \
      -v $PWD/collector-config-test.yaml:/etc/otel/config.yaml \
      -v /tmp/spans:/tmp/spans \
      otel/opentelemetry-collector-contrib:0.153.0 --config=/etc/otel/config.yaml
    sleep 2 && python3 tests/send_test_spans.py && sleep 2
    [ $(wc -l < /tmp/spans/output.jsonl) -gt 0 ]

Pin the image tag (0.153.0) - the latest tag changes component stability levels between releases; the image exposes OTLP over gRPC on 4317 and HTTP on 4318. Full workflow with checkout, startup gating, and teardown: references/ci-integration.md.

Anti-patterns

Anti-pattern	Why it fails	Fix
Only running otelcol validate	Catches syntax errors but not pipeline wiring or processor logic errors	Add a send-and-assert step (Steps 2-4)
Using debug exporter with basic verbosity for assertions	Outputs only count summaries; no attribute values to assert on	Use verbosity: detailed or switch to file exporter
Asserting immediately after sending spans	file exporter flush_interval defaults to 1s - file may be empty	Wait at least 2s after last span
Using latest Docker image tag in CI	Component stability levels change between releases; tests break on unrelated collector upgrades	Pin to a specific version tag
Reusing production exporter in test pipeline	Sends test spans to the live backend	Use a named test pipeline (traces/test) with file or debug exporter

Limitations

• otelcol validate does not check network reachability of exporter endpoints - a valid config may still fail at runtime if the backend is unreachable.
• The file exporter is in the contrib distribution (otel/opentelemetry-collector-contrib), not the core distribution. Verify it is present in the collector build used in CI.
• The default flush_interval is 1 second; very-high-throughput tests may need flush_interval: 100ms to avoid waiting on large batches.
• This skill covers pipeline correctness testing. For sampling-ratio verification or tail-sampling behavior, pair with opentelemetry-trace-assertions which uses in-process SDK exporters for finer-grained span-count control.

References

• OTel Collector overview - pipeline model, receiver/processor/exporter roles
• OTel Collector configuration docs - YAML structure, component naming, service.pipelines, otelcol validate command
• OTel Collector troubleshooting docs - debug exporter setup
• OTel Collector transforming telemetry docs - OTTL, filter/transform processor config
• OTel Collector quick-start docs - Docker image, port mapping
• debug exporter README - verbosity levels (basic/normal/detailed), sampling_initial, sampling_thereafter
• file exporter README - JSON-lines output format, path, flush_interval
• opentelemetry-trace-assertions - in-process SDK in-memory exporter for unit-level span assertions