otel-collector-config-tester
A misconfigured Collector pipeline silently drops or misroutes spans - no
error at deploy time, only missing data at query time. This skill tests two
distinct failure modes:
- Static config errors - invalid YAML, undefined component references,
missing required fields. Caught by
otelcol validate before the process
starts.
- Dynamic pipeline errors - a syntactically valid config that doesn't
actually forward spans (wrong pipeline wiring, processor ordering, filter
that drops everything). Caught only by sending real spans and asserting
on the output.
When to use
- A new receiver, processor, or exporter is added to a Collector config
and the team needs to confirm the pipeline is wired correctly before
deploying.
- A processor chain (batch, filter, transform/OTTL) was reordered and
span loss or attribute mutation must be ruled out.
- CI should gate on "collector config is valid and spans reach the exporter"
rather than relying on post-deploy observation.
Step 1 - Validate config syntax with otelcol validate
Run:
otelcol validate --config=collector-config.yaml
This checks that all components referenced in service.pipelines are
defined in their respective top-level sections, required fields are present,
and the YAML parses cleanly. It does not start the collector process.
The config structure the validator checks:
receivers:
otlp:
protocols:
grpc:
endpoint: "0.0.0.0:4317"
processors:
batch: {}
exporters:
otlp/backend:
endpoint: "https://backend.example.com:4317"
service:
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [otlp/backend]
Components follow type[/name] naming (otlp/backend above), which allows
multiple instances of the same type in one config. Every component referenced
in service.pipelines must
be declared in its top-level section - validate reports undefined
references as errors.
In CI:
- name: Validate collector config
run: otelcol validate --config=collector-config.yaml
Exit code is non-zero on any validation error, so a failing step blocks
the pipeline.
Step 2 - Wire the debug exporter to observe span flow
Add the debug exporter to a test pipeline alongside (or instead of) the
production exporter. This exporter writes span data to the collector process
stdout without requiring a backend.
Three verbosity levels are available:
| Level | Output per batch |
|---|
basic (default) | Single-line count summary: "resource spans": 1, "spans": 2 |
normal | One line per span record |
detailed | Full multi-line dump: TraceID, ParentID, timestamps, status, all attributes |
Config to route a test pipeline through the debug exporter:
receivers:
otlp:
protocols:
grpc:
endpoint: "0.0.0.0:4317"
exporters:
debug:
verbosity: detailed
service:
pipelines:
traces/test:
receivers: [otlp]
processors: []
exporters: [debug]
Multiple pipelines of the same signal type use type/name syntax
(traces/test above), so the test pipeline does not conflict with the
production traces pipeline in the same config.
Send a span to the collector and grep stdout for the trace ID or a known
attribute to assert receipt:
# Send a test span via grpcurl or the OTel SDK
grpcurl -plaintext -d @ localhost:4317 \
opentelemetry.proto.collector.trace.v1.TraceService/Export \
< test-span.json
# Assert the debug exporter emitted the span
docker logs <container> 2>&1 | grep "my.attribute"
Step 3 - Wire the file exporter for machine-readable assertions
The debug exporter writes to stdout, which is inconvenient for assertion
scripts. The file exporter writes each exported batch as a JSON object per
line, making it grep- and jq-parseable:
exporters:
file:
path: /tmp/collector-spans.jsonl
service:
pipelines:
traces/test:
receivers: [otlp]
processors: []
exporters: [file]
After sending spans, assert on the output file:
# Check at least one span was exported
[ $(wc -l < /tmp/collector-spans.jsonl) -gt 0 ] || { echo "No spans exported"; exit 1; }
# Assert a specific attribute value was preserved through processors
jq -e '
.resourceSpans[].scopeSpans[].spans[]
| select(.name == "order.create")
| .attributes[]
| select(.key == "order.item_count")
| .value.intValue == 1
' /tmp/collector-spans.jsonl
Each line in the file is a JSON object, which matches the OTLP/JSON protobuf
encoding. The default flush_interval is 1 second - wait at least 2 seconds
after the last span before asserting on the file in a test script.
Step 4 - Test processor behavior end-to-end
Processors modify spans in transit. A common failure mode: a filter or
transform processor was added but its OTTL condition is wrong, silently
dropping all spans.
Test pattern using the file exporter as the oracle:
processors:
# Filter keeps only spans with http.response.status_code >= 400
filter/errors_only:
error_mode: ignore
traces:
span:
- 'attributes["http.response.status_code"] < 400'
service:
pipelines:
traces/test:
receivers: [otlp]
processors: [filter/errors_only]
exporters: [file]
Send two spans - one with http.response.status_code = 200, one with
http.response.status_code = 500 - then assert the file contains exactly
one span with the 500 status code and zero spans with 200.
The Transform processor uses OTTL (OpenTelemetry Transformation Language) for
advanced mutations. Test attribute mutations the same way: send a known input
span, read the file exporter output, assert the mutated attribute value.
Step 5 - CI integration
Full pipeline: validate config, start the collector in Docker, send test
spans, assert on the file exporter output, stop the container. Minimal shape:
- name: Validate config
run: otelcol validate --config=collector-config-test.yaml
- name: Start collector, send spans, assert
run: |
docker run -d --name otel-test -p 4317:4317 \
-v $PWD/collector-config-test.yaml:/etc/otel/config.yaml \
-v /tmp/spans:/tmp/spans \
otel/opentelemetry-collector-contrib:0.153.0 --config=/etc/otel/config.yaml
sleep 2 && python3 tests/send_test_spans.py && sleep 2
[ $(wc -l < /tmp/spans/output.jsonl) -gt 0 ]
Pin the image tag (0.153.0) - the latest tag changes component stability
levels between releases; the image exposes OTLP over gRPC on 4317 and HTTP on
4318. Full workflow with checkout, startup gating, and teardown:
references/ci-integration.md.
Anti-patterns
| Anti-pattern | Why it fails | Fix |
|---|
Only running otelcol validate | Catches syntax errors but not pipeline wiring or processor logic errors | Add a send-and-assert step (Steps 2-4) |
Using debug exporter with basic verbosity for assertions | Outputs only count summaries; no attribute values to assert on | Use verbosity: detailed or switch to file exporter |
| Asserting immediately after sending spans | file exporter flush_interval defaults to 1s - file may be empty | Wait at least 2s after last span |
Using latest Docker image tag in CI | Component stability levels change between releases; tests break on unrelated collector upgrades | Pin to a specific version tag |
| Reusing production exporter in test pipeline | Sends test spans to the live backend | Use a named test pipeline (traces/test) with file or debug exporter |
Limitations
otelcol validate does not check network reachability of exporter
endpoints - a valid config may still fail at runtime if the backend
is unreachable.- The
file exporter is in the contrib distribution
(otel/opentelemetry-collector-contrib), not the core distribution.
Verify it is present in the collector build used in CI.
- The default
flush_interval is 1 second; very-high-throughput tests may
need flush_interval: 100ms to avoid waiting on large batches.
- This skill covers pipeline correctness testing. For sampling-ratio
verification or tail-sampling behavior, pair with
opentelemetry-trace-assertions
which uses in-process SDK exporters for finer-grained span-count control.
References
- OTel Collector overview - pipeline model, receiver/processor/exporter roles
- OTel Collector configuration docs - YAML structure, component naming,
service.pipelines, otelcol validate command
- OTel Collector troubleshooting docs - debug exporter setup
- OTel Collector transforming telemetry docs - OTTL, filter/transform
processor config
- OTel Collector quick-start docs - Docker image, port mapping
- debug exporter README - verbosity levels (basic/normal/detailed),
sampling_initial, sampling_thereafter
- file exporter README - JSON-lines output format,
path, flush_interval
opentelemetry-trace-assertions -
in-process SDK in-memory exporter for unit-level span assertions