Logging & Observability Standards

Purpose

Logs are the black box recorder of your system. When failures happen at 2 AM in production, logs are your only witness. This skill ensures application state and failures are highly searchable, machine-readable, and traceable across system boundaries WITHOUT leaking sensitive user data.

When to use

• Bootstrapping a new backend microservice or monolithic API
• Refactoring code filled with disorganized console.log or print statements
• Designing a system that spans multiple services/functions
• Setting up monitoring, alerting, and debugging infrastructure

When NOT to use

• Application performance monitoring (APM) - related but different concern
• Security incident response (use SIEM/security tools)
• User analytics (different use case, different tool)

Inputs required

• Backend service with multiple endpoints/functions
• Logging infrastructure (ELK, DataDog, Grafana Loki, CloudWatch, etc.)
• Understanding of structured logging concepts

Workflow

1. Implement Structured Logging: Configure logger to output NDJSON (Newline Delimited JSON) instead of plain text
2. Inject Context: Attach a correlation_id (or trace_id) at HTTP entry point and pass through all downstream calls
3. Standardize Levels: ERROR (system broken), WARN (unexpected but recovered), INFO (lifecycle), DEBUG (verbose tracing)
4. Sanitize Data: Implement redaction middleware to mask credentials, tokens, and PII before logs hit the stream
5. Add Request Context: Log request duration, status code, user context (anonymized), and performance metrics
6. Trace Async Flows: Pass correlation ID through event handlers, message queues, and inter-service calls
7. Monitor Log Health: Alert on high ERROR rates, unexpected patterns, or missing correlation IDs

Rules

• MUST output JSON in production environments (not plain text)
• MUST include Request ID/Correlation ID in all HTTP requests
• MUST NEVER log raw passwords, session tokens, or financial data
• MUST log full stack traces for ERROR level (for debugging)
• MUST log only message and context for WARN/INFO (not verbose)
• MUST redact/sanitize any PII before log emission
• MUST include timestamps in UTC
• MUST standardize key names across all logs

Anti-patterns

• String Concatenation: logger.info("User " + userId + " failed to login") (unqueryable)
• Logging Expected Errors as ERROR: Logging validation failure as ERROR (use INFO/WARN)
• Silent Catching: Catching exception without logging the stack trace
• No Context: Logs with no request/correlation ID (impossible to trace)
• Unstructured Logs: Plain text logs without JSON structure
• Missing Timestamps: Logs without UTC timestamps (ordering problems)
• Over-Logging: Logging every line of execution (noise, storage cost)

Failure conditions

• Logs are plain text (not JSON)
• No correlation ID tracking
• PII or credentials in logs
• Missing stack traces on ERROR logs
• No way to correlate logs across services

Validation checklist

• Logger outputs NDJSON (each line is valid JSON)
• Correlation ID/Trace ID included in all requests
• ERROR logs include full stack traces
• INFO/WARN logs are concise (no excessive detail)
• All timestamps in UTC
• No passwords, tokens, or PII in logs
• Request duration/latency logged
• Correlation ID passed through async/inter-service calls
• Redaction middleware configured and working
• Log levels used correctly (ERROR for failures, INFO for lifecycle)
• Searchable by correlation ID (verified in log aggregator)
• Sampling/retention policy defined (cost management)

Output format

• Log structure: JSON with consistent keys: timestamp, level, message, correlationId, userId (anonymized), duration, error, stack
• Log format: NDJSON (one valid JSON object per line)
• Timestamps: UTC ISO 8601 format
• Levels: ERROR, WARN, INFO, DEBUG
• Middleware: Automatic request ID generation, redaction on output

Security considerations

• All PII (names, emails, phone numbers) MUST be redacted or anonymized
• Tokens, API keys, passwords MUST NEVER appear in logs
• Credit card data MUST NEVER appear in logs
• Database connection strings MUST be redacted
• User IDs may appear but real names MUST NOT
• Logs MUST be encrypted in transit and at rest

Agent execution notes

• Agent MAY: Add structured logging, implement correlation IDs, add redaction middleware, configure log rotation
• Agent MUST NEVER: Log passwords/tokens, use plain text logs, leave PII unredacted, omit stack traces
• Agent MUST ASK: Before adding new log messages that might contain PII, before changing log levels
• Agent MUST VALIDATE: Logs are JSON, correlation IDs flow through system, no PII present

Example

❌ Anti-pattern (String concatenation, no context, no redaction):

// BAD: Unstructured, concatenated
console.log('User ' + req.user.id + ' logged in at ' + new Date());

// BAD: No correlation ID
logger.info('Processing order');
logger.info('Order processed');

// BAD: Leaking PII and secrets
logger.error('Failed to connect: ' + process.env.DB_PASSWORD);
logger.info('User email: ' + user.email + ' password hash: ' + user.passwordHash);

// BAD: Expected error logged as ERROR
try {
  const user = await User.findById(userId);
} catch (e) {
  logger.error('User not found'); // WRONG level
}

✅ Correct pattern (Structured JSON, correlation IDs, redacted):

// CORRECT: Structured JSON with context
logger.info('User login successful', {
  userId: 'user_123', // Anonymized or hashed
  correlationId: req.id,
  duration: Date.now() - req.startTime,
  timestamp: new Date().toISOString()
});

// CORRECT: Correlation ID flows through system
const requestId = req.headers['x-request-id'] || uuid();
req.correlationId = requestId;

// Pass to downstream calls
await orderService.process(order, { correlationId: requestId });

// CORRECT: Redaction middleware
logger.addRedaction([
  process.env.DB_PASSWORD,
  /\d{4}-\d{4}-\d{4}-\d{4}/, // Credit card pattern
  /@\w+\.\w+/, // Email pattern
]);

// CORRECT: Proper error logging with stack trace
try {
  const user = await User.findById(userId);
} catch (error) {
  logger.warn('User not found', {
    userId,
    correlationId: req.correlationId,
    message: error.message
    // Stack trace logged by error handler, not here
  });
}

// CORRECT: Full context on errors
logger.error('Database connection failed', {
  error: error.message, // Not the password!
  code: error.code,
  stack: error.stack,
  correlationId: req.correlationId,
  timestamp: new Date().toISOString(),
  severity: 'critical'
});