App Insights Instrumentation

Implement high-value telemetry with minimal noise and strong trace correlation for production troubleshooting and performance optimization.

Use This Skill When

• The user needs monitoring for apps running in Azure
• The user needs distributed tracing across microservices
• The user needs alerting, dashboards, or root cause analysis visibility
• The user wants to migrate from custom logging to structured telemetry

Context: Observability Maturity

Immature: No monitoring, errors discovered by users
Developing: Logs and metrics collected, no correlation or alerting
Managed: Traces linked across services, alerts firing, dashboards useful → Target
Optimized: Real-time anomaly detection, SLO tracking, automated runbooks

Required Inputs

• Application stack: Node.js, Python, .NET, Java, Go, etc.
• Deployment target: App Service, Container Apps, AKS, Function, VM
• Critical user journeys: Signup → Confirmation, Payment → Order, Search → Results
• SLO targets: Availability (%), latency (ms p99), error budget
• Compliance needs: PII filtering, data residency, retention policy
• Existing telemetry: Custom logging, APM, or greenfield?

Decision Tree

Is this a new application or existing?
├─ New → Start with OpenTelemetry SDK from day one
└─ Existing → Assess current logging, add instrumentation incrementally

Which language?
├─ .NET → Application Insights .NET SDK (auto-instrumentation or manual)
├─ Node.js → OpenTelemetry + @opentelemetry/sdk-node
├─ Python → OpenTelemetry + opentelemetry-sdk
├─ Java → OpenTelemetry agent (zero-code) or manual SDK
└─ Other → OpenTelemetry language-specific SDK

How many services (microservices)?
├─ Single → Simple Application Insights setup, no special tracing
├─ 2-5 services → OpenTelemetry with correlation IDs across calls
└─ 5+ services → Distributed trace propagation (W3C Trace Context)

What's the expected volume?
├─ < 1M events/day → 100% sampling acceptable
├─ 1-10M → 10-25% sampling with head-based decisions
└─ > 10M → 1-5% sampling with tail-based (post-request) sampling decisions

Workflow

Phase 1: Setup Application Insights Workspace

1. Create Application Insights resource:

   az monitor app-insights component create \
     --resource-group $RG \
     --app-name "ai-${app}-${env}" \
     --location eastus
   
   # Get instrumentation key
   IKEY=$(az monitor app-insights component show --resource-group $RG --app-name "ai-${app}-${env}" --query "instrumentationKey")
   

2. Create Log Analytics workspace (for KQL queries):

   az monitor log-analytics workspace create \
     --resource-group $RG \
     --workspace-name "law-${app}-${env}"
   
   # Link to Application Insights
   az monitor app-insights component update --resource-group $RG --app-name "ai-${app}-${env}" \
     --workspace "/subscriptions/$SUBID/resourceGroups/$RG/providers/Microsoft.OperationalInsights/workspaces/law-${app}-${env}"
   

3. Configure retention policy:

   • Default: 30 days free
   • Increase if needed (paid): 90 days, 1 year
   • Archive to Storage: 7+ years for compliance

Phase 2: Add SDK to Application

Example 1: .NET

# Install NuGet packages
dotnet add package Microsoft.ApplicationInsights.AspNetCore
dotnet add package Microsoft.ApplicationInsights.DependencyCollector

// In Program.cs
builder.Services.AddApplicationInsightsTelemetry();

// Instrumentation key from Key Vault
var ikeySecret = new DefaultAzureCredential();
var kvUri = new Uri($"https://{kvName}.vault.azure.net");
var kvClient = new SecretClient(kvUri, ikeySecret);
var ikeySecret = kvClient.GetSecret("APPINSIGHTS_INSTRUMENTATION_KEY");

Example 2: Node.js with OpenTelemetry

npm install @opentelemetry/api @opentelemetry/sdk-node \
  @opentelemetry/auto-instrumentations-node \
  @opentelemetry/exporter-trace-otlp-http @opentelemetry/sdk-trace-node

// instrumentation.js (first require in entry point)
const { NodeTracerProvider } = require("@opentelemetry/sdk-node");
const { getNodeAutoInstrumentations } = require("@opentelemetry/auto-instrumentations-node");
const { OTLPTraceExporter } = require("@opentelemetry/exporter-trace-otlp-http");

const traceExporter = new OTLPTraceExporter({
  url: `https://${process.env.APPLICATIONINSIGHTS_ENDPOINT_URL}/v2.1/track`
});

const sdk = new NodeSDK({
  traceExporter,
  instrumentations: [getNodeAutoInstrumentations()]
});
sdk.start();

Example 3: Python

pip install opentelemetry-api opentelemetry-sdk \
  opentelemetry-exporter-azure-monitor \
  opentelemetry-instrumentation-flask opentelemetry-instrumentation-requests

from azure.monitor.opentelemetry import configure_azure_monitor

configure_azure_monitor(
    connection_string=os.environ.get("APPLICATIONINSIGHTS_CONNECTION_STRING")
)

Phase 3: Add Custom Telemetry

1. Track custom events (business logic):

   // .NET
   telemetryClient.TrackEvent("UserSignup", new Dictionary<string, string> {
     { "signup_source", "mobile_app" },
     { "user_tier", "premium" }
   });
   

   // Node.js
   const tracer = trace.getTracer("app");
   const span = tracer.startSpan("UserSignup", {
     attributes: { "signup_source": "mobile_app", "user_tier": "premium" }
   });
   

2. Track custom metrics:

   // .NET: Track duration
   var stopwatch = System.Diagnostics.Stopwatch.StartNew();
   ProcessOrder(order);
   stopwatch.Stop();
   telemetryClient.GetMetric("ProcessOrderDuration").TrackValue(stopwatch.ElapsedMilliseconds);
   

3. Add correlation context:

   // Automatic: Activity.Current.Id correlates across services
   // Manual: Add request ID to headers
   using (var activity = new System.Diagnostics.Activity("ProcessPayment").Start()) {
     activity.AddTag("order_id", orderId);
     CallDownstreamAPI(); // Downstream receives trace context in headers
   }
   

Phase 4: Configure Sampling

1. Adaptive sampling (default):

   Automatically reduces sample rate when volume spikes
   - Default: 25 events/sec threshold
   - Above: Reduces to 5% sampling
   - Below: Continues 100% sampling
   

   services.Configure<ApplicationInsightsServiceOptions>(options => {
     options.SamplingSettings = new SamplingSettings {
       MaxTelemetryItemsPerSecond = 20,
       EvaluationInterval = TimeSpan.FromMinutes(1),
       SampledItemCount = 5
     };
   });
   

2. Fixed sampling (simpler, for testing):

   Always sample 10% of requests
   Useful for known volume or testing
   

3. Tail-based sampling (advanced):

   • Sample entire traces based on errors, latency
   • Only available in Open Telemetry Collector (not direct AppInsights)

Phase 5: Create Dashboards & Alerts

1. Business KPI dashboard:

   // Failed signups in last hour
   customEvents 
   | where name == "SignupFailed" 
   | where timestamp > ago(1h)
   | summarize count() by tostring(customDimensions.failure_reason)
   
   // Average order processing time
   customMetrics
   | where name == "ProcessOrderDuration"
   | summarize avg(value) by bin(timestamp, 5m)
   

2. Performance dashboard:

   // Request latency (p50, p95, p99)
   requests
   | summarize p50=percentile(duration,50), p95=percentile(duration,95), p99=percentile(duration,99) by bin(timestamp, 1m)
   
   // Dependency failures
   dependencies
   | where success == false
   | summarize count() by target, type, resultCode
   

3. Alert rules (trigger on SLA breach):

   Condition: Error rate > 5% in 5 minutes
   → Severity 2 (warning)
   → Notify: on-call via email, PagerDuty
   
   Condition: Response time p99 > 1000ms in 10 minutes
   → Severity 3 (info)
   → Notify: #alerts Slack channel
   
   Condition: Any database connectivity failures
   → Severity 1 (critical)
   → Immediate escalation to DBA
   

Phase 6: Correlation & Troubleshooting

1. Follow request through services:

   User → API Gateway (trace-id: abc123)
        → Auth Service (inherits trace-id: abc123)
        → Database (inherits trace-id: abc123)
   
   All logs linked via same trace-id; one query finds entire flow
   

2. Root cause analysis query:

   let trace_id = "abc123";
   union
     (requests | where operation_Id == trace_id),
     (dependencies | where operation_Id == trace_id),
     (exceptions | where operation_Id == trace_id),
     (traces | where operation_Id == trace_id),
     (customEvents | where tostring(customDimensions.trace_id) == trace_id)
   | project timestamp, type, message, target, success
   | order by timestamp asc
   

Output Contract

1. Instrumentation Plan

   • SDK/agent selected and justification
   • Sampling strategy (fixed %, adaptive, tail-based)
   • Correlation approach (trace context headers)

2. Implementation Checklist

   • Packages/SDKs to install
   • Code changes (SDK initialization, custom events)
   • Environment variables or Key Vault secrets

3. Telemetry Schema

   • Custom events (name, dimensions, metrics)
   • Business KPIs to track
   • Naming conventions (snake_case, dimensions)

4. Dashboard & Alert Baseline

   • Business KPI dashboard (queries provided)
   • Performance dashboard (queries provided)
   • Alert rules (thresholds, severity, channels)

5. Validation Steps

   • Send test event, verify in portal
   • Trace sample request end-to-end
   • Verify alerts fire on synthetic failures

Guardrails

• Never log secrets, PII, or credentials: Filter sensitive data before sending.
• Use structured logging: JSON/structured dimensions, not free text.
• Avoid high-cardinality dimensions: Don't use user ID or request ID as dimension (too many unique values = high cost).
• Set retention policy: Default 30 days; archive old data to storage for compliance.
• Sample intelligently: 100% at <1M events/day; reduce sampling as volume grows.
• Correlate traces across services: Use W3C Trace Context headers to link requests.
• Monitor the monitor: Ensure Application Insights is receiving data; check quota.
• Use alerts for SLA breaches: Don't rely on dashboards; alerts notify on-call team.