Build Grafana Dashboards
Design and deploy Grafana dashboards with best practices for maintainability, reusability, and version control.
When to Use
- Creating visual representations of Prometheus, Loki, or other data source metrics
- Building operational dashboards for SRE teams and incident responders
- Establishing executive-level reporting dashboards for SLO compliance
- Migrating dashboards from manual creation to version-controlled provisioning
- Standardizing dashboard layouts across teams with template variables
- Creating drill-down experiences from high-level overviews to detailed metrics
Inputs
- Required: Data source configuration (Prometheus, Loki, Tempo, etc.)
- Required: Metrics or logs to visualize with their query patterns
- Optional: Template variables for multi-service or multi-environment views
- Optional: Existing dashboard JSON for migration or modification
- Optional: Annotation queries for event correlation (deployments, incidents)
Procedure
See Extended Examples for complete configuration files and templates.
Step 1: Design Dashboard Structure
Plan dashboard layout and organization before building panels.
Create a dashboard specification document:
# Service Overview Dashboard
## Purpose
Real-time operational view for on-call engineers monitoring the API service.
## Rows
1. High-Level Metrics (collapsed by default)
- Request rate, error rate, latency (RED metrics)
- Service uptime, instance count
2. Detailed Metrics (expanded by default)
- Per-endpoint latency breakdown
- Error rate by status code
- Database connection pool status
3. Resource Utilization
- CPU, memory, disk usage per instance
- Network I/O rates
4. Logs (collapsed by default)
- Recent errors from Loki
- Alert firing history
## Variables
- `environment`: production, staging, development
- `instance`: all instances or specific instance selection
- `interval`: aggregation window (5m, 15m, 1h)
## Annotations
- Deployment events from CI/CD system
- Alert firing/resolving events
Key design principles:
- Most important metrics first: Critical metrics at the top, details below
- Consistent time ranges: Synchronize time across all panels
- Drill-down paths: Link from high-level to detailed dashboards
- Responsive layout: Use rows and panel widths that work on various screens
Expected: Clear dashboard structure documented, stakeholders aligned on metrics and layout priorities.
On failure:
- Conduct dashboard design review with end users (SREs, developers)
- Benchmark against industry standards (USE method, RED method, Four Golden Signals)
- Review existing dashboards in team for consistency patterns
Step 2: Create Dashboard with Template Variables
Build the dashboard foundation with reusable variables for filtering.
Create dashboard JSON structure (or use UI, then export):
{
"dashboard": {
"title": "API Service Overview",
"uid": "api-service-overview",
"version": 1,
"timezone": "browser",
"editable": true,
"graphTooltip": 1,
"time": {
"from": "now-6h",
"to": "now"
},
"refresh": "30s",
"templating": {
"list": [
{
"name": "environment",
"type": "query",
"datasource": "Prometheus",
"query": "label_values(up{job=\"api-service\"}, environment)",
"multi": false,
"includeAll": false,
"refresh": 1,
"sort": 1,
"current": {
"selected": false,
"text": "production",
"value": "production"
}
},
{
"name": "instance",
"type": "query",
"datasource": "Prometheus",
"query": "label_values(up{job=\"api-service\",environment=\"$environment\"}, instance)",
"multi": true,
"includeAll": true,
"refresh": 1,
"allValue": ".*",
"current": {
"selected": true,
"text": "All",
"value": "$__all"
}
},
{
"name": "interval",
"type": "interval",
"options": [
{"text": "1m", "value": "1m"},
{"text": "5m", "value": "5m"},
{"text": "15m", "value": "15m"},
{"text": "1h", "value": "1h"}
],
"current": {
"text": "5m",
"value": "5m"
},
"auto": false
}
]
},
"annotations": {
"list": [
{
"name": "Deployments",
"datasource": "Prometheus",
"enable": true,
"expr": "changes(app_version{job=\"api-service\",environment=\"$environment\"}[5m]) > 0",
"step": "60s",
"iconColor": "rgba(0, 211, 255, 1)",
"tagKeys": "version"
}
]
}
}
}
Variable types and use cases:
- Query variables: Dynamic lists from data source (
label_values(), query_result())
- Interval variables: Aggregation windows for queries
- Custom variables: Static lists for non-metric selections
- Constant variables: Shared values across panels (data source names, thresholds)
- Text box variables: Free-form input for filtering
Expected: Variables populate correctly from data source, cascading filters work (environment filters instances), default selections appropriate.
On failure:
- Test variable queries independently in Prometheus UI
- Check for circular dependencies (variable A depends on B depends on A)
- Verify regex patterns in
allValue field for multi-select variables
- Review variable refresh settings (on dashboard load vs on time range change)
Step 3: Build Visualization Panels
Create panels for each metric with appropriate visualization types.
Time series panel (request rate):
{
"type": "timeseries",
"title": "Request Rate",
"gridPos": {"h": 8, "w": 12, "x": 0, "y": 0},
"targets": [
{
"expr": "sum(rate(http_requests_total{job=\"api-service\",environment=\"$environment\",instance=~\"$instance\"}[$interval])) by (method)",
"legendFormat": "{{method}}",
"refId": "A"
}
],
"fieldConfig": {
"defaults": {
"unit": "reqps",
"color": {
"mode": "palette-classic"
},
"custom": {
"drawStyle": "line",
"lineInterpolation": "smooth",
"fillOpacity": 10,
"spanNulls": true
},
"thresholds": {
"mode": "absolute",
"steps": [
{"value": null, "color": "green"},
{"value": 1000, "color": "yellow"},
{"value": 5000, "color": "red"}
]
}
}
},
"options": {
"tooltip": {
"mode": "multi",
"sort": "desc"
},
"legend": {
"displayMode": "table",
"placement": "right",
"calcs": ["mean", "max", "last"]
}
}
}
Stat panel (error rate):
{
"type": "stat",
"title": "Error Rate",
"gridPos": {"h": 4, "w": 6, "x": 12, "y": 0},
"targets": [
{
# ... (see EXAMPLES.md for complete configuration)
Heatmap panel (latency distribution):
{
"type": "heatmap",
"title": "Request Duration Heatmap",
"gridPos": {"h": 8, "w": 12, "x": 0, "y": 8},
"targets": [
{
# ... (see EXAMPLES.md for complete configuration)
Panel selection guide:
- Time series: Trends over time (rates, counts, durations)
- Stat: Single current value with threshold coloring
- Gauge: Percentage values (CPU, memory, disk usage)
- Bar gauge: Comparing multiple values at a point in time
- Heatmap: Distribution of values over time (latency percentiles)
- Table: Detailed breakdown of multiple metrics
- Logs: Raw log lines from Loki with filtering
Expected: Panels render correctly with data, visualizations match intended metric types, legends descriptive, thresholds highlight problems.
On failure:
- Test queries in Explore view with same time range and variables
- Check for metric name typos or incorrect label filters
- Verify aggregation functions match metric type (rate for counters, avg for gauges)
- Review unit configurations (bytes, seconds, requests per second)
- Enable "Show query inspector" to debug empty results
Step 4: Configure Rows and Layout
Organize panels into collapsible rows for logical grouping.
{
"panels": [
{
"type": "row",
"title": "High-Level Metrics",
"collapsed": false,
# ... (see EXAMPLES.md for complete configuration)
Layout best practices:
- Grid is 24 units wide, each panel specifies
w (width) and h (height)
- Use rows to group related panels, collapse less critical sections by default
- Place most critical metrics in first visible area (y=0-8)
- Maintain consistent panel heights within rows (typically 4, 8, or 12 units)
- Use full width (24) for time series, half width (12) for comparisons
Expected: Dashboard layout organized logically, rows collapse/expand correctly, panels align visually without gaps.
On failure:
- Validate gridPos coordinates don't overlap
- Check that row panels array contains panels (not null)
- Verify y-coordinates increment logically down the page
- Use Grafana UI "Edit JSON" to inspect grid positions
Step 5: Add Links and Drill-Downs
Create navigation paths between related dashboards.
Dashboard-level links in JSON:
{
"links": [
{
"title": "Service Details",
"type": "link",
"icon": "external link",
# ... (see EXAMPLES.md for complete configuration)
Panel-level data links:
{
"fieldConfig": {
"defaults": {
"links": [
{
"title": "View Logs for ${__field.labels.instance}",
# ... (see EXAMPLES.md for complete configuration)
Link variables:
$service, $environment: Dashboard template variables${__field.labels.instance}: Label value from clicked data point${__from}, ${__to}: Current dashboard time range$__url_time_range: Encoded time range for URL
Expected: Clicking panel elements or dashboard links navigates to related views with context preserved (time range, variables).
On failure:
- URL encode special characters in query parameters
- Test links with various variable selections (All vs specific value)
- Verify target dashboard UIDs exist and are accessible
- Check that
includeVars and keepTime flags work as expected
Step 6: Set Up Dashboard Provisioning
Version control dashboards as code for reproducible deployments.
Create provisioning directory structure:
mkdir -p /etc/grafana/provisioning/{dashboards,datasources}
Datasource provisioning (/etc/grafana/provisioning/datasources/prometheus.yml):
apiVersion: 1
datasources:
- name: Prometheus
type: prometheus
access: proxy
# ... (see EXAMPLES.md for complete configuration)
Dashboard provisioning (/etc/grafana/provisioning/dashboards/default.yml):
apiVersion: 1
providers:
- name: 'default'
orgId: 1
folder: 'Services'
type: file
disableDeletion: false
updateIntervalSeconds: 30
allowUiUpdates: true
options:
path: /var/lib/grafana/dashboards
foldersFromFilesStructure: true
Store dashboard JSON files in /var/lib/grafana/dashboards/:
/var/lib/grafana/dashboards/
├── api-service/
│ ├── overview.json
│ └── details.json
├── database/
│ └── postgres.json
└── infrastructure/
├── nodes.json
└── kubernetes.json
Using Docker Compose:
version: '3.8'
services:
grafana:
image: grafana/grafana:10.2.0
ports:
- "3000:3000"
volumes:
- ./grafana/provisioning:/etc/grafana/provisioning
- ./grafana/dashboards:/var/lib/grafana/dashboards
environment:
- GF_SECURITY_ADMIN_PASSWORD=admin
- GF_USERS_ALLOW_SIGN_UP=false
- GF_AUTH_ANONYMOUS_ENABLED=true
- GF_AUTH_ANONYMOUS_ORG_ROLE=Viewer
Expected: Dashboards automatically loaded on Grafana startup, changes to JSON files reflected after update interval, version control tracks dashboard changes.
On failure:
- Check Grafana logs:
docker logs grafana | grep -i provisioning
- Verify JSON syntax:
python -m json.tool dashboard.json
- Ensure file permissions allow Grafana to read:
chmod 644 *.json
- Test with
allowUiUpdates: false to prevent UI modifications
- Validate provisioning config:
curl http://localhost:3000/api/admin/provisioning/dashboards/reload -X POST -H "Authorization: Bearer $GRAFANA_API_KEY"
Validation
Common Pitfalls
- Variable not updating panels: Ensure queries use
$variable syntax, not hardcoded values. Check variable refresh settings.
- Empty panels with correct query: Verify time range includes data points. Check scrape interval vs aggregation window (5m rate needs >5m of data).
- Legend too verbose: Use
legendFormat to show only relevant labels, not full metric name. Example: {{method}} - {{status}} instead of default.
- Inconsistent time ranges: Set dashboard time sync so all panels share the same time window. Use "Sync cursor" for correlated investigation.
- Performance issues: Avoid queries returning high cardinality series (>1000). Use recording rules or pre-aggregation. Limit time ranges for expensive queries.
- Dashboard drift: Without provisioning, manual UI changes create version control conflicts. Use
allowUiUpdates: false in production.
- Missing data links: Data links require exact label names. Use
${__field.labels.labelname} carefully, verify label exists in query result.
- Annotation overload: Too many annotations clutter the view. Filter annotations by importance or use separate annotation tracks.
Related Skills
setup-prometheus-monitoring - Configure Prometheus data sources that feed Grafana dashboardsconfigure-log-aggregation - Set up Loki for log panel queries and log-based annotationsdefine-slo-sli-sla - Visualize SLO compliance and error budgets with Grafana stat and gauge panelsinstrument-distributed-tracing - Add trace ID links from metrics panels to Tempo trace views