elasticsearch-pro-dev

Overview

Elasticsearch is a distributed search and analytics engine: JSON documents indexed into shards, queried with a rich DSL, aggregated at speed. It powers full-text search, log analytics, and metrics — workloads where relational databases struggle. It's also a distributed system with real operational demands and a query language that rewards precision.

The two disciplines are index design (mappings, analyzers, sharding) and query design (relevance, filters, aggregations). Get the mapping wrong and you're reindexing; get the cluster wrong and you're firefighting. This skill covers both, plus production operations.

When to use

• Adding full-text search to an application.
• Designing indexes, mappings, and analyzers.
• Writing relevance-tuned queries and aggregations.
• Choosing between Elasticsearch, OpenSearch, Postgres FTS, or Algolia/Typesense.
• Operating clusters (sharding, replicas, upgrades, snapshots).
• Debugging slow queries, shard problems, or mapping explosions.
• Building log/metrics analytics on the Elastic stack.

Core concepts

• Indexes, shards, replicas. An index splits into primary shards (parallelism unit) distributed across nodes; replicas copy shards for HA and read scale. Shard count is set at creation — oversharding wastes resources, undersharding caps throughput. Size shards ~10-50GB.
• Mappings. Field types (text vs keyword vs numeric vs date) determine what's searchable and how. text for full-text search (analyzed), keyword for sorting/aggregations/exact match. Multi-fields (text + keyword sub-field) give both.
• Dynamic mapping dangers. Auto-detected mappings guess types — sometimes wrong (dates as text, numbers as strings). A new field per document (mapping explosion) can destabilize the cluster. Prefer explicit mappings with dynamic: strict or runtime for unknowns.
• Analyzers. Character filters → tokenizer → token filters turn text into searchable tokens. Language analyzers (stemming, stop words) dramatically affect relevance; custom analyzers for special content (part numbers, code).
• Query DSL: queries vs filters. must/should affect relevance scoring; filter clauses are yes/no (cacheable, faster). Structure searches as: filters for constraints, queries for relevance-ranked text.
• Relevance tuning. bool combinations, multi_match types (best_fields, most_fields, cross_fields), boosting, function_score for business rules (recency, popularity). Tune with real queries and judgments, not guesses.
• Aggregations. Bucket (terms, date_histogram, ranges) and metric (avg, percentiles, cardinality) aggs for analytics. They're powerful and expensive — scope them with filters, limit bucket counts, and watch memory.
• Pagination. from/size for shallow pages; search_after (with a tiebreaker sort) for deep pagination; Scroll API for bulk export. Deep from is expensive — it re-executes across shards.
• Index lifecycle (ILM). Hot → warm → cold → delete policies automate rollover, shrink, and deletion — essential for time-series data (logs, metrics). Rollover by size/age, not by calendar alone.
• Reindexing. Mappings can't change in place (mostly) — reindex into a new index, then alias-swap. Aliases decouple index names from the physical index; always point reads at aliases.
• Snapshots. S3/GCS repository snapshots for backup and recovery — the only supported backup. Test restores; snapshot before upgrades.
• Cluster health. Green/yellow/red refer to shard allocation, not performance. Yellow (missing replicas) is degraded; red (missing primaries) is data-loss territory. Monitor pending tasks and relocation storms too.
• Circuit breakers and memory. Fielddata and large aggregations can OOM nodes; circuit breakers protect the cluster but trip on legitimate heavy queries — size heaps (≤50% RAM, ≤~30GB) and keep aggregations bounded.
• Security. TLS, authentication, role-based index privileges, API keys per application. Never expose Elasticsearch directly to the internet — it's a ransomware magnet.

Practical workflow

1. Design the mapping first. Explicit types per field; text + keyword multi-fields where both search and sort/agg are needed; dynamic: strict to catch surprises.

   { "mappings": { "dynamic": "strict",
     "properties": {
       "title": { "type": "text", "fields": { "keyword": { "type": "keyword" } } },
       "status": { "type": "keyword" },
       "price": { "type": "scaled_float", "scaling_factor": 100 },
       "created_at": { "type": "date" }
     } } }
   

2. Choose analyzers deliberately. Default is fine for English prose; language analyzers per language field; custom analyzers for SKUs/codes (keyword tokenizer + lowercase).

3. Index through aliases. Write to products-write, read from products-read; reindex-and-swap for mapping changes with zero downtime.

4. Structure queries: filters + relevance. Filters for constraints (status, category, price range), multi_match for the text, boosts for business priority.

   { "query": { "bool": {
     "must": { "multi_match": { "query": "wireless headphones", "fields": ["title^3", "description"] } },
     "filter": [ { "term": { "status": "active" } }, { "range": { "price": { "lte": 200 } } } ]
   } } }
   

5. Paginate with search_after. Sort by _score + _id tiebreaker (or timestamp + id); pass the last hit's sort values. Reserve from/size for shallow admin pages.

6. Aggregate carefully. Filter before aggregating; cap terms size; use composite aggs for paginating through buckets; watch the fielddata circuit breaker.

7. Manage lifecycle with ILM. Rollover hot indices by size/age; move to warm (fewer replicas, force-merge), then delete. Automate from day one for time-series data.

8. Operate. Snapshot to object storage on a schedule; monitor shard balance, JVM heap, GC, indexing rate, and query latency; upgrade rolling with compatibility checks.

   PUT _index_template/logs-template
   { "index_patterns": ["logs-*"],
     "template": { "settings": { "number_of_shards": 3 },
       "mappings": { "dynamic": "strict", "properties": {
         "@timestamp": { "type": "date" }, "level": { "type": "keyword" } } } } }
   

Common pitfalls

• Dynamic mapping explosions — a field per document variant destabilizing the cluster; explicit mappings with strict dynamic.
• Wrong field types — text where keyword was needed for sorting/aggs (or vice versa); multi-fields from the start.
• Deep from/size pagination — expensive shard fan-out; use search_after.
• Unbounded aggregations — millions of buckets OOMing nodes; filter first, cap sizes, use composite.
• Oversharding — hundreds of tiny shards wasting heap and slowing everything; aim for 10-50GB per shard.
• Changing mappings in place — impossible for most changes; reindex + alias swap is the path.
• No snapshots — no supported backup; snapshot on a schedule and test restores.
• Exposed to the internet — unauthenticated clusters get ransomed; TLS + auth + network policy.
• Ignoring ILM — time-series indices growing forever; automate rollover and deletion.
• Relevance guessed, not measured — boosting without judgment lists; test with real queries.
• Heap misconfiguration — over 50% RAM or over ~30GB causing GC pain; follow the sizing rules.
• Yellow cluster ignored — missing replicas treated as fine until a node dies; yellow is degraded, fix it.