Elasticsearch Index Design

Elasticsearch Index Design

Design explicit index mappings from access patterns, review existing mappings for type and storage mistakes, and apply corrections through a new index plus reindex when field types must change.

<!-- begin-partial: preamble -->

Environment Configuration

This skill executes Elasticsearch operations through the elastic CLI. If the elastic CLI is not installed, tell the user what it is needed for. Do not guess credentials, call the HTTP API directly, or attempt other workarounds.

This skill references operations in HTTP-shorthand form (e.g., GET /, GET /_cat/indices, GET /{index}/_mapping, GET /{index}/_settings/index.mode, POST /_query). The Operations table at the end of this document maps each shorthand to the equivalent elastic CLI command — always use the CLI rather than calling the HTTP API directly.

<!-- end-partial: preamble -->

Process

1. Gather access patterns per field. Before choosing types, list how each field is used. For every field capture:

   • Search — full-text match, phrase, relevance scoring?
   • Filter — exact term, terms set, prefix?
   • Aggregate — terms, cardinality, histogram, stats?
   • Sort — ascending/d descending in result sets?
   • Retrieve only — returned in _source but never queried?

   The decision: classify each field into one primary access pattern (search, exact, numeric metric, date, boolean, structured object, or retrieve-only). Missing access-pattern data is a blocker — ask the user rather than guessing. Call GET / to confirm connectivity; when reviewing an existing index, call GET /{index}/_mapping to ground the discussion in the current mapping.

2. Choose field types from access patterns. Map each field to the minimal type set that satisfies its pattern. Read Field Type Decisions and Multi-Field Patterns before proposing mappings.

   Key judgments:

   Pattern	Mapping
   Full-text search only	text (no keyword sub-field)
   Filter / agg / sort only	keyword (not text)
   Full-text search and sort or aggregation	text with fields.keyword multi-field
   Decimal price or metric	double, float, or scaled_float — not text or integer
   Timestamp	date
   True/false flag	boolean
   Free-form key/value map with many distinct keys	flattened — not dynamic object

   Multi-field rule: When a field must be searchable and sortable/aggregatable (e.g. product name), map it as text with a keyword sub-field — search on name, sort and aggregate on name.keyword. Mapping as only text or only keyword is wrong for that combined pattern.

   Explicit mapping rule: For new indices, always define mappings explicitly with PUT /{index}. Do not rely on dynamic mapping for production indices — the first document can lock in wrong types (strings as text, ambiguous numbers as keyword).

   Index settings: Set deliberate number_of_shards and number_of_replicas in the same PUT /{index} request when the deployment allows it (Self-Managed / Elastic Cloud Hosted). On Serverless, omit shard and replica counts (Elastic manages them); still supply explicit mappings. State chosen values or document that defaults apply.

   Example — products index optimized for search plus sort/agg on name:

   {
     "settings": {
       "number_of_shards": 1,
       "number_of_replicas": 1
     },
     "mappings": {
       "properties": {
         "name": {
           "type": "text",
           "fields": {
             "keyword": { "type": "keyword", "ignore_above": 256 }
           }
         },
         "price": { "type": "double" },
         "created": { "type": "date" },
         "in_stock": { "type": "boolean" }
       }
     }
   }

   Create with PUT /products passing the settings and mappings blocks. Verify with GET /products/_mapping.

3. Guard against mapping explosion and storage bloat. On high-volume indices, type mistakes multiply cost. Read Mapping Explosion and Storage Bloat and apply these review checks:

   • Analyzed-but-not-searched fields — Fields used only for filter and aggregation (url, HTTP status_code, tags, IDs) must be keyword, not text. text wastes space; aggregations on text require fielddata or a .keyword sub-field that should not exist if the field is not searched.
   • message.keyword without ignore_above — A keyword sub-field on a large full-text body indexes the entire raw string as one term. Flag this anti-pattern; remove the sub-field when only full-text search is needed, or add ignore_above when a bounded exact-match sub-field is truly required.
   • Dynamic free-form objects — object with "dynamic": true on user-supplied key/value data with thousands of distinct keys causes mapping explosion. Recommend flattened (or strict dynamic / allowlist strategy).
   • doc_values: false — On fields retrieved in hits but never sorted, aggregated, or filtered (e.g. display-only session_id), set "doc_values": false on keyword to save disk at scale.
   • scaled_float — For metrics with bounded precision (e.g. response_time_ms), prefer scaled_float with an appropriate scaling_factor over plain float/double when storage dominates.

   Prefer "dynamic": "strict" on the root mapping unless unknown fields are an explicit requirement.

4. Apply design: create new index and reindex when types change. Elasticsearch cannot change an existing field's type in place. When review finds wrong types (text→keyword, object→flattened, float→scaled_float, doc_values changes on existing fields), state clearly that fixes require a new index and reindex — not a mapping update on the live index.

   Workflow for correcting an existing high-volume index such as events:

   1. Design the corrected mapping on a new index name (e.g. events-v2) incorporating all fixes from steps 2–3.
   2. Create the destination with PUT /events-v2 and the full corrected mappings (and settings where applicable).
   3. Copy documents with POST /_reindex — for large indices use wait_for_completion=false and track the task. Source: { "index": "events" }, destination: { "index": "events-v2" }.
   4. Verify with GET /events-v2/_count (compare to source count) and GET /events-v2/_mapping (confirm types).
   5. Cut over reads and writes (index alias swap or application config) after validation.

   Example corrected excerpt for the events review pattern:

   {
     "mappings": {
       "properties": {
         "@timestamp": { "type": "date" },
         "event_id": { "type": "keyword" },
         "session_id": { "type": "keyword", "doc_values": false },
         "url": { "type": "keyword" },
         "status_code": { "type": "keyword" },
         "response_time_ms": { "type": "scaled_float", "scaling_factor": 100 },
         "tags": { "type": "keyword" },
         "message": { "type": "text" },
         "labels": { "type": "flattened" }
       }
     }
   }

   Do not attempt in-place mapping fixes for these type changes — they are rejected or leave data inconsistent. For greenfield indices, a single PUT /{index} before first ingest avoids reindex entirely.