Elasticsearch Reindex

Elasticsearch Reindex

Copy documents from source indices or data streams to a destination using POST /_reindex. An expert reindex workflow prepares the destination explicitly, chooses local versus remote execution, filters at the source when only a subset is needed, runs long copies asynchronously, tracks the task to completion, and verifies the destination document count before reporting results.

<!-- begin-partial: preamble -->

Environment Configuration

This skill executes Elasticsearch operations through the elastic CLI. If the elastic CLI is not installed, tell the user what it is needed for. Do not guess credentials, call the HTTP API directly, or attempt other workarounds.

This skill references operations in HTTP-shorthand form (e.g., GET /, GET /_cat/indices, GET /{index}/_mapping, GET /{index}/_settings/index.mode, POST /_query). The Operations table at the end of this document maps each shorthand to the equivalent elastic CLI command — always use the CLI rather than calling the HTTP API directly.

<!-- end-partial: preamble -->

Process

1. Confirm connectivity and deployment type. Call GET /. Read build_flavor and version.number to know whether shard, replica, and cluster-settings APIs are available (Serverless manages shards/replicas internally and blocks most _cluster/* APIs). The decision: continue only when the cluster is reachable. If the call fails, stop — do not guess endpoints or credentials.

2. Decide local versus remote reindex. Compare where the source and destination live.

   • Same cluster — use local reindex: source.index and dest.index only. Do not add source.remote when both indices are on the cluster you are connected to.
   • Different cluster — use reindex from remote: add source.remote with the remote cluster URL and credentials. Remote reindex does not support slicing; compensate with query-based partitioning (date ranges, term filters) across parallel requests. Confirm the remote host is allowlisted on Self-Managed / ECH (reindex.remote.whitelist in cluster config); Serverless manages allowlisting internally (ECH remotes only, Tech Preview).

   Data needed: source index name(s), destination index name, and whether they share a cluster.

3. Inspect the source — never guess field names or counts. Call GET /{source}/_mapping to ground field names and types. Call GET /{source}/_count (or GET /_cat/count/{source}?h=count on Self-Managed / ECH) to learn how many documents exist.

   The decision: full copy versus filtered subset.

   • Full copy — omit source.query (match-all behavior).
   • Filtered subset — add source.query with Query DSL. For time ranges, use a range filter on the timestamp field (commonly @timestamp), e.g. "gte": "2025-01-01", "lt": "2025-02-01" for January 2025. Do not run a full-index copy when the user asked for a date range or other filter.

   Data needed: the user's filter criteria and the mapping-confirmed field names.

4. Prepare the destination index before copying. _reindex does not copy mappings, shard counts, or analyzers. Create the destination with explicit settings and mappings derived from the source mapping via PUT /{dest}.

   • On Self-Managed / ECH: set number_of_replicas: 0 and refresh_interval: "-1" on the destination during the copy for write throughput; restore production values afterward with PUT /{dest}/_settings.
   • On Serverless: omit number_of_shards and number_of_replicas (managed by Elastic); you may set refresh_interval: "-1" during the copy.
   • For data stream destinations: ensure an index template with data_stream: {} exists, create the data stream, and set dest.op_type to "create" (append-only).

   The decision: create/prepare the target rather than relying on auto-creation with dynamic mapping. Wrong or missing mappings cause partial failures or silent type coercion.

   Data needed: destination name, corrected or compatible mappings, and deployment-specific settings constraints.

5. Build and submit the reindex request. Call POST /_reindex?wait_for_completion=false for any copy that may take more than a few seconds or when the user says the index is large — the response returns a task id immediately instead of blocking.

   Request body essentials:

   • source.index — source index or data stream (correct name, not reversed with dest.index).
   • dest.index — prepared destination from step 4.
   • source.query — include only when step 3 chose a filtered subset.
   • conflicts: "proceed" — when retrying a partially complete reindex.
   • Optional tuning: source.size (batch size), requests_per_second (throttle), slices=auto on local reindex only (parallelize per primary shard — never for remote), scroll (increase keep-alive on slow clusters), max_docs (test runs), script (transform), dest.pipeline (ingest enrichment).

   Example filtered subset (January 2025 only):

   {
     "source": {
       "index": "eval-reindex-src",
       "query": {
         "range": {
           "@timestamp": { "gte": "2025-01-01", "lt": "2025-02-01" }
         }
       }
     },
     "dest": { "index": "eval-reindex-jan" }
   }

   Do not reach for _split, _shrink, or snapshot/restore when the task is a filtered subset copy or a straight document migration — those APIs solve different problems.

6. Track the task to completion. Store the task id from the reindex response. Poll GET /_tasks/{task_id} until completed is true. Read status.total, status.created, and response.failures. On Self-Managed / ECH you may also list active reindex tasks with GET /_tasks?actions=*reindex&detailed; on Serverless, query by task id only (list/cancel are not available). Adjust throttling mid-flight with POST /_reindex/{task_id}/_rethrottle?requests_per_second=N without canceling.

7. Verify and report the destination count. Call GET /{dest}/_count (works on all deployment types). On Self-Managed / ECH you may also use GET /_cat/count/{dest}?h=count. Compare source filter expectations to the destination count. Report the exact count from the destination — do not estimate or guess.

   After a successful full copy, restore production settings on the destination with PUT /{dest}/_settings (replicas and refresh interval on Self-Managed / ECH; refresh interval only on Serverless).