ES索引的生命周期管理

介紹

? ES可用于索引日志類數據，在此場景下，數據是源源不斷地被寫入到索引中。為了使索引的文檔不會過多，查詢的性能更好，我們希望索引可以根據大小、文檔數量或索引已創建時長等指標進行自動回滾，可以自定義將超過一定時間的數據進行自動刪除。ES為我們提供了索引的生命周期管理來幫助處理此場景下的問題。

? 索引的生命周期分為四個階段：HOT->WARM->COLD->DELETE。

? HOT為必須的階段外，其他為非必須階段，可以任意選擇配置。在日志類場景下不需要WARN和COLD階段，下文只配置了HOT與DELETE階段。

環境

• Elasticsearch: 6.8
• Kibana：6.8

配置過程

配置生命周期管理策略

PUT _ilm/policy/datastream_policy {"policy": { "phases": {"hot": { "actions": {"rollover": { "max_docs" : 1}}},"delete": {"min_age": "30s", "actions": {"delete": {} }}}} }

創建一個名為“datastream_policy“的策略，包含HOT和DELETE階段。

• HOT階段：當日志索引文檔數量超過1條時就會發生rollover。
• DELETE階段：超過30秒時執行刪除。

另外，hot->actions->rollover還支持其他維度的控制，比如：

"max_size": "50GB" "max_age": "30d"

配置索引模板

PUT _template/datastream_template {"index_patterns": ["datastream-*"], "settings": {"number_of_shards": 1,"number_of_replicas": 1,"index.lifecycle.name": "datastream_policy", "index.lifecycle.rollover_alias": "datastream" } }

創建一個索引模版，以datastream-開頭的索引自動采用settings的配置。

• "index.lifecycle.name": "datastream_policy"：使用的策略為“datastream_policy”
• "index.lifecycle.rollover_alias": "datastream"：使用該模版創建的索引統一用”datastream“的別名進行管理。

創建索引

PUT datastream-000001 {"aliases": {"datastream": {"is_write_index": true}} }

創建一個起始索引，設置索引別名，并設置索引別名可寫入。

配置lifecycle檢測時間

PUT /_cluster/settings{"transient": {"indices.lifecycle.poll_interval": "10s" } }

es會根據設置時間定期檢測，默認為十分鐘，為了測試看效果，改為10秒。

驗證

使用索引別名，寫入一條數據

PUT /datastream/_doc {"message":"hello" }

根據lifecycle時間配置，寫入數據10秒后，使用以下命令檢測進度

GET datastream-*/_ilm/explain

自動創建了一個新的索引(datastream-000002)，返回數據：

{"indices" : {"datastream-000002" : {"index" : "datastream-000002","managed" : true,"policy" : "datastream_policy","lifecycle_date_millis" : 1631874381522,"phase" : "new","phase_time_millis" : 1631874381659,"action" : "complete","action_time_millis" : 1631874381659,"step" : "complete","step_time_millis" : 1631874381659},"datastream-000001" : {"index" : "datastream-000001","managed" : true,"policy" : "datastream_policy","lifecycle_date_millis" : 1631874367828,"phase" : "hot","phase_time_millis" : 1631874367975,"action" : "rollover","action_time_millis" : 1631874371733,"step" : "attempt-rollover","step_time_millis" : 1631874381440,"phase_execution" : {"policy" : "datastream_policy","phase_definition" : {"min_age" : "0ms","actions" : {"rollover" : {"max_docs" : 1}}},"version" : 1,"modified_date_in_millis" : 1631874359862}}} }

再等20秒后，舊的索引(datastream-000001)被標記為刪除，查看進度返回數據：

{"indices" : {"datastream-000002" : {"index" : "datastream-000002","managed" : true,"policy" : "datastream_policy","lifecycle_date_millis" : 1631874381522,"phase" : "hot","phase_time_millis" : 1631874381863,"action" : "rollover","action_time_millis" : 1631874391733,"step" : "check-rollover-ready","step_time_millis" : 1631874391733,"phase_execution" : {"policy" : "datastream_policy","phase_definition" : {"min_age" : "0ms","actions" : {"rollover" : {"max_docs" : 1}}},"version" : 1,"modified_date_in_millis" : 1631874359862}},"datastream-000001" : {"index" : "datastream-000001","managed" : true,"policy" : "datastream_policy","lifecycle_date_millis" : 1631874381943,"phase" : "delete","phase_time_millis" : 1631874421434,"action" : "delete","action_time_millis" : 1631874421434,"step" : "wait-for-shard-history-leases","step_time_millis" : 1631874421434,"phase_execution" : {"policy" : "datastream_policy","phase_definition" : {"min_age" : "30s","actions" : {"delete" : { }}},"version" : 1,"modified_date_in_millis" : 1631874359862}}} }

再次等待10秒后，舊的索引已被刪除，查看進度返回數據：

{"indices" : {"datastream-000002" : {"index" : "datastream-000002","managed" : true,"policy" : "datastream_policy","lifecycle_date_millis" : 1631874381522,"phase" : "hot","phase_time_millis" : 1631874381863,"action" : "rollover","action_time_millis" : 1631874391733,"step" : "check-rollover-ready","step_time_millis" : 1631874391733,"phase_execution" : {"policy" : "datastream_policy","phase_definition" : {"min_age" : "0ms","actions" : {"rollover" : {"max_docs" : 1}}},"version" : 1,"modified_date_in_millis" : 1631874359862}}} }

其他

如果一個索引的生命周期管理已運行完成，那么對應策略的修改不會重新觸發索引生命周期管理的重新運行。 若想更新后的策略在已完成生命周期管理的索引上生效，需要單獨的命令干預。如下：

• 通過將索引生命周期的運行狀態調整到修改階段之前，以下兩個示例都將狀態調整HOT階段。（官網：https://www.elastic.co/guide/en/elasticsearch/reference/6.8/ilm-move-to-step.html）

  # 調整到hot階段-回滾之前 POST _ilm/move/datastream-000001 {"current_step": { "phase": "completed","action": "completed","name": "completed"},"next_step": { "phase": "hot","action": "rollover","name": "check-rollover-ready"} } # 調整到hot階段-回滾之后 POST _ilm/move/datastream-000001 {"current_step": { "phase": "completed","action": "completed","name": "completed"},"next_step": { "phase": "hot","action": "complete","name": "complete"} }

索引的生命周期管理中，利用到了索引別名，新建的索引會被自動關聯到既定的別名上。向索引中寫入文檔和從索引中查詢文檔，都通過別名指定，所以保證了整個過程對于客戶端透明。

參考：

https://www.elastic.co/guide/en/elasticsearch/reference/6.8/index-lifecycle-management.html

https://www.cnblogs.com/runnerjack/p/12758658.html

總結

以上是生活随笔為你收集整理的ES索引的生命周期管理的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。