Drastic Elastic [Part 4]: Aggregations & Plugins

In an earlier post in this mini-series I mentioned that the aggregated data we persist in ELasticSearch has discrete retention times:

  • 5 minute aggregation => (retention time of) one day
  • hourly aggregations => 7 days
  • daily aggregations => 5 years

This means that we reach well over 50% of our total data retention after one week (the only additions after that point are daily aggregations while data at other aggregation levels gets refreshed/updated) – after 4 or 5 weeks we had something like 8 billion documents in ElasticSearch amounting to 13 TB of data.

In this last article of our four part series we describe how ElasticSearch plugins help us to address appropriate aggregation levels without having to build in extra round trips (adding to latency), or to fetch more data than we need (which would require filtering in the client). Weiterlesen