Elk on Docker (-Compose)

The ELK/Elastic stack is a common open source solution for collecting and analyzing log data from distributed systems. This article will show you how to run an ELK on Docker using Docker Compose. This will enable you to run ELK distributed on your docker infrastructure or test it on your local system. Weiterlesen

Getting started with Kibana [Links]

You have huge data sets to analyze? You want to gain insights into your gigabytes of logs? The Elastic Stack (Elasticsearch, Logstash, Beats, Kibana) offers you a great set of tools for that. After you got your logs or other data into Elasticsearch, Kibana will offer you a great UI to deep dive into your data. But how to get started with Kibana? Weiterlesen

Drastic Elastic [Part 4]: Aggregations & Plugins

In an earlier post in this mini-series I mentioned that the aggregated data we persist in ELasticSearch has discrete retention times:

  • 5 minute aggregation => (retention time of) one day
  • hourly aggregations => 7 days
  • daily aggregations => 5 years

This means that we reach well over 50% of our total data retention after one week (the only additions after that point are daily aggregations while data at other aggregation levels gets refreshed/updated) – after 4 or 5 weeks we had something like 8 billion documents in ElasticSearch amounting to 13 TB of data.

In this last article of our four part series we describe how ElasticSearch plugins help us to address appropriate aggregation levels without having to build in extra round trips (adding to latency), or to fetch more data than we need (which would require filtering in the client). Weiterlesen