{"id":20994,"date":"2015-09-21T11:35:22","date_gmt":"2015-09-21T10:35:22","guid":{"rendered":"https:\/\/www.inovex.de\/\/?p=912"},"modified":"2026-03-17T08:57:53","modified_gmt":"2026-03-17T07:57:53","slug":"apache-mesos-an-introduction","status":"publish","type":"post","link":"https:\/\/www.inovex.de\/de\/blog\/apache-mesos-an-introduction\/","title":{"rendered":"Apache Mesos: An introduction"},"content":{"rendered":"

One of the biggest challenges in data centers is to maintain multiple clusters for different workloads. Say\u00a0you want to run Hadoop<\/a>, Kafka<\/a> and Storm<\/a> which means that you have to maintain 3 different clusters. These different clusters are hardly utilized most of the time\u00a0so for example when you run Hadoop you need many resources to get the\u00a0job done but the rest of the day these resources stay idle. With a very simple calculation you can see how much time your resources are idle and only waste space and money (and we didn\u2019t talk\u00a0about hardware replacements at this point!). Read on for the nitty gritty details in this first article in our Mesos mini series.<\/p>\n

Say a\u00a0Hadoop cluster runs a 3 hour job, this\u00a0means the cluster is idle 21 hours a day. Looking at the\u00a0whole year we have\u00a0a cluster which is idle for 7665 hours (!). And\u00a0this calculation does not even take into account that we\u00a0usually\u00a0want to overprovision our cluster so that we can handle\u00a0load peaks.<\/p>\n

In 2009 Benjamin Hindman and his team at UC Berkley had the same problem with multiple Hadoop clusters. So they created Apache Mesos which is a distributed system kernel and works as a cluster scheduler. Mesos abstracts the resources (CPU, memory, I\/O, network) of a cluster for end users. So Mesos abstracts the whole cluster resources into one big computer and allows the user to have OS functionalities on a cluster-level. \u00a0Mesos provides\u00a0a Web UI as well as\u00a0an API for resource and scheduling management. One of the first and biggest Mesos adopters is Twitter<\/a>.<\/p>\n

\"Classical<\/a>
Figure 1: Classic\u00a0cluster setup<\/figcaption><\/figure>\n

Apache Mesos runs on any\u00a0POSIX oriented operating system (e.g. Linux and OSX) and allows you to share resources between multiple frameworks, which are handled kind of like an application.\u00a0 With Mesos you\u00a0are able to combine all these resources into one\u00a0big cluster and run different workloads on it. When we introduce an intelligent time scheduling for example Hadoop runs only at a specific time and we can reduce the size of the cluster. Another approach is to run best effort jobs which run when some resources are idle and can be killed if a high priority job comes in. This oversubscription<\/a> approach for Mesos is in development by Mesosphere<\/a>, Intel<\/a> and some other big companies.<\/p>\n

\"A<\/a>
Figure 2: A cluster with Mesos as scheduler<\/figcaption><\/figure>\n

It\u2019s pretty helpful to be able to\u00a0run different frameworks on your cluster without the need to provision new resources. So for example when you have some Tasks which can be better done with a special framework, e.g. Storm, you can simply run it on your cluster.<\/p>\n

\n

<\/p>\n<\/div>