{"id":21032,"date":"2016-08-29T10:12:24","date_gmt":"2016-08-29T09:12:24","guid":{"rendered":"https:\/\/www.inovex.de\/?p=2103"},"modified":"2026-03-17T07:59:25","modified_gmt":"2026-03-17T06:59:25","slug":"hbase-and-phoenix-on-azure-adventures-in-abstraction","status":"publish","type":"post","link":"https:\/\/www.inovex.de\/de\/blog\/hbase-and-phoenix-on-azure-adventures-in-abstraction\/","title":{"rendered":"HBase and Phoenix on Azure: adventures in abstraction"},"content":{"rendered":"

One of my favourite essays by Joel Spolsky (he of Stack Overflow fame) is \u201cThe law of leaky abstractions\u201c<\/a>. In it he describes how the prevalence of layers of abstraction \u2013 be it coding languages or libraries or frameworks \u2013 have helped us accelerate our productivity. We don\u2019t have to talk directly to a database engine because we can let our SQL do that for us; we don\u2019t have to implement map reduce jobs in java anymore because we can use Hive; we don\u2019t have to\u2026 well, you get the idea.<\/p>\n

But he also points out that even the best frameworks and languages are less than perfect, and when things go awry, these frameworks \u201cleak\u201c the details of their abstraction out to the observing world. We are then confronted with all the nuts-and-bolts of the implementation that had been hitherto kindly hidden from us, and we often have no alternative but to busy ourselves with a depth of detail that we had not expected.<\/p>\n

Hadoop is a good case in point: consider my first Hadoop project in 2011, shown below on the left (\u201con premise\u201c) where we implemented most of the map-reduce jobs in java with a sprinkling of pig scripts.<\/p>\n

\"on-premise-vs-cloud\"<\/p>\n

Compare that to the project we have just completed, running on Microsoft Azure, shown on the right (\u201ccloud\u201c).<\/p>\n

Note that we have added two extra layers of abstraction ourselves (HBase and Phoenix), but that the cloud stack has added another 2 (virtualization and azure storage).<\/p>\n

This is all fine\u2026 until things are not quite so fine \u2013 and then it is a non-trivial task finding the cause. In fact, on rare occasions you may even end up needing to know almost as much about the underlying implementation as if there had been no abstraction layer in the first place! With this as our background, I\u2019d like to offer the following:<\/p>\n

    \n
  1. Some reflections on using Phoenix as an SQL layer over HBase<\/li>\n
  2. Some comments on HBase compaction settings in the context of Azure<\/li>\n<\/ol>\n
    \n

    <\/p>\n<\/div>