{"id":33943,"date":"2022-03-09T10:53:26","date_gmt":"2022-03-09T09:53:26","guid":{"rendered":"https:\/\/www.inovex.de\/?p=33943"},"modified":"2023-11-02T07:57:34","modified_gmt":"2023-11-02T06:57:34","slug":"getting-started-with-the-rapids-accelerator-for-apache-spark-on-azure-databricks","status":"publish","type":"post","link":"https:\/\/www.inovex.de\/de\/blog\/getting-started-with-the-rapids-accelerator-for-apache-spark-on-azure-databricks\/","title":{"rendered":"Getting Started with the Rapids Accelerator for Apache Spark on Azure Databricks"},"content":{"rendered":"

In this blog post, we will look at Nvidia’s Rapids Accelerator for Apache Spark, a plugin for distributed data processing on the GPU. You will learn, how to set it up on Azure Databricks and run a toy example.
\n<\/p>\n

In the last two decades, we have witnessed the rise of big data and data science. Data is getting bigger and transformations are becoming more complex while CPU processing power is reaching its limits. Due to their high parallelism and bandwidth, GPUs are getting more and more adopted in high-performance computing and data science to keep up with the ever-growing demand for computing power.<\/p>\n

Apache Spark is currently the de-facto standard framework for building distributed data processing pipelines for big data. Nvidia recently released and open-sourced the \u201cRapids Accelerator for Apache Spark\u201c, a plugin to accelerate Spark applications on the GPU. This is done completely transparent, i.e. the user defines his Spark data frames or SQL queries as on regular (CPU) Spark and the plugin performs the respective operations on the GPU or falls back to the CPU if an operation is not supported.<\/p>\n

In this series of blog posts, we will get to know and evaluate the Rapids Accelerator . This first blog post will be about:<\/p>\n