{"id":27825,"date":"2021-04-27T19:21:48","date_gmt":"2021-04-27T17:21:48","guid":{"rendered":"https:\/\/www.inovex.de\/blog\/?p=20785"},"modified":"2025-09-15T11:30:45","modified_gmt":"2025-09-15T09:30:45","slug":"set-the-kernels-free-remote-kernels-for-jupyter-notebooks-spark","status":"publish","type":"post","link":"https:\/\/www.inovex.de\/de\/blog\/set-the-kernels-free-remote-kernels-for-jupyter-notebooks-spark\/","title":{"rendered":"Set the Kernels Free \u2013 Remote Kernels for Jupyter Notebooks & Spark"},"content":{"rendered":"

According to the Data Science Survey conducted by JetBrains in 2018, Jupyter\/IPython notebooks are the most popular tool in the category IDEs and Editors<\/em> among data scientists. Whether notebooks can be seen as true IDEs is debatable, however, there is no doubt that they offer several advantages \u2013 especially for exploratory data analysis. Incremental execution of code, quick access to visualizations as well as capturing transient computations and the ease of sharing results with others are the major success factors of notebooks.<\/p>\n

As for today, there are above 100 compatible kernels<\/a>, enabling programming in various languages, like Python, Julia, R or even c#. But notebooks offer much more. It is possible to use them for data science on distributed resources, for example by using notebooks in conjunction with a well-known and widely used framework for distributed data processing like Apache Spark. However, traditional notebooks are limited in accessing distributed cluster resources. Furthermore, while working with Spark in combination with Notebooks several challenges must be overcome, especially regarding scalability and stability.\u00a0 Why? Well, it all comes down to the kernel, the unit responsible for interpreting and executing code from the notebook cells.<\/p>\n

A traditional notebook container includes the notebook UI and also the kernel itself. Meaning: the notebook and its kernel are running as one unit. To understand the impact this has on working with Spark we first have to take a look at a typical Spark application:<\/p>\n

In general, each application runs as an independent group of processes within the cluster and is coordinated by the SparkContext <\/em>object which exists in the Driver Program<\/i>. The SparkContext<\/i> connects to the Cluster Manager<\/i> (E.g. YARN or Mesos for Hadoop), which then allocates needed resources for the application. Finally, SparkContext provides the Executors<\/i> with the application code, e.g. as Python code or JAR artefacts and sends tasks <\/i>to execute.<\/p>\n

Now, Spark itself offers two possible deploy modes: Cluster and Client Mode. The main difference between them is where the Spark Driver Program <\/i>runs. In Client Mode, it is launched directly within the Spark-Submit <\/i>process, and so it lives within the client, e.g. within a notebook container. In Cluster Mode, the Spark driver <\/i>is launched outside the client. Speaking of a regular YARN\/Hadoop cluster, the Spark driver <\/i>runs in the application master inside its own YARN container on one of the cluster nodes. After deployment, the client is no longer necessary for further execution. The Spark driver<\/i>, in contrast to Client Mode, is then subject to the resource manager (YARN) and can be scheduled automatically on nodes with available resources.<\/p>\n

After discussing Spark deploy modes, we can now have a look at the combination of both: Spark and Jupyter notebooks. As previously mentioned, in a traditional Jupyter notebook, the kernel exists within the\u00a0notebook container.<\/p>\n

\"Depiction
A traditional notebook setup: kernel running within the notebook container.<\/figcaption><\/figure>\n

Looking at the image above and remembering how Spark works and what Client and Cluster Mode are, it is obvious that this notebook architecture only allows us to work with Spark in Client Mode. The Spark-Driver <\/em>and SparkContext<\/em> are located in the notebook kernel, which is bound to the client process. Unfortunately, the above combination of Spark and notebooks is not flawless. There are several issues that can be improved.<\/p>\n

\n

<\/p>\n<\/div>