{"id":21083,"date":"2018-04-10T13:30:43","date_gmt":"2018-04-10T11:30:43","guid":{"rendered":"http:\/\/www.inovex.de\/blog\/?p=13029"},"modified":"2023-09-18T07:45:15","modified_gmt":"2023-09-18T05:45:15","slug":"managing-isolated-environments-with-pyspark","status":"publish","type":"post","link":"https:\/\/www.inovex.de\/de\/blog\/managing-isolated-environments-with-pyspark\/","title":{"rendered":"Managing isolated Environments with PySpark"},"content":{"rendered":"

With the sustained success of the Spark data processing platform even data scientists with a strong focus on the Python ecosystem can no longer ignore it. Fortunately, it is easy to get started with PySpark<\/a>\u2014the official Python API<\/span> for Spark\u2014due to millions of word count tutorials on the web. In contrast to that, resources on how to deploy and use Python packages like Numpy, Pandas, Scikit-Learn in isolated environments with PySpark are scarce. A nice exception to that is a blog post by Eran Kampf. Being able to install your own Python libraries is especially important if you want to write User-Defined-Functions (UDFs) as explained in the blog post Efficient UD<\/span>(A)Fs with PySpark<\/a>.<\/p>\n

<\/p>\n

For most Spark\/Hadoop distributions, Cloudera in my case, there are basically two options for managing isolated\u00a0environments:<\/p>\n

    \n
  1. You give all your data scientists SSH<\/span> access to all your cluster\u2019s nodes and let them do whatever they want like installing virtual environments with virtualenv<\/a> or conda<\/a> as detailed in the Cloudera documentation<\/a>.<\/li>\n
  2. Your sysadmins install Anaconda Parcels using the Cloudera Manager Admin Console to provide the most popular Python packages in a one size fits all fashion for all your data scientists as described in a Cloudera blog post<\/a>.<\/li>\n<\/ol>\n

    Both options have drawbacks which are as severe as they are obvious. Do you really want to let a bunch of data scientists run processes on your cluster and fill up the local hard-drives? The second option is not even a real isolated environment at all since all your applications would use the same libraries and maybe break after an update of a\u00a0library.<\/p>\n

    Therefore, we need to empower our data scientists developing a predictive application to manage isolated environments with their dependencies themselves. This was also recognized as a problem and several issues (SPARK<\/span>-13587<\/a> &<\/span> SPARK<\/span>-16367<\/a>) suggest solutions, but none of them have been integrated yet. The most mature solution is actually coffee boat<\/a>, which is still in beta and not meant for production. Therefore, we want to present here a simple but viable solution for this problem that we have been using in production for more than a\u00a0year.<\/p>\n

    So how can we distribute Python modules and whole packages on our executors? Luckily, PySpark provides the functions sc.addFile<\/a> and sc.addPyFile<\/a> which allow us to upload files to every node in our cluster, even Python modules and egg files in case of the latter. Unfortunately, there is no way to upload wheel files which are needed for binary Python packages like Numpy, Pandas and so on. As a data scientist you cannot live without\u00a0those.<\/p>\n

    At first sight this looks pretty bad but thanks to the simplicity of the wheel format it\u2019s not so bad at all. So here is what we do in a nutshell: For a given PySpark application, we will create an isolated environment on HDFS<\/span> with the help of wheel files. When submitting our PySpark application, we copy the content of our environment to the driver and executors using sc.addFile<\/a>. Simple but\u00a0effective.<\/p>\n

    \n

    <\/p>\n<\/div>