{"id":21066,"date":"2017-10-17T07:58:59","date_gmt":"2017-10-17T06:58:59","guid":{"rendered":"https:\/\/www.inovex.de\/blog\/?p=3539"},"modified":"2024-08-29T08:04:31","modified_gmt":"2024-08-29T06:04:31","slug":"efficient-udafs-with-pyspark","status":"publish","type":"post","link":"https:\/\/www.inovex.de\/de\/blog\/efficient-udafs-with-pyspark\/","title":{"rendered":"Efficient UD(A)Fs with PySpark"},"content":{"rendered":"

Nowadays, Spark surely is one of the most prevalent technologies in the fields of data science and big data. Luckily, even though it is developed in Scala and runs in the Java Virtual Machine (JVM), it comes with Python bindings also known as PySpark, whose API was heavily influenced by Pandas<\/a>. With respect to functionality, modern PySpark has about the same capabilities as Pandas when it comes to typical ETL and data wrangling, e.g. groupby, aggregations and so on. As a general rule of thumb, one should consider an alternative to Pandas whenever the data set has more than 10,000,000 rows which, depending on the number of columns and data types, translates to about 5-10 GB of memory usage. At that point PySpark might be an option for you that does the job, but of course there are others like for instance Dask<\/a> which won\u2019t be addressed in this post.<\/p>\n

If you are new to Spark, one important thing to note is that Spark has two remarkable features besides its programmatic data wrangling capabilities. One is that Spark comes with SQL as an alternative way of defining queries and the other is Spark MLlib<\/a> for machine learning. Both topics are beyond the scope of this post but should be taken into account if you are considering PySpark as an alternative to Pandas and scikit-learn for larger data sets.<\/p>\n

But enough praise for PySpark, there are still some ugly sides as well as rough edges to it and we want to address some of them here, of course, in a constructive way. First of all, due to its relatively young age, PySpark lacks some features that Pandas provides, for example in areas such as reshaping\/pivoting or time series. Also, it is not as straightforward to use advanced mathematical functions from SciPy within PySpark. That\u2019s why sooner or later, you might walk into a scenario where you want to apply some Pandas or SciPy operations to your data frame in PySpark. Unfortunately, there is no built-in mechanism for using Pandas transformations in PySpark. In fact, this requires a lot of boilerplate code with many error-prone details to consider. Therefore we make a wish to the coding fairy, cross two fingers that someone else already solved this and start googling\u2026 and here we are \ud83d\ude09<\/p>\n

The remainder of this blog post walks you through the process of writing efficient Pandas UDAFs in PySpark. In fact, we end up abstracting all the necessary boilerplate code into a single Python decorator, which allows us to conveniently specify our PySpark Pandas function. To give more insights into performance considerations, this post also contains a little journey into the internals of PySpark.<\/p>\n

\n

<\/p>\n<\/div>