{"id":71148,"date":"2026-10-23T11:21:22","date_gmt":"2026-10-23T09:21:22","guid":{"rendered":"https:\/\/www.inovex.de\/?p=71148"},"modified":"2026-10-23T11:21:22","modified_gmt":"2026-10-23T09:21:22","slug":"local-pyspark-testing-with-databricks-connect-devcontainers","status":"publish","type":"post","link":"https:\/\/www.inovex.de\/en\/blog\/local-pyspark-testing-with-databricks-connect-devcontainers\/","title":{"rendered":"Local PySpark Testing with Databricks Connect &#038; Devcontainers"},"content":{"rendered":"<p>Testing serves as a cornerstone for any production-ready application, boosting reliability across local development as well as pipelines. Yet, the transfer to cloud-native ecosystems like Databricks can lead to unexpected difficulties. We discovered such a difficulty by the use of the Databricks-owned client library <a href=\"https:\/\/docs.databricks.com\/aws\/en\/dev-tools\/databricks-connect\/\">Databricks Connect<\/a>.<\/p>\n<p>Why? The integration of the library can break local testing workflows.<\/p>\n<p>At Real I.S. AG, we are developing a data integration platform powered by Databricks. As part of our toolkit, we utilize Databricks Connect. However, we identified that this specific package requires special handling of running tests.<\/p>\n<p>This post explores the technical nuances of this error, provides a deep dive into the mechanics of Databricks Connect, and outlines our solution for maintaining a seamless, working test environment.<!--more--><\/p>\n<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_88 counter-hierarchy ez-toc-counter ez-toc-custom ez-toc-container-direction\">\n<div class=\"ez-toc-title-container\"><p class=\"ez-toc-title\" style=\"cursor:inherit\"><\/p>\n<\/div><nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/www.inovex.de\/en\/blog\/local-pyspark-testing-with-databricks-connect-devcontainers\/#Databricks-Connect-as-glue-between-you-and-Databricks\" >Databricks Connect as glue between you and Databricks<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/www.inovex.de\/en\/blog\/local-pyspark-testing-with-databricks-connect-devcontainers\/#Devcontainers-to-the-rescue\" >Devcontainers to the rescue<\/a><ul class='ez-toc-list-level-3' ><li class='ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/www.inovex.de\/en\/blog\/local-pyspark-testing-with-databricks-connect-devcontainers\/#Integrating-pytest\" >Integrating pytest<\/a><\/li><\/ul><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/www.inovex.de\/en\/blog\/local-pyspark-testing-with-databricks-connect-devcontainers\/#An-excursion-into-the-Databricks-Connect-Python-package\" >An excursion into the Databricks Connect Python package<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/www.inovex.de\/en\/blog\/local-pyspark-testing-with-databricks-connect-devcontainers\/#Minimal-working-example\" >Minimal working example<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-6\" href=\"https:\/\/www.inovex.de\/en\/blog\/local-pyspark-testing-with-databricks-connect-devcontainers\/#Summary\" >Summary<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-7\" href=\"https:\/\/www.inovex.de\/en\/blog\/local-pyspark-testing-with-databricks-connect-devcontainers\/#Resources-for-further-reading\" >Resources for further reading<\/a><\/li><\/ul><\/nav><\/div>\n<h2><span class=\"ez-toc-section\" id=\"Databricks-Connect-as-glue-between-you-and-Databricks\"><\/span>Databricks Connect as glue between you and Databricks<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Apache Spark is one of the leading distributed processing and analytics engines, being used in a variety of our projects. The PySpark package refers to the standard, native way of interacting with Spark using Python. Under the hood, it installs in its package Java dependencies to create local SparkSessions directly on your development machine. This approach is powerful for local development and testing, as it allows you to run Spark computations without connecting to a remote cluster:<\/p>\n<pre class=\"lang:python decode:true\">from pyspark.sql import SparkSession\r\n\r\nspark = SparkSession.builder.getOrCreate()\r\ndf = spark.createDataFrame(\r\n\u00a0\u00a0\u00a0\u00a0[(1, \"foo\"),(2, \"bar\"),],\r\n\u00a0\u00a0\u00a0\u00a0[\"id\", \"label\"]\r\n)<\/pre>\n<p>A modern Databricks setup often utilizes the client library <a href=\"https:\/\/docs.databricks.com\/aws\/en\/dev-tools\/databricks-connect\/\">Databricks Connect<\/a>. This package enables the integration of various IDE\u2019s such as PyCharm or VSCode. This enables features like access to Databricks compute, debugging functionality, and more. Connect can be installed as usual as package:<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-69326 size-full aligncenter\" src=\"https:\/\/www.inovex.de\/wp-content\/uploads\/1-5.png\" alt=\"Onboarding Databricks Connect\" width=\"1581\" height=\"416\" srcset=\"https:\/\/www.inovex.de\/wp-content\/uploads\/1-5.png 1581w, https:\/\/www.inovex.de\/wp-content\/uploads\/1-5-300x79.png 300w, https:\/\/www.inovex.de\/wp-content\/uploads\/1-5-1024x269.png 1024w, https:\/\/www.inovex.de\/wp-content\/uploads\/1-5-768x202.png 768w, https:\/\/www.inovex.de\/wp-content\/uploads\/1-5-1536x404.png 1536w, https:\/\/www.inovex.de\/wp-content\/uploads\/1-5-400x105.png 400w, https:\/\/www.inovex.de\/wp-content\/uploads\/1-5-360x95.png 360w\" sizes=\"auto, (max-width: 1581px) 100vw, 1581px\" \/><\/p>\n<p>Attempting to use the native PySpark approach (creating a local SparkSession) in an environment configured for Databricks Connect will result in a RuntimeError, as Databricks Connect only supports remote sessions:<\/p>\n<pre class=\"lang:python decode:true\" title=\"Unable to create a local session\">from pyspark.sql import SparkSession\r\n\r\nspark = SparkSession.builder.getOrCreate()\r\n# RuntimeError: Only remote Spark sessions using Databricks Connect are supported. Use DatabricksSession.builder to create a remote Spark session instead.<\/pre>\n<p>The reason for this: Databricks Connect will replace your existing PySpark package with a specialized version for Databricks. As such, it is designed for remote sessions only, removing the capability to host local sessions.<\/p>\n<ul>\n<li>Digging into the Connect package clarifies this behaviour. Only remote configuration settings are evaluated. If those are missing, the above RuntimeError is raised by default.<\/li>\n<\/ul>\n<p>Instead of running Spark locally, it must establish a connection from your IDE to a remote Spark cluster, e.g. a high-performance cluster hosted on Databricks.<\/p>\n<ul>\n<li>For the sake of readability, we will continue with a setup working with remote sessions locally. If you are interested in the details, how and why Databricks Connect replaces certain packages such as PySpark, jump to \u201cAn excursion into Databricks Connect\u201d below.<\/li>\n<\/ul>\n<p>To connect to a Spark server, you can still use the remote session builder from the replaced PySpark package:<\/p>\n<pre class=\"lang:python decode:true\" title=\"Creating a remote session\">from pyspark.sql import SparkSession\r\n\r\n# Remote session via Spark Connect (sc)\r\nspark = SparkSession.builder.remote(\"sc:\/\/&lt;SPARK-SERVER&gt;\").getOrCreate()<\/pre>\n<p>The implication for running local tests is that you have to provide a Spark server. Containers will simplify this drastically.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Devcontainers-to-the-rescue\"><\/span>Devcontainers to the rescue<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p><a href=\"https:\/\/containers.dev\/\">Devcontainers<\/a> provide a modern approach to keeping the development environment reproducible and consistent. A typical approach is a single container, mirroring the runtime and libraries of the production system.<\/p>\n<p>The requirement of a remote SparkSession can be solved by the introduction of a second devcontainer: an open-source Spark server, provided as Docker image.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-69324 size-full\" src=\"https:\/\/www.inovex.de\/wp-content\/uploads\/2-6.png\" alt=\"Devcontainers with spark master\" width=\"1726\" height=\"546\" srcset=\"https:\/\/www.inovex.de\/wp-content\/uploads\/2-6.png 1726w, https:\/\/www.inovex.de\/wp-content\/uploads\/2-6-300x95.png 300w, https:\/\/www.inovex.de\/wp-content\/uploads\/2-6-1024x324.png 1024w, https:\/\/www.inovex.de\/wp-content\/uploads\/2-6-768x243.png 768w, https:\/\/www.inovex.de\/wp-content\/uploads\/2-6-1536x486.png 1536w, https:\/\/www.inovex.de\/wp-content\/uploads\/2-6-400x127.png 400w, https:\/\/www.inovex.de\/wp-content\/uploads\/2-6-360x114.png 360w\" sizes=\"auto, (max-width: 1726px) 100vw, 1726px\" \/><\/p>\n<p>After configuring the SparkSession Builder to reference the container, a remote SparkSession can finally be established using Spark Connect.<\/p>\n<pre class=\"lang:default decode:true\" title=\"Creating a remote session to a spark container using the container name\">from pyspark.sql import SparkSession\r\n\r\n# Reference the name of the spark container\r\nspark = SparkSession.builder.remote(\"sc:\/\/spark\").getOrCreate()<\/pre>\n<h3><span class=\"ez-toc-section\" id=\"Integrating-pytest\"><\/span>Integrating pytest<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>A parameterized test using an agnostic Spark as fixture might be:<\/p>\n<pre class=\"lang:default decode:true \" title=\"Test example\"># tests\/test_spark.py\r\n\r\nfrom collections import namedtuple\r\nimport pytest\r\n\r\nMetadata = namedtuple(\"Metadata\", [\"description\"])\r\n\r\nPySparkTestParameter = namedtuple(\r\n\r\n\u00a0\u00a0\u00a0\u00a0\"PySparkTestParameter\", [\"metadata\", \"data\", \"schema\", \"expected_count\"]\r\n\r\n)\r\n\r\n@pytest.mark.parametrize(\r\n\u00a0\u00a0\u00a0\u00a0\"params\",\r\n\u00a0\u00a0\u00a0\u00a0[\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0PySparkTestParameter(\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0metadata=Metadata(\"simple\"),\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0data=[\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0(1, \"foo\"),\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0(2, \"bar\"),\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0],\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0schema=[\"id\", \"label\"],\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0expected_count=2,\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0)\r\n\u00a0\u00a0\u00a0\u00a0],\r\n\u00a0\u00a0\u00a0\u00a0# more verbose logs\r\n\u00a0\u00a0\u00a0\u00a0ids=lambda p: p.metadata.description,\r\n)\r\n\r\ndef test_pyspark(spark, params):\r\n\u00a0\u00a0\u00a0\u00a0df = spark.createDataFrame(\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0data=params.data,\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0schema=params.schema,\r\n\u00a0\u00a0\u00a0\u00a0)\r\n\u00a0\u00a0\u00a0\u00a0df.show()\r\n\u00a0\u00a0\u00a0\u00a0assert df.count() == params.expected_count<\/pre>\n<p>Using the multi devcontainer setup, the fixture can be constructed as follows:<\/p>\n<pre class=\"lang:default decode:true\" title=\"conftest\"># tests\/conftest.py\r\n\r\nimport pytest\r\nfrom pyspark.sql import SparkSession\r\n\r\n@pytest.fixture(scope=\"session\")\r\ndef spark() -&gt; SparkSession:\r\n\u00a0\u00a0\u00a0\u00a0_spark = SparkSession.builder.remote(\"sc:\/\/spark\").getOrCreate()\r\n\r\n\r\n\u00a0\u00a0\u00a0\u00a0# add missing configurations\r\n\u00a0\u00a0\u00a0\u00a0_spark.conf.set(\"spark.sql.pyspark.legacy.inferMapTypeFromFirstPair.enabled\", True)\r\n\u00a0\u00a0\u00a0\u00a0_spark.conf.set(\"spark.sql.execution.pandas.inferPandasDictAsMap\", False)\r\n\u00a0\u00a0\u00a0\u00a0return _spark<\/pre>\n<p>Running pytest now leads to the desired output: A locally running SparkSession using Databricks Connect!<\/p>\n<pre class=\"lang:default decode:true\">((python_venv) ) vscode \u279c \/workspace (master) $ pytest -v\r\n==================================================================== test session starts =====================================================================\r\nplatform linux -- Python 3.12.12, pytest-9.0.2, pluggy-1.6.0 -- \/python_venv\/bin\/python\r\ncachedir: .pytest_cache\r\nrootdir: \/workspace\r\nconfigfile: pyproject.toml\r\nplugins: anyio-4.12.1\r\ncollected 1 item\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\r\n\r\ntests\/test_spark.py::test_pyspark[simple] PASSED<\/pre>\n<h2><span class=\"ez-toc-section\" id=\"An-excursion-into-the-Databricks-Connect-Python-package\"><\/span>An excursion into the Databricks Connect Python package<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>TL,DR: Databricks-Connect will replace your native PySpark package with a specialized version for the Databricks platform, unable to host a local SparkSession. Both packages are mutually exclusive to each other.<\/p>\n<p>Let\u2019s take a look into the package of Databricks Connect itself:<\/p>\n<pre class=\"lang:default decode:true\" title=\"Part of the content of the databricks connect wheel\">&gt; pip download databricks-connect -d download --no-dependencies\r\n&gt; cd download\r\n&gt; unzip databricks_connect-*.whl\r\n&gt; tree -L 2 .\r\n.\r\n\u251c\u2500\u2500 databricks\r\n\u2502 \u00a0 \u251c\u2500\u2500 __init__.py\r\n\u2502 \u00a0 \u2514\u2500\u2500 connect\r\n\u251c\u2500\u2500 databricks_connect-16.1.7-py2.py3-none-any.whl\r\n\u251c\u2500\u2500 databricks_connect-16.1.7.data\r\n\u2502 \u00a0 \u2514\u2500\u2500 scripts\r\n\u251c\u2500\u2500 databricks_connect-16.1.7.dist-info\r\n\u2502 \u00a0 \u251c\u2500\u2500 entry_points.txt\r\n\u2502 \u00a0 \u251c\u2500\u2500 METADATA\r\n\u2502 \u00a0 \u251c\u2500\u2500 RECORD\r\n\u2502 \u00a0 \u251c\u2500\u2500 top_level.txt\r\n\u2502 \u00a0 \u2514\u2500\u2500 WHEEL\r\n\u251c\u2500\u2500 delta\r\n\u2502 \u00a0 \u251c\u2500\u2500 __init__.py\r\n\u2502 \u00a0 \u2514\u2500\u2500 ...\r\n\u2514\u2500\u2500 pyspark\r\n\u251c\u2500\u2500 __init__.py\r\n\u251c\u2500\u2500 ...\r\n\u251c\u2500\u2500 sql.py\r\n\u251c\u2500\u2500 session.py\r\n\u2514\u2500\u2500 ...<\/pre>\n<p>Besides databricks.connect itself, two additional packages can be found: delta and pyspark. When further investigating into the wheel folder `databricks_connect-16.1.7.dist-info` (see <a href=\"https:\/\/packaging.python.org\/en\/latest\/specifications\/binary-distribution-format\/#binary-distribution-format\">https:\/\/packaging.python.org\/en\/latest\/specifications\/binary-distribution-format\/#binary-distribution-format<\/a> for more information), we identify that Databricks Connect takes priority over any prior installed delta\/pyspark package and replaces them.<\/p>\n<pre class=\"lang:default decode:true\" title=\"Metadata of wheel\">&gt; cat databricks_connect-16.1.7.dist-info\/top_level.txt\r\ndatabricks\r\ndelta\r\npyspark\r\n\r\n&gt; cat databricks_connect-16.1.7.dist-info\/METADATA\r\n...\r\nProvides: pyspark\r\nObsoletes: pyspark\r\n...\r\n\r\n&gt; du -s -m pyspark\r\n11 pyspark # 11 Megabyte. Much less than native PySpark as Java dependencies are missing (required for local session)<\/pre>\n<h2><span class=\"ez-toc-section\" id=\"Minimal-working-example\"><\/span>Minimal working example<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Clone the example and run it yourself in the IDE of your choice: https:\/\/github.com\/inovex\/databricks-pyspark-local-testing-demo<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Summary\"><\/span>Summary<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Databricks Connect will break your ability to create a local Spark server using the PySpark package. However, you can still create remote sessions which are hosted on Databricks or otherwise available to you (such as a locally running Spark server). Using devcontainers, you are able to host both your development environment and a Spark server in a developer-friendly way. And continue to run your tests familiar to you.<\/p>\n<p>The drawback of this approach is a limited set of functionality compared to the Databricks ecosystem. Some features may not work for your individual use case: the Spark Connect server (and thus the remote session) provides a limited subset of the full Databricks ecosystem&#8217;s functionality. Features that rely on deep integration with Databricks-specific services \u2013 such as special data types or other proprietary utilities \u2013 may not be fully available in this standalone environment. Developers must validate that this approach covers all essential features required for their individual use case.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Resources-for-further-reading\"><\/span>Resources for further reading<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<ul>\n<li><a href=\"https:\/\/docs.databricks.com\/aws\/en\/dev-tools\/databricks-connect\/python\/\">Databricks Connect<\/a><\/li>\n<li><a href=\"https:\/\/docs.databricks.com\/aws\/en\/dev-tools\/databricks-connect\/python\/\">Use Spark Connect Server with Databricks Connect<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/inovex\/databricks-pyspark-local-testing-demo\">Minimal working example<\/a><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>Testing serves as a cornerstone for any production-ready application, boosting reliability across local development as well as pipelines. Yet, the transfer to cloud-native ecosystems like Databricks can lead to unexpected difficulties. We discovered such a difficulty by the use of the Databricks-owned client library Databricks Connect. Why? The integration of the library can break local [&hellip;]<\/p>\n","protected":false},"author":334,"featured_media":71194,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"inline_featured_image":false,"ep_exclude_from_search":false,"footnotes":""},"tags":[],"service":[455,926],"level":[1280],"coauthors":[{"id":334,"display_name":"Jan Niklas Steeg","user_nicename":"jsteeg"}],"class_list":["post-71148","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","service-data-engineering-en","service-software-craftsmanship-en"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v28.5 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Local PySpark Testing with Databricks Connect &amp; Devcontainers - inovex GmbH<\/title>\n<meta name=\"description\" content=\"Learn how Databricks Connect affects local PySpark testing workflows and how to set up a reproducible test environment using Docker devcontainers.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.inovex.de\/en\/blog\/local-pyspark-testing-with-databricks-connect-devcontainers\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Local PySpark Testing with Databricks Connect &amp; Devcontainers - inovex GmbH\" \/>\n<meta property=\"og:description\" content=\"Learn how Databricks Connect affects local PySpark testing workflows and how to set up a reproducible test environment using Docker devcontainers.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.inovex.de\/en\/blog\/local-pyspark-testing-with-databricks-connect-devcontainers\/\" \/>\n<meta property=\"og:site_name\" content=\"inovex GmbH\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/inovexde\" \/>\n<meta property=\"article:published_time\" content=\"2026-10-23T09:21:22+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.inovex.de\/wp-content\/uploads\/Local-PySpark-Testing-with-Databricks-Connect-Devcontainers.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1500\" \/>\n\t<meta property=\"og:image:height\" content=\"880\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Jan Niklas Steeg\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:image\" content=\"https:\/\/www.inovex.de\/wp-content\/uploads\/Local-PySpark-Testing-with-Databricks-Connect-Devcontainers-1024x601.png\" \/>\n<meta name=\"twitter:creator\" content=\"@inovexgmbh\" \/>\n<meta name=\"twitter:site\" content=\"@inovexgmbh\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Jan Niklas Steeg\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"6 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/en\\\/blog\\\/local-pyspark-testing-with-databricks-connect-devcontainers\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/en\\\/blog\\\/local-pyspark-testing-with-databricks-connect-devcontainers\\\/\"},\"author\":{\"name\":\"Jan Niklas Steeg\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/en\\\/#\\\/schema\\\/person\\\/566ec94bc4810235927584df193f7fe3\"},\"headline\":\"Local PySpark Testing with Databricks Connect &#038; Devcontainers\",\"datePublished\":\"2026-10-23T09:21:22+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/en\\\/blog\\\/local-pyspark-testing-with-databricks-connect-devcontainers\\\/\"},\"wordCount\":872,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/en\\\/#organization\"},\"image\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/en\\\/blog\\\/local-pyspark-testing-with-databricks-connect-devcontainers\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.inovex.de\\\/wp-content\\\/uploads\\\/Local-PySpark-Testing-with-Databricks-Connect-Devcontainers.png\",\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/www.inovex.de\\\/en\\\/blog\\\/local-pyspark-testing-with-databricks-connect-devcontainers\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/en\\\/blog\\\/local-pyspark-testing-with-databricks-connect-devcontainers\\\/\",\"url\":\"https:\\\/\\\/www.inovex.de\\\/en\\\/blog\\\/local-pyspark-testing-with-databricks-connect-devcontainers\\\/\",\"name\":\"Local PySpark Testing with Databricks Connect & Devcontainers - inovex GmbH\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/en\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/en\\\/blog\\\/local-pyspark-testing-with-databricks-connect-devcontainers\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/en\\\/blog\\\/local-pyspark-testing-with-databricks-connect-devcontainers\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.inovex.de\\\/wp-content\\\/uploads\\\/Local-PySpark-Testing-with-Databricks-Connect-Devcontainers.png\",\"datePublished\":\"2026-10-23T09:21:22+00:00\",\"description\":\"Learn how Databricks Connect affects local PySpark testing workflows and how to set up a reproducible test environment using Docker devcontainers.\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/en\\\/blog\\\/local-pyspark-testing-with-databricks-connect-devcontainers\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/www.inovex.de\\\/en\\\/blog\\\/local-pyspark-testing-with-databricks-connect-devcontainers\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/en\\\/blog\\\/local-pyspark-testing-with-databricks-connect-devcontainers\\\/#primaryimage\",\"url\":\"https:\\\/\\\/www.inovex.de\\\/wp-content\\\/uploads\\\/Local-PySpark-Testing-with-Databricks-Connect-Devcontainers.png\",\"contentUrl\":\"https:\\\/\\\/www.inovex.de\\\/wp-content\\\/uploads\\\/Local-PySpark-Testing-with-Databricks-Connect-Devcontainers.png\",\"width\":1500,\"height\":880},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/en\\\/blog\\\/local-pyspark-testing-with-databricks-connect-devcontainers\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/www.inovex.de\\\/en\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Local PySpark Testing with Databricks Connect &#038; Devcontainers\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/en\\\/#website\",\"url\":\"https:\\\/\\\/www.inovex.de\\\/en\\\/\",\"name\":\"inovex GmbH\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/en\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/www.inovex.de\\\/en\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/en\\\/#organization\",\"name\":\"inovex GmbH\",\"url\":\"https:\\\/\\\/www.inovex.de\\\/en\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/en\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/www.inovex.de\\\/wp-content\\\/uploads\\\/2021\\\/03\\\/inovex-logo-16-9-1.png\",\"contentUrl\":\"https:\\\/\\\/www.inovex.de\\\/wp-content\\\/uploads\\\/2021\\\/03\\\/inovex-logo-16-9-1.png\",\"width\":1921,\"height\":1081,\"caption\":\"inovex GmbH\"},\"image\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/en\\\/#\\\/schema\\\/logo\\\/image\\\/\"},\"sameAs\":[\"https:\\\/\\\/www.facebook.com\\\/inovexde\",\"https:\\\/\\\/x.com\\\/inovexgmbh\",\"https:\\\/\\\/www.instagram.com\\\/inovexlife\\\/\",\"https:\\\/\\\/www.linkedin.com\\\/company\\\/inovex\",\"https:\\\/\\\/www.youtube.com\\\/channel\\\/UC7r66GT14hROB_RQsQBAQUQ\"]},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/en\\\/#\\\/schema\\\/person\\\/566ec94bc4810235927584df193f7fe3\",\"name\":\"Jan Niklas Steeg\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/5ae0a2538acacff99a3f1e991ef70218483b9ff5483f2a238032cfd56d7d31da?s=96&d=retro&r=g440aceff9f79687ee7c88007b607c39f\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/5ae0a2538acacff99a3f1e991ef70218483b9ff5483f2a238032cfd56d7d31da?s=96&d=retro&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/5ae0a2538acacff99a3f1e991ef70218483b9ff5483f2a238032cfd56d7d31da?s=96&d=retro&r=g\",\"caption\":\"Jan Niklas Steeg\"},\"url\":\"https:\\\/\\\/www.inovex.de\\\/en\\\/blog\\\/author\\\/jsteeg\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Local PySpark Testing with Databricks Connect & Devcontainers - inovex GmbH","description":"Learn how Databricks Connect affects local PySpark testing workflows and how to set up a reproducible test environment using Docker devcontainers.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.inovex.de\/en\/blog\/local-pyspark-testing-with-databricks-connect-devcontainers\/","og_locale":"en_US","og_type":"article","og_title":"Local PySpark Testing with Databricks Connect & Devcontainers - inovex GmbH","og_description":"Learn how Databricks Connect affects local PySpark testing workflows and how to set up a reproducible test environment using Docker devcontainers.","og_url":"https:\/\/www.inovex.de\/en\/blog\/local-pyspark-testing-with-databricks-connect-devcontainers\/","og_site_name":"inovex GmbH","article_publisher":"https:\/\/www.facebook.com\/inovexde","article_published_time":"2026-10-23T09:21:22+00:00","og_image":[{"width":1500,"height":880,"url":"https:\/\/www.inovex.de\/wp-content\/uploads\/Local-PySpark-Testing-with-Databricks-Connect-Devcontainers.png","type":"image\/png"}],"author":"Jan Niklas Steeg","twitter_card":"summary_large_image","twitter_image":"https:\/\/www.inovex.de\/wp-content\/uploads\/Local-PySpark-Testing-with-Databricks-Connect-Devcontainers-1024x601.png","twitter_creator":"@inovexgmbh","twitter_site":"@inovexgmbh","twitter_misc":{"Written by":"Jan Niklas Steeg","Est. reading time":"6 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.inovex.de\/en\/blog\/local-pyspark-testing-with-databricks-connect-devcontainers\/#article","isPartOf":{"@id":"https:\/\/www.inovex.de\/en\/blog\/local-pyspark-testing-with-databricks-connect-devcontainers\/"},"author":{"name":"Jan Niklas Steeg","@id":"https:\/\/www.inovex.de\/en\/#\/schema\/person\/566ec94bc4810235927584df193f7fe3"},"headline":"Local PySpark Testing with Databricks Connect &#038; Devcontainers","datePublished":"2026-10-23T09:21:22+00:00","mainEntityOfPage":{"@id":"https:\/\/www.inovex.de\/en\/blog\/local-pyspark-testing-with-databricks-connect-devcontainers\/"},"wordCount":872,"commentCount":0,"publisher":{"@id":"https:\/\/www.inovex.de\/en\/#organization"},"image":{"@id":"https:\/\/www.inovex.de\/en\/blog\/local-pyspark-testing-with-databricks-connect-devcontainers\/#primaryimage"},"thumbnailUrl":"https:\/\/www.inovex.de\/wp-content\/uploads\/Local-PySpark-Testing-with-Databricks-Connect-Devcontainers.png","inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.inovex.de\/en\/blog\/local-pyspark-testing-with-databricks-connect-devcontainers\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.inovex.de\/en\/blog\/local-pyspark-testing-with-databricks-connect-devcontainers\/","url":"https:\/\/www.inovex.de\/en\/blog\/local-pyspark-testing-with-databricks-connect-devcontainers\/","name":"Local PySpark Testing with Databricks Connect & Devcontainers - inovex GmbH","isPartOf":{"@id":"https:\/\/www.inovex.de\/en\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.inovex.de\/en\/blog\/local-pyspark-testing-with-databricks-connect-devcontainers\/#primaryimage"},"image":{"@id":"https:\/\/www.inovex.de\/en\/blog\/local-pyspark-testing-with-databricks-connect-devcontainers\/#primaryimage"},"thumbnailUrl":"https:\/\/www.inovex.de\/wp-content\/uploads\/Local-PySpark-Testing-with-Databricks-Connect-Devcontainers.png","datePublished":"2026-10-23T09:21:22+00:00","description":"Learn how Databricks Connect affects local PySpark testing workflows and how to set up a reproducible test environment using Docker devcontainers.","breadcrumb":{"@id":"https:\/\/www.inovex.de\/en\/blog\/local-pyspark-testing-with-databricks-connect-devcontainers\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.inovex.de\/en\/blog\/local-pyspark-testing-with-databricks-connect-devcontainers\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.inovex.de\/en\/blog\/local-pyspark-testing-with-databricks-connect-devcontainers\/#primaryimage","url":"https:\/\/www.inovex.de\/wp-content\/uploads\/Local-PySpark-Testing-with-Databricks-Connect-Devcontainers.png","contentUrl":"https:\/\/www.inovex.de\/wp-content\/uploads\/Local-PySpark-Testing-with-Databricks-Connect-Devcontainers.png","width":1500,"height":880},{"@type":"BreadcrumbList","@id":"https:\/\/www.inovex.de\/en\/blog\/local-pyspark-testing-with-databricks-connect-devcontainers\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.inovex.de\/en\/"},{"@type":"ListItem","position":2,"name":"Local PySpark Testing with Databricks Connect &#038; Devcontainers"}]},{"@type":"WebSite","@id":"https:\/\/www.inovex.de\/en\/#website","url":"https:\/\/www.inovex.de\/en\/","name":"inovex GmbH","description":"","publisher":{"@id":"https:\/\/www.inovex.de\/en\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.inovex.de\/en\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.inovex.de\/en\/#organization","name":"inovex GmbH","url":"https:\/\/www.inovex.de\/en\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.inovex.de\/en\/#\/schema\/logo\/image\/","url":"https:\/\/www.inovex.de\/wp-content\/uploads\/2021\/03\/inovex-logo-16-9-1.png","contentUrl":"https:\/\/www.inovex.de\/wp-content\/uploads\/2021\/03\/inovex-logo-16-9-1.png","width":1921,"height":1081,"caption":"inovex GmbH"},"image":{"@id":"https:\/\/www.inovex.de\/en\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/inovexde","https:\/\/x.com\/inovexgmbh","https:\/\/www.instagram.com\/inovexlife\/","https:\/\/www.linkedin.com\/company\/inovex","https:\/\/www.youtube.com\/channel\/UC7r66GT14hROB_RQsQBAQUQ"]},{"@type":"Person","@id":"https:\/\/www.inovex.de\/en\/#\/schema\/person\/566ec94bc4810235927584df193f7fe3","name":"Jan Niklas Steeg","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/5ae0a2538acacff99a3f1e991ef70218483b9ff5483f2a238032cfd56d7d31da?s=96&d=retro&r=g440aceff9f79687ee7c88007b607c39f","url":"https:\/\/secure.gravatar.com\/avatar\/5ae0a2538acacff99a3f1e991ef70218483b9ff5483f2a238032cfd56d7d31da?s=96&d=retro&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/5ae0a2538acacff99a3f1e991ef70218483b9ff5483f2a238032cfd56d7d31da?s=96&d=retro&r=g","caption":"Jan Niklas Steeg"},"url":"https:\/\/www.inovex.de\/en\/blog\/author\/jsteeg\/"}]}},"_links":{"self":[{"href":"https:\/\/www.inovex.de\/en\/wp-json\/wp\/v2\/posts\/71148","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.inovex.de\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.inovex.de\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.inovex.de\/en\/wp-json\/wp\/v2\/users\/334"}],"replies":[{"embeddable":true,"href":"https:\/\/www.inovex.de\/en\/wp-json\/wp\/v2\/comments?post=71148"}],"version-history":[{"count":3,"href":"https:\/\/www.inovex.de\/en\/wp-json\/wp\/v2\/posts\/71148\/revisions"}],"predecessor-version":[{"id":71149,"href":"https:\/\/www.inovex.de\/en\/wp-json\/wp\/v2\/posts\/71148\/revisions\/71149"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.inovex.de\/en\/wp-json\/wp\/v2\/media\/71194"}],"wp:attachment":[{"href":"https:\/\/www.inovex.de\/en\/wp-json\/wp\/v2\/media?parent=71148"}],"wp:term":[{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.inovex.de\/en\/wp-json\/wp\/v2\/tags?post=71148"},{"taxonomy":"service","embeddable":true,"href":"https:\/\/www.inovex.de\/en\/wp-json\/wp\/v2\/service?post=71148"},{"taxonomy":"level","embeddable":true,"href":"https:\/\/www.inovex.de\/en\/wp-json\/wp\/v2\/level?post=71148"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.inovex.de\/en\/wp-json\/wp\/v2\/coauthors?post=71148"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}