{"id":56335,"date":"2024-07-22T09:47:26","date_gmt":"2024-07-22T07:47:26","guid":{"rendered":"https:\/\/www.inovex.de\/?p=56335"},"modified":"2024-07-22T09:47:26","modified_gmt":"2024-07-22T07:47:26","slug":"effective-snowflake-data-ingestion-with-java","status":"publish","type":"post","link":"https:\/\/www.inovex.de\/de\/blog\/effective-snowflake-data-ingestion-with-java\/","title":{"rendered":"Effective Snowflake Data Ingestion with Java"},"content":{"rendered":"<p>Snowflake is becoming increasingly popular as a data platform, and thus the need for ways to integrate Snowflake into production systems is also growing. As type-safe programming languages like Java lead to more stable productive environments and are often preferred over untyped languages (e.g. Python), this implies the challenge to connect Snowflake databases in those type-safe languages. While Snowflake comes with dedicated methods for data loading from Cloud providers (e.g. <a href=\"https:\/\/docs.snowflake.com\/en\/user-guide\/data-load-snowpipe-intro\">Snowpipes<\/a>), we have to rely on other implementation approaches when ingesting data directly from our type-safe Java apps.<\/p>\n<p>So in this blog post, we will cover the following native data ingestion methods for Java: Batch and Merge insertion using the JDBC template and Stream insertion via the <strong>Snowflake Streaming API<\/strong>.<\/p>\n<p><!--more-->We will also compare their performance in terms of ingestion time and Snowflake compute usage. This provides you with a practical guide to choosing the right implementation for your application and is applicable to all languages that provide a Snowflake JDBC driver (in particular it should be applicable to other JVM-based languages like Kotlin or Scala).<\/p>\n<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_82_2 counter-hierarchy ez-toc-counter ez-toc-custom ez-toc-container-direction\">\n<div class=\"ez-toc-title-container\"><p class=\"ez-toc-title\" style=\"cursor:inherit\"><\/p>\n<\/div><nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/www.inovex.de\/de\/blog\/effective-snowflake-data-ingestion-with-java\/#Preliminary-remarks\" >Preliminary remarks<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/www.inovex.de\/de\/blog\/effective-snowflake-data-ingestion-with-java\/#Snowflake-Connection-in-Java\" >Snowflake Connection in Java<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/www.inovex.de\/de\/blog\/effective-snowflake-data-ingestion-with-java\/#Data-Ingestion-with-Batch-Insert\" >Data Ingestion with Batch Insert<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/www.inovex.de\/de\/blog\/effective-snowflake-data-ingestion-with-java\/#Data-Ingestion-with-Merge-Insert\" >Data Ingestion with Merge Insert<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/www.inovex.de\/de\/blog\/effective-snowflake-data-ingestion-with-java\/#Data-Ingestion-using-Snowflake-Streaming\" >Data Ingestion using Snowflake Streaming<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-6\" href=\"https:\/\/www.inovex.de\/de\/blog\/effective-snowflake-data-ingestion-with-java\/#Performance-Comparison\" >Performance Comparison<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-7\" href=\"https:\/\/www.inovex.de\/de\/blog\/effective-snowflake-data-ingestion-with-java\/#Recommendations\" >Recommendations<\/a><\/li><\/ul><\/nav><\/div>\n<h2><span class=\"ez-toc-section\" id=\"Preliminary-remarks\"><\/span>Preliminary remarks<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Throughout this post, we will use a Snowflake database with a table named INGEST_TABLE, which is created in SnowSQL with the following command<\/p>\n<pre class=\"lang:tsql decode:true \" title=\"Creation of Ingestion Table\">CREATE OR REPLACE TABLE INGEST_TABLE(\r\nid int,\r\nname varchar,\r\nprice double,\r\njsonData variant);<\/pre>\n<p><span style=\"font-weight: 400;\">The Variant column, which represents JSON formatted data, is used to create proper load on the database. We implement a corresponding Java class named \u201cSnowflakeTableEntity\u201c:<\/span><\/p>\n<pre class=\"lang:java decode:true \" title=\"Java Entity of Database Entry\">public class SnowflakeTableEntity {\r\n   private int id;\r\n   private String name;\r\n   private Double price;\r\n   private final String jsonData = \"{'someBigJson':0}\";\r\n\r\n   @JsonIgnore\r\n   private final ObjectMapper objectMapper = new ObjectMapper();\r\n\r\n   public String toJson() {\r\n       try {\r\n           return objectMapper.writeValueAsString(this);\r\n       } catch (JsonProcessingException e) {\r\n           log.error(e);\r\n           return null;\r\n       }\r\n   }\r\n}\r\n<\/pre>\n<h2><span class=\"ez-toc-section\" id=\"Snowflake-Connection-in-Java\"><\/span>Snowflake Connection in Java<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Like most other database systems, Snowflake provides us with a JDBC driver for Java, which we will use throughout this blog post. To connect our JDBC driver to Snowflake, we first have to authenticate it to Snowflake.<br \/>\nFor that, we have to create an RSA key pair (e.g. by invoking ssh-keygen) and assign the public key to a Snowflake User with the SnowSQL command<\/p>\n<pre class=\"lang:tsql decode:true\" title=\"Creation of RSA Key\">ALTER USER sf_user SET rsa_public_key='MII...';<\/pre>\n<p>where sf_user is the Snowflake User in our example. With the private key of our generated key pair which will be located in a private key file, let\u2019s say \u2018rsa_key.p8\u2019, we can configure our data source which we use to instantiate the JDBC driver.<br \/>\nTherefore, we have to convert the private key, which is saved in base64 encoded text into the PrivateKey Java Interface:<\/p>\n<pre class=\"lang:java decode:true\" title=\"Java Implementation of Key Read\">import java.security.KeyFactory;\r\nimport java.security.PrivateKey;\r\nimport java.security.spec.KeySpec;\r\nimport java.security.spec.PKCS8EncodedKeySpec;\r\nimport java.util.Base64;\r\n\r\n  \/**\r\n   * Creates a RSA private key from a P8 file\r\n   *\r\n   * @param file a private key P8 file\r\n   * @return RSAPrivateCrtKey instance\r\n   * @throws Exception arises if any error occurs\r\n   *\/\r\n  private static PrivateKey readPrivateKey(File file) throws Exception {\r\n    String key = Files.readString(file.toPath(), Charset.defaultCharset());\r\n\r\n    String privateKeyPEM =\r\n        key.replace(\"-----BEGIN PRIVATE KEY-----\", \"\")         .replaceAll(System.lineSeparator(), \"\")\r\n            .replace(\"-----END PRIVATE KEY-----\", \"\");\r\n\r\n    byte[] encoded = Base64.getDecoder().decode(privateKeyPEM);\r\n\r\n    KeyFactory keyFactory = KeyFactory.getInstance(\"RSA\");\r\n    PKCS8EncodedKeySpec keySpec = new PKCS8EncodedKeySpec(encoded);\r\n    return keyFactory.generatePrivate(keySpec);\r\n  }<\/pre>\n<p>Note, how here the &#8212;&#8211;BEGIN PRIVATE KEY&#8212;&#8211; and &#8212;&#8211;END PRIVATE KEY&#8212;&#8211; parts have to be removed from the private key.<br \/>\nWith that, we can already create our JDBC template that uses the corresponding Datasource:<\/p>\n<pre class=\"lang:java decode:true \" title=\"Java Implementation of the Data Source\">SnowflakeBasicDataSource dataSource = new SnowflakeBasicDataSource();\r\n\r\ndataSource.setPrivateKey(readPrivateKeyFile(file));\r\ndataSource.setSchema(schema);\r\ndataSource.setDatabaseName(databaseName);\r\ndataSource.setRole(role);\r\ndataSource.setUser(user);\r\ndataSource.setWarehouse(warehouse);\r\ndataSource.setUrl(sfUrl);\r\n\r\nHikariDataSource pooledDataSource = new HikariDataSource();\r\npooledDataSource.setDataSource(dataSource);\r\n\r\n     SnowflakeConnection\r\n sfConnection = pooledDataSource.getConnection().unwrap(SnowflakeConnection.class);\r\n     JdbcTemplate sfJdbcTemplate = new JdbcTemplate(pooledDataSource);\r\n<\/pre>\n<p><span style=\"font-weight: 400;\">where the parameters of the Datasource are the corresponding snowflake objects and the file input is your private key file. The<\/span><span style=\"font-weight: 400;\"> sfUrl<\/span><span style=\"font-weight: 400;\"> should be of the form <\/span><\/p>\n<pre class=\"lang:default decode:true \" title=\"Snowflake URL\">jdbc:snowflake:\/\/{snowflake-instance-locator}.{snowflake-instance-region}.snowflakecomputing.com)<\/pre>\n<p>Having set up the Snowflake connection correctly, we should now be able to communicate with our Snowflake database. Here is a test selection query:<\/p>\n<pre class=\"lang:java decode:true \" title=\"Java Example Query\">String query = \"SELECT id FROM INGEST_TABLE;\";\r\nList&lt;Integer&gt; idList = sfJdbcTemplate.queryForList(query, Integer.class);<\/pre>\n<h2><span class=\"ez-toc-section\" id=\"Data-Ingestion-with-Batch-Insert\"><\/span>Data Ingestion with Batch Insert<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>This method should be rather familiar to users who have already worked with JDBC. It relies on traditional OLTP insert and update methods and uses the JDBC template batchUpdate functionality:<\/p>\n<pre class=\"lang:java decode:true \" title=\"Java Batch Insert\">public void insert(List&lt;SnowflakeTableEntity&gt; entities) {\r\n   String sql = \"INSERT INTO INGEST_TABLE (ID, NAME, PRICE, JSONDATA) SELECT ?, ?, ?, PARSE_JSON(?);\";\r\n\r\n   sfJdbcTemplate.batchUpdate(sql, new BatchPreparedStatementSetter() {\r\n       @Override\r\n       public void setValues(PreparedStatement preparedStatement, int i) throws SQLException, SQLException {\r\n           SnowflakeTableEntity entity = entities.get(i);\r\n           preparedStatement.setInt(1, entity.getId());\r\n           preparedStatement.setString(2, entity.getName());\r\n           preparedStatement.setDouble(3, entity.getPrice());\r\n           preparedStatement.setString(4, entity.getJsonData());\r\n       }\r\n\r\n       @Override\r\n       public int getBatchSize() {\r\n           return entities.size();\r\n       }\r\n   });\r\n}\r\n<\/pre>\n<p>Here, we used Snowflake\u2019s PARSE_JSON function to convert our json string to a variant object.<br \/>\nIf we wanted to upsert entries instead of just inserting them, we would have to determine existing entities and update those, while inserting the non-existing ones.<br \/>\nAlthough the batch insert works theoretically, we\u2019ll see in the Performance Comparison section that one should use the other methods for effective data ingestion.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Data-Ingestion-with-Merge-Insert\"><\/span>Data Ingestion with Merge Insert<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>This solution is based on native Snowflake ingestion methods rather than traditional JDBC methods. Here, we use Snowflake stages for intermediate data storage and then merge these stages into our table. In general, Snowflake stages are intended as gateways to a variety of data sources, such as cloud storage and uploaded data files. Thus, we first create an internal Snowflake stage (alternatively one could use the table stage of our ingest table which Snowflake creates by default, see the <a href=\"https:\/\/docs.snowflake.com\/en\/user-guide\/data-load-local-file-system-create-stage\">Snowflake documentation<\/a>):<\/p>\n<pre class=\"lang:tsql decode:true \" title=\"Snowflake SQL Ingest Stage Creation\">CREATE OR REPLACE STAGE INGEST_STAGE FILE_FORMAT = (TYPE = JSON);<\/pre>\n<p>The Snowflake JDBC driver provides us with an API extension to directly load data into this internal stage from a Java stream:<\/p>\n<pre class=\"lang:java decode:true \" title=\"Java Input Stream Creation\">public void uploadDataToSnowflakeStage(List&lt;SnowflakeTableEntity&gt; entities) throws SQLException {\r\nString json = entities\r\n           .stream()\r\n           .map(SnowflakeTableEntity::toJson)\r\n           .collect(Collectors.joining(\"\\n\"));\r\n\r\n   InputStream inputStream = toInputStream(ByteArrayInputStream(json.getBytes(StandardCharsets.UTF_8)\r\n);\r\n\r\n   String stageName = \"@\".concat(\"INGEST_STAGE\");\r\n   String pathPrefix = LocalDate.now().toString();\r\n   String destinationFile = String.format(\"%s.json\", LocalDateTime.now());\r\n   boolean compressData = true;\r\n\r\n   connection.uploadStream(stageName, pathPrefix, inputStream, destinationFile, compressData);\r\n}\r\n<\/pre>\n<p><span style=\"font-weight: 400;\">Next, we can insert the uploaded data into our table by merging the stage:<\/span><\/p>\n<pre class=\"lang:default decode:true\" title=\"Java Stage Merge\">public void mergeDataIntoTable() {\r\n   String sql = \"\"\"\r\n           MERGE INTO INGEST_TABLE USING\r\n           (SELECT $1:id id, $1:name name, $1:price price, PARSE_JSON($1:jsonData) jsonData FROM @INGEST_STAGE) tempTable\r\n           ON INGEST_TABLE.id = tempTable.id AND INGEST_TABLE.name = INGEST_TABLE.name\r\n           WHEN MATCHED THEN\r\n               UPDATE SET jsondata = tempTable.jsonData\r\n           WHEN NOT MATCHED THEN\r\n               INSERT (id, name, price, jsondata)\r\n               VALUES (tempTable.id, tempTable.name, tempTable.price, tempTable.jsonData);\r\n           \"\"\";\r\n\r\n   jdbcTemplate.execute(sql);\r\n}\r\n<\/pre>\n<p>Note, that there is an integrated mechanism of upserting via the matched and not matched clauses. This is a feature that distinguishes this method from the others and makes it the preferred one if an upsert is necessary in the data model.<br \/>\nOne thing to consider, however, is the file size of the resulting uploaded data file. As the <a href=\"https:\/\/docs.snowflake.com\/en\/user-guide\/data-load-considerations-prepare\">Snowflake documentation<\/a> suggests, this file should be around 100-250 MB. We will experiment with different upload sizes in the Performance Comparison section.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Data-Ingestion-using-Snowflake-Streaming\"><\/span>Data Ingestion using Snowflake Streaming<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>This method promises low-latency data ingestion and is suitable for real-time data streams according to the Snowflake documentation. We can ingest data using the Snowflake streaming API. For this, we first have to create a dedicated Snowflake Channel:<\/p>\n<pre class=\"lang:default decode:true \" title=\"Java Streaming Channel Creation\">Properties props = new Properties();\r\n\r\nprops.put(\"private_key\", readPrivateKeyFile(file));\r\nprops.put(\"user\", user);\r\nprops.put(\"database\", database);\r\nprops.put(\"schema\", schema);\r\nprops.put(\"warehouse\", warehouse);\r\nprops.put(\"role\", role);\r\nprops.put(\"connect_string\", url);\r\nprops.put(\"host\", account);\r\nprops.put(\"scheme\", \"https\");\r\nprops.put(\"ssl\", \"on\");\r\nprops.put(\"port\", 443);\r\n\r\nSnowflakeStreamingIngestClient client = SnowflakeStreamingIngestClientFactory.builder(\"CLIENT\").setProperties(props).build();\r\n\r\nOpenChannelRequest request =\r\n       OpenChannelRequest.builder(channelName)\r\n               .setDBName(database)\r\n               .setSchemaName(schema)\r\n               .setTableName(tableName)\r\n               .setOnErrorOption(\r\n                       OpenChannelRequest.OnErrorOption.CONTINUE)\r\n               .build();\r\n\r\nSnowflakeStreamingIngestChannel channel = client.openChannel(request);\r\n<\/pre>\n<p>Using this channel, we can now ingest data by transforming entities to map objects:<\/p>\n<pre class=\"lang:default decode:true \" title=\"Java Streaming Channel Insert \">public void insert(List&lt;SnowflakeTableEntity&gt; entities) {\r\n   List&lt;Map&lt;String, Object&gt;&gt; rows = entities\r\n           .stream()\r\n           .map(entity -&gt; toRow(entity))\r\n           .toList();\r\n\r\n   InsertValidationResponse response = channel.insertRows(rows, \"offsetToken\");\r\n\r\n   if (response.hasErrors()) {\r\n       throw response.getInsertErrors().get(0).getException();\r\n   }\r\n}\r\n\r\npublic static Map&lt;String, Object&gt; toRow(SnowflakeTableEntity entity) {\r\n   Map&lt;String, Object&gt; row = new HashMap&lt;&gt;();\r\n\r\n   row.put(\"ID\", entity.getId());\r\n   row.put(\"NAME\", entity.getName());\r\n   row.put(\"PRICE\", entity.getPrice());\r\n   row.put(\"JSONDATA\", entity.getJsonData());\r\n\r\n   return row;\r\n}\r\n<\/pre>\n<p>This method does not provide any possibility to update existing data rows, so it is limited to insertion only, which is the major drawback compared to the other methods when considering Snowflake in an OLTP context.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Performance-Comparison\"><\/span>Performance Comparison<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>To evaluate which method is the most suitable for ingesting data into Snowflake with Java, we tested each method by ingesting several 1000 data entries into an empty table with various batch sizes and two different warehouses (xs and s). In the resulting graph, we omit the run times of the batch insert, as it was slower than the other methods almost by a factor of 100!<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-56336\" src=\"https:\/\/www.inovex.de\/wp-content\/uploads\/plot_warehouses_all.png\" alt=\"a graph showing the execution time by insertion method in milliseconds.\" width=\"1000\" height=\"600\" srcset=\"https:\/\/www.inovex.de\/wp-content\/uploads\/plot_warehouses_all.png 1000w, https:\/\/www.inovex.de\/wp-content\/uploads\/plot_warehouses_all-300x180.png 300w, https:\/\/www.inovex.de\/wp-content\/uploads\/plot_warehouses_all-768x461.png 768w, https:\/\/www.inovex.de\/wp-content\/uploads\/plot_warehouses_all-400x240.png 400w, https:\/\/www.inovex.de\/wp-content\/uploads\/plot_warehouses_all-360x216.png 360w\" sizes=\"auto, (max-width: 1000px) 100vw, 1000px\" \/><\/p>\n<p>The other two methods, however, performed comparably well while the stream insert was a little bit faster than one might expect. Also note how the insertion speed decreases with higher batch sizes and increases again, after an optimal batch size is reached. This optimal batch size corresponds to a compressed file size of ~100 MB, which is consistent with the Snowflake documentation. In our case the size of the warehouse did not play a significant role, however this might change for different data schemas or sizes.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Recommendations\"><\/span>Recommendations<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Based on the results of the previous section, we would recommend using Stream insertion wherever possible. As Stream insertion is an append-only method, you can use Merge insertion for cases where you must consider upsertion for entries with unique values. In each case, you should experiment with different batch sizes to determine the optimal one for your use case. Also, try out different warehouse sizes and use the smallest if you do not notice any performance difference.<\/p>\n<p><em><strong>Useful Links:<\/strong><\/em><\/p>\n<ul>\n<li><a href=\"https:\/\/docs.snowflake.com\/en\/developer-guide\/sql-api\/authenticating\">https:\/\/docs.snowflake.com\/en\/developer-guide\/sql-api\/authenticating<\/a><\/li>\n<li><a href=\"https:\/\/docs.snowflake.com\/en\/user-guide\/data-load-local-file-system-create-stage\">https:\/\/docs.snowflake.com\/en\/user-guide\/data-load-local-file-system-create-stage<\/a><\/li>\n<li><a href=\"https:\/\/docs.snowflake.com\/en\/developer-guide\/jdbc\/jdbc-using#label-jdbc-upload-from-stream-to-stage\">https:\/\/docs.snowflake.com\/en\/developer-guide\/jdbc\/jdbc-using#label-jdbc-upload-from-stream-to-stage<\/a><\/li>\n<li><a href=\"https:\/\/docs.snowflake.com\/en\/user-guide\/data-load-snowpipe-streaming-overview\">https:\/\/docs.snowflake.com\/en\/user-guide\/data-load-snowpipe-streaming-overview<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/snowflakedb\/snowflake-ingest-java\/blob\/master\/src\/main\/java\/net\/snowflake\/ingest\/streaming\/example\/SnowflakeStreamingIngestExample.java\">https:\/\/github.com\/snowflakedb\/snowflake-ingest-java\/blob\/master\/src\/main\/java\/net\/snowflake\/ingest\/streaming\/example\/SnowflakeStreamingIngestExample.java<\/a><\/li>\n<li><a href=\"https:\/\/docs.snowflake.com\/en\/user-guide\/data-load-considerations-prepare\">https:\/\/docs.snowflake.com\/en\/user-guide\/data-load-considerations-prepare<\/a><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>Snowflake is becoming increasingly popular as a data platform, and thus the need for ways to integrate Snowflake into production systems is also growing. As type-safe programming languages like Java lead to more stable productive environments and are often preferred over untyped languages (e.g. Python), this implies the challenge to connect Snowflake databases in those [&hellip;]<\/p>\n","protected":false},"author":287,"featured_media":56352,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"inline_featured_image":false,"ep_exclude_from_search":false,"footnotes":""},"tags":[77,71,385,723,1108],"service":[411],"coauthors":[{"id":287,"display_name":"Adrian Westermeier","user_nicename":"awestermeier"}],"class_list":["post-56335","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","tag-big-data","tag-cloud","tag-data-engineering","tag-java","tag-snowflake","service-data-engineering"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.5 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Effective Snowflake Data Ingestion with Java - inovex GmbH<\/title>\n<meta name=\"description\" content=\"We will cover the following native data ingestion methods for Java: Batch and Merge insertion via the Snowflake Streaming API.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.inovex.de\/de\/blog\/effective-snowflake-data-ingestion-with-java\/\" \/>\n<meta property=\"og:locale\" content=\"de_DE\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Effective Snowflake Data Ingestion with Java - inovex GmbH\" \/>\n<meta property=\"og:description\" content=\"We will cover the following native data ingestion methods for Java: Batch and Merge insertion via the Snowflake Streaming API.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.inovex.de\/de\/blog\/effective-snowflake-data-ingestion-with-java\/\" \/>\n<meta property=\"og:site_name\" content=\"inovex GmbH\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/inovexde\" \/>\n<meta property=\"article:published_time\" content=\"2024-07-22T07:47:26+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.inovex.de\/wp-content\/uploads\/Snowflake_Java.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1500\" \/>\n\t<meta property=\"og:image:height\" content=\"880\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Adrian Westermeier\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:image\" content=\"https:\/\/www.inovex.de\/wp-content\/uploads\/Snowflake_Java-1024x601.png\" \/>\n<meta name=\"twitter:creator\" content=\"@inovexgmbh\" \/>\n<meta name=\"twitter:site\" content=\"@inovexgmbh\" \/>\n<meta name=\"twitter:label1\" content=\"Verfasst von\" \/>\n\t<meta name=\"twitter:data1\" content=\"Adrian Westermeier\" \/>\n\t<meta name=\"twitter:label2\" content=\"Gesch\u00e4tzte Lesezeit\" \/>\n\t<meta name=\"twitter:data2\" content=\"9\u00a0Minuten\" \/>\n\t<meta name=\"twitter:label3\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data3\" content=\"Adrian Westermeier\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/effective-snowflake-data-ingestion-with-java\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/effective-snowflake-data-ingestion-with-java\\\/\"},\"author\":{\"name\":\"Adrian Westermeier\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/#\\\/schema\\\/person\\\/f5c1b24448d1a9107694327373ef0884\"},\"headline\":\"Effective Snowflake Data Ingestion with Java\",\"datePublished\":\"2024-07-22T07:47:26+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/effective-snowflake-data-ingestion-with-java\\\/\"},\"wordCount\":1143,\"commentCount\":2,\"publisher\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/#organization\"},\"image\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/effective-snowflake-data-ingestion-with-java\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.inovex.de\\\/wp-content\\\/uploads\\\/Snowflake_Java.png\",\"keywords\":[\"Big Data\",\"Cloud\",\"Data Engineering\",\"Java\",\"Snowflake\"],\"inLanguage\":\"de\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/effective-snowflake-data-ingestion-with-java\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/effective-snowflake-data-ingestion-with-java\\\/\",\"url\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/effective-snowflake-data-ingestion-with-java\\\/\",\"name\":\"Effective Snowflake Data Ingestion with Java - inovex GmbH\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/effective-snowflake-data-ingestion-with-java\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/effective-snowflake-data-ingestion-with-java\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.inovex.de\\\/wp-content\\\/uploads\\\/Snowflake_Java.png\",\"datePublished\":\"2024-07-22T07:47:26+00:00\",\"description\":\"We will cover the following native data ingestion methods for Java: Batch and Merge insertion via the Snowflake Streaming API.\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/effective-snowflake-data-ingestion-with-java\\\/#breadcrumb\"},\"inLanguage\":\"de\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/effective-snowflake-data-ingestion-with-java\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"de\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/effective-snowflake-data-ingestion-with-java\\\/#primaryimage\",\"url\":\"https:\\\/\\\/www.inovex.de\\\/wp-content\\\/uploads\\\/Snowflake_Java.png\",\"contentUrl\":\"https:\\\/\\\/www.inovex.de\\\/wp-content\\\/uploads\\\/Snowflake_Java.png\",\"width\":1500,\"height\":880,\"caption\":\"Grafik einer Data Platform (Snowflake), die an einen Monitor (Java) angebunden ist.\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/effective-snowflake-data-ingestion-with-java\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Effective Snowflake Data Ingestion with Java\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/#website\",\"url\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/\",\"name\":\"inovex GmbH\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"de\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/#organization\",\"name\":\"inovex GmbH\",\"url\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"de\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/www.inovex.de\\\/wp-content\\\/uploads\\\/2021\\\/03\\\/inovex-logo-16-9-1.png\",\"contentUrl\":\"https:\\\/\\\/www.inovex.de\\\/wp-content\\\/uploads\\\/2021\\\/03\\\/inovex-logo-16-9-1.png\",\"width\":1921,\"height\":1081,\"caption\":\"inovex GmbH\"},\"image\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/#\\\/schema\\\/logo\\\/image\\\/\"},\"sameAs\":[\"https:\\\/\\\/www.facebook.com\\\/inovexde\",\"https:\\\/\\\/x.com\\\/inovexgmbh\",\"https:\\\/\\\/www.instagram.com\\\/inovexlife\\\/\",\"https:\\\/\\\/www.linkedin.com\\\/company\\\/inovex\",\"https:\\\/\\\/www.youtube.com\\\/channel\\\/UC7r66GT14hROB_RQsQBAQUQ\"]},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/#\\\/schema\\\/person\\\/f5c1b24448d1a9107694327373ef0884\",\"name\":\"Adrian Westermeier\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"de\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/wp-content\\\/uploads\\\/Bewerbungsbild-96x96.png56de6a0de06c3e036720d584bc3842fd\",\"url\":\"https:\\\/\\\/www.inovex.de\\\/wp-content\\\/uploads\\\/Bewerbungsbild-96x96.png\",\"contentUrl\":\"https:\\\/\\\/www.inovex.de\\\/wp-content\\\/uploads\\\/Bewerbungsbild-96x96.png\",\"caption\":\"Adrian Westermeier\"},\"description\":\"Hi, my name is Adrian and I am working as a Data Scientist at inovex. I am interested in the newest Deep Learning trends, especially in NLP and Reinforcement Learning where I wrote my Bachelor's and Master's thesis.\",\"url\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/author\\\/awestermeier\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Effective Snowflake Data Ingestion with Java - inovex GmbH","description":"We will cover the following native data ingestion methods for Java: Batch and Merge insertion via the Snowflake Streaming API.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.inovex.de\/de\/blog\/effective-snowflake-data-ingestion-with-java\/","og_locale":"de_DE","og_type":"article","og_title":"Effective Snowflake Data Ingestion with Java - inovex GmbH","og_description":"We will cover the following native data ingestion methods for Java: Batch and Merge insertion via the Snowflake Streaming API.","og_url":"https:\/\/www.inovex.de\/de\/blog\/effective-snowflake-data-ingestion-with-java\/","og_site_name":"inovex GmbH","article_publisher":"https:\/\/www.facebook.com\/inovexde","article_published_time":"2024-07-22T07:47:26+00:00","og_image":[{"width":1500,"height":880,"url":"https:\/\/www.inovex.de\/wp-content\/uploads\/Snowflake_Java.png","type":"image\/png"}],"author":"Adrian Westermeier","twitter_card":"summary_large_image","twitter_image":"https:\/\/www.inovex.de\/wp-content\/uploads\/Snowflake_Java-1024x601.png","twitter_creator":"@inovexgmbh","twitter_site":"@inovexgmbh","twitter_misc":{"Verfasst von":"Adrian Westermeier","Gesch\u00e4tzte Lesezeit":"9\u00a0Minuten","Written by":"Adrian Westermeier"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.inovex.de\/de\/blog\/effective-snowflake-data-ingestion-with-java\/#article","isPartOf":{"@id":"https:\/\/www.inovex.de\/de\/blog\/effective-snowflake-data-ingestion-with-java\/"},"author":{"name":"Adrian Westermeier","@id":"https:\/\/www.inovex.de\/de\/#\/schema\/person\/f5c1b24448d1a9107694327373ef0884"},"headline":"Effective Snowflake Data Ingestion with Java","datePublished":"2024-07-22T07:47:26+00:00","mainEntityOfPage":{"@id":"https:\/\/www.inovex.de\/de\/blog\/effective-snowflake-data-ingestion-with-java\/"},"wordCount":1143,"commentCount":2,"publisher":{"@id":"https:\/\/www.inovex.de\/de\/#organization"},"image":{"@id":"https:\/\/www.inovex.de\/de\/blog\/effective-snowflake-data-ingestion-with-java\/#primaryimage"},"thumbnailUrl":"https:\/\/www.inovex.de\/wp-content\/uploads\/Snowflake_Java.png","keywords":["Big Data","Cloud","Data Engineering","Java","Snowflake"],"inLanguage":"de","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.inovex.de\/de\/blog\/effective-snowflake-data-ingestion-with-java\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.inovex.de\/de\/blog\/effective-snowflake-data-ingestion-with-java\/","url":"https:\/\/www.inovex.de\/de\/blog\/effective-snowflake-data-ingestion-with-java\/","name":"Effective Snowflake Data Ingestion with Java - inovex GmbH","isPartOf":{"@id":"https:\/\/www.inovex.de\/de\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.inovex.de\/de\/blog\/effective-snowflake-data-ingestion-with-java\/#primaryimage"},"image":{"@id":"https:\/\/www.inovex.de\/de\/blog\/effective-snowflake-data-ingestion-with-java\/#primaryimage"},"thumbnailUrl":"https:\/\/www.inovex.de\/wp-content\/uploads\/Snowflake_Java.png","datePublished":"2024-07-22T07:47:26+00:00","description":"We will cover the following native data ingestion methods for Java: Batch and Merge insertion via the Snowflake Streaming API.","breadcrumb":{"@id":"https:\/\/www.inovex.de\/de\/blog\/effective-snowflake-data-ingestion-with-java\/#breadcrumb"},"inLanguage":"de","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.inovex.de\/de\/blog\/effective-snowflake-data-ingestion-with-java\/"]}]},{"@type":"ImageObject","inLanguage":"de","@id":"https:\/\/www.inovex.de\/de\/blog\/effective-snowflake-data-ingestion-with-java\/#primaryimage","url":"https:\/\/www.inovex.de\/wp-content\/uploads\/Snowflake_Java.png","contentUrl":"https:\/\/www.inovex.de\/wp-content\/uploads\/Snowflake_Java.png","width":1500,"height":880,"caption":"Grafik einer Data Platform (Snowflake), die an einen Monitor (Java) angebunden ist."},{"@type":"BreadcrumbList","@id":"https:\/\/www.inovex.de\/de\/blog\/effective-snowflake-data-ingestion-with-java\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.inovex.de\/de\/"},{"@type":"ListItem","position":2,"name":"Effective Snowflake Data Ingestion with Java"}]},{"@type":"WebSite","@id":"https:\/\/www.inovex.de\/de\/#website","url":"https:\/\/www.inovex.de\/de\/","name":"inovex GmbH","description":"","publisher":{"@id":"https:\/\/www.inovex.de\/de\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.inovex.de\/de\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"de"},{"@type":"Organization","@id":"https:\/\/www.inovex.de\/de\/#organization","name":"inovex GmbH","url":"https:\/\/www.inovex.de\/de\/","logo":{"@type":"ImageObject","inLanguage":"de","@id":"https:\/\/www.inovex.de\/de\/#\/schema\/logo\/image\/","url":"https:\/\/www.inovex.de\/wp-content\/uploads\/2021\/03\/inovex-logo-16-9-1.png","contentUrl":"https:\/\/www.inovex.de\/wp-content\/uploads\/2021\/03\/inovex-logo-16-9-1.png","width":1921,"height":1081,"caption":"inovex GmbH"},"image":{"@id":"https:\/\/www.inovex.de\/de\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/inovexde","https:\/\/x.com\/inovexgmbh","https:\/\/www.instagram.com\/inovexlife\/","https:\/\/www.linkedin.com\/company\/inovex","https:\/\/www.youtube.com\/channel\/UC7r66GT14hROB_RQsQBAQUQ"]},{"@type":"Person","@id":"https:\/\/www.inovex.de\/de\/#\/schema\/person\/f5c1b24448d1a9107694327373ef0884","name":"Adrian Westermeier","image":{"@type":"ImageObject","inLanguage":"de","@id":"https:\/\/www.inovex.de\/wp-content\/uploads\/Bewerbungsbild-96x96.png56de6a0de06c3e036720d584bc3842fd","url":"https:\/\/www.inovex.de\/wp-content\/uploads\/Bewerbungsbild-96x96.png","contentUrl":"https:\/\/www.inovex.de\/wp-content\/uploads\/Bewerbungsbild-96x96.png","caption":"Adrian Westermeier"},"description":"Hi, my name is Adrian and I am working as a Data Scientist at inovex. I am interested in the newest Deep Learning trends, especially in NLP and Reinforcement Learning where I wrote my Bachelor's and Master's thesis.","url":"https:\/\/www.inovex.de\/de\/blog\/author\/awestermeier\/"}]}},"_links":{"self":[{"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/posts\/56335","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/users\/287"}],"replies":[{"embeddable":true,"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/comments?post=56335"}],"version-history":[{"count":5,"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/posts\/56335\/revisions"}],"predecessor-version":[{"id":56474,"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/posts\/56335\/revisions\/56474"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/media\/56352"}],"wp:attachment":[{"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/media?parent=56335"}],"wp:term":[{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/tags?post=56335"},{"taxonomy":"service","embeddable":true,"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/service?post=56335"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/coauthors?post=56335"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}