{"id":51496,"date":"2024-02-22T09:04:45","date_gmt":"2024-02-22T08:04:45","guid":{"rendered":"https:\/\/www.inovex.de\/?p=51496"},"modified":"2024-05-23T11:33:08","modified_gmt":"2024-05-23T09:33:08","slug":"updating-embedded-systems-via-usb-yocto-mender","status":"publish","type":"post","link":"https:\/\/www.inovex.de\/de\/blog\/updating-embedded-systems-via-usb-yocto-mender\/","title":{"rendered":"Updating Embedded Systems via USB"},"content":{"rendered":"<p>When it comes to managing and updating embedded and IoT devices, over-the-air system updates (OTAs) have become the gold standard over the years. While that holds true, some situations and use cases may require an alternative, stand-alone solution for robust updates.<!--more--><\/p>\n<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_79_2 counter-hierarchy ez-toc-counter ez-toc-custom ez-toc-container-direction\">\n<div class=\"ez-toc-title-container\"><p class=\"ez-toc-title\" style=\"cursor:inherit\"><\/p>\n<\/div><nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/www.inovex.de\/de\/blog\/updating-embedded-systems-via-usb-yocto-mender\/#USB-Updates-%E2%80%93-When-and-Why-to-Consider\" >USB Updates \u2013 When and Why to Consider?<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/www.inovex.de\/de\/blog\/updating-embedded-systems-via-usb-yocto-mender\/#The-update-mechanism-%E2%80%93-Which-factors-are-decisive\" >The update mechanism \u2013 Which factors are decisive?<\/a><ul class='ez-toc-list-level-3' ><li class='ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/www.inovex.de\/de\/blog\/updating-embedded-systems-via-usb-yocto-mender\/#Failure-resilience-and-roll-back-mode\" >Failure resilience and roll-back mode<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/www.inovex.de\/de\/blog\/updating-embedded-systems-via-usb-yocto-mender\/#Security\" >Security<\/a><ul class='ez-toc-list-level-4' ><li class='ez-toc-heading-level-4'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/www.inovex.de\/de\/blog\/updating-embedded-systems-via-usb-yocto-mender\/#Signing\" >Signing<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-4'><a class=\"ez-toc-link ez-toc-heading-6\" href=\"https:\/\/www.inovex.de\/de\/blog\/updating-embedded-systems-via-usb-yocto-mender\/#Encryption\" >Encryption<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-4'><a class=\"ez-toc-link ez-toc-heading-7\" href=\"https:\/\/www.inovex.de\/de\/blog\/updating-embedded-systems-via-usb-yocto-mender\/#Key-Management\" >Key Management<\/a><\/li><\/ul><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-8\" href=\"https:\/\/www.inovex.de\/de\/blog\/updating-embedded-systems-via-usb-yocto-mender\/#Granularity\" >Granularity<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-9\" href=\"https:\/\/www.inovex.de\/de\/blog\/updating-embedded-systems-via-usb-yocto-mender\/#Server-side\" >Server-side<\/a><\/li><\/ul><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-10\" href=\"https:\/\/www.inovex.de\/de\/blog\/updating-embedded-systems-via-usb-yocto-mender\/#Hands-on-USB-updates-with-Yocto-Mender-and-systemd\" >Hands-on: USB updates with Yocto, Mender and systemd<\/a><ul class='ez-toc-list-level-3' ><li class='ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-11\" href=\"https:\/\/www.inovex.de\/de\/blog\/updating-embedded-systems-via-usb-yocto-mender\/#Prerequisites\" >Prerequisites<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-12\" href=\"https:\/\/www.inovex.de\/de\/blog\/updating-embedded-systems-via-usb-yocto-mender\/#When-should-the-update-be-installed\" >When should the update be installed<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-13\" href=\"https:\/\/www.inovex.de\/de\/blog\/updating-embedded-systems-via-usb-yocto-mender\/#Preparing-and-signing-the-update\" >Preparing and signing the update<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-14\" href=\"https:\/\/www.inovex.de\/de\/blog\/updating-embedded-systems-via-usb-yocto-mender\/#Optional-Encrypt-the-update\" >Optional: Encrypt the update<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-15\" href=\"https:\/\/www.inovex.de\/de\/blog\/updating-embedded-systems-via-usb-yocto-mender\/#Installing\" >Installing<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-16\" href=\"https:\/\/www.inovex.de\/de\/blog\/updating-embedded-systems-via-usb-yocto-mender\/#udev-and-systemd-magic\" >udev and systemd magic<\/a><ul class='ez-toc-list-level-4' ><li class='ez-toc-heading-level-4'><a class=\"ez-toc-link ez-toc-heading-17\" href=\"https:\/\/www.inovex.de\/de\/blog\/updating-embedded-systems-via-usb-yocto-mender\/#Introducing-systemdmount\" >Introducing systemd.mount<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-4'><a class=\"ez-toc-link ez-toc-heading-18\" href=\"https:\/\/www.inovex.de\/de\/blog\/updating-embedded-systems-via-usb-yocto-mender\/#The-systemd-service\" >The systemd service<\/a><\/li><\/ul><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-19\" href=\"https:\/\/www.inovex.de\/de\/blog\/updating-embedded-systems-via-usb-yocto-mender\/#Debugging\" >Debugging<\/a><\/li><\/ul><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-20\" href=\"https:\/\/www.inovex.de\/de\/blog\/updating-embedded-systems-via-usb-yocto-mender\/#Conclusion\" >Conclusion<\/a><\/li><\/ul><\/nav><\/div>\n<h2><span class=\"ez-toc-section\" id=\"USB-Updates-%E2%80%93-When-and-Why-to-Consider\"><\/span>USB Updates \u2013 When and Why to Consider?<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>It is not an easy task to keep the logistics of OTAs sorted. There is an update server that needs to be up-to-date and free from security issues to avoid the whole update chain becoming compromised. You also need mechanisms to manage which updates are rolled out to certain devices, e.g., to realize rolling out a new software in phases or to update just a few devices to allow additional paid features. But depending on the actual use case and the situation, this server part and the associated effort are not really needed.<br \/>\nIn an early product stage, for example, it is definitely necessary to think about updates and prepare the software to allow them in general, e.g., to ship updates to early customers quickly but also to comply with legal regulations.<br \/>\nNevertheless, implementing and maintaining a full-blown over-the-air update solution is not necessarily appropriate in this situation.<\/p>\n<p>Instead, enabling system updates via USB could be a viable solution. This allows updating the system in general without the need for an update server and caring for the infrastructure in the first place. But offers the possibility to change to the over-the-air approach later \u2013 via update.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"The-update-mechanism-%E2%80%93-Which-factors-are-decisive\"><\/span>The update mechanism \u2013 Which factors are decisive?<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>When building embedded and IoT devices, using an embedded build system is highly recommended. The Yocto project, as a de-facto industry-leading solution to do so, has a pretty good assortment of supported mechanisms. Also, they provide a recommendable<a href=\"https:\/\/wiki.yoctoproject.org\/wiki\/System_Update\"> comparison<\/a> between them. Nevertheless, most of them can be used in the competing build system buildroot or custom-tailored contexts as well.<\/p>\n<p>Likewise, all listed systems \u2013<a href=\"https:\/\/swupdate.org\/\"> swupdate<\/a>,<a href=\"https:\/\/mender.io\/\"> Mender<\/a>,<a href=\"https:\/\/github.com\/ostreedev\/ostree\"> OSTree<\/a> and<a href=\"https:\/\/rauc.io\/\"> RAUC<\/a> \u2013 allow updating from local files, and therefore updating via USB, even if it is mostly not the default case the update systems are designed for.<\/p>\n<p>When deciding on one system, key points to consider are:<\/p>\n<h3><span class=\"ez-toc-section\" id=\"Failure-resilience-and-roll-back-mode\"><\/span>Failure resilience and roll-back mode<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>In my opinion, modern systems should utilize A\/B updates, which means there are two full-blown system partitions on the device. A, for example, is active and in use when an update arrives. The update is installed on partition B. When rebooting, the bootloader switches to B as an active partition. Once the device comes up, you can run a short self-test to check everything works as expected and mark the update as successful. If something goes wrong in this process and the update is not confirmed within a certain time or does not even boot, the whole system is rebooted and switched back to partition A.<\/p>\n<p>As this approach uses two system partitions, it requires more disk space. But in return, it is pretty robust and pleasant for users. The savings from building a system with less memory can not make up for the inconvenience and maintenance cost of using the older system\/recovery implementation or similar approach anymore. In this case, there\u2019s only one fully operational system partition. If the update fails, the device is stuck in the minimal, not operational, recovery system and restoring it requires manual actions from a user or engineer.<\/p>\n<h3><span class=\"ez-toc-section\" id=\"Security\"><\/span>Security<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>System updates are probably one of the most rewarding attack surfaces on embedded systems. By their nature as self-contained devices with limited user interaction, the attack surface is significantly limited compared to a standard computer. But system updates are an attractive target for two distinct reasons:<\/p>\n<ol>\n<li aria-level=\"1\">If the device itself is well secured, examining the update file or blob is a less complex target. There is no need for advanced tooling to hack the hardware, e.g., to read out the eMMC storage. It is not even necessary to own a device to attack it. On the other hand, a full system update may contain intellectual property (IP) or credentials, certificates and similar items that allow for the compromise of devices in the field. For example, the actual software within a device that does sophisticated data processing and really adds value to the product.<\/li>\n<li aria-level=\"1\">An update mechanism that does not check if an update is signed correctly or has leaked certificates allows attackers to distribute compromised software. As a result, devices may become unusable at all, compromised or a botnet. Depending on the level of such a scenario, called a supply chain attack, it is nearly impossible to restore a trustworthy state.<\/li>\n<\/ol>\n<h4><span class=\"ez-toc-section\" id=\"Signing\"><\/span>Signing<span class=\"ez-toc-section-end\"><\/span><\/h4>\n<p>The second point, namely verifying the update file has a proper signature, is an absolute must-have for an embedded update system. By using signed updates, the update mechanism can check if it is actually a valid update from the right source. An update must only be installed if the signature is correct. That is supported by all the mechanisms mentioned above.<\/p>\n<p>Additionally, proper key management is needed on the manufacturers side in order to avoid supply chain attacks, for example someone stole the signing keys and acts as a legit update provider. This also involves being able to revoke keys on devices in the field.<\/p>\n<h4><span class=\"ez-toc-section\" id=\"Encryption\"><\/span>Encryption<span class=\"ez-toc-section-end\"><\/span><\/h4>\n<p>When it comes to USB updates, the first point in particular becomes more important to consider. All discussed update mechanisms on the Yocto site are built for over-the-air updates in the first place. Uploading the update to a trustworthy server and distributing it to the actual devices via an authenticated and encrypted HTTPS connection is a different scenario than USB. Although all of them support a local update file case in general (and thus USB), additional security efforts needed, for example, encrypting the update file, are mostly up to the developers using the mechanism.<br \/>\nBut in the end, whether encrypting your update is actually needed depends on the very specific threat scenario of a product and whether the software rather than the hardware adds the business value to be protected.<\/p>\n<h4><span class=\"ez-toc-section\" id=\"Key-Management\"><\/span>Key Management<span class=\"ez-toc-section-end\"><\/span><\/h4>\n<p>Key management is, as mentioned, one often neglected, but important topic when it comes to embedded and IoT devices. This involves handling signing and encryption keys on the manufacturer side as well as securely storing and handling verifying and decryption keys on the devices itself.<br \/>\nBest practice is using distinct hardware security modules (HSM) to store keys and execute cryptographic operations. The specialized modules are not only faster and more secure, they are also protected against side-channel attacks, e.g., restoring keys by analyzing the power consumption of the CPU or the electromagnetic fields.<\/p>\n<h3><span class=\"ez-toc-section\" id=\"Granularity\"><\/span>Granularity<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>The granularity at which a system can be updated could be a crucial factor when deciding on an update mechanism, too. While all the named ones support the system image use case, some can go further and allow updating system components, e.g., containers or files, only. If this is relevant to you, take a closer look at the options, as they differ in possible components, underlying mechanisms and guarantees when it comes to failures.<\/p>\n<h3><span class=\"ez-toc-section\" id=\"Server-side\"><\/span>Server-side<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>As this post targets standalone system updates via USB, the server-side of the update mechanism should not be the focus. Nevertheless, being able to switch to over-the-air updates if necessary is an important factor. For this reason, considering the server side right from the beginning is mandatory from my perspective.<\/p>\n<p>There are different options for update servers, for example the<a href=\"https:\/\/mender.io\/how-it-works\"> Mender management server<\/a> or<a href=\"https:\/\/github.com\/eclipse\/hawkbit\"> Eclipse hawkBit<\/a>. It should not be overlooked that operating and maintaining an update server is plenty of work, which is why USB updates may be a solution when starting with a limited number of devices. If you prefer the over-the-air approach anyway, maybe consider a managed Mender server.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Hands-on-USB-updates-with-Yocto-Mender-and-systemd\"><\/span>Hands-on: USB updates with Yocto, Mender and systemd<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<h3><span class=\"ez-toc-section\" id=\"Prerequisites\"><\/span>Prerequisites<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>To avoid losing focus, we assume having a Yocto system with the Mender client integrated, a proper setup bootloader which is capable to switch between both Mender partitions, and systemd as the init system ready for this post.<\/p>\n<p>If you need help with this, get in touch!<\/p>\n<h3><span class=\"ez-toc-section\" id=\"When-should-the-update-be-installed\"><\/span>When should the update be installed<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>It is not always a good time to install updates, in particular when rolling out over-the-air updates. Depending on the usage scenario of a device, updating at an arbitrary time without user consent could lead to annoyed users or serious damage in the worst case.<br \/>\nWhen updating via USB, it is a bit different. As this process requires user interaction anyway \u2013 a user or engineer needs to acquire a USB stick with the update or flash it on a stick and put it into the device \u2013 we can assume it is a safe and appropriate time once the USB stick is attached. This allows sophisticated and elegant solutions utilizing the udev mechanism.<\/p>\n<p>On Linux, the udev device manager supervises and controls hot-plugging devices. Using udev rules, we can script what\u2019s happening when a USB device is attached, e.g., starting the update process.<\/p>\n<h3><span class=\"ez-toc-section\" id=\"Preparing-and-signing-the-update\"><\/span>Preparing and signing the update<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>When presuming to have a Yocto with Mender ready to go, a .mender update file should already be created as a build artifact.<br \/>\nNow, it is needed to create a set of keys to sign and verify the update with <a href=\"https:\/\/www.openssl.org\/\">OpenSSL<\/a>. Today, using an ECDSA key with curve P-256 is <a href=\"https:\/\/www.feistyduck.com\/library\/openssl-cookbook\/online\/openssl-command-line\/key-generation.html\">recommended<\/a> and can be generated as follows:<\/p>\n<pre># Generate a private key and params\r\nopenssl ecparam -genkey -name prime256v1 -out private-and-params.key\r\n\r\n# Derive a private key file\r\nopenssl ec -in private-and-params.key -out private.key\r\n\r\n# Derive a public key file\r\nopenssl ec -in private-and-params.key -pubout -out public.key<\/pre>\n<p>The private key should be protected and stored in a safe location while the public key is installed in the Yocto system. Rename it to artifact-verify-key.pem and install it, preferably by extending the mender-client recipe in your own layer with a mender-client_%.bbappend file, e.g. containing<\/p>\n<pre>FILESEXTRAPATHS:prepend := \"${THISDIR}\/files:\"\r\n\r\nSRC_URI:append = \" file:\/\/artifact-verify-key.pem\"<\/pre>\n<p>If everything is in place, sign the .mender artifact using the <a href=\"https:\/\/docs.mender.io\/downloads#mender-artifact\">mender-artifact utility<\/a>:<\/p>\n<pre>mender-artifact sign artifact.mender -k private.key -o artifact-signed.mender<\/pre>\n<p>Although it is possible to sign the artifact as a Yocto build step, it is not recommended from a security perspective.<\/p>\n<h3><span class=\"ez-toc-section\" id=\"Optional-Encrypt-the-update\"><\/span>Optional: Encrypt the update<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>As mentioned before, updating via USB exposes the update data and makes it an easy target. Thus, I would recommend encrypting the update file if your chosen update mechanism does not do this out of the box, depending on your threat scenario.<br \/>\nThere are several options to solve this. Without non-standard tools needed on the board, using OpenSSL is my way to go, and as a best practice, working with asymmetric key encryption, e.g., RSA.<br \/>\nHowever, RSA does not support large files, so it is a common approach to encrypt the actual large file, in this case the update, with symmetric key cryptography using a one-time password that is encrypted itself using the asymmetric RSA and packed together with the actual update. If you want to know more about this, you may take a look at<a href=\"https:\/\/www.czeskis.com\/random\/openssl-encrypt-file.html\"> this<\/a> blog post, for example, as the OpenSSL documentation is rather confusing.<br \/>\nAs a result, you should have created yourself some scripts or helper applications for encrypting and decrypting the update and have them, together with the keys, in place.<\/p>\n<p>Reminder: Using a hardware security module (HSM) is the gold standard when it comes to proper on-device key management. As there are several HSM implementations, using them cannot be covered in this article.<\/p>\n<h3><span class=\"ez-toc-section\" id=\"Installing\"><\/span>Installing<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>Once the update is signed, maybe encrypted and all keys are in place, we need to make sure it gets installed.<br \/>\nUsing the Mender client, installing a local update is easy. It is just<\/p>\n<pre class=\"decode-attributes:false lang:default decode:true \">\/usr\/bin\/mender install &lt;path\/to\/update&gt;<\/pre>\n<p>So we can create a small script that searches for the update file on a known mount path and handles the update. The example below shows a basic variant:<\/p>\n<pre class=\"lang:default decode:true \">#!\/bin\/bash\r\n\r\nUPDATED=\"0\"\r\nSEARCH_PATH=\"\/media\"\r\nSEARCH_ARG=\"${SEARCH_PATH}\/*.mender.tar.gz\"\r\nKEY_PATH=\"\/path\/to\/decrypt-key.pem\"\r\n\r\nif test -f ${SEARCH_ARG}; then\r\n       echo \"Update file found!\"\r\n       UPDATE_PATH=$(realpath ${SEARCH_ARG})\r\n       MENDER_FILE=$(basename ${UPDATE_PATH} .tar.gz)\r\n\r\n       \/usr\/bin\/decrypt-mender-update.sh ${UPDATE_PATH} ${KEY_PATH}\r\n\r\n       \/usr\/bin\/mender install ${SEARCH_PATH}\/${MENDER_FILE}\r\n       STATUS=$?\r\n\r\n       if [ ${STATUS} -eq 0 ]; then\r\n              echo \"Mender install was successful! Rebooting ...\"\r\n              UPDATED=\"1\"\r\n       else\r\n              echo \"Installing the update failed!\"\r\n       fi\r\n\r\n       rm &lt;decrypted\/update\/file&gt;\r\nfi\r\n\r\nMOUNT_PATH=${SEARCH_PATH%.*}\r\necho ${MOUNT_PATH}\r\numount ${MOUNT_PATH}\r\n\r\nif [ ${UPDATED} -eq 1 ]; then\r\n       reboot\r\nfi<\/pre>\n<p>The script assumes the USB stick is already mounted in a certain location; we will later see how this is done.<br \/>\nBesides, it is rather straight-forward. If a file with the expected extension is found, the update is decrypted using a dedicated script, as mentioned above. Afterward, the decrypted file is passed to the Mender client. Then Mender does the actual updating magic. It checks if the update file is signed and the signature matches the expected one on the device \u2013 only in this case does the update get installed.<br \/>\nNext, the decrypted update file is deleted again to avoid leaking. Let\u2019s have a short digression on the reasons:<\/p>\n<p>To avoid another attack vector, careful handling of the decrypted update file is necessary. If it is temporarily stored on the USB drive itself, there is a potential risk that an attacker will remove the drive during the update installation. This would allow easy access to the system, including certificates or application codes with intellectual property. To avoid this, one possibility is storing the decrypted update at a location that is considered safe but writable, e.g., an encrypted data partition. Nevertheless, the decrypted update file should be deleted once it is no longer needed or not written on disk at all if possible.<\/p>\n<p>Finally, the script monitors the return code of Mender and reboots the device upon success.<\/p>\n<h3><span class=\"ez-toc-section\" id=\"udev-and-systemd-magic\"><\/span>udev and systemd magic<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>When thinking about udev and researching how to write an udev rule, your first idea will probably be utilizing RUN+= and calling the mender client or the script right off &#8211; at least it was mine. But reading the<a href=\"https:\/\/www.freedesktop.org\/software\/systemd\/man\/latest\/udev.html\"> docs<\/a> points to two reasons we cannot simply utilize RUN.<\/p>\n<p>1. It is only allowed for very short-running foreground tasks. Longer tasks will block the device itself and dependent devices. Installing the update takes some time, so it\u2019s not applicable.<\/p>\n<p>2. A program started by RUN is not allowed to access the network or mount\/unmount file systems. As the script should unmount the USB stick when finished, there\u2019s a second reason why this idea won\u2019t work.<\/p>\n<p>But the documentation points to the correct solution:<\/p>\n<p>Starting daemons or other long-running processes is not allowed; the forked processes, detached or not, will be unconditionally killed after the event handling has finished. In order to activate long-running processes from udev rules, provide a service unit and pull it in from an udev device using the SYSTEMD_WANTS device property. See<a href=\"https:\/\/www.freedesktop.org\/software\/systemd\/man\/latest\/systemd.device.html#\"> systemd.device(5)<\/a> for details.<\/p>\n<h4><span class=\"ez-toc-section\" id=\"Introducing-systemdmount\"><\/span>Introducing systemd.mount<span class=\"ez-toc-section-end\"><\/span><\/h4>\n<p>Simply starting a systemd service will not work either, but utilizing the systemd.mount mechanism together with SYSTEMD_WANTS does.<br \/>\nThe idea is quite sophisticated.<br \/>\nFirst, the udev rule mounts the USB device to a certain location and sets the variable ENV{SYSTEMD_WANTS}+= to a distinct systemd service file, which calls the actual update script.<br \/>\nAfterwards, the systemd service uses<a href=\"https:\/\/www.freedesktop.org\/software\/systemd\/man\/latest\/systemd.mount.html\"> systemd.mount<\/a> to get started. systemd-mount, which is used within the udev rule to mount the drive, also creates a .mount unit configuration file containing information about the mount point.<\/p>\n<p>systemd checks automatically for implicit dependencies of services (in systemd units) with, e.g., AFTER=&lt;mountpoint&gt;.mount and starts them once the matching configuration file is created.<\/p>\n<p>To get the assignment between mount point and service right, we need to pay attention to the naming scheme.<\/p>\n<p>The script above assumed \/media as the mount path and searched there for the update file.<br \/>\nThus, we need to mount an attached USB drive to \/media within the udev rule first.<br \/>\nAn example rule could look like this:<\/p>\n<pre>ACTION==\"add\", SUBSYSTEMS==\"usb\", SUBSYSTEM==\"block\", \\\r\nENV{ID_FS_USAGE}==\"filesystem\", \\\r\nENV{SYSTEMD_WANTS}+=\"mender-usb-update.service\", \\\r\nRUN{program}+=\"\/usr\/bin\/systemd-mount --no-block --automount=yes --collect $devnode \/media\"<\/pre>\n<p>It basically tells udev if there is a USB block device (aka a USB drive) added, sets the usage ID to file system, and adds a systemd WANTS dependency to the service to be run.<\/p>\n<p>Lastly, it mounts the drive using systemd-mount instead of legacy mount (recommended, see<a href=\"https:\/\/github.com\/systemd\/systemd\/issues\/11982#issuecomment-472335224\"> more<\/a>) to the mount point \/media.<\/p>\n<p>In this case, the naming scheme to recognize the mount point as a systemd.mount configuration file is simple. It is just media.mount.<br \/>\nOn larger paths, the \/ is replaced by &#8211; in the naming scheme: \/home\/anna\/usb would, for example, result in the home-anna-usb.mount configuration file.<\/p>\n<h4><span class=\"ez-toc-section\" id=\"The-systemd-service\"><\/span>The systemd service<span class=\"ez-toc-section-end\"><\/span><\/h4>\n<p>At last, we\u2019ll take a look at the service definition as a counterpart to the udev rule to actually start our update script:<\/p>\n<pre>[Unit]\r\nDescription=Handles Mender USB Updates\r\nRequires=media.mount\r\nAfter=media.mount\r\n\r\n[Service]\r\nExecStart=\/usr\/bin\/install-usb-update.sh\r\n\r\n[Install]\r\nWantedBy=media.mount<\/pre>\n<p>It is quite basic and simple. The systemd service binds to the media.mount configuration file, as already mentioned, and its only task is to start our update script.<br \/>\nThat is it! Our update should run now!<br \/>\n(If encryption and signing are in place and nothing gets mixed up.)<\/p>\n<h3><span class=\"ez-toc-section\" id=\"Debugging\"><\/span>Debugging<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>The shown approach, utilizing systemd and udev is sophisticated, solid and does not involve much code. But due to its massive dependency on systemd, understanding what is going on under the hood and investigating bugs becomes more difficult.<br \/>\nA helpful starting point in case of hiccups are systemctl status and journalctl.<br \/>\nThe first mentioned prints out an overview of a service&#8217;s status and some logs from execution.<\/p>\n<p>For full logs, journalctl is the tool of choice.<\/p>\n<p>Use<\/p>\n<pre>systemctl status systemd-udevd.service -l<\/pre>\n<p>or<\/p>\n<pre>journalctl -u systemd-udevd.service -f<\/pre>\n<p>to query the systemd-udev service and check if the rule is triggered correctly.<br \/>\nCalling the update script directly in the udev\u2019s rule RUN command would lead to an error visible in these logs, for example.<\/p>\n<p>To inspect the update service or the script called, you can use these tools with the name of your systemd update service.<br \/>\nThe -f option with journalctl keeps the log output open and waits for new incoming messages, which is great for monitoring the installation process with Mender.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Conclusion\"><\/span>Conclusion<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>While over-the-air updates are the state-of-the-art, some situations and use cases may require an alternative, stand-alone solution for robust updates. The most update mechanisms for embedded systems consider and offer an easy solution for this application. This blog post has demonstrated one exemplary way to implement it. This is not the perfect solution for any use case, it is rather a sophisticated approach with minimal interaction required. But it is a starting point that can be adjusted to several needs. Maybe you want an additional button to start the update or a web UI to upload an update file to the device instead of a managed update server? Feel free to contact us if you need help updating your embedded system. There\u2019s no reason to not update!<\/p>\n","protected":false},"excerpt":{"rendered":"<p>When it comes to managing and updating embedded and IoT devices, over-the-air system updates (OTAs) have become the gold standard over the years. While that holds true, some situations and use cases may require an alternative, stand-alone solution for robust updates.<\/p>\n","protected":false},"author":35,"featured_media":51890,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"inline_featured_image":false,"ep_exclude_from_search":false,"footnotes":""},"tags":[1147,585,74],"service":[505,712],"coauthors":[{"id":35,"display_name":"Anna-Lena Marx","user_nicename":"amarx"}],"class_list":["post-51496","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","tag-edge-embedded","tag-embedded-systems","tag-iot","service-embedded-systems","service-internet-of-things-iot"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v26.5 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Updating Embedded Systems via USB - inovex GmbH<\/title>\n<meta name=\"description\" content=\"&quot;Discover When USB Updates Trump Over-the-Air in IoT&quot; - This article examines the critical factors for choosing an update mechanism, including failure resilience, roll-back mechanisms, and general security in the embedded and IoT landscape. Learn how to implement USB updates on a Yocto Linux platform using Mender and systemd.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.inovex.de\/de\/blog\/updating-embedded-systems-via-usb-yocto-mender\/\" \/>\n<meta property=\"og:locale\" content=\"de_DE\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Updating Embedded Systems via USB - inovex GmbH\" \/>\n<meta property=\"og:description\" content=\"&quot;Discover When USB Updates Trump Over-the-Air in IoT&quot; - This article examines the critical factors for choosing an update mechanism, including failure resilience, roll-back mechanisms, and general security in the embedded and IoT landscape. Learn how to implement USB updates on a Yocto Linux platform using Mender and systemd.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.inovex.de\/de\/blog\/updating-embedded-systems-via-usb-yocto-mender\/\" \/>\n<meta property=\"og:site_name\" content=\"inovex GmbH\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/inovexde\" \/>\n<meta property=\"article:published_time\" content=\"2024-02-22T08:04:45+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-05-23T09:33:08+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.inovex.de\/wp-content\/uploads\/usb_update_method_v3.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1500\" \/>\n\t<meta property=\"og:image:height\" content=\"880\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Anna-Lena Marx\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:image\" content=\"https:\/\/www.inovex.de\/wp-content\/uploads\/usb_update_method_v3-1024x601.png\" \/>\n<meta name=\"twitter:creator\" content=\"@inovexgmbh\" \/>\n<meta name=\"twitter:site\" content=\"@inovexgmbh\" \/>\n<meta name=\"twitter:label1\" content=\"Verfasst von\" \/>\n\t<meta name=\"twitter:data1\" content=\"Anna-Lena Marx\" \/>\n\t<meta name=\"twitter:label2\" content=\"Gesch\u00e4tzte Lesezeit\" \/>\n\t<meta name=\"twitter:data2\" content=\"16\u00a0Minuten\" \/>\n\t<meta name=\"twitter:label3\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data3\" content=\"Anna-Lena Marx\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.inovex.de\/de\/blog\/updating-embedded-systems-via-usb-yocto-mender\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.inovex.de\/de\/blog\/updating-embedded-systems-via-usb-yocto-mender\/\"},\"author\":{\"name\":\"Anna-Lena Marx\",\"@id\":\"https:\/\/www.inovex.de\/de\/#\/schema\/person\/b7d6fd8c3ec8972f3b14f0205e25d022\"},\"headline\":\"Updating Embedded Systems via USB\",\"datePublished\":\"2024-02-22T08:04:45+00:00\",\"dateModified\":\"2024-05-23T09:33:08+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.inovex.de\/de\/blog\/updating-embedded-systems-via-usb-yocto-mender\/\"},\"wordCount\":2983,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/www.inovex.de\/de\/#organization\"},\"image\":{\"@id\":\"https:\/\/www.inovex.de\/de\/blog\/updating-embedded-systems-via-usb-yocto-mender\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.inovex.de\/wp-content\/uploads\/usb_update_method_v3.png\",\"keywords\":[\"edge-embedded\",\"Embedded Systems\",\"IoT\"],\"articleSection\":[\"Applications\",\"English Content\"],\"inLanguage\":\"de\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.inovex.de\/de\/blog\/updating-embedded-systems-via-usb-yocto-mender\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.inovex.de\/de\/blog\/updating-embedded-systems-via-usb-yocto-mender\/\",\"url\":\"https:\/\/www.inovex.de\/de\/blog\/updating-embedded-systems-via-usb-yocto-mender\/\",\"name\":\"Updating Embedded Systems via USB - inovex GmbH\",\"isPartOf\":{\"@id\":\"https:\/\/www.inovex.de\/de\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.inovex.de\/de\/blog\/updating-embedded-systems-via-usb-yocto-mender\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.inovex.de\/de\/blog\/updating-embedded-systems-via-usb-yocto-mender\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.inovex.de\/wp-content\/uploads\/usb_update_method_v3.png\",\"datePublished\":\"2024-02-22T08:04:45+00:00\",\"dateModified\":\"2024-05-23T09:33:08+00:00\",\"description\":\"\\\"Discover When USB Updates Trump Over-the-Air in IoT\\\" - This article examines the critical factors for choosing an update mechanism, including failure resilience, roll-back mechanisms, and general security in the embedded and IoT landscape. Learn how to implement USB updates on a Yocto Linux platform using Mender and systemd.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.inovex.de\/de\/blog\/updating-embedded-systems-via-usb-yocto-mender\/#breadcrumb\"},\"inLanguage\":\"de\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.inovex.de\/de\/blog\/updating-embedded-systems-via-usb-yocto-mender\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"de\",\"@id\":\"https:\/\/www.inovex.de\/de\/blog\/updating-embedded-systems-via-usb-yocto-mender\/#primaryimage\",\"url\":\"https:\/\/www.inovex.de\/wp-content\/uploads\/usb_update_method_v3.png\",\"contentUrl\":\"https:\/\/www.inovex.de\/wp-content\/uploads\/usb_update_method_v3.png\",\"width\":1500,\"height\":880,\"caption\":\"Grafik: USB Stick und dunkel blauer Hintergrund\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.inovex.de\/de\/blog\/updating-embedded-systems-via-usb-yocto-mender\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.inovex.de\/de\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Updating Embedded Systems via USB\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.inovex.de\/de\/#website\",\"url\":\"https:\/\/www.inovex.de\/de\/\",\"name\":\"inovex GmbH\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\/\/www.inovex.de\/de\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.inovex.de\/de\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"de\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.inovex.de\/de\/#organization\",\"name\":\"inovex GmbH\",\"url\":\"https:\/\/www.inovex.de\/de\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"de\",\"@id\":\"https:\/\/www.inovex.de\/de\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.inovex.de\/wp-content\/uploads\/2021\/03\/inovex-logo-16-9-1.png\",\"contentUrl\":\"https:\/\/www.inovex.de\/wp-content\/uploads\/2021\/03\/inovex-logo-16-9-1.png\",\"width\":1921,\"height\":1081,\"caption\":\"inovex GmbH\"},\"image\":{\"@id\":\"https:\/\/www.inovex.de\/de\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/inovexde\",\"https:\/\/x.com\/inovexgmbh\",\"https:\/\/www.instagram.com\/inovexlife\/\",\"https:\/\/www.linkedin.com\/company\/inovex\",\"https:\/\/www.youtube.com\/channel\/UC7r66GT14hROB_RQsQBAQUQ\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.inovex.de\/de\/#\/schema\/person\/b7d6fd8c3ec8972f3b14f0205e25d022\",\"name\":\"Anna-Lena Marx\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"de\",\"@id\":\"https:\/\/www.inovex.de\/de\/#\/schema\/person\/image\/a8cfb531252ec2ef604ef3033c45111b\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/44ab0d3916e62a17054b8e4cea92702db304b9dcb6dd28fb9915484c1830409b?s=96&d=retro&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/44ab0d3916e62a17054b8e4cea92702db304b9dcb6dd28fb9915484c1830409b?s=96&d=retro&r=g\",\"caption\":\"Anna-Lena Marx\"},\"url\":\"https:\/\/www.inovex.de\/de\/blog\/author\/amarx\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Updating Embedded Systems via USB - inovex GmbH","description":"\"Discover When USB Updates Trump Over-the-Air in IoT\" - This article examines the critical factors for choosing an update mechanism, including failure resilience, roll-back mechanisms, and general security in the embedded and IoT landscape. Learn how to implement USB updates on a Yocto Linux platform using Mender and systemd.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.inovex.de\/de\/blog\/updating-embedded-systems-via-usb-yocto-mender\/","og_locale":"de_DE","og_type":"article","og_title":"Updating Embedded Systems via USB - inovex GmbH","og_description":"\"Discover When USB Updates Trump Over-the-Air in IoT\" - This article examines the critical factors for choosing an update mechanism, including failure resilience, roll-back mechanisms, and general security in the embedded and IoT landscape. Learn how to implement USB updates on a Yocto Linux platform using Mender and systemd.","og_url":"https:\/\/www.inovex.de\/de\/blog\/updating-embedded-systems-via-usb-yocto-mender\/","og_site_name":"inovex GmbH","article_publisher":"https:\/\/www.facebook.com\/inovexde","article_published_time":"2024-02-22T08:04:45+00:00","article_modified_time":"2024-05-23T09:33:08+00:00","og_image":[{"width":1500,"height":880,"url":"https:\/\/www.inovex.de\/wp-content\/uploads\/usb_update_method_v3.png","type":"image\/png"}],"author":"Anna-Lena Marx","twitter_card":"summary_large_image","twitter_image":"https:\/\/www.inovex.de\/wp-content\/uploads\/usb_update_method_v3-1024x601.png","twitter_creator":"@inovexgmbh","twitter_site":"@inovexgmbh","twitter_misc":{"Verfasst von":"Anna-Lena Marx","Gesch\u00e4tzte Lesezeit":"16\u00a0Minuten","Written by":"Anna-Lena Marx"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.inovex.de\/de\/blog\/updating-embedded-systems-via-usb-yocto-mender\/#article","isPartOf":{"@id":"https:\/\/www.inovex.de\/de\/blog\/updating-embedded-systems-via-usb-yocto-mender\/"},"author":{"name":"Anna-Lena Marx","@id":"https:\/\/www.inovex.de\/de\/#\/schema\/person\/b7d6fd8c3ec8972f3b14f0205e25d022"},"headline":"Updating Embedded Systems via USB","datePublished":"2024-02-22T08:04:45+00:00","dateModified":"2024-05-23T09:33:08+00:00","mainEntityOfPage":{"@id":"https:\/\/www.inovex.de\/de\/blog\/updating-embedded-systems-via-usb-yocto-mender\/"},"wordCount":2983,"commentCount":0,"publisher":{"@id":"https:\/\/www.inovex.de\/de\/#organization"},"image":{"@id":"https:\/\/www.inovex.de\/de\/blog\/updating-embedded-systems-via-usb-yocto-mender\/#primaryimage"},"thumbnailUrl":"https:\/\/www.inovex.de\/wp-content\/uploads\/usb_update_method_v3.png","keywords":["edge-embedded","Embedded Systems","IoT"],"articleSection":["Applications","English Content"],"inLanguage":"de","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.inovex.de\/de\/blog\/updating-embedded-systems-via-usb-yocto-mender\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.inovex.de\/de\/blog\/updating-embedded-systems-via-usb-yocto-mender\/","url":"https:\/\/www.inovex.de\/de\/blog\/updating-embedded-systems-via-usb-yocto-mender\/","name":"Updating Embedded Systems via USB - inovex GmbH","isPartOf":{"@id":"https:\/\/www.inovex.de\/de\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.inovex.de\/de\/blog\/updating-embedded-systems-via-usb-yocto-mender\/#primaryimage"},"image":{"@id":"https:\/\/www.inovex.de\/de\/blog\/updating-embedded-systems-via-usb-yocto-mender\/#primaryimage"},"thumbnailUrl":"https:\/\/www.inovex.de\/wp-content\/uploads\/usb_update_method_v3.png","datePublished":"2024-02-22T08:04:45+00:00","dateModified":"2024-05-23T09:33:08+00:00","description":"\"Discover When USB Updates Trump Over-the-Air in IoT\" - This article examines the critical factors for choosing an update mechanism, including failure resilience, roll-back mechanisms, and general security in the embedded and IoT landscape. Learn how to implement USB updates on a Yocto Linux platform using Mender and systemd.","breadcrumb":{"@id":"https:\/\/www.inovex.de\/de\/blog\/updating-embedded-systems-via-usb-yocto-mender\/#breadcrumb"},"inLanguage":"de","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.inovex.de\/de\/blog\/updating-embedded-systems-via-usb-yocto-mender\/"]}]},{"@type":"ImageObject","inLanguage":"de","@id":"https:\/\/www.inovex.de\/de\/blog\/updating-embedded-systems-via-usb-yocto-mender\/#primaryimage","url":"https:\/\/www.inovex.de\/wp-content\/uploads\/usb_update_method_v3.png","contentUrl":"https:\/\/www.inovex.de\/wp-content\/uploads\/usb_update_method_v3.png","width":1500,"height":880,"caption":"Grafik: USB Stick und dunkel blauer Hintergrund"},{"@type":"BreadcrumbList","@id":"https:\/\/www.inovex.de\/de\/blog\/updating-embedded-systems-via-usb-yocto-mender\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.inovex.de\/de\/"},{"@type":"ListItem","position":2,"name":"Updating Embedded Systems via USB"}]},{"@type":"WebSite","@id":"https:\/\/www.inovex.de\/de\/#website","url":"https:\/\/www.inovex.de\/de\/","name":"inovex GmbH","description":"","publisher":{"@id":"https:\/\/www.inovex.de\/de\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.inovex.de\/de\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"de"},{"@type":"Organization","@id":"https:\/\/www.inovex.de\/de\/#organization","name":"inovex GmbH","url":"https:\/\/www.inovex.de\/de\/","logo":{"@type":"ImageObject","inLanguage":"de","@id":"https:\/\/www.inovex.de\/de\/#\/schema\/logo\/image\/","url":"https:\/\/www.inovex.de\/wp-content\/uploads\/2021\/03\/inovex-logo-16-9-1.png","contentUrl":"https:\/\/www.inovex.de\/wp-content\/uploads\/2021\/03\/inovex-logo-16-9-1.png","width":1921,"height":1081,"caption":"inovex GmbH"},"image":{"@id":"https:\/\/www.inovex.de\/de\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/inovexde","https:\/\/x.com\/inovexgmbh","https:\/\/www.instagram.com\/inovexlife\/","https:\/\/www.linkedin.com\/company\/inovex","https:\/\/www.youtube.com\/channel\/UC7r66GT14hROB_RQsQBAQUQ"]},{"@type":"Person","@id":"https:\/\/www.inovex.de\/de\/#\/schema\/person\/b7d6fd8c3ec8972f3b14f0205e25d022","name":"Anna-Lena Marx","image":{"@type":"ImageObject","inLanguage":"de","@id":"https:\/\/www.inovex.de\/de\/#\/schema\/person\/image\/a8cfb531252ec2ef604ef3033c45111b","url":"https:\/\/secure.gravatar.com\/avatar\/44ab0d3916e62a17054b8e4cea92702db304b9dcb6dd28fb9915484c1830409b?s=96&d=retro&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/44ab0d3916e62a17054b8e4cea92702db304b9dcb6dd28fb9915484c1830409b?s=96&d=retro&r=g","caption":"Anna-Lena Marx"},"url":"https:\/\/www.inovex.de\/de\/blog\/author\/amarx\/"}]}},"_links":{"self":[{"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/posts\/51496","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/users\/35"}],"replies":[{"embeddable":true,"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/comments?post=51496"}],"version-history":[{"count":6,"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/posts\/51496\/revisions"}],"predecessor-version":[{"id":53974,"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/posts\/51496\/revisions\/53974"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/media\/51890"}],"wp:attachment":[{"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/media?parent=51496"}],"wp:term":[{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/tags?post=51496"},{"taxonomy":"service","embeddable":true,"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/service?post=51496"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/coauthors?post=51496"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}