In this article I will show you some of the options how you can deal with binary bloat in your Git repository. They each come with unique pros and cons that I will point out. Additionally, you’ll get suggestions for further information on the topic.
The situation: A project has several binaries that need to be tracked and versioned along with the code. The binaries are small enough and rarely change, so versioning them directly in Git is appropriate.
That was some months ago, and in the meantime, the project evolved, binaries grew and changed. Now, you are left with a bloated repository of multiple gigabytes that takes a long time to clone. This is significantly slowing down the CI pipeline, and it is time to fix it.
Dealing with symptoms directly
There are many options to solve this problem, all with unique pros and cons. The simplest solution is dealing only with problematic symptoms like a slow CI pipeline. This can be overcome with one of the following:
- Shallow clone
git clone --depth 1 URL: Limiting the depth of the clone means that history is not pulled.
- Clone without binary files
git clone --filter=blob:none URL: The filter option is more powerful and can be used to filter only specific files that meet specific criteria. See docs.
- Finally, if the large binary file size is not your problem but the large number of files, you should look into
git sparse-checkout. See docs.
But sometimes these workarounds are not enough and you need to rewrite the repo history. Below I will discuss the two solutions that should cover most of the use cases. The first option is migrating to the
git lfs. The second option is pruning the binaries from the repo history using
git filter-repo and versioning the binaries in a separate versioning system.
Migrating to git lfs
The main advantage of the migration to
git lfs is that the repo history is left intact and no information is lost. This method is recommended when the binaries are really tightly coupled with the code. This is easily done with the following steps:
- Install the git-lfs using this guide or in conda like so:
conda install -c conda-forge git-lfs
- In your repo, run:
git lfs install
- Rewrite history. For example, migrating all bz2 files:
git lfs migrate import --include-ref=master --include="*.bz2"
For more details on how to match your use case see this guide.
The downside of using
git lfs is that it adds more complexity to the workflow of developers. Decisions need to be made: Which files are to be versioned with Git and which with
git lfs? Do you decide based on the file size or the file format/extension? The
.gitattributes file needs to be maintained etc.
You get the gist, there is overhead and nobody likes overhead. But if the binary files are really tightly coupled with the code and need to be versioned, this is the way to go.
Pruning and migrating out of Git
When the binaries are not tightly coupled with the code, there is another solution: simply prune away all the binary files from the repo history and version them in a separate, more appropriate system.
The pruning part is simple thanks to the amazing
git filter-repo tool. For example, pruning away all the binary files larger than 10 MB can be done using the guide from GitLab. For more elaborate cases please refer to the official documentation with examples.
After the pruning, your repo should be very lean, and you are free to version or track the binaries outside of Git. For example, in the project I was working on the
mlflow was the appropriate tool we decided on after some trial period. Using the
mlflow Python API, it was easy to track any artefacts and results along with the meta-data which matched the needs of the project well.
The obvious downside is that you lose the history of the binary files. This was the trade-off we were willing to take as the history of the binary files was not very valuable and we still had access to it in the backups of the repo that we made before the prune.
I hope this was helpful and you were able to get some useful pointers and links that will help with your Git endeavours.