GitLab Object Storage Working Group, 5 Nov 2021

Previous Meeting

⏯

youtube image

►

From YouTube: Object Storage WG Async kickoff meeting

Description

- Working Group Page: https://about.gitlab.com/company/team/structure/working-groups/object-storage/
- GitLab uploads development documentation - https://docs.gitlab.com/ee/development/uploads.html
- Speed up the monolith - A conference talk about Workhorse and direct upload https://www.youtube.com/watch?v=M74pzdp9xr4
- Defining a generic upload internal API - https://gitlab.com/gitlab-org/gitlab/-/issues/213288#note_325358026
- Shipping MinIO as part of Omnibus Gitlab - https://gitlab.com/groups/gitlab-org/-/epics/6099

- Photo 1 by Sixteen Miles Out on Unsplash - https://unsplash.com/@sixteenmilesout?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText
- Photo 2 by Linh Ha on Unsplash - https://unsplash.com/@lintin9095?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText
- Photo 2 by Vladislav Babienko on Unsplash - https://unsplash.com/@garri?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText

A

Hello, everyone and welcome to the object, storage working group asynchronous kickoff meeting. My name is alessio and I am the facilitator for this working group. I'm recording this short video to introduce some of the high level aspect that we are going to cover. So, let's get to it.

A

Object storage is a fundamental part of gitlab. It's our distributed highly available file storage with direct upload workers can intercept incoming file and upload them directly to object storage before the request hits the rails application.

A

If you are not familiar with direct upload, I encourage you to watch a presentation I made a couple of years ago. The title is speed up the monolith and talks about workers and how we implement a direct upload to object storage.

A

Unfortunately, direct upload cannot be enabled for every incoming file upload in its current state, and this is one of the main problem we are going to cover in this working group.

A

Over the course of the next few slides I'll go through some of the areas of improvement that we should take into consideration during this working group.

A

Starting from this different aspects of the product, those improvements will contribute to a better object: storage experience for customers, users and contributors, reducing development cost operation, cost security risk also making it easier to adopt new features for on-premise installation.

A

Each feature has its own bucket. Because of this we don't have direct upload enabled everywhere, as each upload requires a custom worker's code, contributing a new uploads requires coding it both in rails and go, and each new feature requires a new bucket that requires omnibus and chart support. That requires sre works to provision that new bucket and provisioning new buckets, slow down features adoptions on on-premise installation and also creates a complex first installation experience with 13 bucket. At the time of this recording with one default catch-all bucket.

A

There is no need for omnibus and chart support, merge, request, no need for custom workers code and no need for sre intervention to provision those new buckets. This will give us a faster development and review process, and every on-premise installation will immediately have access to new features without the need to provision. The new bucket point number two code complexity.

A

This picture is a good representation of our object storage code. It all started with carrier wave, a gem that provides a simple and extremely flexible way to upload files from ruby application, and it was about installation back then. However, this is no longer our use case as we upload files from workers in go, and we had to patch carrierswave's internals to support direct upload, giving us a very hard development experience.

A

Our implementation is built on top of third-party framework, where every object, storage client is another third-party library. Unfortunately, some of them are a maintained and we have to fork them to support our users with the removal of carrier wave and implementing a new generic internal api for uploads.

A

There will be no need for workers custom code on each upload that, together with the catch old bucket, will finally allow us to enable direct upload globally.

A

There will be no need for coping file from a temporary bucket to the final bucket, which is a technical details of the current implementation that is causing troubles to users and customers where they are trying to upload large file.

A

And, finally, we will be able to rely on officially supported native clients for object, storage.

A

Point number three supporting both object, storage and local storage is a big cost in the beginning. Object storage was only available as gitlab enterprise edition feature, so we had to support local storage for the community edition.

A

However, local storage is based on the assumption of a shared disk between components, which is only true on a single box installation without ha or with nfs, which we no longer recommend.

A

We also have testing gap on object, storage, because workers and an object, storage provider are not present on our unit tests. So too much is replaced by a mock implementation and both ci and local development runs on a shared disk, because it's a single machine, often hiding broken implementations until we deploy on an ha environment.

A

The removal of local storage support shipping main io as part of the product as a local object, storage provider will standardize our file storage on a single technology. Reducing the complexity of development will also mitigate security risk, removing and attack vectors, as we no longer write user provided files on the application, server.

A

Thank you for your time. I'm looking forward to your ideas on this topic. So please take a look at the agenda and write down your items and I hope to see you in the next working group meeting.