GitLab Pipeline Security team meetings, 30 May 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Object Storage Direct Upload Optimization (skip Carrierwave copy step)

Description

Demo of how we improve the direct uploads to object storage by skipping Carrierwave's copy+delete mechanism.

Link to slide: https://docs.google.com/presentation/d/19JjF8lCwY4nVIiBkrK8AwggIzBPRobEhd9-f1KB-Q1Q/edit?usp=sharing

Related links:
Main Issue
- https://gitlab.com/gitlab-org/gitlab/-/issues/285597

Feature Flag Roll-out Issue
- https://gitlab.com/gitlab-org/gitlab/-/issues/385447

MRs
- https://gitlab.com/gitlab-org/gitlab/-/merge_requests/106315
- https://gitlab.com/gitlab-org/gitlab/-/merge_requests/112843
- https://gitlab.com/gitlab-org/gitlab/-/merge_requests/106740
- https://gitlab.com/gitlab-org/gitlab/-/merge_requests/117432
- https://gitlab.com/gitlab-org/gitlab/-/merge_requests/119972
- https://gitlab.com/gitlab-org/gitlab/-/merge_requests/117746

A

Hello, everyone- this is Eric from pipeline security, so today I'm going to talk about um object, storage, direct upload, optimization, um mainly about um keeping that copy step. Every time a file is uploaded to object storage.

A

um So the problem to solve here is that we were encountering problems with large files being uploaded to object storage, because um after Workhorse uploads to a temporary location in a backup bucket, um the rails, which are more specifically care wave, would move copy plus delete this file into its final location.

A

So it's like- and this is problematic in case we're in the fog driver, for the specific object storage like AWS is not optimized for multi-part uploads, so for large files this would be very slow and eventually you can fail the rails process so as a high level sequence diagram of the current flow without the skip copy, um so Runner would upload um specifically this for the artifacts right.

A

This fix for now that we're doing is we're gonna roll out for artifacts, specifically for now um so, for example, the the runner would upload the artifact to post. It will upload to the API artifacts endpoint, which will be intercepted by Workforce Workhorse. Would then first initiate and authorize call to drills and then rails would return all the object, storage, related information and, more importantly, the upload path?

A

Where object, storage would then upload to I mean where Workhorse would then upload to object storage, so in this case, um rails would return the object, storage, temporary path and then Workhorse would use that file upload. That I mean upload pad.

A

um It would upload the artifact to the temporary uploads that upload path to object, storage um and then, after it completes the upload to object, storage Workhorse would then hit the rails endpoint to finalize the upload, which would then internally the rails would create the job artifact record in the database and whatnot.

A

But at that point in time, when it receives the job artifact record the database, it would trigger carrier waves default behavior of moving the file from the temp uploaded back to its final destination, which is the part that we're trying to eliminate in cases for object, storage, direct upload.

A

um So how do we improve this? um So in many cases for object storage? This extra copy delete step is unnecessary because first, it's the file is uploaded in the end to object storage. It's just that right now. Workforce is uploading the file to a temporary path.

A

So our goal is to use that same direct upload Behavior, but instead of going to a temporary paths, we would apply. We will upload to the final path instead, so it is how it would work without the copy step. So the same as Runner post artifacts workers authorizes uploads to object storage, but in this case um the rails authorized endpoint would be in the turn the final upload path instead of a temporary path, so that once Workforce finalizes the upload rails will determine is this.

A

If this file has been uploaded to a final destination already, then it won't um trigger the care waves default behavior of copying and moving to a final destination anymore.

A

So at a high level implementation detail, um I won't go through all the code here. um I would I would include the link to the slide so that you could choose anyone's interested to look at the code and more specifics can can visit these Links at a high level.

A

um This is what we did here overall, so first we added a new feature flag so that we can control this and observe and slowly roll out the feature for artifact uploads. So we have this new feature flag and if this feature flag is enabled, then when Workforce authorizes for the upload rails would use the final um upload path pattern. Instead of like using the old, temporary random path generated.

A

um We also introduce a new upload that hashback pattern, which will I would show later in the demo of how it looks like and then upon authorizing an upload to find a location. We will track that upload I'm using a redist hash.

A

So um once the upload is being finalized on the rail side right, we will check in the redis hash. If there's a matching entry for this given upload, and if there is one, then we would bypass carrier waves default behavior of moving the file from temp to destination.

A

You can see um the details about how that was implemented, new links here um and then because of this change, who had to prevent Workhorse default implementation or default behavior of deleting the uploaded object from other sorry, it was initially without this change.

A

um Workforce would always issue a delete to object storage because he would assume it was attempt upload anyway, but in this case we don't want Workforce to delete the file uploaded because it's already uploaded to the file destination.

A

So this can result to stale objects in rare locations wherein, for some reason, Workhorse completed the upload to object storage, but then, on the rail side upon finalized step, it failed for some reason and that would result to like a stale object.

A

So we you guys over these changes. We added the Cron job that would like um regularly check for scale uploads and delete um using the reddish hash, delete the redis entry for that still upload and delete the related if ever object from object, storage, so I would do a quick demo of how this works. So first, um as you can see, let's see I'm not sharing my.

A

Let's start with the share for now share screen.

A

Okay, so um I have a test project here locally and we will just read: write this job that would have upload a generic artifact, but on my rails console um the feature flag is disabled, so making sure like what how it behaves right now without this optimization and we would um on Nino, you would observe the calls are commands that it would generate to see that is actually indeed copying and deleting the file from temp to file destination.

A

So I would re retry this job.

A

And hit their um create a artifact and upload it, and you can see here in the the trace of commence sent to the object storage. We have the copy here and then there's the delete. So this is this is Carrier way of doing this and um we can check for the the path.

A

So that's the like the hash pad generated in So. Currently, this is the the path pattern for artifacts, where it includes a project ID the job, artifact, ID and whatnot.

A

um Maybe we can browse the artifacts. So in this case this is 75 7858.

A

A

So there there's the artifact that has been uploaded, um so that's how it works right now without the optimization. Now, let's enable the feature flag.

A

Okay, um I'll stop so here I'll watch again this time we will try the job. We should I'm, expecting we're expecting to not see the copy and delete anymore. In this case.

A

um So there um no more copy and delete and if you wanna see the path generated.

A

I want to put that one against the metadata. Sorry about that.

A

So here's the new pattern. It includes, like the final prefix just to like um indicate that this file was created through that new code.

A

Note that we still create the file or the object under the the root hash of the project ID, so that we can, in future cases wherein there's a need to like issue an object, storage delete based on a prefix pattern. We can still delete um manually all objects from under a specific project and um to see that this thing still works, we can browse the file is still there and, um let's see the so there's here's the newly uploaded artifact.

A

That's final ca5 F6 F6, so you notice the the file name is not artifacts that zip anymore. um We don't use the actual file name anymore. We always generate a random, unique um file, name for each upload and then, but in the database. The job artifact record will still have the original file name so that, if we decide to, for example, download the file still show the original file name, which is artifact texture and that's it um for anyone who's rented to look into this more deep.

A

um The more specifics and history here are links to the main issue, the next step right now. It's not enable jet in production, so I will I'm still working on the rollout issue and then here are the rated Mrs for this. So yes um thank you and if there's any questions feel free to post um comments or whatnot on the issue or on our um slack group, Channel bye.