Cloud Native Buildpacks Working Group, 29 Apr 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Working Group: 2021-04-29

Description

BuildKit + Buildpacks

A

A

Okay, sorry got distracted by the recording message, um so let's look over what we'll be talking about, so I will briefly talk to you about docker build and how that ties into the build kit world in the future, we'll go through the demo and then we'll dive deeper a bit into the implementation of how it works and discuss some of the challenges that I've encountered when trying to marry, build kit and buildbacks from both buildkitsite and buildbackside, and I will also talk about potential editions and future work that can happen to this integration and we'll have hopefully time at the end.

A

If anyone has any questions or for discussion, if anyone is interested so uh without further ado, let's jump in so docker build. Probably a lot of people on this call will be familiar about with this one, but I just wanted to set the stage, because it will be interesting to see how this solution has evolved into something like buildkit.

A

So if we look at the traditional docker build, then the way you could think about it is pretty much in the following way. You have your docker file, which is your cookbook. The instructions on how to turn your source code into a container image and you have your context, which is usually your source code, but can be something else as well. Maybe it's a table or some binaries as well.

A

Maybe if you build something already so as a result of that, you get the container image and you can control the build behavior where various toggles you can, for example, do things like change the network mode you can also very commonly use feature is to change the behavior by injecting build dogs.

A

So essentially that lets you do some form of templating of the docker files by you could potentially change the base image or change how your container image is created, but obviously it's not nothing nearly as powerful as something that build kit can offer or buildbacks can offer, and if you're interested in want details, then there's docker docks there and I'll share the slide as well. So you can explore after the fact about more about the how this works and what it is.

A

If we look at buildkit, basically we went from the fact that there was a single implementation inside the docker demon, which was responsible for taking a dockerfile and executing that cookbook into something called frontend.

A

So essentially, frontend is this new intermediate layer and what it's responsible for it's responsible for taking the source code if it wants to as well and executing some arbitrary actions on it. So it's no longer tied to just the dockerfile.

A

You could potentially have anything that can be executed and expressed as a certain layer, certain directional graph of actions, and you can basically potentially create any form of process to turn your source code into a container image, so that basically gives users much more freedom in terms of what they can do and there's a number of uh different front-end implementations kicking about both for making container images, but also for things like making reproducible builds and so on so forth.

A

So if you want, if you're interested, to learn more about how exactly the build kit front-end front-end works, then you can uh click on this link. It's essentially, it describes the pa and draws the parallels between how buildkit works and how, for example, a c service would be compiled in terms of translation into intermediate layer, producing binaries spawning tasks from binary spawning processes from binaries and so on and so forth.

A

So, if you think about what this means for buildbacks, that means that potentially there can be some sort of front-end image here that will run the buildbacks actions and serve as a way to integrate the two. So instead of having the dockerfile, you could potentially have something that kicks off the buildbacks front-end and then that operates in in the same way that the pax cli would do where it would execute your build detects and so on so forced uh stages until it gets a container image as the output.

A

So at this point you might be wondering well, what's the point of integrating with build kit, if there is something like pack already well, the benefits that this brings is that there's a much tighter integration.

A

So if you have docker docker compose, then you can just issue a build command and what will happen is build. Kit will kick in and execute those actions execute the buildbacks frontend instead of having to break out into a different cli and then go back into the docker tooling.

A

It also means that by the virtue of having this integration, buildbacks potentially can extend its reach from just being uh potentially a way to just make images for deployment to maybe being a bit closer to the developer edit cycle, where it can potentially be a way to make images for development and testing, and so on so forth, because compassibility of buildbacks can be used beyond just making production deployment images right, and that also means that anything that integrates with the docker world can now also work on buildback images as well and buildback build process.

A

So there are some common integrations out there that you might have heard about which is vs code, remote containers and that's essentially the elevator pitch there is you can develop inside a container with your vs code, editor or pycharm remote interpreter, which basically is a similar idea, but for python, where again you point your ide into a container and that provides you with a production-like environment for your application.

A

Finally, there is a lot of work going on on the docker side to make the integration with various clouds more straightforward.

A

So, potentially with something like this, you could, with the compose integration, you could potentially do something like docker compose build, which would use build bugs to produce an image, and then you could do issue a command which will actually from your docker compose, which will actually just deploy to either amazon cloud or azure cloud for you without you know, breaking away again into different toolings or been worried about orchestrating these together.

A

So let's try to have a look at how something like this would actually work. So can someone confirm that they can see a shell and please, let me know if the font size is sufficiently big. It looks great. Okay, awesome. Thank you very much. So, basically, we've. If you remember from the from the diagram earlier, there was something called front end. So again, this is just a container image.

A

This one is not particularly optimized, it's quite big, but don't worry about it. Can it can be smaller and contain? Basically what frontend is it's a container image that exposes a particular interface so that buildkit knows how to communicate with it? It essentially means that you can use the same workflows that you would normally do for distributing it and referencing. It is also as easy as just saying my fronted image is and given the docker reference, so we are currently in a folder inside the samples repository from cloud native buildbox.

A

So let me just quickly show you what's inside there. uh I think I can just look at status. Can't I so. Basically, the only thing that we've done is we've added a docker file and you might be thinking well. This is just you know, nullifies the efforts, because if we have a dockerfile, then what's the point, but this is not actually the dockerfile that you may be thinking of.

A

So let me just quickly show you what that docker file actually is and I'll get vs code and drop the docker file for you, so that you can see it. So it's absent java.

A

Okay, so basically, you can see that this dockerfile actually doesn't have much context. It just specifies the name of the builder image, so this is similar to what you would have potentially done. If you called paccli, you would have given it an image of the builder to sorry, the name of the builder image to use, and essentially what it's using. Is this syntax comment. So this is the special comments that uh basically tells buildget that it needs to kick in and treat this build as something different than the dockerfile.

A

So this means that you can basically switch the different switch the behavior and choose the front end, even though you still have this docker file or you can even do some other setups, but this one is probably the one that's easiest to go with and let me just zoom in as well so that it's visible. So this is what our docker file is and if you look back this exactly matches the name of our container image, which is the frontend container image. Okay.

A

So, let's try to build this, so I'm just issuing the docker build command and for reference I have in my daemon I have the uh build kit enabled. So that's why you can see that build kit is kicking in here. If you're familiar with the older docker build, then you know that the output is quite quite different.

A

So by seeing a lot of this blue, you can uh tell this is actually build kit doing this thing and you can see that what's actually happening is that it detected that this is yes, there is a docker file, but it's using a custom frontend now the cloud native buildbox frontend and you can see that it picked up the builder image and now it's going through all the steps of the buildbacks lifecycle.

A

So it's found the sources it run, the detection it run the analyze and restore. Unfortunately, there wasn't much to restore because I'm doing a fresh build just for you and from that point on, you can see that now it's doing the build, so this will take a few moments and because this is a java app, so we can see that the paccato java buildback got activated and what's currently doing, is fetching release of jre just so that the application can be assembled to run and on the output of this.

A

What we will see is that we will have gotten an image which would be quite similar to what you would get from uh pack cli, uh but it's slightly different because of not complete conformance to the spec, but essentially it will be the image that contains the sample java app from the cloud natives repository.

A

So let's give it a few moments until it finishes doing its thing and then we can run the image and actually see it in action.

A

Okay, so you can see that once it's done now, it has finished the build, and now it's just saving some cash for future rebuilds to be slightly faster.

A

And it's it's exporting the image now, so it expo it's exported all the layers cool. So let's now try to run it. So what I'm doing here now, I'm just running this image and I'm manually, invoking the lifecycle launcher we'll get into details for why that happens a bit later, but you can see that it appears to be running and if we go into localhost 80 80 you can actually there we go. You can see that it works, so we are successfully serving the application that was built by buildback for the bill kit frontend.

A

So, but that's not just that. I wanted to also show you an example of the docker integration, so we mentioned that potentially we can have tight integrations with something like this. So let's go into the another sample example.

A

So this is going to be jungle up and basically again here there were some changes that were done, but they were quite quite minor again, so essentially I'm again adding a dockerfile, and you can see that in this case the docker file is actually the using the google container platform builders and I'm also, I also have a small setup. You can see I'm using the google entry point, so this is basically demonstrating the bill. Docs functionality.

A

uh You can see that basically build, but if you are familiar with the way that google container builders are working, then you know that this system builder and essentially what this one does. It's translating build dogs into the pack environment variables that that get injected into the build environment and the the rest of it is just the vs code. Remote containers things.

A

So this one is this one is just a big file that was generated by vs code to get the remote containers to work, and similarly this is just uh this startup for remote containers.

A

So, let's try to reopen this in the container and see if this I expected to see is that first of all, this will reload and what it will do is that it will try to build the container which again will use the build kit front-end.

A

So essentially, we are building a development container using the build kit front-end for cloud native, build box, and when it's, when it succeeds, what we should be able to see is that we have a vs code window into that container, which is a cloud native, build box, build container a similar functionality. Here, you can see that there was docker compose that was called and docker compose in its uh regard.

A

What's called um delegated the build to the build kit frontend because again, build kit is enabled, so this one again will take some time. So I think we can come back to it later when we we can, in the meantime, discuss how the how this is actually implemented and how it actually works. So why is that chip in a way?

A

Let's discuss what we've just seen. So how does the bill kit work essentially high level build kits as we've discussed, is a special container image. uh There are tons of libraries and apis available in go to make the development of frontends easier, but, strictly speaking, nothing.

A

There is nothing special about go and there can be a build confronted in rust and there is a rust library and any really language that supports grpc can serve as a way to create a front-end.

A

Go is just easiest because it has lots of libraries available, so you'll find that you don't need to reinvent as many things or implement as many things and grpc is needed, because that's how the front-end communicates for the build daemon, so build kits, communicate, runs as a daemon and communicates with the front end using jrpc using std and st out, and the way that high level the functionality works is that you have a graph directed graph of action that is get generated and that is called an llb and then that graph gets sent to the daemon and the daemon executes on that graph.

A

So it executes on that recipe and that is called usually or in the code at least it's referred to as solve, so it solves that graph and as a result of that, you get a reference. A reference is just basically a pointer to say, hey your output of your build is here and usually it's just an image, or it can be multiple images if you're doing a multi-platform build like if you're doing, arm and intel build at the same time.

A

So if we were to apply that to the build kit world, then you would come up with some graph that looks some similar to this. It doesn't cover everything, but it essentially would look something like this, so you have your builder image, which would then be combined for the context and a bunch of inputs.

A

Apart from the your source code, there will also be probably caches in input and some user environment variables, which we've previously seen specified as build dogs and some platform environment variables, and then we would run through the classic life cycle for buildbox, where we would find. What are the actions that need to be called see if there is anything to reuse from the previous builds and then do reuse it?

A

If there is anything and then we perform a build after we performed a build, we just need to construct the final image, which would be a combination of the runtime image, the launcher, which is what we've seen invoking earlier explicitly and some metadata, along with the layers that actually belong to your application. So these would be.

B

Oh, I was at the edge of my seat.

C

B

C

C

Riveted concerned that this will ruin my ability.

D

To bash on docker file, if it's still in docker file, I don't think it has to be docker fine.

B

D

B

I'm working on the project, tamil consumption of it, so it doesn't have to be.

D

So docker supports another way of telling it to use a different front end.

B

uh Well, yeah: I think that was the syntax level part of the dockerfile, but then to any like, let's say to the dr daemon or through docker compose, you could specify the file right. So you could just specify a project tomo file and then that project tamil file, as long as it has that syntax part at the top it'll execute this different front end, and that's the part that I was experimenting with and hoping to pr into the poc.

D

So you'd have to when you do your docker build, you have to specify different, you could go dash f, project tunnel and the project. I'm always is that right, yeah.

B

So it's clunky but yeah it's an alternative. If we really wanted.

D

B

To otherwise the docker file could essentially look just like a project tamil file.

D

This is a pr we can make it to build kit, a slightly different interface.

D

The front end match against the files in the.

D

Directory somehow, like could you invert that, since the front-end image could decide what what the correct file is.

B

No, at least not from the dr damon side, because they called I mean upstream.

D

Like like, would that be a way something we could pr upstream that wouldn't be too built back.

D

Specific, like could the image front end image have a label on it that says match against these files.

D

That would be difficult because the front end is defined in the file itself at the top.

B

I mean, I think anything is possible. It would just require a lot of valid reasoning to why we would be pushing for that direction. Welcome.

A

Back eric hello, sorry, I've, I my zoom decided to crash for some reason. I don't know at which point I've gotten broken up. So please do. Let me know.

A

C

The diagram uh that you showed, I mean you were somewhere.

D

In the middle everything.

B

Oh yeah a slide back. Okay.

A

Diagram, so this one yes, okay, I mean, I think by now, actually have has this finished? No, it's not okay. Let's let it take, then, um okay, so I'll start somewhere halfway. Then uh okay, so we have a director graph of actions which is done uh solved by the build kit, demon and essentially the output of that the output of the solution is a reference and the reference can be either a single image or multiple images.

A

So if it's multiple images, usually it means that you're doing a cross-platform builds you're doing like an arm and intel build at the same time, but a lot of the times you just have a single image as the reference of your output.

A

So essentially this is how it functions and the way that uh high level it would work is that you create this graph and then you send it to the build kit daemon to execute it. So that's why grpc is needed and uh if I don't know again, if at what point I broke up so feel free to stop me. If I'm repeating myself, uh that means that any language or any ecosystem that supports grpc could theoretically be used to implement a custom front-end, because it's really nothing special.

A

Apart from being a container image that communicates with a process over grpc.

A

A

I think again, we've covered I've covered that, but I don't know at what point I broke up. So the challenges of build kits are that it's quite docker-centric in some cases, especially if you're looking for the most seamless integration, you have some sort of metadata file. That needs to be the entry point to that world unless you want to go through the more custom setup which doesn't work as seamlessly, and that means that there needs to be some sort of metadata file. That says: hey build kit.

A

You need to use this front end instead of the default one, the way that people usually treat it based on the example frontends that I've seen is that they pick up a metadata file from usually one of their build systems.

A

So, for example, if you had a let's say rust project, maybe you would have a special comment at the beginning of your cargo.tomo, which then can be used as that input metadata file.

A

uh There were also some talks that, potentially you know, project automl could work similarly in the world of buildkit, but we'll see how it's going to work. So another issue is that build kits.

A

If you want to go for that workflow of integration, seamless integration and producing container images, it can be quite uh boilerplating and again I won't go into the detail, because I want to leave more questions more time for questions, and it has also some interesting ideas about how the cache and sharing of artifacts should work.

A

So, for example, for the cache model, build kit works by having pc machine wise or server wide cache versus. If we look at something like buildbugs, the caches are basically scoped to your application.

A

So it's quite a big difference and the sharing of functionality of sharing binaries is a bit difficult because you can't easily inject some binary from your front end into the build process, so that makes it a bit harder to share some functionality that you might otherwise want to just mount a magic binary. Let's say like again for build some of the buildbacks functionality into the target image and just run it, but these are kind of implementation details that as users of buildback, you may not necessarily concern yourself with.

A

Finally, uh the big styling feature of buildbacks is the ability to replace individual layers so again, something that build kit, unfortunately lexington today, and you can sort of approximate that performance and that experience by basically doing something which would translate into the dockerfile world as a bunch of multi-stage builds and copy froms.

A

But it's not really the same experience and again you wouldn't get that nice feature or nice property of later layers not been invalidated by the previous layer change. So if we go into the challenges of buildbacks, then I've been focusing mostly as a kind of platform creator, because essentially front-end is a platform of sorts.

A

So a platform spack is a bit tightly coupled to the docker demon in some cases, because it has, it appears to have two modes of variation where you have either a docker daemon or you have a remote registry which neither really applies in this case.

A

So hopefully it's something that you know in the future can involve and change. Again. We've discussed the cache scoping so just to reinforce it's slightly different and pak. Cli also has much more flexibility in terms of what it offers, because you can basically mount anything from the host which uh docker has been historically quite opposed to and even with build kit.

A

Whilst you do have some functionality, it's a functionality that is treated as you can mount secrets from the host, but not necessarily any host off, which is what pax cli appears to allow in terms of what can be done in the future. So there is still a few things missing. As you've seen, I had to run the launcher as commands manually, because because basically, the entry point of the image is not set automatically, so that's something that can be improved, there's also some talks.

A

There was some suggestions on the issue uh from the community that the id from project automall could be potentially used as the way to namespace the caches. So that would be an interesting enhancement to add as well, and there is also some uh things that don't necessarily affect functionality, but would be good to do for this pack completeness.

A

There is some kind of sanity checks fail, fast checks that can happen. There is timestamps that can be reset for the build reproducibility and so on so forth.

A

Finally, there might be better integrations as as I've mentioned, if there are potentially additions or changes in the spec, then that means that it might be easier to integrate the something like build kit or some other platform into buildbox.

A

So maybe, if the builder image is specified in project automall, then there is a possible way to start to use that as the metadata file that single metadata file that can kick off the build kit process.

A

If oci is used more commonly as an interface between the platform and the life cycle, that that means that it might be easier to integrate because, as I mentioned earlier, buildbacks of today sort of have two modes of operation and that's either you have a docker demon or you have a remote registry so having something which is uh sort of less opinionated would make it easier to integrate something like bill. Confronted into this and that's pretty much it, I can check whether the vs code is still checking away.

A

It might still do I'm not sure why I'm sorry, I think it must be because I've lost the connection halfway through so I'll open the floater questions and, in the meantime I can try to get this vs code thing to work.

D

So I have some questions around caching. I guess sure first thing to say is: I think this is a great idea. I've talked to tonis about it a bit like I'm a hundred percent sold that we should, um you know, kind of at least to replace local builds for sure uh use, build kit instead of implementing our own. You know parts of the life cycle to do that. I think just even looking at how fast this is. You know it seems like an improvement from what we have. You know locally for sure.

D

The question I have around caching is specifically: uh there are environments where builds may happen on different. You know from different containers or vms, where we don't want to download the previous image before we do the rebuild uh you know and then upload the layers that changed. We just want to rebuild the layers that changed um without having to do that. Initial pull and I think, there's actually an issue in build kit.

D

I'm not sure if you see if you've seen this one, it wasn't linked in the presentation um that tonis actually kind of encouraged us to implement. When I had a chat with them, um I'd link to the zoom for a merge lb that would let you do builds locally and then not have layers present unless they're accessed, essentially so that, when you can, you know, do the push up to the registry uh it'll, you know kind of do the same thing.

D

Our current exporter does where it doesn't care that the layer isn't present locally when it stitches it together, and so I'm wondering if you've seen that or thought about strategies like you know,.

D

Does it, I guess, should we position this as something that is, you know just a option for local builds, but when you're building against a registry we keep our current integration. You know: should we try to implement this first before we roll out build kit, support and then get rid of the current exporter implementation kind of curious where people are thinking there.

C

Can I make sure I'm understanding you, um so this merge lb would be for creating basically partial images in the demon that could then be pushed to the registry.

C

So then, like our local, like our registry implementation, would go through the demon first and always be pushed rather than having a publish that works directly against the registry.

C

A

C

That makes sense to me on some level our registry implementation. The way it is now is probably about as efficient as it could be and as efficient as this would be, because we don't need to recreate layers that don't need to exist, but I think it runs into problems with how users want to use the tool. Most people do not want to just directly build things into the registry. They want to build it locally and then push it. So if we could get the same characteristics.

C

Without having to, while still doing a local, build, that'd, be pretty cool.

B

Yeah, because isn't it my understanding that basically, this workflow, this build kit workflow, is all making operations within the docker daemon so like there is no registry concept right, it's not like publishing or pushing anything remotely until you do a doctor push.

B

So I don't. I basically wonder if they're separate conversations.

A

Yeah, essentially the what you can do from inside uh built-in front-end in terms of accessing the daemon is quite limited, so you don't have the ability to, for example, um access the docker sockets or do things like perform, for example, find out what is the tag of the image that you're building?

A

You are supposed to give out a reference which is basically here's the contents of an image you asked me to build, and then this is where the rest of the build kit demon takes all and says. Okay, I took the contents. I took the metadata that you produced. I'm gonna target is this: maybe I'm gonna export it into a towel, or maybe I'm gonna save it in the demon or maybe you want to build and push at the same time.

A

So this is where sort of that handle happens and the front end doesn't participate in in those actions anymore.

C

I have one question based on that about a point you raised, that would be nice people to export to like an oci layout or tar file. um I'm not sure. I totally understand how that interacts with bill kit sort of based on this.

C

uh Based on that explanation, where it seems like, if like we don't need from the sounds of it, doesn't sound to me like we need to build a tarball first and then give it to bill kit to finish our build. Instead, we're you know giving a set of instructions to the demon, then it runs it and ends up with a with a docker demon.

A

Yeah, so you don't necessarily need to give it a double. The difference is that um if we look at the way that the life cycle today works particularly export a bit, it has two modes of export and one is I'm going to export to a docker demon and therefore I need access to docker, sockets or I'm going to export to a registry and therefore I need registered credentials.

A

So this is the bit that sort of clashes with the bill kit way of doing things, because for the bill kit, you just say here's a file system, it doesn't necessarily need to be even tables. It's just you create an image file system and then build. It knows how to export it. So I think the clash that's happening is that bilkit wants to do its own exports. When uh the cloud natives platform spack has quite rigid ideas about how the export should function and what are the modes of operation.

A

I hope that made it a bit clearer.

C

Yeah, I guess I'm not at all surprised that our current spec doesn't account for this and that we would need to do something different here in order to um target bill kit as an endpoint. I guess I'm like. Maybe we don't need to do this all here, but curious to take this offline, to learn like in detail how you're sort of creating this file system for the final app image um in build kit, because our export step like right now, the state of the container that's running export, isn't the state of the app container.

C

So somehow you have to gather the exact set of layers you want to make it into a file system. On top of the run image right, I'm so curious about how that how that's working in this demo.

A

Yeah sure I mean we can, if we have time we can dive into that even now, but if there are other questions and I'm happy to go, whichever way.

D

Well, I think, for that file system that gets generated right. The problem is that build kit. Doesn't let you generate a partial, oci image layout right and you have to have the full image that gets exported locally on the file system before it'll? Do the export and that's the big feature- that's missing, comparing our exporter to build kit's exporter right, um but it seems like they're open to you know a feature in build kit.

D

That would let you create a partial layout and upload that to the registry, um just just from talking to tonis and reading that particular issue. You know we may not need our exporter to achieve the same outcome if we're willing to make some contributions upstream. It's my understanding.

C

I think it's worth calling out that when we export to the daemon, we do essentially have to construct the full image.

D

For sure, I'm just thinking about the registry case adjusting.

C

The registry case and the registry, through bill kit, so you're saying export to build kit, partial.

D

So might build could build some layers right because it runs through the llb.

D

I assume that ends up as oci on disk at the end and that gets sent to the registry, but right now build kit has to process and generate all of those layers right and it has to come up with a full image before it gets exported to the registry if, instead it was comfortable, you know like, as in that github issue, you know keeping pointers, two layers that aren't actually present through, that build process. Saying: okay, this is cached.

D

I don't even have this, but you know in the end, I'm gonna write a reference uh and I'm gonna keep an empty reference in that file system based layout to generate and then upload those layers to the registry. I I'm not convinced that this 1131 gets us all the way there it may just. Let us do the local build and have those empty references, I'm not sure if docker would be comfortable.

D

You know there might be a little more work on top of this to get that working, but it seems like they're open to you know. Implementing that amount of functionality upstream- and that would let us get rid of our you know, kind of custom logic, for you know partial image generation that we're doing in the exporter in the end.

D

um There's another problem with this, though, that I was going to bring up, which is that in a lot of the platforms that run our builds, they run with without user name spacing, you know like no capabilities whatsoever, and so in those cases I think if we tried to do the build during build kit- and you know, even if we had that partial export functionality, you know those platforms would need to give more privileges to their build containers like if it's kpak running on kate's, for instance, so that'd be another reason to maybe keep our.

D

You know. Direct registry exporter implementation around anyways.

B

Yeah I mean I totally agree with the registry case being something that's important.

B

I think when emily- and I spoke- we were talking about keeping the registry, the oci registry implementation and then replacing the daemon implementation with an oci layout right and then pac, for instance, could then take the outcome of that ocl layout export and put it into the daemon in a quote: unquote, uh efficient manner, and then that would be the same implementation that we could use in build kit, because you know, I think this might lend into like the the code that eric was about to show.

B

But during the steps that you know you create this llb graph or whatever, during the execution or after the execution of the exporter like that happens in sequence, right and so then, from there you could specify the next step, which is to look at the outcome of that oci layout, whatever the exporter produced and then use those files to generate your final image, that's then referenced outward.

B

I don't know if that helps, but I don't know eric if you want to speak a little bit more to that.

A

I mean I can, I can show how it it is implemented today. So it's not quite the same again: efficiency as potentially the pax cli does, but essentially you can see here this code again. Let me zoom in a bit. This is where the export basically procedure begins, so we get the run image and then we do the export.

A

Where we add the launcher, add the buildback launch layers and you can see basically the operations that mostly happen here is copy, so it copies from the file system that produced the build and it copies the particular layer that we want to to copy, and basically these layers they the way that again, this happens today is basically just by running the solving the llb, which again gives us the file system and then looking at the doctorals, which were produced and checking whether they are launch layers or not so basically, work in the file system.

A

Saying you know this is a launch layer. I need to copy it. This is not a launch layer. I can skip it so this is then feeds into this llb copy operation, and this is what, together, when combined with the all the other bits like run image like launcher like the up layer, produces that output image. So you can see at the end. Basically, it has an llb state which has all of this layered on top of each other and with solid solvent to get a reference which then gets used to export an image.

D

So in this case, the build happens, first generates all the layers and then, when you're doing the, what you call the export phase is actually running copy instructions to copy all the layers in separately. So it's it's not like build kit is orchestrating the build process itself. Build kit is just orchestrating the construction of the you know like image in the end right, yeah.

A

D

After each buildback runs or something or after the you know, each layer gets generated. Yeah.

A

Okay, yeah essentially, so, if you think about uh every of this every box of this graph in an llb state, basically they all reference each other. So there's a builder state that then gets referenced when it's all combined. Then there is an llb run operation, which is represents the detect stage. Then we copy any results from that, then we do analyze, which again it does a bunch of llb, runs and then the output of that when you solve this graph, this middle bit is a file system which then, as mentioned. Basically, you walk through.

A

You can, and this is where the export process that we've just seen happens where it finds. What are the launch layers? What are the layers that need to be included, and so on so forth? So this is the bit that has that I had to basically mimic the life cycle in because again because of how life cycle operates today, so I had to basically mimic how it works by looking at the launch layers and constructing the that file system, that build kit can understand and can export into a docker demon as the image.

B

So this might be a terrible analogy, but I I'm thinking of this along the lines of like futures and promises in javascript right where you're just like creating this stream via llb of like what should happen during these steps and then, when you pass it to solve it, does everything internally right, so you could put in between these steps or states. What you want to happen in those particular instances is that.

A

B

A

That's a really good analogy, because essentially you go through the process and frontend can either do it once it can. Do it many times where you make an llb, you make a graph, you give it to build kids to solve. It returns a reference to you and then you can do something with that reference, so you can either export it or you can look inside it. You can look at the files inside you can maybe inspect.

A

You know, read a single file, look through the directories and so on so forth, so you can basically chain these, and in this case there would be two of these cycles. That happened because the first cycle happens for the build, and then this is when the front end inspects the output of the build says. What are the launch layers and then the cycle happens again where an lb is constructed.

A

For that particular build to say pick this layer pick this layer combine it all together into a new reference that band will get exported. So I think promises is actually a very good analogy.

D

I was just going to ask a simple one like these images that you're producing they don't have the zero dot timestamps. Do they right because you're copying it from the built image instead of the tar. I guess that's one piece that the export's probably missing, unless I missed that in the export.

A

Yeah, so this is some of this is something that I mentioned on the future work. I don't see why that couldn't happen, because we do have access to that file system. So, conceivably, we can't just say: hey uh reset all the timestamps.

D

There are two places where there are timestamps there's in the files themselves, so we zero the time stamps for all the files in the disk, then there's also in the image that gets generated. There's like a created at time that we also zero so that it's possible to rebuild. Like literally the same. You know, image has the same digest in the registry.

D

At the end, if you build with the same inputs, um I think you're talking about the ones in the file system, but we'll build could also let you zero the timestamp of the image that gets generated. I.

A

D

A

You you do have access to the metadata which or image config, which is separate from the file system, but I am I've never tried to actually set that to see if buildkit would respect it. uh I would go off on a limb saying that probably bill kit sets it, because that sort of falls into the whole build kit wanted to take care of exports story.

D

That'd be another feature upstream just turn off the.

C

Something I'm curious is about it's like right now we do things like take what the diff id of different layers are going to be and like put them in a label in the same image, and with this llb copy interface, can we get back the um the digest of the layer from that and then use it in another step to make these labels.

A

I'm I don't know enough to to answer that, uh because, from the interfaces I've seen, you don't seem to get access to the intermediate layers. You only get access to the output. There might be some hidden api somewhere that I'm not aware of.

D

For the thing you generate for this oci on disk format, you generate um what are the layers, tar balls or the layers tgzs? What are you generating directly and then passing to build kit? Could we calculate the checksums ahead of time.

C

I think you're just generating a file system, one step at a time right, like the actual font, the file system of the final image. If I'm understanding this correctly.

D

Yeah the snapshots, so you just you see how do you end up with separate layers for each like once the build pack runs, it's going to generate, let's say 10 layers right. How do you separate those into separate things? Is it later you're doing copy instructions on them.

A

So basically, every llb command you're on produces a layer, so you can see here. This is an example of a run command. So this is where we are executing the builder part of the platform. So you can see it's not actually run in it. It's again coming back to that idea of promises. This is basically just saying hey as a as a part of that graph. What I want you to do is I want you to insert a run operation in there, and this would be done corresponding to this build box over here.

A

So essentially we would say hey. I want you to take the file system that I've had before from the previous states, and I want you to execute a run command on top of it. So every single llp command that you add on top of this, would produce a layer and then, when we come round to export sorry, sorry I didn't interrupt and then, when you come back to the export, you can see what we do here is we do the solve.

A

So then, the output of that is the build image when we execute on all of those actions. I click it on all of those promises.

A

So then, again, if you want to get multiple layers out of it, this is where this thing is happening, where basically right now, the front-end is looking at the build file system and it's looking for individual layers that buildback produced, and it basically translates each layer into a separate copy which then, because it's a separate, llb operation. It then produces a new layer of the image.

C

So if you think you can really see how this came out of docker file, I'm going to do a command and each command I do is a layer. It's like a programming interface for that kind of yeah and the end result is the file system. That's been created by all of the commands.

D

If we wanted the diff ids back, you know I'm not sure exactly what context this export's running in, but if this exports running with the files currently on disk, we could just you know the dev ids are uncompressed, so we could stream everything through tar pre-calculate, the diff ids in the stage take a little time. That's kind of.

C

D

C

We like calculate all of our own diff ids, like I was just sort of this- is not a blocker at all, I'm just asking, because a lot of the time we spend in export is spent calculating diff ids, and if we're like doing this step by step against the demon, if it could somehow because it's going to recalculate them anyways, if it could somehow just give us that back instead of everyone calculating it twice, it would be a lot faster than it is now.

C

Do we actually.

D

C

Why it looks so fast in this demo because probably nobody's gonna, like only the demons calculating it, we're not spending time recalculating it.

A

Yeah, so it's completely reliant on build cadas, so that that also means that if you were, for example, to run it because against the exact same file system, because it knows that the input has not changed, it will just return all cached as well. So yeah, it's not doing anything sort of magic around layers by itself. It defers to build kit completely. That.

D

Do we need the diff ids um if, if we're not going to support like rebuilding from a remote image, or something like that, are they really are those labels really necessary if you're just scoped to rebuilding locally.

C

uh We use them for restoring layers, but if we were restoring layers differently in this world, it could be different right, like you can imagine doing it by in this mode. I mentioned by like file path like you'd, have to you, have an image, and then you copy the files from that path, rather than you know, pulling a blob out of the registry by diff id or docker saving and pulling a blob out by diff id. You would instead tell buildkit to copy everything at a path during restore back into your build.

B

B

All right, so I know we're at time. Are there any last-minute questions.

A

Before we we go, I just wanted to also show that vs code did eventually complete and if we look inside here, we can see that we have our build kit generated file system and I have vs code running into it, and I can for example, see I get all the autocomplete from my application.

A

So on so forth, I can potentially even run things here if I had the correct path set up and so on so forth. So this is the the bit reference in the integrations that you would potentially also get if you integrate with something like build kit.

B

Awesome- and this is using the google builder- is that right.

A

Yeah, so if you look in inside here, let me just zoom in a little bit uh you can see that there's google python paper, google, python runtime, so it's uh google containers, buildbox builder,.

C

I just want to know who's built packs for that I'm solo kidding.

B

Cool awesome. Well, thank you again eric. I think it was really insightful presentation. I do appreciate it and uh yeah we'll definitely keep working on it. um Like I said I have a. Maybe you dropped off, but I have a pr coming for project hummel that hopefully we could collaborate on.

A

Nice yeah, that's that'd, be awesome.

B

All right anything else.

B

Cool. Thank you all see you next time.

A

Thank you very much, bye-bye.