Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2022 [End User Sessions], 2 Jun 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Unraveling the Magic Behind Buildpacks - Sambhav Kothari, Bloomberg & Natalie Arellano, VMware

Description

Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Unraveling the Magic Behind Buildpacks - Sambhav Kothari, Bloomberg & Natalie Arellano, VMware

Cloud Native Buildpacks makes building container images a breeze. It comes with out-of-the-box support for rebasing, reproducibility, multiple entrypoints and more! In this talk we’ll uncover the magic that the lifecycle - the binary at the heart of CNB - uses to convert source code into OCI images.

A

Hello, everyone uh welcome to our kubecon eu talk on unraveling the magic behind buildbacks, um I'm samba kothari, I'm an ml platform engineer at bloomberg. I'm also a toc member in the buildbacks project. uh Joining me today is natalie ariano.

A

She is the team lead for the implementation team, which maintains the life cycle, which is the heart of the buildbacks project. That's what makes the magic possible so what's on. uh What's on our agenda for today, uh we're starting with the basics, we'll start with what container images actually are uh giving a brief overview of the oci spec.

A

We'll then talk about how container builds typically work, we'll take a look into two case studies, one with docker build and one using basic shell scripts without any root to demonstrate that container images can be really created from first principles with basic utilities and then natalie will take over and talk about how build packs and lifecycle actually apply this in practice and make complex applications possible through a really simple api.

A

So, let's get started what exactly are container images? Well, like everything else in the unix world, they're just files more specifically structured files, along with just some configs described as json bobs,.

A

So, to put that into perspective throughout the talk we'll be using this example application uh it's a fairly simple python web application with some requirements for text that describe the dependencies for the application, and I also have here with me a simple dockerfile uh which we'll be using for a first case, study uh we're using dockerfile, because I assume that's what most people would be familiar with, and it also helps us understand various concepts. So, let's take a deeper look into this dockerfile.

A

It's fairly typical starts off with a base image that contains the python interpreter sets the working directory for the rest of the commands to execute, to slash up explicitly copies out just the requirements requirements.txt file and then installs it using pip before copying the rest of the source code. If you're familiar with dockerfile and how caching works, you know exactly why we're doing this and then finally, we're setting the launch entry point uh so that python actually spins up a flash server and opens our web page cool.

A

So if we try and build all of this uh we'll end up with something that looks like this, uh you might be wondering what so. This is actually the output or cleaned up output from docker inspect, it's a large json block. But what we'll be focusing on? Are these two things right here? First up the config blob and next the layers object.

A

So what exactly is the config blob? It's a set of key value, prepares that describe the runtime container environment, so you'll think you'll see things here, such as the user. That's going to be used for the runtime, the base environment that needs to be set, the entry point for your container and the working directory.

A

Now, if you look closely, you can map the actual docker file instructions to each to some of these keys in the config map.

A

So typically, when you're using these instructions in docker files, that's exactly what's happening, they're just getting mapped to various keys in the config block. Similarly, if you take a look at the layers, blob you'll see a bunch of checksums. So what exactly are these layers they're just compressed tarpauls, containing the file system? Bundle that will form your container root fs. So all of these blobs you see here are content addressable.

A

What that means is that they can be located and they're named according to their content, so they're named based on the checksum of their entire tar.

A

If we were to pretty print this whole set of layers, we'll see that again, we can map these certain instructions that create layers from dockerfile back to the layers that we ended up, seeing in the layers blob. So the first five layers that you see on the top come from the from instruction.

A

Then we have the work directory instruction copy run, and so on so put that to in order to put that into perspective, here's what's actually happening in the container image, layerable layer. If you use a tool like dive to inspect it, so a work directory instructor instruction actually created an empty folder called app or copy requirements for text file, actually created a new layer with just the requirements.txt file with something like a run instruction.

A

Something special happens: docker will take a snapshot before and after compare the changes in the file system and any changes are then put on that specific layer. So this is also why you'll often see run instructions with lots of lines of shell commands combined together, because they all end up in one layer, although not that flexible. The simple assumption has like really worked out well for dockerfiles, so now that we know what layer and config objects are, let's take a deeper dive into how docker actually uses them and how docker build works.

A

Before that, I want to talk about a more primitive docker command that we might not use as often to build the container images which is docker commit. So what docker commit allows us to do is to take a running container and create a container image out of the state of that container, and this is fairly powerful. For example, let's start with the simple busy box image I'm going to make some file system changes to that container. Keep in mind. This is a running container.

A

This is not an image and then I'm going to use docker commit to actually convert the state of the container into an image and just to make sure that this was actually working and the image persisted my changes. I'm going to use docker run again to cut the demo file, and this is uh this is the core principle behind docker build.

A

So when you're, seeing a dockerfile like this, what and you're running it with docker build what docker is actually doing behind the scenes in a very oversimplified manner?

A

Is it's actually creating various container images running the equivalent instruction inside that container, using docker, commit to capture the changes and then then create intermediate containers until uh container images until you get to the final output image that you want, so if you've run docker build you've, you might have often seen these things like creating intermediate container for fd or creating this image, one three three so on so when you're running this what's actually happening is docker is spinning up multiple containers running these instructions, calling comment on it and creating intermediate images that it then uses to run the next step.

A

So that's how dockable essentially works, but just to recap, container images are just a few file system layers and as tarballs and a few set of json files, config files and docker build, isn't necessarily the only way to construct these things.

A

So in order to really drive the point home, we're going to create a simple busy box image using nothing but basic shell utilities with no docker daemon, uh no road, nothing, ah let's get started whoops.

A

Okay, so I just have a handy assistant to help me type so that I don't make mistakes, but it's all running live. So we have a simple function: that downloads busybox from the internet and creates a few sim links um so that we can actually use these commands in the output container image we'll be creating it's nothing, special, just busy books from the internet and symlinks to these commands.

A

Then the first thing we need to do to create the container image is follow the ocl layout specification, and that requires us when using it in folder mode to create certain directories with certain files uh that are laid out in a content addressable fashion.

A

So the first thing we need to do is create the directory that will contain our entire oc image in the ocean image layout format, then we'll actually create the layer, so in this case I'll be starting up this entire bin folder, which will be the only layer in my output image, then uh just to make sure that everything's working correctly, I'm just checking what the tower file actually contains and we've got. This is going to be our final uh file system that we see on the container image. We don't have a lot of layers.

A

We just have one layer with this entire set of files and then, as I mentioned earlier, all these layers and everything in oci wall needs to be content addressable. So what I'm doing is calculating the checksums and the sizes for these layers so that we can actually name them accordingly, so just to make sure here's our original checksum and then what we'll be doing is making a blobs directory where we'll be storing this output layer and renaming it so that it fits the appropriate content, addressable structure that oci expects.

A

So now it's ready and it's named properly. So let's next check that the name actually matches the digest that we calculated and everything looks fine, then we'll be creating the config blob which will describe the runtime properties of the container.

A

So let's do that so again, the things that we need here are the operating system and the architecture which we expect the final container image and the binaries within to work on the base. Runtime environment, in this case we're just setting the path variable to include the slash bin directory so that all the executables that we've just put in the image are available to us when we're trying out shell commands and I've also set the working directory to the root directory.

A

Finally, I've also included the root file system, the the layers blob that I referred to earlier and I have addressed the layer checksum here to make sure that we are capturing this layer correctly in the config block. So right now these are just uncompressed layer, checksums that we've put here next up again, the config blob itself needs to be content addressable, so we're just running the same commands again to calculate the checksum and size and moving the config block to its appropriate location in the blobs directory.

A

Next, there are a couple of other concepts in the oci spec that are not directly exposed in docker. uh The first one is the manifest which makes which actually captures the content, addressable location of the config and compressed or uncompressed digests of the layers.

A

In this case, I want to make things easier for myself and I'm using uncompressed layers, but typically when you're pushing out to the registry, these would be compressed and the digest that you see here will be different than the ones you have in your config and then I'm going to do the same thing with the manifest make it content, addressable and one last important file that we have is the index so typically, when you're pulling ubuntu bionic or focal on different architectures or different operating systems, docker pulls the right version of that image.

A

The way it knows to pull the right version is something known as the oci index, which contains pointers to various different uh manifests for targeting different platforms and architectures. In this case, I'm just building an image for my laptop and I'm not doing a cross architecture build. So that's all I need to specify and then finally, we need one extra piece of information to make it an appropriate oci directory and that's pretty much it. We have a.

A

We should have a working image, I'm going to push it to a local registry uh using a nifty little tool called crane, so it just allows you to take an image in the ocl layout format and push it to a registry. I have a registry running locally and looks like everything worked, fine and just to make sure that this is actually working. Let's try and run it so I'm gonna docker on and pull the latest version of the image that I just pushed up and try to exec into a shell and it pulled.

A

The right thing looks like everything is working. Ls shows we have a bin directory with the appropriate things. If we check the environment it's set properly. If I check the user, it's set to uid 0, because we didn't specify something and everything seems to be working fine.

A

So this really goes to show that container images are nothing but just tarballs and json files and you can construct them without root or any other magical tool like a docker daemon, and this is really powerful.

A

And this is exactly what buildbacks uses to create container images without root or without the need for a docker daemon.

A

So, finally, we have natalie, who will be explaining the most important bits of this talk, which is how cloud native build packs utilize. These first principles to build container images over to you.

B

B

So we are now going to take a closer look at cloud native, build packs. um Before we can talk about a cloud-native build packs build from beginning to end. We should probably cover what a build pack actually is so very simply put build. Packs are software that know how to analyze source code and determine the best way to build it, they're, actually, a collection of two binaries one called detect and one called build.

B

They detect binary. It serves to look at application source code and determine if that build pack is actually relevant for the application. So you might look for a particular file that indicates that the application is written in a particular language that could be what relates to the build pack. The second binary is called build, and that is what actually does the work to turn source code into a runnable application.

B

Multiple build packs can work together, so you could have a collection of build packs that each provide one piece of the total steps. That would be necessary to create something. That's runnable, and this allows great flexibility and interoperability between build packs, which we'll see.

B

So, looking a little more carefully on what build packs actually do this is you know tying back to what we saw with a dockerfile example? Buildpacks can pull in dependencies, they can run compilation if needed, they can define processes to run when the application is started, they can configure the environment among other things, and they can also generate a software bill of materials or s-bomb to describe any dependencies that they've added.

B

So build packs when they're doing all of these things, they must follow the cnb specification, um which is not loading, there's actually a nice graphic. Okay there. It is so the cmv specification uh it it indicates. What build packs are allowed to do. We've actually placed some pretty strict limitations um on the parts of the file system that build packs are able to write to, as sam alluded to build packs are running entirely unprivileged.

B

So that's a limitation and this specification, while limiting allows for very powerful capabilities which we'll see.

B

So we're just going to give um a brief tour of a buildpack's view of the world. This is what a build pack might see in the file system when the build is running. We're, assuming that the detect phase has already happened, so the build pack has opted into the build and now is performing its its piece of the of the puzzle.

B

So we have three directories of interest. The first, most importantly, is the workspace which contains the application source code, there's als, so a layers directory that is designated for that build pack. So this is some build pack id that's running right now, and it has its own child of the layers directory where it can make changes.

B

Finally, we have a platform directory where platforms can provide their own specific configuration and build packs might know to look for things there so build packs, let's say they're providing an application dependency. They would provide that in a subdirectory of their layers directory. So this is some layer which contains some dependency and they've also provided a configuration file which is in tamil format.

B

Our project uses tomml across the board to provide configuration at different points in the process, so this some layer.tamul is an instruction for the lifecycle about how the layer directory should be handled and we'll get into more examples of different ways that the life cycle can handle these layers.

B

So what goes into a layer directory? We, it's sort of open-ended build packs can do whatever we they want, but there are some conventions that build packs can follow. That will make things easier if they, for example, want to provide dependencies that will be available on the path or in the environment for subsequent build packs. So we'll give an example later of how these directories are special, and there are other things that build packs can do like set environment variables.

B

They can also provide. They can provide multiple layers, they're not limited to just one. They can provide build time, configuration and runtime configuration. So we mentioned the process to start at runtime.

B

A sort of unique feature of build packs is that we allow multiple process types to be defined on an image, so it could be convenient to have a single image with you know, two or more different processes to be started. For example, a web or a worker process in the same image. Build packs. Allow you to do that, and we've really just shown.

B

You know in the interest of time and making this a comprehensible talk, uh just sort of a subset of all of the things that build packs are capable of doing, in particular, there's a lot of flexibility to customize the environment for the process at runtime, some more advanced capabilities that you don't actually have to know about.

B

If you don't want to, but just suffice to say that there's more things available and last but not least, uh you know every build pack follows this pattern, so they are all able to make changes in their own layers directory, and you know adding layers as children of that directory or make changes in the workspace directory, which contains the source code, and that's really it that's kind of all. Bill packs are allowed to do which, which again is limiting, but for a very important reason.

B

So now that we've kind of given a build packs view of the world, we want to step back a little bit and give a view of the world from a platform perspective.

B

So, as we mentioned, build packs are executables right, so they need an environment to run in and the best way to provide that environment is via an oci image that we call a builder. So it contains a base image the build image which has all of the os level dependencies that are required by the build packs. In that builder, it has one or more build packs and a binary called the lifecycle which, as sam mentioned, is sort of the heart of the cnb project.

B

So, as a platform operator, you can decide which build packs are supported by your platform by choosing which build packs go into your builder.

B

To perform a build, you would provide source code, then execute the life cycle and the output would be your application image so just to sort of describe. What's in the image, we have a base image which you'll notice is the run image, not the same as the build image. Necessarily in order to for security reasons right. We want to minimize the attack surface and provide a more minimal image containing only the dependencies needed at runtime.

B

In our final image, then we have application dependencies, which you know dependency a b and c. These are layers within the image that directly correspond to those subdirectories of the build pack layers directory and then finally, the app as a as another layer on the top.

B

So, just to look um a little bit closely on what the life cycle actually does um at a high level, it is preparing the environment for build packs and then running bill packs themselves. So the first phase, I didn't say the life cycle actually executes in a series of distinct phases that each have its own responsibility.

B

So the first phase is analyze where we look in, let's say we're exporting to an oci registry. We look in the registry and see if there's any image already there that we built already, and we do that so that we can, by knowing, what's in the registry, avoid re-uploading things that didn't change, then, is the detect phase, where we run a series of build pack groups to determine which group is actually needed to build the application, and this can be useful.

B

You know if you're a platform that supports a wide variety of language families. You might not want to have a separate builder for each one. You can just put all of the build packs that you support into one builder and the build packs by analyzing. The source code will know which group is actually needed for the build.

B

Then we have restore that's just copying dependencies that were cached in a previous, build back into the build container, so that they're available.

B

Finally, is the build phase where each build packs build executable is invoked to actually do the work, that's necessary to make a runnable application.

B

And, finally, the export phase, where we are creating oci layers by making tarballs from those directories that we talked about calculating their checksums and creating configs, just like sam showed so we're taking advantage of these first principles to actually make an image without needing to use docker.

B

So we just showed five phases. It's um you know it's a lot for an introduction, but uh just to just to show that as an end user, you don't have to think about all of that right. You don't have to think about the life cycle. You don't have to think about the different phases.

B

There are many ways to perform a build packs build, but one way that we provide as a project is a cli tool really intended for local development, that you can use to build applications and have them be exported either into a docker daemon or into an oci registry.

B

So here you can see that we provide my app image, which is the desired tag of the image. We give a path to application source code and we provide the builder that we want to use.

B

But this can be made even simpler, because we can infer that the application directory is the current working directory. You can set your default builder as a project. We provide the specification and tooling to run, build packs builds, but we don't actually provide build packs. So there's an ecosystem of build pack providers, google, heroku salesforce and the paquetto project. Being you know, among others, in providing you know, production ready, build packs uh that you can choose from. You can also write your own build packs right.

B

That's the whole point of having a specification is that you can know exactly how to create your own, build pack executables to follow this logic that we're showing.

B

So let's show a demo.

B

B

This, because we have a bad internet, is a a can demo, but it's just to show you know that simple invocation of the pack command here uh we're using the paquetto builder, but you can again you can use any um you can see from the output.

B

Oops I scrolled up too far, so you can see from the output those life cycle phases that I mentioned here. We have analysis. We have detection. In this case, five build packs, opted into the build nothing to restore. Now we have the build step where each build pack is doing its thing.

B

Here's export, where we're creating layers from those directories and adding them to the image. And finally, we have the image being saved.

B

So let me go back.

B

Okay, so this is just going to show in a little more detail what actually happened with that build we're focusing on the build packs that actually provided the python dependencies.

B

So here we have three build packs, they're, each doing a separate thing: the cp, the c python build pack installs, see python the pip build pack installs pip, then the pip install build pack actually runs pip to populate the pip cache and install packages, and the reason it's helpful to kind of break it up into these steps is that you know again thinking about having a minimal runtime image. We only want to include the things that are necessary for the application to run. So in this case, we need see python.

B

We need the installed packages uh in the final image, but we don't want our package manager there. So the build packs have designated these layers as being exportable.

B

However, see python and pip are necessary for pip install to run so we can designate these layers as being accessible to subsequent build packs through the environment and we'll show how that works in just a second.

B

Finally, all of the layers could be useful on a subsequent build right to make the next build faster, so the build packs designate them as cachable.

B

Then, to show it in a little bit more detail just this is the view of the file system uh before any bill packs have executed right. So we have the workspace directory containing our app.pi and requirements.txt.

B

We have the layers directory, which is empty because no build packs have run, and then we have the environment with the path variable. That's inherited from the build time base image.

B

So after the c python build pack has run, we have a c python layer, see python.tamul. This tamil file inside will be instructing the lifecycle. This is a cacheable layer. This is an exportable layer and this layer should be exposed to subsequent build packs.

B

This is getting back to those special subdirectories that I mentioned, so anything in the lib directory is going to be added to the ld library, path and library path variables.

B

Similarly, with include and bin, this is all dot automatically by the life cycle, so the build pack doesn't have to do anything special just put stuff in there.

B

The build pack can also configure the environment. In this case, the contents of the python path.override file would be the path to the see python layer directory.

B

Similarly, with pip, we have a pip layer, we update the environment because this again is exposed to subsequent build packs with those special directories.

B

Then we have the pip install build pack after it runs, it created a layer for the pip cache another layer for the install packages.

B

So just to recap, we have three build packs that created four layers. Two layers are exported in the final image. Two layers are made available to subsequent build packs through the environment, and all four layers are cached. At this point, we didn't have to do any snapshotting.

B

If we go back to the dockerfile example, you know creating four layers with docker file might involve creating four separate run instructions which would be four separate containers, and you know four separate uh instances of running a snapshot to determine the file system diff, which we didn't have to do in this case because of the build pack specification, the life cycle, knowing exactly where the build packs, are writing their changes.

B

We alluded to the software bill of materials, so this is just showing how build packs, because they know exactly what they installed can provide standardized structured, s-bom files to describe what was added. So here we have um describing the three layers: uh cyclone dx spdx and sift json files uh describing the dependencies that were added- and this is a typical of an image config that you would see in a buildpack's produced image.

B

You'll notice that it's very very similar to what sam showed earlier. um So you know just to reiterate images that are built with build packs are really you know, they're oci images. They can be run with any tool that you could use to run, an image that you built with docker, so they can be run with docker. They can be run with kate's.

B

um So this is just to recap: what did we see build packs they're, just software, they analyze source code and determine the best way to build it. They follow a specification that really limits what they're able to do, because of that there is a logical mapping of the dependencies that they install to the layers in the application image that is produced that can be really useful.

B

The life cycle is a binary that orchestrates the build. It sets everything up for build packs. It invokes their executables at the right time and creates the final application image using the oci concepts that sam illustrated and finally just to linger on the limitations of the build pack api, because we know that build packs only make changes in very specific locations. There is a strict separation of the base image and build pack provided layers.

B

This allows us to do something very powerful, which.

B

We've demonstrated.

B

In other talks, but not this one, so just to show you uh what we mean um we can imagine if we have an application image, there's a security vulnerability, but that vulnerability is just in the uh runtime base image, because there we know, there's no intermingling of the run image and the build pack provided layers. We can simply swap out the vulnerable base image with a newly patched run image, and this is something that can be done just by manipulating a pointer in the registry.

B

It doesn't require a rebuild, it doesn't require new running containers and it it's just that uh metadata manipulation, and because of that that can allow you to update many applications very quickly and easily. um You know on the order of thousands of applications in just a few minutes.

B

So there are advantages we think there are many advantages to doing things this way. First and foremost, it concentrates the understanding that is required to build a secure. You know efficient optimized container image, just to build pack authors and not to application developers right as application developers. You just run pack build my image or you know the equivalent, and you don't have to think about all of these things, like you know, minimizing the number of layers or you know logically, grouping dependencies.

B

This means that dependency updates are also very easy and straightforward. So if you have a vulnerable vulnerable dependency, you can just bump the version of the build pack that you're using in your pipeline, and you know, rerun things you will get an image that you know is fully up to date.

B

um As we mentioned, the layers within the image are logically mapped to the dependencies that they provide, which is useful just to understand. What's in the image, as well as to when doing a rebuild, only replace the bits that actually need to change, uh making things faster, you can get more accurate, s-bombs because build packs again, they know exactly what they put in the image.

B

So you can get an s-bomb. That tells you more than something that just came from a scanner and finally, rebase allows for fast patching of os level vulnerabilities at scale.

B

There are limitations to this model. You know by design, so we mentioned that build packs, can't write changes just anywhere. They have to write in that workspace and the layers directory- and this is you know, of course, limiting on some of the things that you can do again by design. They run completely unprivileged.

B

But again you can't do something like install a system package with a build pack.

B

However, our project is currently looking at ways that we can allow for these kinds of extensions in a safe way, so where you would use a build pack like mechanism to extend your build time or your runtime based image before build packs, do their thing giving sort of a more complete way of doing certain things without having to completely remake your base. Images from scratch.

B

So that is our talk. We love to hear from bill pack's users or, if you're, just thinking about build packs, or if you have questions about build packs, please reach out we're available everywhere.

B

There were icons that will show up in a second.

B

That's all. Thank you.