Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2018 (Copenhagen), 14 Nov 2015

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Source2Image Deep Dive – Ben Parees, Red Hat & Matt Moore, Google (Intermediate Skill Level)

Description

Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).

Source2Image Deep Dive – Ben Parees, Red Hat & Matt Moore, Google (Intermediate Skill Level)

This session is dedicated to an in-depth understanding of building images on or for Kubernetes. We will present some best practices, give some short demos and discuss some improvements that can be made to help developers repeatably convert source code into compiled apps running on Kubernetes. In this session, we will discuss what building images means to us as a community, and where we should focus our efforts in future. We will explore current channels for collaboration like sig-apps and dev-tools-wg, as ways to evolve the community discussion on these topics.

About Ben
Ben Parees is a Principal Engineer on OpenShift. He serves as the lead for the Developer Experience team which focuses on workflows that take developers from source code to PaaS deployed Docker image based applications, including continuous integration and delivery flows.

About Matt
Matthew co-founded and was the original Tech Lead for the Google Container Registry product, and has been involved with Docker at Google since the beginning. These days, Matthew is focused on Uber TLing container development and firmly believes that we can do better than Dockerfiles. He is the principal author of github.com/google/containerregistry and github.com/bazelbuild/rules_docker which enable the construction and manipulation of Docker images without a Docker daemon or root. As a part of building better container images, Matthew believes we should kick the Distro out of our containers. Come see why at his talk!
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

I'm been furries, I lead the developer, experience team on open shift from Red Hat, and so we contribute the s2i project as well as the build functionality on open shift, and this is matt moore as your mic off.

B

Testing yeah, so I'm I'm, Matt, Moore I work in container tools at Google, and you know I sort of have my fingers and a lot of pies, including container builds container registry and so on. Just.

A

Quick poll how many people were at our session, the intro session yesterday, good, we didn't scare too many people away all right so today, we're gonna be following up on that and talking about how to take those technologies that we discussed yesterday, all the different build tools.

A

How do you build images and talk about what it would mean to bring them to a kubernetes cluster, so we're gonna go through kind of what are the options today that exist and then we're gonna talk through some proposals about where we see the space going in the future and then open the floor for discussion, get some thoughts from you guys as well about what you think you'd like to see and build capabilities on kubernetes.

A

So today, one of your options- and this is probably what most people are doing today if you're even doing image builds on cluster at all. A lot of people are probably doing their image builds off the cluster, but if you're doing a non cluster, then you're probably running a privileged pod, which, if you watch loses session this morning, you know why not to do that, but you're running a foolish pod, you're mounting the docker socket into that privileged pod and then you're. Invoking docker build with your doctor file.

A

You're, also, obviously mounting all your build content into that pod or injecting it, and then you push the resulting image up to your registry and tada. You have built an image in a very insecure way. So what's good well, you can do that today on any cluster doesn't require anything you don't have out of the box as long as you have a doctor, daemon available.

A

So with things like cryo, you might be starting to run clusters where you don't have a docker daemon on your host nodes, and so that's enemy limitation and as far as using this approach, and you get really great layer sharing and reuse again, we talked some about this yesterday, but because you're building effectively on the host machine, all the layers that you built during that docker build process are going to be in that local storage.

A

And so, if you come back later and do another build on that machine, all those layers are available in the cache and get reused to get good build performance. As a result of that bad part is see losses discussion this morning, I think she covered everything that I was going to say about this, but it's your your opening up your host essentially to anybody who can create those pods to do anything they want on that host machine.

A

So, if you're running a development cluster- or you trust everybody who's running on that cluster, both trust them not to be malicious and trust them not to do something dumb. Then it's okay, but if you want to do a multi-tenant environment, then it's not gonna work for you, and it also means that developers need to learn docker files, not necessarily worst thing in the world, but again from our discussion yesterday. If you want to insulate your developers from docker files, let them just work in source code. Then this approach doesn't give you that abstraction.

A

So the other option today and I apologize if this feels a little bit like a product plug, but there is it's going to lead to. Where do we go as a community in this space? So there is the open ship build API, so you have to be running an open ship cluster, but it is an API resource defined according to the kubernetes spec that lets you define a build.

A

You want to run on the cluster and there is a controller that recognizes you've created this build API object and creates a privilege pod on the user's behalf, but that the user does not actually have direct control over. So the operations that are performed within that privilege pod are completely controlled by the build object, nothing that the user is able to control. So it's going to be safe logic. That's either going to do the s2i build or a docker build, but without again user injection of their own logic in that privileged environment.

A

So it's nice that it's a first-class API for a new build, so you can use your client commands to query the list of builds. They show up as resources. Just like pods or deployments or anything else, you can create the build. You can watch the state of them. You can view their logs. All that looks just like working with any other kubernetes resource and users. Don't get access to the previous pods. They don't need access through pods.

A

They also don't get access to the docker daemon and if you're using us to I builds, you don't even need a docker file, so you've abstracted that away and because it is effectively still doing that docker build on hosts you're, getting that layer reuse as long as your build lands on the same node again or a future build what's bad, it's still using privileged pods.

A

So if there is a vulnerability, it could be an exposure for the cluster and it is an open shift, specific feature of the API and the controller that make all this work do not exist on kubernetes today, and then it also still requires the doctor daemon, both s2i and the docker build approach are interacting with the docker Damon. So if you're trying to use cryo- and you want to get rid of the docker name entirely- you can't you can move to cryo, but you still have to also have a docker daemon running on your hosts.

A

A

So what does this API look like again just sort of setting the table for the future of a build API? In general, we provide an input for your git repository for your source code. You can provide input from other images. So if you have an image that contains some dependent jars or some drivers, we can pull down that image, extract the content from that and provide it as another input that the build will consume secrets. Kubernetes secrets can be provides content.

A

So if you have a secret that you need to access a maven repository or something you can make that available to your build through the kubernetes secrets. Api, you can provide your own environment variables for proxy settings or configuration inputs, and then you can also do secrets for accessing the input, so there's secrets that are inputs to the build itself and there are secrets for cloning, your source.

A

So if you have a specific secret to access your git repository or a secret to pull down the images that you're getting content from, you can supply those as well. When these builds get scheduled, we offer serial and parallel scheduling by default at serial. You know, if you, you define what we call build config and then each time you trigger that build you'll get a build.

A

Object will run one at a time until they're all done or you can run them all in parallel and you can configure the resources, because these things all run in pods how many resources, just like any other pot you want that build to be able to consume and you can trigger your builds via web hooks. Both you know: github, git, git, lab, etc. Web hooks also a generic web hook and also on image change. So if your base image that your build is going off of changes, we can automatically trigger a build.

A

So if there's a security, vulnerability and rel and your images based on rel, you get, the new rel image will rebuild your application image on top of it without you looking at anything and same thing, if you're getting content from other images that can also trigger your build to rerun and then finally, where do you want to push your application image when you're done so your your output target any registry you want to push to and if necessary, a secret that enables you to push there.

A

Wouldn't be a kubernetes presentation without some yeah Mille. So here is an example of the build config API expect out, and you can basically just see the various inputs that I discussed. There is more than what is seen here, but you can see here. We've got the run policy as serial we've got a github trigger so that when your github repository changes, we will trigger a build here. We also have an image change trigger so that if the base image changes we'll run a build, we've got our git source repository my app slash Ruby.

A

We also have a git ref, so we're actually going to build a specific branch of that repository. We've got our git source secrets so that we can actually clone that repo in case it's private and then, as far as on the on the other side, we've got these strategy, so this is where we decided we're using s2i. Are we using docker? Are we going to use something else?

A

And so here we're using the s2i strategy so strategy and we're saying the Builder image is going to be this rui 2.4 and you see it says image stream tag, that's a abstraction that OpenShift wraps around access to images, but basically you can think of you're saying I want to use the Ruby version, 2.4 tag and then, lastly, we have the output target again, wrapped with an image stream. But basically saying this is the image that we're gonna push out when this build finishes.

A

So that's the OpenShift build API.

A

The another technology that I just recently became familiar with is a and I'm not affiliated with this project, but is the container builder interface project, and this is one of the first attempts to define a CR D for builds on kubernetes. So it attempts to actually abstract a few of the different build technologies. It lets you define a build configuration in a custom resource and provide wrappers and plugins so that you can do a docker based build, build, get billed or Conoco I think those are the four it covers today.

A

But it's intended to be a very pluggable system where, if you have other ways to build images in the future, it would be able to wrap those as well again runs those on the kubernetes cluster. It's got all the same security problems as I mentioned at the beginning, in the sense of needing access to the docker socket and then, as far as its inputs, you can provide the context for your docker, build from either a git repo or a kubernetes config map.

B

So another CRD worth calling out that I I know some projects are using for build. Is the agro proj workflow CRD?

B

This example is largely lifted from one of the sample amell's in their repo, but you can see that it's basically launching a darker and darker sidecar and there's a little script in here that actually will run a docker file based build against it. You know again this it's. This is not necessarily purpose-built for builds. It I think it's intended to be sort of a broad workflow solution and build. Is you know one one of the types of workloads people have used this for because you know it can do. Dockerfile builds on cluster.

B

But it you know it certainly an examples like this: it's prone to some of the security problems that you know we've been talking about, because you can see it's running a privileged sidecar and interacting with that docker demon of basically directly through this inline shell script. Here.

B

This is actually not a kubernetes on Cluster build, but what we've been using for Google container builder, if you squint and look at it, there are a lot of parallels to it and today, because of a lot of the security complications with running on cluster builds we're actually using VM sandboxes to execute each individual. Build part of the reason recently we've been releasing.

B

Projects like Conoco is to try and move in a direction where you know these individual steps, which are actually containers that run over some build context, can be made more secure because the API as it exists today, is basically giving you the daemon socket from the host and say go to town right. Do do your whatever you do normally docker billdocker push and whatnot, because we've sort of given given up and said: okay. Well it the container boundaries, not just not secure.

B

If we need to hand out a docker daemon socket, you know we need to assume the user owns. The host and I've played around with it, and it's actually really easy to do the kinds of things Liz does in her demo. But what's really nice about this model? Is you can you can plug virtually any builder?

B

You want into it right, you can you can build and publish it, Kocher image either publicly or to GCR, and the VM is authenticated with your project that would host the image, and so you can pull down the Builder container and execute it in the build context to do nearly arbitrary things, but we would like to move towards a model where we model some of the privileged aspects like kubernetes does today and can block some of the privileged access which really the one the one use case I can think of where we mainly use it today is docker file builds that go through the docker daemon, and so it seems silly to have this privileged exception.

B

For just that one use case, and so with things like you know, Konica, which we talked about yesterday. It would be great to try and replace that and move to a world where we could run these in an unprivileged setting, and it would be great if we could do that on cluster and shed this vm overhead.

B

So you know we talked about a bunch of methods. I don't want to clean this as exhaustive, but you know one of the things sort of squinting and looking at this, a lot of them feel like they have common elements emerging right. You know all of them had to find some sort of source. You know get seems to be pretty popular one or more steps on sort of how to build that source context and I.

B

Think a relatively common pattern is expressing that how, in terms of sort of a builder container abstraction and then oftentimes volumes are useful, I didn't show it on the container builder one it really don't, because it's not kubernetes, you don't have all of those options, so it gives you basically empty dirt, but having access to powerful volumes is actually really interesting and then you know modeling, outputs and authentication with resources that are, you know like source and whatnot are all sort of part of this right.

A

Yeah, so what we are hoping to have a discussion about here, then, is you know with what we've seen the landscape come up with so far and we're starting to see more in the community around these tools? What are your experiences with what you're doing with builds? What do you see in these API? Is that's good? What do you see? Maybe that's bad or missing.

A

You know we think. As as mattre said, you know, input content is certainly a key part of any API pushing the image seems like it's going to be a key part and the mechanism itself between yesterday and today, we've covered a lot of different techniques for building images, both dockerfile based ones, ones that don't aren't based on docker file, ones that are based on docker Damons ones that aren't based on docker Damon's seems likely that we're not going to pick one of those so probably kind of like the direction the CVI interface is going in.

A

To some extent the open shift, API is probably needs to be a pluggable system as well is at least what what we are thinking.

A

So questions or experience that you'd like to share with what you've done, building images either off cluster or on cluster or directions. You're thinking about going.

C

Hi I'm Artie, he rose to the author of the CD I sent to her I work. Great.

D

C

This is not question, and just just expresso and CBI currently only supports Turk of our best builders, but I'm also trying to implement other languages such as basil, or associate image or Google's kalbi, dot, yml or anything. So maybe, if we can add many language C's to CBI interface, maybe we can use CBI as the point to hoser your proposal. Cool.

E

D

Guess, it'll all unfair, but because you won't have known this before you wrote the slides. But do you think that the G Weiser stuff that was released this week from Google around having sandbox kernels? You know actually restricting that what that docker, Keeton kernel can actually do will actually go some way towards solving those security issues that we've been talking about earlier. So.

A

I'll give the short answer and then I'll, let Matt give the long answer so yeah I think it can I have only read through the blog post on it, but it definitely seems like an exciting technology that would make things like you know today. There are build systems that spin up a VM so that they can do builds on cluster or effectively on cluster in a in a safe way, and of course they lose the caching reuse, but they get that security and the idea of a lightweight way to do that sounds awesome and yeah.

A

So I think, as we see how that project involves, there's definitely a place for it. I.

B

B

Yes, we certainly thought about it.

B

You know, as I mentioned yesterday, one of the I think really interesting things about G Weiser and one of the considerations when we were thinking about Konica was running it in that context, and so you know, if you wanted to so one of the interesting things from this morning, and you know from some of the examples, is a typical way of getting access to the daemon socket is to use the host path to access the host UNIX socket that you used to speak to the doc or demon as been pointed out, if you're, using something like G visor to do isolation on your cluster that might not be there.

B

Crio is the same way right so having the capacity to do these in a daemon 'less way with something like Conoco build, IMG potentially are interesting, I, don't know, I, don't know what the current limits are on gee visor in terms of running that set of things under it is, but it's one of the reasons that you know we we pursued Conoco is to run it in context like that. To get dockerfile builds in context where there isn't a demon.

E

So my question: is you mentioned that you would like to roll the VMS out, exclude them from the picture because of the overhead, but maybe that's not what's always preferable. In some cases, the users may want the reservation that the VMS provide, and there are several projects that support running VMs on kubernetes clusters, including Red Hat scoobert, and got the containers and vert led project that I am working on the latter.

E

One just makes it possible to run usual VM images as their pods and maybe will something be considered along the lines of using some of their projects to provide desolation that may be needed by some users.

B

So I, don't I, don't know that this sentiment was necessarily that using VMs as the isolation technology at the the pod level was the problem.

B

I think the certain systems use sort of single-use VMs to isolate the darker daemon that is or the container runtime that's performing the build and I think some of the patterns we've seen are problematic, even even in those environments right, because, if you're using something like cata containers, which runs a light weight VM to isolate your pod, your pod is reaching out to the host and interacting with a privileged process on the host. You might might as well not have put that pod in a cotta container to start right, because you.

C

B

Now own the host, pretty much does that. Does that answer your question.

E

Docker diamond that their forms they built itself inside the VM, that's yeah, so.

B

You're talking about like in the example where you launch something like a darker and darker sidecar, launch that in a cotta container as a sidecar and do the build against that I think that's I, don't know it's certainly a possibility. I, don't know if who's tried it but I think.

A

It's it's a possibility. What I've seen so far from at least just watching the community. There seems to be a lot more interest in finding a way to do proved. Unprivileged docker builds through things like Conoco, so, which is just gonna, be lighter weight than having to spin up a vm each time. So is it a possibility?

A

It certainly it gives you great isolation and I think it's got a lot of benefits there, but if we can get to the point of actually being able to do, unprivileged docker builds first I think that's preferable from a lightweight efficiency perspective.

B

If you, if you're looking closely to and the example, that's spun up, the doctor and docker sidecar had like a little loop, where it was waiting for the doctor daemon to be ready to which I thought was sort of funny. I think this man.

A

Had a question.

F

Hi, just a one small question about this source I built: does this open shift internal registry required for this source image builds? Sorry is the open, shipped internal registry, which is complete? It comes which comes with open shift. Does it required the build source image? No.

A

Source image needs basically I'll see if I go through. There's four things he's a darker socket because it has to be able to launch containers, it needs the Builder image and it can pull that builder image from any registry.

A

It doesn't have I mean if you're running on open shift, it's probably coming from the internal registry, but it doesn't have to, and then it needs your source code from somewhere and then it needs somewhere to push the resulting image which again on open shift, will probably be your internal registry, but it does not have to be no yeah so, and there is, there is also a command line version of s2 I.

A

If you just want to build on your machine where you don't even have open shift, you just do as I build tell us your source, repo, your builder image and the output image. It doesn't push it anywhere, it just builds it locally and then you can dock or push it yourself. I.

G

In the spirit of composability I'm kind of surprised here, but this focus on source right there are lots of build pipelines that already take source and produce. Binaries seems to be the reigning step to take the binaries and produce the image I.

B

Think there's a I think in that case I think the what a lot of these models do is they have a source that is a binary input. I mean one one of the options that, for instance, container builder supports, is getting the source context from Google, Cloud storage and versus a git repo I know that openshift supports a lot yeah.

A

I think that's a great point, I'm the image inputs that I referred to earlier in the build API. That's sort of that concept with the idea of maybe you earlier built some source into a set of jars that you baked into an image and now you're doing another, build where you're gonna just take those jars out and put them into a runtime image. That's lightweight! So that's certainly one way and source also, probably just an overused term it. It really is build inputs frequently get source, not necessarily yeah.

H

D

H

Summarize the reasons why we won't do need the talker daemon. Besides the run statement, or if you want to do something like that, you.

B

B

You said quickly.

H

So if I don't have run statements and I just want to copy my go binary into the into the image I'm good, all.

B

A

Yeah I mean if.

B

You don't have run statement in your docker file.

B

They started growing the docker file syntax again, so this may be dated, but basically every every directive in it amounts to things like add and copy which will modify the file system and pretty much. Everything else is some sort of metadata change, exposing ports declaring volumes the user, etc.

B

So if you, you know, know how to use tar, you can construct the file system elements and then the rest is a little bit of JSON manipulation. There's a variety of libraries that can do this and so run run, makes everything trickier because run is executed by sorry. There's a weird echo over there run is executed by activating the container up to that point, entering it executing that command and then snapshotting it right.

B

So, if you don't have run you don't need to launch and snapshot containers, and so you can do it without a docker daemon.

G

But isn't don't we have the ability, with whatever container runtime, we're using to launch the container X Q step and then snapshot it without docker doing it? For us.

B

Who says we have a container runtime, so it one at one of the things you can do. If you construct things this way is effectively from a Chromebook or OS X, not running a VM. In order to run a docker daemon is effectively cross-compile your go binary and slap it on the image right. You because you're, not activating the container, you don't need to do that. It actually makes situations like producing fat images in from a single environment with a single tool tractable, because.

A

A container an isolated container.

B

We're in the business of running containers, typically on orchestrators, right, I, think three of the things we talked about yesterday were build a IMG and Conoco. None of that. None of those use, a daemon, bilder and IMG are effectively activating containers, I think more or less directly. The run C, please correct me and Conoco expects some sort of Orchestrator to put it in an empty container, and then it basically emulates the docker file evaluation from by downloading the file system. Contents run by executing the command in that context, and then manually snapshotting it itself.

B

So, yes, I, think to execute run. Write like you more or less me some way of running the container because of how it is imperative, Lee built, but doing that through the darker demon traditionally means over exposing yourself to get at that sort of limited functionality. I.

A

Think there's another yeah, so there's another aspect of us which is in an ideal world. You know kubernetes already perfectly capable of running containers for you right. Why are we jumping through all these hoops to run a container in a safe way when, in theory, kubernetes is already running container safely right? The technical obstacle to that today is yes. I could start up a kubernetes container from my base. Image and I could exact in and run all my docker file commands there's no way to commit that container.

B

A

Push it sure so that's kind of the blocker to a build system based on that, if there were an API that allowed you to commit a run in kubernetes pod and get that image ID and push that image, then that would be another approach to this. But that's not a possibility today and.

B

You couldn't tweak the metadata right. Kubernetes will start it. It may start it with some metadata, but that I, don't think, would be committed to the final image.

A

Yeah you would, you would actually need a community API that says commit this pod and by the way, also tweak all this metadata about it.

I

So my question is more for the audience, so I'm I'm curious. How many people think docker file base builds is important for their future usage like? Is it a part of the experience what they already have in tech debt as a critical or meaning? If you didn't have docker files in your life.

A

Sample size of one.

B

Hey I'd argue, run, I'd argue runs not declared, but.

G

So I'm from a service provider, so I've got. Customers are gonna, want docker files as long as there are people that want docker files.

B

Me to you, that's and, and that's part of the reason for projects like build a Conoco right like we recognize I the custom I, don't want to speak for you guys, but we recognized the customer need- and you know this. This also goes to why GCB, the container builder was basically like. Well, we got a support, docker files and if we've got to support talker files until something better comes along, we have to do things a certain way and now that some of those better things are coming along.

B

We want to consider ways of reevaluating that and so I recognize the need for it. But you know I'm, I'm, personally, very interested in pursuing better ways of building images that you know don't need that and advocating for better ways that you know. Don't need that and I'm really happy that approaches have emerged that make docker file based, builds better, but I also, you know, I have a higher bar than that I think.

A

We're actually out of time, um we.

B

Were encouraged to one more minute, are you guys to jump us in the hall if we run out of time, yeah I.

J

Would just say that I think that when it comes to large platforms, where you're running kubernetes or openshift really at that end, you don't care what spelling it right as long as it's compliant and it works and everything's great, so the best and most efficient way. Whether that's darker today are some other technology. Like builder tomorrow's one thing, it's really only becomes a doctor fight only becomes something that might be important to developers on laptops who are learning about this technology and stuff like that.

J

But but once you get into a scalable environment, it really doesn't matter as long as the containers are compliant and run in an efficient way and are built in an efficient way. I.

B

Think we're at a time.

A

Yeah he's gonna ultimately be around and I feel thank.

B

You thank you. Everyone.