OCI Weekly Discussion, 12 Aug 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: OCI Weekly Discussion - 2020-08-12

Description

The OCI weekly developer's meeting recording from 12 Aug 2020; notes/agenda here: https://hackmd.io/El8Dd2xrTlCaCG59ns5cwg#August-12-2020

A

Wow justin joining our call.

B

I thought I might for a change you're, going to put it slightly unsociable hours.

A

Yeah, I guess so.

C

Probably not the only one here late so.

A

D

A

Nice to meet you darren.

B

A

A

Let me throw the hackmd in the chat, looks like we have a poop peeler call this week.

A

That's because I built the cloud.

A

E

For a second darren, I thought you were almost in an airplane. It almost looked like one of those overhead cubbies above your head. There, oh yeah, but I see that it's like an angle or something.

F

Yeah, no I'm in a closet, yeah.

E

You can come out, we don't care.

F

F

Yeah no too many kids at home, so I set up my office in the closet.

E

Was it at least a walk-in closet.

F

Yeah yeah, you know it's funny because I actually I I worked at home before a couple years ago and I also worked in the closet, but it was a much smaller one. So this time I just I upgraded to the master bedroom.

F

E

Does that mean you're responsible for folding laundry.

F

Well, no, there's no laundry in here.

A

Virtual background, or is that a real background that is really my office with a gentle blur applied, so you can't tell if there's a if it's messy or not.

E

uh You might want to check your blur factor. We can't.

F

A

Actually, I can switch now now that amy's here I'll I've got the kubecon. uh Yes.

G

Those were actually a hit, my god.

A

And of course, the uh we've got some starbucks backgrounds here to fake everyone out that virginia's open for business or something.

E

Yes, how about.

A

E

Loading masks on your face.

A

Oh yeah, yeah that'd be good.

C

G

G

Chat where you can get your own virtual background, come.

A

G

The ambassador's thing for github.

E

G

Options, someone.

E

G

3D, like you, could be in like the lobby and things.

A

Yeah I have that here somewhere yeah, I'm at kubecon. I.

G

Did not realize this were going to be so popular all alright good to know.

A

uh Well, darren came all the way to attend this call, so we should probably give him some time.

A

So how can be in the chat for those that aren't regulars? uh It's a easy place to add yourself to attendance and follow the agenda, um and so we have a few things darren reached out.

A

uh I guess yeah a week ago and um he's here to complain about multi-architecture images, because we couldn't think we couldn't think of a better place. So we said the user yeah yeah. So I don't know.

F

I don't know if this whole yeah- I don't know so. This is a generic uh here's, some pain points and no particular solutions. I guess um but uh yeah. So I made a couple notes like. Let me share.

F

F

E

F

This works, um oh wait. I think I shared okay. This will work um so basically kind of boils down to like three kind of areas that are kind of like pain points.

F

um So uh before I get into the specifics of those um like just to give context is like so like at rancher. We we support, like primarily it's just like intel and you know, and uh arm 64.

F

uh for various projects, so it's like rancher, longhorn, um rancho, west k3, os uh and then for like for k3s. We also do like arm v7, basically just raspberry pi, um so I mean I think we have like a decent amount of experience of doing trying to do multi-arc, um but one of the things like with rancher is like it encompasses so many projects.

F

I mean we're, probably talking about like around 100 different docker images across so many different projects. um It's not like. We have like a uh like the legacy of having a lot of consistency between everything. It's like we've developed a consistent pipeline, but the projects were all kind of different and built in slightly different ways and whatnot.

F

um So I think we have a decent amount of experience of doing this, um and it's not easy and just from what I've seen is trying to get other people to support. Multi-Arc is also not easy, um and so these are just some of like the gotchas or whatever.

F

um So the first one like I'll just read for those three and then uh I'll go into detail on each one, but like basically the first one just has to do with the fact that, like the manifest is one like immutable thing that kind of joins together are like all the architectures, um so it's kind of a step that has to be done at the end. uh The the second thing is like push-pull tag.

F

Oddities I can go into detail those, but, like there's just a lot of odd behavior, you know this one, you could probably say is like. Oh, this is tooling issues but honestly, like I, don't even think it's tooling issues, because it's not exactly sure what the right behavior is like. We really became very evident when we started trying to implement k3c, which was like our own little weird container, runtime experiment of like well. What's the right behavior for all these manifest things like what's the right hash, I should be showing um and kind of.

F

The last point there is is the digest that you're? Looking at it's like it's kind of inconsistent and odd, which you know it's like what digest gets printed and displayed for a lot of things? Is it the manifest or the image? And so, if you're looking at, like you know the output from like a a pull operation or a push operation or um a runtime like a docker, inspect or cri output, it's like which hash is it it's?

F

It's always kind of odd, um but um I'll kind of come back to those points, but like so a couple things that, like we kind of learned in cdi, is like how to address this because we had to do so.

F

Many different projects is like, like first off like the docker build is like really it's just it's one step in a larger process, so anything anything that kind of looks at is like assuming a solution where it's like I build all architectures at once is not super feasible like that's just not very like you know, I think, like kind of like buildex does this style of like I build, and I do all the architectures at once and then I can like assemble that all together, like it's just, not really feels feasible for a lot of projects, it's significantly easier to build on intel to build on arm like to do these operations, um and so, like the second point, there is like cross, compiling sucks.

F

It's like it's so difficult to cross, compile just in general, that's a well-known thing, um but because, like the architectures are so like arm is so fast these days it's like you know, those are the primary architectures, at least we're supporting, and we see it's. um You know important. You know phil I'll, throw in their uh power pc is, but that one.

F

You know that one's fast too but like, but the point is, is like people are, it's it's just way easier to build on the actual architecture, and so what we've kind of come up with and like what we've seen this repeated over and over again in other project projects or whatever is if you want to do multi-arc. It's really like you just take the pipeline, you have, and you just run it on the architecture.

F

So you don't cross compile. You just run multiple versions of your pipeline and you build kind of everything in parallel, but that's like where the big, the the annoying part comes with the manifest is, if you do everything in parallel, you kind of need one step at the end to join it all back together to create the manifest- and I know like it's possible- to amend the the manifest.

F

But it's like that's kind of racy because, like you, have to pull, modify push it back and then also it's just the thing of like you. Don't necessarily if you want to like the end result is the tag you don't necessarily want to be tagging a manifest that you're then modif like you, keep basically modifying that tag.

F

um So we, like all of our ci pipeline, like kind of what we do is everything's kicked off through a github of when we want to do a release. We create a tag in github um and then that will kick off ci.

F

Then ci will run in parallel across um all the architectures and then, when all the architectures are done, we then have one step which will then create the manifest and push that up there, but that, like that, last step of like kind of joining together is where a lot of ci systems can't really do that, and also the other thing is like it's hard to find one single ci system that will do all architectures, um namely windows. Like you know, it's like you might be able to get. um You might be able to get like.

F

You know linux, intel and arm in the same ci system, but then windows is always a different beast. um So if you want to support windows, you know so it's like you're, usually dealing. So it's like we've.

F

We currently use drone, um which gave us a lot of flexibility and and honestly, like we'd, like to move off the drone to something else, but because of this one little thing of like how we have to kind of structure, the the builds all in parallel and then kind of do one step when they're all done at the end makes it a little harder to just move to some other off-the-shelf ci thing.

F

um So I just wanted to like point that out like like this is like just like just what we found is the easiest ways to build in ci and do multi-architecture is really just in parallel and on the actual, like running full pipelines in parallel on the actual architecture.

F

um So that's mostly like why, like the first point, ends up being like a hard to deal with, um I don't know how you guys, like I think, like maybe I'll, just go through these others too, and then I'll just open up, and we can discuss I'm not just like talking forever.

F

um So the thing with like push pull tag like we run into this a lot where it's like, um basically like. If you pull a manifest, um it depends on the docker virgin, but I just checked yesterday, if you pull a manifest he'll say like the digest of the manifest, and then the image gets reported as the digest of the manifest.

F

But then, if you tag it and push it, you'll actually tag the actual docker image, and then it will push a different thing like that. It'll push the image so like this is a common thing that we we've seen where it's like. People um like our developers, are screwed up where it's like, because all of our images are multi-arc where it's like. Oh like tag, this one is this label, and basically nobody can ever figure out how to tag a manifest.

F

um They end up tagging the intel image, not the the you know the actual manifest. So then our tags get screwed up.

E

Aaron, when you say manifest, you actually mean the the index right. The multiple sorry.

F

Sorry, the manifest list- yeah, sorry, okay, sorry.

E

Just making sure.

F

Yeah sorry, the manifest list- yes wrong, word um yeah, so it will actually tag the manifest list, um yeah um yeah and I already kind of whatever what what digest. So, it's like there's all these kind of, so it's kind of like so it kind of goes into two points of like one. The the one thing to point to everything is kind of hard.

F

um The the second is that, like we've, effectively created like with manifest list a pointer and it kind of comes with all the problems of the pointer is like, depending on the use case, it's like do you actually want a reference to that, or do you want a thing that it's pointing to and, like you see this odd, like these complexities get like manifested in, like, let's say the pull code in container d, it's like resolving these lists and stuff has weird side effects um because, like you want to pull, but you only want one architecture, but then, if you pull- and you only want one architecture, then you end up with a manifest list that then points to references of a different architecture.

F

You didn't want to pull so you cannot have an income chain disk. So there's there's all these kind of weird audit, but like at the end of the day. It's like really what people expect and want is like if you just like tag and push on one architecture, and then you pull on another, it's not found it's like that's, like kind of the user experience people get they they want it to happen, but it's like not like that. You know like today. It's really you know the the you know have to do.

F

The second step of the man like doing the manifest or manifest tool or whatever. um But then it's kind of like this is all kind of tags are all in the same name space. You know it's like if you, if you look at parallels of any other uh packaging system, you know rpms and debians or whatever is they? They effectively have separate name spaces, uh different contexts for each architecture.

F

So it's like it's very obvious to most people. It's like well yeah that one exists in arm and it doesn't exist intel because they're like completely separate repositories or or whatever, um but we've kind of with the manifest lists.

F

uh It's you know, put everything kind of in the same place and you know it's like if there was something like architecture-specific tags, that would probably be cool um if there was some way to do this client-side, uh you know resolving it. You know not client-side but like having some behavior. Where you look, you know, look for something it's just like. I don't.

F

I question the at the end of the day, the value of the one artifact to point to multiple architectures, and I assume that probably has a lot to do with signing and whatnot, um but like those benefits, have yet to come, like actually be seen in the wild, because there's really no flow yet for like at least the kubernetes world of how to validate all the stuff and signatures. And you know that's what like notary, v2 and stuff, I guess they're going so that's kind of like my basic points.

F

I don't know if any of that's useful or helpful. Anyone wants to comment on anything.

B

um One thing about signing is that it turned out the other day that no one ever agrees, whether we should be signing the manifest list or the individual images at all. So actually, this turned out to be an open question last week that we hadn't really discussed, because it turns out that a bunch of people are expecting to sign images and another bunch of people are expecting to sign manifest lists.

F

Yeah it- and I just I would almost I would because I just think, because it's not even clear to me, um because I just look at what what's the the the behavior today is like everything just really relies on it. On a tag and like I- and you know you put that like in your definition your manifest like, I don't even understand how signing is gonna fit into all this.

F

um But again I would prefer to sign the images because, like again, that's gonna make more sense to me, because, even if I don't sign the images like if I sign the manifest list, I have to sign the image too, because I can't create the manifest list without knowing that the images are signed um because, again, like I'm not going to do it all at the same time, they're going to be done at different times and on different systems.

F

So the images are going to be signed. If I care about signing um yeah.

E

Yeah I mean the notary stuff is like a whole nother thing and it's a good question. But what I think what was interesting is your last slides and what your expected thing was: docker pull a particular tag and it would just work on the various architectures, and that obviously, is the goal of what index is supposed to be for, but is your pr point that it's just too hard to get there? So it's not working.

F

Well, so it's it's like the behavior is that, basically, if I push a tag on an architecture, then that tag exists for that architecture, not like the polling, so so this is. The thing is like technically, like manifest lists, are working fine from the consumer side right now, like that's kind of the like, you know, for the most part that work it's it's all on the kind of the publishing building side. That's the complexity! That's.

E

The problem yeah and then because.

F

It's so complex.

E

What we've seen is various people do things differently. So, like we went through this a lot with the microsoft images we had this raging debate with net images and separate repos all in one repo and drove everybody kind of nuts and everybody's, pointing see what they do. It's like everybody's, just doing whatever hell they want, there's no standard, that's like right! So we debated should we set a standard, but we saw that with the different architectures and even the cube.

E

Images are split in various ways and I is the question that should index to be able to support both or just if we made the index tooling and just set the standard say hey if you're doing a foo that the architecture, the various architecture food would be in the same repo, and if we fix the index tooling, that people can do it easier and that pattern would just work.

F

Well, I mean I definitely I don't think people want to run multiple repos. You know if like if there's some way to name space within a repo, they still want one logical storage thing they don't want to. You know.

A

F

Then that's going to be like some administrative operational overhead, um but if they can, you know logically separate out, because because I mean most of this, it just comes.

F

I mean a lot of it, at least for me, as it comes down to the tag. It's not so much the you know the the actual digest, um but it's it's for me yeah, it's just mostly the the tag it's like if it was possible for a tag to be scoped to an architecture.

F

uh I I think that would alleviate like a lot of things.

E

But again you kind of go back to tooling, because technically you can have a tag point to its pic like, and I have to double check with. The latest thing we did with net was, but in the net ones. We did come up with like the standard. That said, and I'm not trying to say that everybody should follow. I'm just saying that the technology is there, like.

E

You can have a tag for an architecture and we kind of had this thing, like you know: dot net debian, yeah, blah blah blah version, and that was all in the the tag and then the index would still point to the underlying digest. But the interesting thing is, I think, what you're kind of getting at is because there's no great tooling for the whole thing and an index points at digest.

E

Doesn't it it doesn't pay attention to tags that you can easily get your tag and your index your tag for specific architecture and the multi-arc index out of sync, because you have to make sure that these are all manually assembled every time.

F

Yeah yeah, it's really yeah, it's really easy to mess it up, but the so that was like. So when I first brought this up on twitter or whatever was that I made the comments like I don't you know, I don't even know if there's a spec issue, you know it's like, but even if it's a client, runtime issue, it's like they're. You know it's like developing some standard of how to how to do that.

F

So I don't know if necessarily a spec needs to change or if everyone or like clients need to like agree on some standard or something.

E

um I think it's just tooling because from a spec perspective, all the pieces are there like if you had tooling, because to your point and your point about building and multi architectures and multiple build environments is an excellent one, because you're basically building these things and just because you build one like day: zero right, you're, building, all four or six architectures, and you have to build them on separate environments or they're, just built asynchronously.

E

You want to push them individually, but then at the end, and you want to push them probably with a specific tag. But then you want to be able to say all right now I want an index, I know what the tags are.

E

Can I use some tooling where I pass the four tags to this thing, specify what I want the index to be tagged and just figure out what the digest are and make it all happen. And then, if I want to update one of those, I can you know kind of roll. That update forward too, like I guess, I'm just getting the spec is in is supports all of that flexibility.

A

Yeah I mean so, I don't think darren. I mean manifest tool. Does exactly that. I mean you tell it which images you want to assemble and the final tag. I think two two kind of key things that came out of you. What you shared darren I mean one, is that the development of multi-arc or multi-platform images happened iteratively.

A

While there was already an expectation for what certain tools did so docker pull, you know had a history of what it did and when we introduced manifest list it. The engine hid the fact that it pulled this set of pointers and then made a decision for you based on the cpu architecture of your engine's. You know system and then threw that away and basically reflected in all future commands. Docker run docker start whatever yeah that you know you've now dereferenced to your local architecture uh and the digest has changed on.

E

You like, for those.

A

A

Representation of a change but.

F

Oddly, the digest doesn't change like I just tried this yesterday. If you pull it says the digest of the manifest yeah. When you do docker images yeah, it's weird yeah.

A

So so, there's that sort of use case and existence of tooling in container d, we decided to to sort of go oci native and and still represent, at least in the api. You know that you are pulling an index, and so we you can pull parts of it. You can pull the whole thing. You can push parts of it. You can push the whole thing. So it's a little bit of a swiss army knife that now you know, when you embed container d, you have to make that decision.

A

How you want to you know, represent your interaction with index uh dereferencing for lack of a better term. um So I you know I get that I think that's mostly yeah. I mean I don't know what the solution is. As far as you know, docker added the manifest experimental command.

A

That is, is more the natural interface point to what is this thing and what are the parts of it and how can I modify it? Docker pull and push have never necessarily changed to to reference that reality of manifest lists or indexes. So there's a bit of a disconnect there. That, I think, is just a long-term long-term, like decision about how manifest sub command and pull push run should interact.

F

Yeah, but so I, but I just generally question: what's the value of the manifest list.

A

Yeah so yeah- and I that was the second thing I meant to say- or I should have said it first, but you know the the whole impetus for the start of these ideas. Was you know I'll skip, ibm architectures? Let's just talk arm you're, an arm enthusiast and you're following some online tutorial that says: docker run alpine colon 3.7.

A

Well that doesn't work on arm.

A

So you know we had all kinds of ways to mess around with, oh we'll add underscore armed and we'll post the the r variant of it over here with this tag name, um but you keep adding architectures power and z and whatever else is out there window well, microsoft came along at the same time. So now you have windows, images and some you know my sequel. Could you know docker run? My sequel could can work on windows um so that idea of we want one name representation for an image.

A

Somehow behind the scenes you have to represent that that could be a very different image based on you know, obviously, on the binary requirements for the cpu architecture running on or platform.

A

So you know there are two options. One is you you hide all that behind a registry that now has to know about these details and the the design and decision was no. The registry should be a blob store. It shouldn't have to know these details because there were some early patches to the v1 registry that tried to play that game. Like oh I'll, know your architecture and os and I'll hand you what you want, which takes away all the problems that that you know are now on the packager to solve.

A

You know today, um so the other. The other option is to find a way to and yeah. I I think I don't t. I don't know if you have any great wisdom. I mean he's been doing this for a couple years now, building all the official images multi-arc and handling the the synchronization of building across a build farm and then doing the manifest list, but yeah there's no magic there. It's difficult.

H

I I would say that darren, you sound exactly like me three years ago, some of the words you're saying are verbatim what I was saying three years ago.

F

Then you learned better no or you just accepted the world.

H

um We we just accepted that this is the world we live in and that we didn't have the power to influence that change three years ago. So we found workarounds and we're not happy with them to be clear.

F

Yeah, so I think I definitely agree with the idea like they should not be on the registry side. The registry should not be dynamic and trying to make a decision and giving you what you need. So I think the the behavior should be in the client.

F

I like that approach to me the I kind of feel like it's kind of like we have a human issue of like well, we want a human to be able to like just put in foo, and I get the right thing um so to me, like that's kind of it falls in the realm of tags. It's like, I, I feel like there.

F

There might be a solution, that's more around attaching metabate and metadata or scoping or some, because I don't think we should just do name mingling in tabs tags, because we already know there's multiple fields for an architecture, but if there was some way to have like kind of an enhanced version of a tag that has some metadata on it or whatnot.

F

So I can say: okay, I want this tag for this architecture and, if that's not found, then I fall back to a different one, because today, a client kind of does that are already with the manifest list of like okay. Well, I have this and, like I have this information from the manifest list. I have to pick what's the right one, so the clients already have kind of the behavior picking the right one, but what I'm saying is like is is basically instead of there's one thing tying that all together.

F

It's still just like a logical name- and I know this like when we get into signing- I don't know how any of this makes sense, but from like the kind of like the user experience and and making the tools simpler and whatnot, I it would make you know it's like. Basically, when I create a tag and I push it, it just attaches the architecture to the tag. And then, when I pull it, I try to find the tag that matches my architecture like yeah.

B

What what if the I mean, because at the moment the pull works? Fine, because it trickles through and the client knows what I mean. Are you kind of asking for the push to be the same where the registry you would if, in effect, just create the manifest list for you from everything, pointed at the same tag? So the only problem is the poll doesn't completely work when it doesn't.

F

Completely work, but it it.

B

B

Sorry, if, in an image-centric world, it works as long as if you didn't have to do any operations on the manifest. So if the registry could manage the manifest list creation for you automatically so every time you pushed something with the same tag, it would create. It would add, to the manifest list, say: hypothetically.

F

Oh, oh yeah, I mean, if you dynamically, so we'll I would just say, because the only other problem with the poll is the you know it's kind of like um there's some loss of information. A lot of you know when you pull and then you tag and you push back.

F

You know, what's what's the right behavior there of what you actually want to tag the image, but.

B

Yeah, I was kind of hypothesizing an image only world where you would always refer to the image, but.

A

B

There's something the rest of it: the manifestless bit would be hidden from you and you'd. Only ever see, image tags say.

F

Correct that would yeah. So, if like I could basically tag an image and push that and well I guess when I push. I say this is the: is the architecture in the image I don't know um but like if, if yeah, if the server was able to know how to do that and then dynamically create the manifest.

H

Yeah one of one of the challenges here is that the manifest list or the indexes introduce the platform object that has os architecture variant and then also the the windows.

G

H

Right the the kernel version, because that has to match as well, so there's not just a bucket for windows, there's multiple buckets based on which windows kernel you support where the the image object itself only has os and architecture. It doesn't have a full platform object.

F

H

Which is also a pain point.

F

Yeah, that's that's why I'm like in my mind, I was thinking like basically taking that platform object and somehow associating it to a tag.

F

D

Can I jump in with a question, uh so I like justin's image, centric idea, um I guess the one drawback I said: what do we do in a world where we need to work with content from multiple different architectures on a single local client like what does what do the images look like? How do we interact with them? That would have to change if not drastically, but like we would need some sort of local discriminator. Oh no. I want to actually work with this.

D

I want to work with the arm64 bits and bytes for this particular operation as opposed to the emd64 or whatever it happens to be. um You know how do we present those? How do we navigate that content.

E

I, if it's the host saying when the host can support multiples, or are you saying when there's more generic tooling, because you're trying to pull it down not related to a docker run kind of command.

D

Right, like you're, just working with the content.

H

Okay, I would say both because there are cases where a host can support multiples, especially say you have an arm64 chip like say the aws arm chips that can support, arm, 32 images and not just arm 32, but arm32v7, v6 and v5.

H

So there are potentially four different image variants for the same image that you could run on that chip and might actually be useful and meaningful to run on that chip, not to mention the qmu user mode, emulation that docker desktop does by default, which opens up a whole larger world of possibilities.

E

So I'm curious from those that are building multi-arc images, because we one of the things we found, at least for the microsoft images, is that we had to publish the the platform specific ones, because people wound up with very weird scenarios when they went to the multi-arc for various, including windows like in acr tasks we actually had to.

E

We make sure we update those very regular to the within a week of the new windows versions, because if you use a multi-arc and in the multi-arc tag they put the new one and the host we saw this with devops doesn't support the new version. It just breaks, which is just broken, and that's why I was like apologizing for on behalf of windows. But it is what it is the.

E

So what we. The first thing we tell customers is like no, actually, you should pin to the specific one you want, but it's helpful like there are places where it's definitely helpful. It's kind of like there are as much as we bash on latest having it available is helpful for like walkthroughs and so forth. So I'm curious for the one for people that are building the multi-arc ones. What do you find your? Where do you find your customers use the multi-arc tag versus they actually want the platform specific one.

H

In in official images, the the biggest place that we see users reaching for those platform, specific tags is when their tooling has failed them, because they're using docker without experimental and thus can't use dash dash platform to explicitly specify to to de-ambiguate the tags or where the platform detection has failed, and they actually wanted something else or they're, using something like the qmu user mode emulation and don't want to have to look up the manifest list to find the digest of the particular architecture that they're actually interested in running.

H

But for for us we ended up with the the platform specific image tags. We have a separate repo for each architecture. We push both because it's useful for users, but also as a quirk of how we have to push manifest lists because we're doing exactly what darin described of each architecture has its own pipeline.

H

That's totally divorced of all the other pipelines, because that's the the easiest way for us to manage our disparate infrastructure and the number of images that we support concurrently and we needed some way to to reference those individual tags after the fact, in a consistent way. So that we could construct the manifest list, because if, if we just did something clever like push by digest to the correct repo, we then don't have somewhere persistent to store that digest, so that we can construct the manifest list later.

H

Unless we created a unique tag for each architecture, combination which, at that point, we're bloating the tags of the official repo instead of having those off elsewhere, which is why we went with individual repo per architecture.

E

Okay, so he likes the individual repo just because the tag listing was becoming too complex.

H

Well that end, because it it made it simpler, there's there's a one-to-one mapping.

H

If you you, you can always go to an official image and know that if, if you want a particular architecture of that image, you just add the name of the repo architecture specific repo to the beginning, and it will be the exact same tag and it will just work.

F

So we I like, I just ran and killed rci um for an issue related to this. This is my own problem, but just shows a complexity is that we do the opposite of you know we put the architecture in the tag, but the software version is the tag version and but it gets confusing because well the tag of the image for the specific uh one we're building is actually the software version plus the architecture.

F

But then the manifest list is the software version, so so, like different stages of our ci pipeline, the the tag equals the software version or the or the the you know or not you know like it might have. So it's like you know it's very easy to screw up. So it's like, I kind of appreciate how you did it with the different docker repos, putting different architectures, because you can keep that tag consistent.

F

um That makes a lot of things much easier because you know it switches in our case and we mess it up all the time.

E

Yeah we I mean just we struggled with that also, and then we were trying to get a good formatting in the output. I put some. I pasted some things there to try to help with at least where we wound up on.

E

It was because we just felt, like the multiple the.net team at one point, had everything in one repo, including the sdks and the runtimes, and the uh the the build dependencies and everything, and it was just this monstrous list, and then there was this: no, let's go the opposite direction and we were trying to figure out what the right balance was um yeah, but.

B

Yeah I mean it, it definitely makes thing I mean like those of us who do this. Quite a lot have trouble with it. God help the people who are just trying to do it with a couple of repos and just generally build stuff for arm. That's, I think, that's part of the problem with this is like yeah, you it's manageable like we all.

B

We all build multi-age images and and live and hey and get annoyed by it occasionally, but it works, but I I I definitely agree that not everyone in the world is able to build yeah for us too hard.

F

Yeah- and I think well and and the reason why I kind of started chiming up about this- was just because honestly, like I'm, super impressed with arm 64 the speed of it these days.

F

You know like the graviton, 2 and stuff like that and in and it's becoming actually more feasible for me to run cheaper on arm, but it's just a pain to build all these images, and I don't want to go like if it was easier for me to build multi-arc images, um then I can move more of my infrastructure over to harm and then there's the impending mac os. You know arm 64 thing, which I don't know how we'll work out.

H

You mean apple silicon right. Yes, of course, it's not arm.

F

But but so the thing is like when I look at this is like no matter how you're doing this, let's say: you're signing you're tagging, whatever you're always going to do the architecture specific one. So it's like, I always have architecture, specific tags, and then I pull those tags together to then create the manifest list. um So you know unless you're building everything in one spot, but that just doesn't seem very feasible um for a lot of a lot of use cases whatever.

F

So I keep kind of going back to that. It's like it seems like there's something you can do with with just putting um the architecture on the tag.

F

But I don't honestly don't even know what the api in the structure looks like or even, if that's in your guys realm.

E

Yeah, I'm still kind of hearing it's a tool like I'm not trying to oversimplify, say it's a tooling issue. It's just that simple, because it's clearly a big pain point, but I I especially with the experimental flag where you can pass in the the disambiguation issue of which architecture to pull it seems like darren. What do you think? What have you seen from the from in the multi-arc places where you're pulling with continuity? Is that part resolved in really is just the tooling on what you do for push and update.

F

Well, hmm what do you think about that? Oh.

E

Sorry I was actually talking. I I said darren. I meant derek, not nothing. You shouldn't answer as well, but I was. I realized. Okay,.

I

um I mean we, we tried to solve it in container d by always preferring the the digest of either the index or the manifest. Whatever was was received from the registry, so try to be like less opinionated um docker not matching is kind of complicated, and I spent some time trying to fix some of those back end issues, but that was never uh never merged back. So, like I yeah I would like to like.

I

I I'm I'm very aware of like what the issues are related to docker and that if you pull and push something that has the architecture as it gets stripped, but with container d, we basically had to build up this whole platform. Matcher and windows definitely made it more complicated than just like simple, like string matching um but yeah. It's it's it's kind of complicated, but yeah. We definitely want to know like if using the container d interface is really hard like.

I

We definitely know, there's like usability issues like if you were to pull and then try to push, but you only pulled one architecture. We've tried to work around that through stuff like cross repository push, but that doesn't work too well. If you're going to a different, a completely different registry and yeah, it's it does make the the client tooling more complicated um but yeah. I would definitely like to know like what part of that we can make better, but polling is actually a much simpler problem here. Building is really the hard part.

E

I

Long as somebody didn't.

E

Screw up and push the latest and overwrite the the platform specific one yeah I like.

F

Because I I don't know how you can easily solve this with tooling, with, like without the user always having to assemble the manifest list. Is that not the right name? Is it called index.

I

Oh, it's called both there's the docker manifest list, there's the oci index. Okay, the same thing depends.

E

On where you are yes,.

F

So so the problem is, is I mean, and we you have to kind of accept the react because like because the other thing is like well, we say it's. A tooling issue is like if for better for worse, docker is really the only tool that the majority of people are interacting, interacting with like um container d, you know as much I love container d will probably not really make its way to the most end users.

F

You know it's, it's it's much more lower level runtime um so, but, but even if you know I just don't know you know, if you say it's just a tooling issue of how how do I actually fix this um without just basically trying to build a better manifest tool, but still fundamentally the manifest tool has kind of the problems of you know. I need to basically pull down a manifest augment it and push it back, um which is difficult to automate.

H

And can be done, atomically.

F

Yeah yeah and because, like we, we don't like because I know some people are are have accepted and they're like oh fine, I'll, just always amend and they're fine. With that. From our perspective, we don't do that, like I'm, not going to amend a manifest list, I'm only going to push it once.

F

um You know, maybe that we're too pure there or something, but but but that makes it a lot harder and and the only way we get a we can really easily do. That is with consistent tagging, um because we we just basically know because we don't actually know what the digests are.

A

We have to I mean so I mean that that aspect of it I mean tnot, actually contributed or suggest. I forget which, but you know the the manifest tool uh usage. um You can say hey. I want these five architectures they're gonna be named this uh but fail if they don't all exist already in the registry, and so that gives you the option. I mean it's sure it's it's ugly.

A

In the you end up pulling or you know, retrying until it works, uh tiana ended up, adding a succeed if not exist, and so, but that's the amend option you don't actually have to pull an edit. You just keep pushing and if there's more available you get a new manifest list that has more entries.

B

The problem with the problem with that we discovered with things like official images, is that sometimes ubuntu, you do docker run ubuntu latest and it fails because any arm has been pushed and you're on x86 and you can no longer run ubuntu latest, which makes people very unhappy.

A

F

A

I

Yeah, I just think I think that the distribution api here I mean, I think, very early on we kind of established that the distribution api is kind of eventually consistent in nature.

I

It's not really a good place to like try to do atomic operations, um but I I think I think maybe there's there's some room here where we could add something in the future to the distribution api to at least making the creation of the the manifest list. Slash index easier, um maybe even having like a separate endpoint that could handle like creation of it from multiple tags that, like atomically, like remove those tags and replace them or something.

E

Like update here's, the amd64, please update any index that references this tag or something like that.

D

The maven world does that with artifacts right you can upload various different. um I forget what they call it, but it's essentially like a suffix that allows you to accumulate more and more bits, whether it's architecture, specific and then you've got this qualifier yeah, and then you and then I guess what it's missing is the opportunity to to uh to lock it to prevent further data from accumulating. But you know there is prior art.

F

But but I still fundamentally go back to it's like we know this introduced some complexity. You can try to overcome it with tooling and whatnot, but I still can't really justify its existence. It's like I, you know. I know man like index, probably useful for other oci artifact things of like I actually want to create more of a bundle of artifacts like that seems interesting.

F

But in this use case I I don't know what we're exactly yeah like this specific design like like what it gains us or whatever.

E

Well, is it so from the consumer? Well, that's. The part is that tion was kind of referring to, I think was tiana. Maybe jacob was like when it works like if you're able to get it work- and you know you've got a good index, then for certain scenarios the pull is actually kind of clean because it just works. It figures it out like if I'm trying to ship an accounting software package, and um so as with many of these things that can we optimize on the consumer side.

E

The problem is, it sounds like we're getting in there's enough edge cases in the consumer side that even there it's frustrating but um and that's what I.

F

E

If we made the tool so that you're so frustrated with the tooling you're kind of questioning the consumption- and we again, I totally get it- I'm not saying you're wrong.

A

I mean there are aspects to the way it works today that I think I mean sure I guess you can go back to the drawing board with new solutions, but helm charts default, kubernetes, dml files. All these things, reference image names that are multi-arch, so istio just runs on our you know you can install.

B

A

On any kubernetes cluster, so you, if you don't solve it this way, you have to solve that some other way, because you can't.

B

A

People aren't going to accept, I I got a different yaml file for power pc and a different one for x86 totally.

F

B

I don't like these.

F

Web assembly or something terrible yeah, but but but um but honestly, it's like what what it really comes down to is you know, and I definitely don't want to push anything more on the user. What I'm saying is the actual solution, or whatever is it? Is it really just you know comes down to like basically, you know a dirty solution as a convention in the name, you know it's like that's how we did multi-arc before um we did multi-arc before uh manifestless became a reality.

F

Was you know in platforms where we could just intercept the call from the user we just put in a an architecture in there.

E

But then it doesn't work like the helm scenarios, where I just want to reference something and it it figures it out, like there's certain images that really do benefit. Otherwise, then they have to do some kind of convention to just like at some point, you have to rephrase that if you are trying to reference multiple things with one name and a mirror, quoting name, then there has to be some solution either. There's an embedded thing in the way the index works or there's a convention that gets slapped on after the fact.

F

Yeah so yeah, so I think, like the I'm proposing, is to basically embed more metadata on the tag. So you know the tag doesn't so the user still uses the same name always, but the runtime can know that that tag is not actually, which would that mean you can actually have multiple tags of the same name like with different um uh or whatever I mean I don't know exactly how to be implemented.

F

But that's the point is that, like it's, you know it's kind of like name angling, but you do it in a proper data structure, but I mean because the fundamental difference, I'm just saying is that you don't tie one artifact, which then refers to everything. That's where I can't seem to, in my mind, get get to a better solution is where I always need to create one artifact, which then points to all the others.

A

I think I mean I, I think that's kind of where justin was heading with I mean it definitely would impact. I don't. I think it has to change anything about how indexes manifest configs, all those sort of spec spec-out pieces of of how these things are represented, but I mean it almost sounds like you're saying, distribution api could be augmented to accept more information and therefore I could docker push the same image name same tag, but with this additional information I could let the registry handle the the whole concept of indexes behind the scenes.

E

Basically, the registration.

A

E

The tooling part of it is what I'm kind of hearing.

F

The only complexity with that is is the it's is then like. What's the digest that you're dealing with is the deep like because you have an intermediate digest of you know, because you're you're still you're, basically going to dynamically, be building some new uh manifest list or index.

F

um So you still have the complexities of like deciding what to do from the client and runtime perspective of yeah.

A

Yeah throw it away, I mean, but we accepted that you know apt and rpm have to deal with that as well. I mean they have to. They have to have enough smarts to know. Oh, I should actually go yeah, it's the same name package as everyone else, but oh I'm on x86. So I'm going to this repo to get that, and so there is, I mean it's not the same. I understand yeah.

F

No because I'm saying there's actually this artifact that that exists, you know so it'd be one thing if, like basically the um I guess, maybe the.

F

Yeah, it would be one thing: I guess: if maybe the the index always lived on the registry and was never pulled, I mean it was never like stored as an asset. You would never really see that digest, um because that that's you know kind of part of the complexity today is that you have an additional layer and it's kind of shown to the user.

A

Yeah, I mean to me: that's that's, maybe a smaller issue. I mean if we need a different visibility level, so to speak about how these things operate. That's that's kind of a minor.

B

It would mean you could never pull by digest um multi-arch, which is a feature that you do have now. I guess that's only that's the only difference, but um that but which you don't get um on debian say I mean there's no kind of equivalent.

B

Of that I mean the debian thing is just a an additional path component for the architecture at the beginning that you don't see and effect which you you could definitely do as a as a kind of registry convention that all registries, dot, amd, 64, slash and then everyone gets their name spaces. For that and things I mean it, windows, slash, amd, 64, slash, justin, cormac, slash thing and they all get pre-populated for you. But it's.

E

So, just from a time check I mean this is good. It's really good. I think we've been banting around this for a long time.

E

I think, like I don't know, if there's an entity that is listening and taking all the notes saying: okay, I've got this I'll, go fix it all like this.

A

E

Become a typical oss project, where phil's got some great tooling in place, docker's got the docker manifest tool. Is there some energy that somebody wants to take this on and and fix it? You know and kind of like become the the the lead on on this and run with it.

E

Who's got the bandwidth. The interest.

F

Hey jacob, um so I mean in terms of if it like comes down to like to uh work in coding.

A

F

Implementing uh we can definitely do that in terms of design and what's the right approach, I don't know how to go. Go about that.

E

We just get some breakout groups like we've, been doing at some other things. It's the problem with the problem, isn't so much the breakout groups. The problem has been there's lots of people, lots of ideas, who's got the bandwidth to actually start committing, whether it be markdown or go files or whatever, like phil's, got some great tooling in place. You know it might be, it might be the right start. I don't I actually I'm ashamed to say. I don't know that.

E

Well, um you know so it can it be somebody that he jumps on what he's got and starts saying, hey, look! This is what we're thinking and then people are reviewing things, but it's at the end of the day. It's who wants to take it and who I can actually can run it through to attend.

A

E

Because I think.

A

We'd all like it.

E

But we'd all love to see this get better. It's just it wears fit on our priority list of where everybody is working on.

H

A

Here to support you, man.

H

It sounds to me like the lowest hanging. Fruit is actually adding that platform object to the image specification.

C

Is it already there, I'm just checking my code and I'm seeing a platform object in the index. Descriptor yeah.

H

But in the index, yes, but not in the image specification.

E

Manifest you mean in manifest itself.

A

Yeah, so it's today the config object has an os and an arch. I think, but that's it so it doesn't fully map the full platform in the image.

H

So if say you, you have a manifest list, it specifies a platform object and you then resolve that platform object to an individual image. You have now lost details of configuration that only exists in that manifest list object, because that platform object contains details that don't exist in the image config.

E

Oh, so they got out of sync a little.

F

C

F

Yeah, I think the config doesn't have os version, at least.

E

Was that the windows.

I

Let me fix that somewhere and like build kit or something as well. Yeah.

E

I think docker adds it.

I

That's off spec, though,.

A

An ocr level.

I

A

C

I'm not seeing os version os features and variant in the image.

A

A

The most experienced with manifest lists, so he probably he probably wants to get involved.

E

You just did about, did you just volunteer him? Is that what I was trying to get darren to volunteer.

H

I mean if, if darren's in I, I will absolutely have as many conversations with darren as he can stand to sit through.

F

I mean yeah so like I, um I would like to volunteer I'm trying trying to think of like um yeah, I mean if we can john, if, if I I mean, maybe we can have some type of follow-up discussion or something um on what seems what seems feasible. I don't I don't know so I you know I. I think that the hard thing is just for me trying to come to like what I what a solution is. I can.

F

I can just mostly complain to what are the what are the issues so, but maybe we can get there.

C

Yeah, as I'm probably one of the most impacted by this by the guy that does the illumor stuff, which is another os, I would also like to join and help out whatever I can.

D

I'm interested in helping as.

D

E

All right, I I'm taking accountability of typing your names right. At least we have some notes that are written in addition to the recording, um but anyway I think we'd all love to see it. It's just a matter of like a divide and conquer thing, and I um I'm sure you'll get. You know constructive feedback from others. um So, let's run with it, okay, cool all right, I gotta run to the next.

E

I will uh so we're just as a point of order or whatever it's called we're off next week, because we all have kubecon because of all these backgrounds and we'll pick up the following week, and um hopefully we can.

B

Discuss the items.

E

On the agenda that we're yours that we're missing today, yeah we can, we could pick it up following I'm sure there'll be some content, but I'm hoping you know as if I understand it right.

E

Justin we're going to be on like a chat session with a whole bunch of people have a whole bunch of questions, so I'm hoping the people that are on this will join in answer all the other people as many questions as we have you can just imagine all the other people have even more questions so hoping you guys can help jump on chat sessions and help others out as well.

E

C

Outside of kubecon, because I'm not in the whole thing there.

E

How does that work and amy's gone? Let me ask amy what that looks like um for those that are so. Basically what I'm hearing is a bunch of you aren't officially at kubecon, so you don't know how to get in yeah. Okay, let me poke around and see if there's some back door, okay, an official backdoor, wait! It's recorded anyway, all right I'll, see you guys thanks later thanks darren for the time really.

A

E

E