Cloud Native Computing Foundation App Delivery Special Interest Group, 17 Jun 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: CNCF App Delivery 2020-06-17

Description

CNCF App Delivery 2020-06-17

A

A

A

A

B

Hello, everyone.

B

So, let's give people a couple minutes to join here if you're already here feel free to add your name already to the list to the.

A

A

B

B

B

B

B

B

B

Right, we have five minutes past the hour, so let's get started here. uh First, gender item: there's always this introduction to newcomers. So this is obviously something that is totally voluntary. But if somebody is new here and just wants to say, hi to the group feel free to do so.

B

Okay, now, let's move on ah yes, definitely want to just quickly, maybe because I just that new. That would be great just to have some of the background.

C

Can you hear me.

B

Yeah we can hear you.

C

Yeah hi, I'm stefan, I'm a product manager at spotify, I'm responsible for the recently open sourced project called backstage that we're looking to donate to the cncf and I'm here to kind of listen in and you know understand the cncf game. I guess.

B

Good to have you thank you. Do you want to like share, please like a couple minutes to share what backstage is about for people who might not have heard about it yet.

C

um Sure I can do that. You want to do it now or later on.

B

uh Whatever you feel more comfortable with we'll have some time today to go over, it's not like a big presentation. We can schedule a deeper presentation, but just in case some people are not familiar with it's just a very brief high-level intro. Maybe.

C

Yeah, I can take a couple of minutes at the end.

B

Yeah I'll just put it just put it in the sun.

B

So then, let's start with some organizational topics. We have still our local discussion, which was now shut down. Okay, then the boat was closed and.

B

I think we should have a winner now, which is 21.

B

B

B

So it seems we have a logo I'll, let em in though who couldn't definitely join us today.

B

So now we have the next cube con that we meet in person which is hopefully norm. We cannot have our own socks.

B

So I also put on the agenda. What I want to do more is also uh have the working groups also present back on the work that they are doing uh for those interested in any of the existing working groups.

B

They have usually separate meetings where they dive deeper into to some topics, uh but I also want to give them more time that we can share and that they can share some of their their progress here and also, if you want to move discussions to a domain, sick uh feel free to do so as well, um because, like example, I think, with the air gap working group, they were a bit striking. Also was the the timing which you know, especially for european folks, was incredibly hard, because I think it was friday.

B

Eight pm, which is a bit of a tough point. There.

B

Okay, so I think we have mark here right.

D

I can uh give everybody a quick update on the work we've been working on in the operator working group. um I don't have a presentation or anything, but I'm I'm happy to kind of walk through what we've been doing um and where we are. What the conversation is is around right now in in the current operator working uh group document, um so we've been meeting um bi-weekly um starting to get a consensus on a few points of trying to define what a kubernetes operator is. um We've narrowed down a few things. I think we have.

D

You know general um consensus, yeah I'll, definitely link to the dock. Here. Sorry.

D

So here's the dock for everybody- um you can pull it up on the screenshot that'd be great yeah.

B

I'll do it in a second just decorating here as well.

D

um So we've narrowed down a few things like you know what what an operator is it you know it extends the kubernetes api to provides application domain specific knowledge, um an in cluster reconciliation loop. um You can definitely see in this document. There's a lot of conversation going on still most recently we've been discussing. You know who the target of this document is.

D

Is it specifically, you know written to the app developer or is it to the cluster operator at the end, somebody else or some combination of those those audiences and there's been some interesting kind of technical conversation around trying to define.

D

Is there a requirement that an operator modifies in cluster resources or does what you know does a does a controller? That's that's modifying aws or gcp resources. Does that count as an operator so trying to like think through the scenarios of what people are calling operators today and making sure that our definition isn't um excluding anything, but it's putting a more.

D

True definable definition of an operator on it, um the uh the calls every every other week. You know the document's been going around on the mailing list. Also, there's still like. I mentioned a lot of unresolved items in here.

D

We still, I think, need to dive into the uh the operator life cycle and whether that's even included as part of the operator definition or not. um It's down in this document a little bit more we're. Definitely anybody it's every other tuesday uh starting next week um and we'd, welcome anybody to join or or participate in the mailing list. um I know there's a few people I've seen in the in the um participants list here today that that do join. That operator working group call every other week also.

D

That's kind of a quick update as to that for the operator working group.

B

Yeah, thank you. I think that the keys, so, if you haven't read it, I would definitely give it a read, and the next step here really is to get ahead of the comments and to join the last call. I think it's it's good that we are starting to close down on a certain number of definitions uh that have been circling around for a while that question on whether um what that I really find interesting is the discussion about.

B

Does it need to modify insurance, the resources, because we have a couple of examples where we have operators that actually don't modify in cluster resources, which is an interesting use case, because in this case, kubernetes is really more.

B

The platform for building the automation, where the one that that's running is yeah, so uh make everybody, give it give it a read, and if you're not joining the meeting and still working with operators, it would be good, although there you are because you're here, I think it's also good to get uh some of red hat um this chart up here is the I think, modified one here, but to get us some some of the written expertise.

E

Michael rev redneck is the red hat representative who's participating.

E

Is there someone from the helm um universe um participating in the calls.

D

um Does join uh the calls pretty regularly and he's pretty active in it, so yeah.

E

All right so there's some balance in the universe. Okay, yes,.

D

We've the conversation has, has you know, you know the love? The call last week we actually did include. Does the installation of an operator should that be defined as part of the operator definition? I don't know if we reached a conclusion on that, but you know that was definitely the the you know, part of that. The helm contribution making sure that we're thinking about the entire life cycle of an operator.

B

Yeah, I think, for me, the key goal would still be able to get with this document to a level where we can update the current company. This documentation about the operator pattern. uh There's a little more specific one. I think that's the goal. We should eventually have.

B

So, okay, uh yeah, if you haven't, read it yet, uh I would recommend having a look at it. I think.

A

B

Nobody joining today from the.

B

uh From the air-gapped working group um overall, the aircap one is an interesting one, so they started to look at air-gapped installations of uh kubernetes, because if you can install kubernetes in an aircraft way, uh it will be hard to install applications. There were several meetings there um I talked to them to the working group leads and in case you are interested.

B

uh Please declare your interest and it seems like right now. People are like kind of dropping off uh the meeting, so it was a lot of popularity in the beginning that it seems to be how the people have other work to do as well, but it was the highest ranked uh request when we started to uh when we discussed it last year at cubecon. What people wanted to work on was for this aircraft situation that we wanted to get involved in.

B

um Let me just also get more involved with the telecom uh user group here as well. The cylinder connection either because it came a lot from the telecom industry who wants to use it for edge installation between all the air gap installations. Here.

B

Okay last item here on the agenda that I have in here, so this was a bit of an overview follow-up. We discussed the topics that people um want us to work on are interested in. This is like okay. What's like next, what's in the agenda, what should we start working on, and initially I put together this google doc with questions that did not result in a lot of feedback honestly, there's just only one question there or on one response there.

B

So what I didn't! What I have here is the link. I took a number of questions that we thought are a good starting point for having a discussion with people and just asking them what they want us to to focus on, and I just want to take today to just go over those issues and you know fix it.

B

Sorry, uh so the first one is: uh do we have like a simple or unified way for application definition? This was something we had also discussed as part of the charter in the beginning. Like this ongoing question, how do I define what my application in kubernetes is, and you could argue, okay, helm can do it, but how do you handle that secrets?

B

How do you handle third party uh dependency and services how to handle these kind of things, and I would at least everybody in here to get started- ask provide obviously their feedback on this one here and the way all these questions are structured. Sorry, right now, is that, like a general question, what do they think is an issue? It's a live issue uh or it's a big issue for them that they can eventually have a ranking. What are the most the most critiquing topics um and then there's.

B

Obviously, the open ended answer of what people are using and we deliberately went for um open-ended here so that we don't say: okay, just people just have like a set of choices that they can make, and this is a little.

A

B

Work um to to go through this and then categorize it, but honestly, I don't expect like hundreds and hundreds of responses of this, it would just be a matter of work.

B

uh The next topic to that we put on data on the on the delivery survey is how do you manage dependencies and also especially certain particle tendencies so dependencies that go beyond your existing applications, like other services, how they even model dependencies, which are what it ties into delivery like this service depends on another service.

B

How can you express this even how will handle this, uh which ties into the next one like delivering complex applications, um ties ability to the operator discussing custom, but in general like how can you ship applications of a certain complexity uh with dependencies, and how do you model that they can easily be installed? I think this is something that we from our experience put in there.

B

It's something we see more and more, that people are shipping applications in kubernetes that are more and more complex than they used to be, and it's still struggling with how to ideally do this. um There's other things coming in here, obviously as well like how do you, for example, if you have to define like certain service routes or configure certain things which handles your party that are installed in the cloud server, whether they are there or not, and how you want to handle.

B

Like and again next question here, packaging of applications, you can get a kind of like off-the-shelf application experience.

B

uh This is also something we see like an interesting point: how people are handling this today and something we start to?

B

Yes, there's like one ism, I'm working on the titles here uh topic here is again: how do you can let's get this like off-the-shelf experience of installing an application beneath this? um This is just the idea, so you have like your application. How do you package it and how do you ship it like in the old days where you had like your installer and you shipped the binary and then installed the application? How do you really do this like on kubernetes, especially with a lot of the external dependencies like?

B

Where do you, for example, put your uh container images and how do you get them in the registry? That is a lot of companies, although from from our personal experience, we see very often issues that, obviously people are not downloading from public registry, so they have to export it. We import it into their own registry, which kind of makes the experience for actually using them.

B

A bit hard and that's why we put the question here: how much of an issue how much of an issue this is for people and uh how they are doing it today, then going yeah, and this is one like how we install applications with specific infrastructure requirements uh into multiple different configured clusters uh again same issue. You have, for example, for your application, a dependency on a service mesh or certain configuration. It should be provided for a series match. Obviously, there is smi, but there's also other dependencies and different clusters might have different capabilities.

B

So a lot of these questions on the ground, interaction that you have that you built the application once and have to install it into multiple clusters rather than building an application. Just just for yourself again. How do you do it and then the whole topic around gene has custody uh something we also start to see more.

B

Do you even know what you're installing or how do you even know what you're installing and where these things are coming from and how they are validated.

B

um Again, we were discussing this before it's like booster here.

B

If I'm getting a 2 000 uh lines, um manifest file like what's everything in there, where do these things come from and what are the dependencies that are pulling into my application and how can I being responsible for a production environment actually validated what people are installing or what I'm trying to install here actually fulfills my security requirements and other requirements.

B

And then some of the automations operations topics so take a calculator.

B

Yeah, caroline we've all put off-the-shelf application pieces in there. So caroline just mentioned you should define concept, applications.

B

So let me just fix my typo here.

B

Yeah this is okay. Operators are right now really custom built uh the questions. How can I get to more, like reusable operational concepts that I want to use and maybe also how? How can I handle situations like? Oh something that we, for example, internally learned when I have to suddenly have to run multiple operators at the same time, because I'm using multiple infrastructure components and how do I connect them together, say I'm like using a design where I'm using an elastic I'm using my own application and I'm using a redis?

B

So how can I get like this? Whole application stack uh properly work together with its independent operational components,.

B

So this is a set of questions uh that we have available right now. This is a starting point, so uh the reason why I brought it up today is just I want to walk you through. It would be great to get your feedback and obviously responses on the question. If you think that there is something missing in those questions that we should asking people as well, uh we can obviously add a couple more questions. Everything like okay, this question is a bit off topic and we should refine it. Please let us know as well.

B

So then opening up four four comments on this. One.

F

So I think this is this: the survey is great, um who is the survey getting sent to.

B

So we will work with. uh We have to work with the cncf and reach it out, reach out to the end user uh community and also for the members some have like diane. Maybe we can have maybe some feedback from the the openshift comments community and, like others, have access to their communities as well. I know that we try to get it by the cncf, but the response rate is usually not that high.

B

What I also would like to do like for new members coming in or putting this on the landing guitar page uh for people like if you're here for the first time, that's the thing that you should familiarize yourself with and also like being able to constantly add it. But I think once you're ready I'd like to reach out to the cncf whether we can get that story out there.

E

Yeah, um if you put it on the github page, with a link to it, um I think one of the things we can do is we can. um I might share it at the on through the commons mailing list, because there's about 20 000 people there not that they're all going to answer, um but also we could tweet it and socialize it. You know your own social channels too, to get some feedback.

B

Yeah that that will be great. um Obviously we have the cncf mailing list as well. We have the delivery mailing list. um We can also share with some of the kubernetes milling with people to give us feedback. I think there's a number of them so.

B

Interestingly, what I've seen in in in the past is that if you get like 100 200 replies, you're, actually pretty good, because people get a lot of service that they would have to fill out and especially if you're not offering up the free t-shirts, uh uh sometimes less might be able to answer them, but we can definitely work on this. For me, the most important thing is that we all feel comfortable with the questions.

B

um So what I would ask everybody just please provide feedback on this one if you think this makes sense, so it doesn't make sense. So if you miss an important question that you think should be in there or if something is unclear, so my plan would be, let's give it two week of a review period, and then we use the cncf and other channels to to distribute it. To get. Some feedback sounds good for everyone.

B

By the way, just learned that there are even virtual sump right now in soon, which is kind of interesting.

B

Good yeah, so yeah, please provide feedback I'll share it. The link the links in the city agenda dot. The link will also be available um in the chat later on as well.

B

So then, I think we are through the official agenda for today. So definitely want to share a bit more about your project for a couple minutes.

C

I can definitely do that. I can't on um share my. I can't share my video, unfortunately, because my karen, my camera, doesn't work apparently, um but I'd be happy to to share my screen and run through a shortcut. Obviously.

C

I just wanted to give my my colleague remy a chance to introduce himself. He was he was here as well. I think he's the first joiner there. He is.

G

Luckily, my camera does work, so I can I can sneak in here, but thank you everybody for having us. I see some familiar faces here, it's great to be here at the sig. My name is remy and I'm the newly minted head of open source here at spotify uh we're starting to get more involved with the cncf, both through backstage, as well as just generally spinning up the open source program and getting more involved in the community.

G

We appreciate the the time today, as we sort of go through the the sandbox proposal process and uh we're getting excited about engaging more fully and being around welcome. Thank you. Thank you.

C

So I'll share my screen here.

C

Can you all see my screen yeah yeah, so backstage? What is that um so backstage is essentially an open platform for building what we call uh like loosely developer portals and I'm gonna. Do this?

C

I didn't plan to run a demo, I mean. Maybe we want to come back and do like a longer demo at some point. Yeah.

B

We can do it in the next meeting if you want to do that.

C

Sure, but I'll just walk you through sort of what it is and what the problem we're trying to solve is, and then we can come back later for a more formal demo, um so yeah. What is the problem that backstage is trying to solve? um Like the infrastructure landscape is pretty complex, I mean, if you look at like the cncf alone.

C

It's huge there's a lot of different open source projects, and we have certainly built like our fair share of like internal infrastructure projects internally at spotify, and that might not be like a problem in itself but like if you look at like the I mean, this is just an example of like the overall complexity of the landscape and um when you, the problem that you have kind of at spotify with that we did have, was that as we were growing our infrastructure and adding more and more tools and more and more like infrastructure teams building, you know distinct uh pieces of our infrastructure like the overall.

C

You know the overall thing or the ecosystem of tools started to become super complicated to a point where actually, like our engineers were starting to slow down, because they couldn't either like find the tools or the tools that we were building like they didn't really connect with each other. There was no, like you know, connecting tissue between them or, like you know, every tool was built like for one specific purpose, which is you know, that's kind of how you build. I guess a successful open source project you solve one problem, you solve it really well.

C

The problem is, though, then, when you sort of try to put all these pieces together and ask engineers to kind of use all of these tools, they will need to become kind of experts, or at least operators of very many of these different infrastructure things, and we don't necessarily think that an engineer at spotify should really have to be an expert in infrastructure and underlying tooling to be productive at spotify.

C

So what did we do? Then? The solution like you know, air quotes. I guess is here to you know uh what we opted for was to bring build one single, consistent, ui for all of our infrastructure, and that is backstage. So rather than having multiple multiple kind of infrastructure teams building, you know their tools as distinct islands, of infrastructure, entire ecosystem.

C

We opted for a model where we would have one central team, my team, to build sort of a platform layer where these different infrastructure teams kind of plugged in their infrastructure and made it into a consistent experience for, for the end users, who are the engineers and the developers at spotify.

C

So, essentially, what started to happen or what we encouraged them was kind of almost to create kind of an app store marketplace for for these tools, and we were seeing metrics as such as like onboarding time, which was getting longer and longer for new engineers, as we were growing as a company.

C

Those metrics started to go down quite dramatically and we're now at the point where we're like, after releasing backstage and kind of consolidating and centralizing all of our technical.

C

Like all the developer tools, as well as technical documentation into one place, uh we actually cut that developer like onboarding time with 55. We measured that, like time until you've sent your tenth pull request, which is, you know, not a perfect metric of productivity by any means, but at least something to solve a proxy. For you know the complexity here that we have been able to reduce.

C

So when we, as we were kind of you, know talking to other infrastructure teams inside companies similar to spotify uh they kind of or in demo backstage.

C

They were like pretty amazed by how many different things we have been able to sort of put into one central place and sort of create a more consistent developer experience, and uh that's when we started to think about like is this something that you know is this a problem that we've sold, that kind of isn't unique just to spotify, but maybe you know something that other companies are struggling with and after doing you doing even more kind of you know, user research- or you know, talking to even more teams. We kind of decided that yeah.

C

This seems to be like a industry wide problem, and we think that we have a pretty interesting solution to it so long term. What we are hoping now with releasing backstage as open source is kind of start building a more standardized engineering.

C

You know ui layer essentially for different infrastructure tools and with the same kind of pluggable architecture, so that you know developers and open source project maintainers from various projects can can sort of make it easy for people to use their products by building an integration like a building, a plug-in as we call it in backstage and we're hoping that we can create kind of a big community around this.

C

So that's why we're here to like you know we're really excited about we're really eager to kind of get more people contributing to this, because we think that if we can create kind of an open source marketplace for like developer experience, kind of layer, things very wishy-washy and where are multiple companies kind of uh you know in joining in the community and building these different plugins, so that you know in a couple of months, you walk up to backstage and we're using jenkins we're using kubernetes we're using you know: grafana we're using circle ci, whatever you're using like there is a plugin for that, and you can just you know, configure your version of backstage to match the infrastructure that you have.

C

That's kind of the long-term ambition we have with the project and the reason why we're kind of starting to engage with the cncf, because we think that we really want to create a good community around this, because we don't think that spotify can necessarily achieve this kind of bigger vision. Ourselves.

C

I think that's uh that's it. Hopefully you have a an overview of like what we're trying to do um we're pretty we're off to a good start like we have had some like excellent response from the community, like more than 200. Companies have reached out to us and wanted like a personal demo, and very many of them are already trying it out and, like you know, running proof of concepts internally and starting to build plugins and starting to kick the tires of it.

C

So we're pretty happy about the first like couple of months of progress and yeah excited for the future.

C

A

F

I do have a question, I'm looking for my notes, um so you use terms like portal and and user interface. Do you consider it mostly uh kind of at that layer at the ui layer? um Because when you talk about developer productivity, you can start to think about. You also talked about platform team, and so is, is backstage predominantly about creating kind of a as you said, portal or plugable ui um over infrastructure, and that is not part of back backstage.

C

uh The way we look at it uh like philosophically at spotify is that we want engineers to kind of have three interfaces that they primarily interact with. Like first is their code editor where they, you know, write the code. The second is like github or you know similar uh and the third like for everything that doesn't. You know like that, doesn't like fit into that. First, two workflows. You know we want there to be backstage like you, shouldn't have to go.

C

Hunt around like you know, looking for you know, debugging your data pipelines or looking at your deployments, or you know, checking out what security issues your software has right now that should all be one system, and that is backstage that's kind of how we think about it.

C

Okay, but but backstage is pretty unopinionated when it comes to some of the lower levels, so at spotify we now have over 130 different plugins and they span from you know all different parts of like the software development stack like there's experimentation, machine learning, you know, following your code from you know, you know pushing into production and, like you know following it out to like with you, you know monitoring and everything. Basically, so backstitch is pretty unopinionated and we cannot let the different teams own. You know a plug-in and the use case.

C

So, for example, we have our ci team they own and operate like a jenkins fleet under the hood, but we don't expose that to our engineers. They they build a plug-in in backstage instead and sort of attaches that to the service definitions and the service catalog that we have, I think it might be related to you know the previous conversation about applications, and- and uh you know we have a we kind of have a model for how we model our software and all that is represented in backstage use.

C

So a team- as you can see here, for example this, like the overview page of things, my team owns uh all of those things- are kind of integrated into backstage, and then we have. The tooling is kind of connected on top of that service catalog. If that makes sense.

F

Yep yeah, that's helpful. Thank you looks.

C

F

C

Like really cool, there's, nothing like kubernetes, specific or anything like that. Like we, for example, managed you know our transition from our homegrown version of kubernetes into kubernetes or gte. Actually, you know without the users kind of changing the interface in backstage, so the infrastructure team responsible for that migration can sort of you know abstract the way the differences between you know: different container systems or even virtual machines or data centers. When we were running there kind of behind this glue layer, if you.

C

B

So the way I can imagine this is like I provide some type of service and I install it and I write a plug-in for for backstage and then it's available to other people like I have my ci pipeline, I have my machine learning, type of environment and I provided and then a developer can say. I just want to use this for my project. Is this how it works.

C

uh Depends on who you mean the developer is okay. There are different players here, like there's someone, you know in the platform or infrastructure organization, that kind of deploys backstage and provides. You know a developer portal that has a lot of different tools and those developers they build the plugins and kind of integrate, different tools and environments into backstage, and then you have on the other side, you have the users right like the engineers developers who are building. You know you know they're just users on backstage and they use it.

C

You know for doing any any kind of support they need during you know, product development in general. Does that answer your question.

B

Yeah, um I think I have a better understanding. I think it would be good like to see the next time. Obviously we can schedule a demo where we can like walk through the workflow for for the different stakeholders uh what they're doing, and that would even have to clarify it more absolutely. I I do think it is an interesting uh project to look at there's a look at that. There was the other initiatives going on. I just wanted to put them into more context.

B

I was in the operator side we have operator up, which is very focused, obviously in operators and how you can install them, manage them having the life cycle management in.

A

B

This is definitely going in a bit of a different direction.

B

The cncf was, or is already we're, also working on a different project where they expose um right now, it's mostly helm charts, but the plan is to go further with with the artifact in case you haven't heard about it. That might also be interesting to look at just what's it wrong. Okay, how do you relate to these other projects and, if there's an overlapping, which is not an issue so that it can also be overlapped anyway, but just that people put into context, but I can't see it so questions from anybody else here in the.

B

B

Okay, maybe then we really schedule this for a demo in two weeks from now. If this is fine for you, then we can like really see it in action and that's usually when more questions come up when people say absolutely.

C

Thanks for I'm short.

B

Notice just walking people through, but I think it's just helpful for people to have uh some sort of understanding here.

C

Yep I mean, if you're, do you can't wait for the two weeks, there's a microsite that we have a backstage dot io, where we have like a demo section, a couple of youtube videos showing how this is used internally at spotify.

A

C

B

I feel free to also post this in this like channel, so if people are interested, there's, usually more people on the slack channel as well, uh that are those who participate in the.

A

B

So feel free to also post it. There.

B

All right, this actually gets us on top of the 45 minutes for this uh meeting today. So thanks everybody for joining again for the next time. Please have a look at the questionnaire, so we can send it out and schedule um a scale of sending it out uh by the different channels which we have so looking forward to everybody's feedback, and we also will put them backstage demo on the agenda. Unless somebody has anything else they want to bring up today.

B

All right then, enjoy the rest of your day. Hopefully.

F

Thank you. You too, thank you much.

C

See you next meeting bye.

C