Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2022, 2 Jun 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: The Risks of Single Maintainer Dependencies - John McBride, VMware

Description

Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

The Risks of Single Maintainer Dependencies - John McBride, VMware

John McBride is a single maintainer for Cobra; a Go command line bootstrapping library and core dependency for many CNCF projects, including Kubernetes, Helm, Etcd, Istio, Linkerd, and many more. John will discuss the challenges of being a single maintainer on such an important project, the lottery factor, the need for contributor community, and the secure software supply chain implications this has for the entire CNCF ecosystem.

A

uh This is the risks of single maintainer dependencies um bear with me still dealing with the jet lag, but in this talk we are going to chat about the lottery factor, contributor communities and the secure software supply chain. So one of the things I always ask myself whenever I show up to one of these talks is okay. Well, who the heck? Are you? And yes, I promise it is relevant for this talk.

A

uh So my name is john mcbride, I'm a senior software engineer at vmware and I work on our open source kubernetes platform, which is tanzania community edition, along with a bunch of ancillary, open source projects around that kubernetes kind, and one of those projects is spf 13 cobra. Now, if you don't know what cobra is, is it's a cli, bootstrapping library for go? That gives you a bunch of really nice apis for creating commands and a really modern, cli application tons of stuff uses it all over the cncf all over kubernetes?

A

So just some quick statistics here on cobra itself. It has over 26 000 stars on github.

A

It has over 900 commits and again it is a key dependency in the cncf kubernetes ecosystem, uh just to touch on a few projects that use it. Kubernetes itself cube cuddle, helm, tanzu at vmware, docker, cd, istio lingard, the github cli. I mean it's everywhere. If it was written and go, it probably uses cobra.

A

Let's uh touch on a brief history of cobra here, cobra's first commit was in 2013 and it came out of sort of that same group at google that was doing that was doing core go, so it sat right at home within kubernetes and with cube cuddle and a lot of the stuff that came out of kubernetes in the early days of go.

A

For example. Here is the default cube cuddle command that returns us a cobra command. It really is the entry point for your actual implementation of your actual features and all the stuff you're actually trying to do cobra wraps around all of that stuff.

A

Let's fast forward to today- and this is the 1.4.0 release that I cut in march- and something interesting to I guess realize about where we're at today with cobra- is that I more or less maintain cobra by myself. Now before I go any further, let's define what maintaining means here. I have admin privileges on the repository.

A

I can merge code, so merge pull requests. I can triage issues close issues and I can cut releases they're, sending those packages sending those apis into kubernetes into vmware into red hat into google all over and again before. I continue here a huge shout out to the cobra community. It's actually gotten much better recently, some other maintainers have stepped up, there's people doing code contributions and still a part of the community. But one of the questions that I get asked pretty frequently you know about the last few years is okay.

A

How did how did I get here? What happened? Well, I was at pivotal, which was acquired by vmware, and there were a number of people who were also doing cobra dev uh that left attritions. You know that sort of normal thing and then the pandemic, and you know I often ask myself this question: how the hell did I get here, but I found myself as the solo maintainer for cobra. I am a solo maintainer for cobra. Let's define what solo maintainer is a solo.

A

Maintainer is an individual or a small group of individuals, maintaining a project with little to no support. These are people on their nights and weekends they're not getting paid just a passion project, it's not really part of their job. Those are those solo, single maintainers of these projects, uh a quick show of hands uh who would define themselves as a single or solo maintainer. If you have a project like that, okay, that's like 20 of the room. Okay, uh there's a flip side to that there are people who consume those projects.

A

Now raise your hand if you would consider yourself a consumer of one of these should literally be everybody, because everybody at kubernetes cloud native con consumes kubernetes or one of these projects in some way which consumes cobra, and here we are so, I think, a lot about my experience of maintaining cobra there. It is- uh and you know this talk- is really philosophical. It's not a deep dive on the cobra technology or really how the technology integrates into the kubernetes and cncf ecosystem.

A

It's really a deep dive on the solo, maintainer experience and a lot of the pitfalls that you, solo, maintainers may find yourself in uh and some of the things that you can do to mitigate risks that are associated with these kind of prog projects and challenges. uh The flip side of that are the consumers of these projects and what you can do to sort of analyze these risks and approach these risks and hopefully uh make things a little better.

A

So first I want to talk about contributor communities and I'm not the first person to talk about how important contributor communities are. I definitely won't be the last person but contributor communities, as I define them, are a group dedicated to the success of an idea, and I think you can almost look at all of us and this contributor community people in kubernetes and the cncf are really dedicated to the group of ideas around a distributed. Containerized workflow world- that is a contributor community, more generally, in the open source.

A

This might be people contributing back code, contributing to the discussion in some way contributing issues, feature requests bugs or security reports. So why are contributor communities so important?

A

Well, they really are the lifeblood of success and longevity and stability in your projects. From a solo maintainer perspective. I think you know anybody who's doing this kind of passion project would want to see their thing get used and be successful communities. Give you that kind of support to give your project legs. One person, two people.

A

Three people really could never scale to make something like kubernetes or any of the cncf projects as successful as they are today from a consumer perspective through providing that support and through being a part of the contributor community, you sort of give a way to give those projects, longevity stability and support.

A

So I want to tell a story- and I think this story illustrates really really well uh just how powerful contributor communities can be. This is java and the first browser wars. So, let's date ourselves way back to 1995., uh there was a quite contentious war for the browser and the early web. At the time the internet wasn't programmable. It was just static, content being served and microsoft. Internet explorer was sort of in this fight with netscape, navigator and netscape navigator wanted to bring programmability into the web.

A

The way they envisioned this was through bringing the java vm into the browser to enable java applets to run right on your web page, and one of my favorites back in the day was runescape. I played this so much when I was a kid, but yes, this ran on the java vm in your browser, so the idea was that netscape was gonna was gonna partner with sun microsystems to bring java in enter brandon ike. This is a screenshot from the lex friedman. Podcast number 160 really really good episode.

A

It's like three hours long so buckle up, but highly recommend this, uh but brandon ike was a early employee to netscape and was tasked with uh sort of having this happen. By bringing java into the netscape navigator browser, he said they netscape started a deal with sound microsystems.

A

The idea was to put the java vm right in the browser. The opportunity was for a companion language, so sort of what he was tasked with was creating this sort of glue, duct tape, language that maybe designers or product people could use that wasn't too low level at the actual java vm level.

A

At the time it was called mocha and today it has become javascript uh brandon. I created this thing in 10 days. He said the uh the internet meant there was a huge first mover advantage being fast getting out. First, it mattered a lot, so they got something out really fast in 10 days and it had a ton of sharp edges. It was almost unusable.

A

uh He also said, worse is better. I don't know what that says about javascript, but there you go, but javascript really took off and people loved it frameworks and snippets and different libraries, and it grew fast beyond brandonike and the people at netscape. Today, javascript is the most popular programming language in the world. This is the stack overflow 2021 developer. Survey uh java on the web is basically no more. You know and there's a lot. We could dive into this story, but the contributor communities were so important in making this happen without a community.

A

You risk success without a community. You risk longevity. So the lesson learned here communities are an necessary, unstoppable force for solo maintainers. uh You need to invest in your communities and I would argue you need to invest in your communities almost as much as you invest in code.

A

That way, you can build a system of support around you to give you that longevity to give you that support that you need in your project, for consumers of these single maintainer dependencies be a part of the community.

A

That's it be a part of the community giving that support to the maintainers being there for them when they need reviews or any number of things within the community. You become a part of that system of support all right. Next, let's talk about the secure software supply chain uh and the secure software supply chain always makes me chuckle a little bit because I never really know what it is.

A

But the way I define it in this talk is it's a dedicated automatic process for consistent replication of deliverables, and I think this is actually best illustrated by some animations I made here, uh oh whoops nope. uh Before I talk about that, we also need to talk about the maintainers of these projects.

A

I think oftentimes oftentimes, we forget about the people, the people as part of the secure software supply chain, there's a lot of really interesting things happening with with six store and s-bombs within kubernetes and cncf, but we forget about the people, sometimes, okay, why? Why are the people so important?

A

Well, really, in my opinion, maintainers are the secure software supply chain without maintainers. It all falls apart. You don't have people maintaining your s-bombs, you don't have people maintaining your cicd systems, okay, here's the animations! So let's illustrate this with some of these animations I made here. These are some packages, maybe a product, maybe a library like cobra, and let's look inside of these packages that we're shipping to our consumers to our our our customers.

A

Everything's, good everything is as expected and we can ship that away and all is good. Now, let's say we come to the next release: some more packages. These are going to go out to people, consumers, whoever we look inside and there's some bad middleware or something that gets shipped out to people. Maybe it's a cve or a hack that happened and people are not happy. Maybe we have to make a statement or do a nasty patch release or something it's. uh It gets messy.

A

What if we had some dedicated automatic system that could detect these things and stop them from happening? We detected some bad middleware, something got injected. We can stop it. There all is good. Now again, I think oftentimes we forget about the actual people. These people have to maintain these cicd systems. The s-bombs sig store itself, and you know in this case it's being maintained all as well, but unfortunately, people fall asleep. People leave projects, sometimes people just completely abandon projects left to dust.

A

In this case, this person is gone, they've fallen asleep, we have another release, but our whole secure software supply chain system has gone into disarray. We have another bad middleware, we can't catch it. Things are not right, it gets shipped off to people, uh and I want to illustrate this with another story here, and I think this really illustrates illustrates well how important the single maintainer the solo maintainers, are within the broader secure software supply chain.

A

So this is mpm event stream and the crypto bandit a number of years ago on this project event stream, which was a popular javascript, npm streaming library. Had this issue opened up and a quick note on this library in npm it was used everywhere, but not necessarily on purpose. It was one of those. It was one of those transitive dependencies that got kind of pulled into everything.

A

The issue says I don't know what to say, and the user opened this up and is basically asking the owner. Why was this new person given ownership access to the repository and why were they also given npm publishing access? So essentially, this person was publishing new packages to npm as part of their new ownership of the repository.

A

That person had also made a very, very, very strange, commit it was decoding and encoding a bunch of random stuff and then getting injected into the actual module. So clearly something very weird was happening. The owner came back and said he emailed me and said he wanted to maintain this module. So I gave it to him. I don't get anything from maintaining this module and I don't even use it anymore and haven't for years then. Next he said note. I no longer have publishing rights to this module on npm, yes, hackers of the world.

A

It's that easy. You just ask, and then you have it.

A

What this thing ended up being was a crypto wallet stealer. So if it was running alongside a crypto wallet, it would attempt to steal all that crypto within there and it got shipped out as part of an npm package that then was consumed by a bunch of other npm packages, and some people thought that maybe this affected hundreds of thousands of people developers companies. It was all over.

A

I I think about this story. Excuse me, I think, about this story. A lot like all the time uh trust is so hard within this ecosystem, especially as a solo. Maintainer code is hard to trust.

A

The packages you consume are hard to trust, but oftentimes I find it's the people, it's it's really hard at the people level as well. So the lesson learned here invest engineering resources.

A

First, solo maintainers, please ensure your own security. If you don't have a physical security key, yet to do two-factor, authentication to github or whatever, please please go get a physical security key. You have to first ensure your own security, so you're not uh being pride to some of these hacks or attacks.

A

uh Also don't be the lone hero riding off into the sunset. You need to bring other people into the inner circle. You need to bring other people into the maintainer track so that you can start handing off responsibilities that there are other people there to help along the way, so that it's not the one person giving up the package abandoning the package like we saw in the mpm story for consumers.

A

The same goes for that you need to get on the maintainer track, invest some of your engineering resources, those people into these important projects. Look at your stack and look where there might be these projects that are at this kind of risk, because this happens and it unfortunately happens all the time all right. Next, let's chat about the lottery factor.

A

The way I define the lottery factor in this talk is: it is a spectrum of risk correlated with the personnel on a team uh more anecdotally. What it is is you know what would happen if I or some really important person on your team, your product team, your open source team? What if they won the 100 million dollar big buck, bonus or something, and they never touched a keyboard again their life? They were just they were just gone.

A

They did not care, they didn't want to work anymore, that's the lottery factor, and it really is a spectrum. I don't think anything falls too far on one way or another.

A

Sometimes you end up with solos here or there, but in the solo maintainer perspective, that really is the highest level of the lottery factor like we saw in that last story. If people leave that is catastrophic. So why is this important? Well, attrition is normal. It happens everywhere. Attrition of critical people is scary.

A

Attrition of solo maintainers can be absolutely catastrophic. These are the people maintaining projects all by themselves that have these. You know massive projects underneath them uh npm uh the npm event stream cobra.

A

So let's tell another story, and this one is a little bit more hypothetical it's about cobra and how cloud native could fall apart and really from my perspective, cobra is just it's: it's a library, it's a bunch of apis that we ship out uh important apis that wrap around a bunch of really important stuff.

A

But what if I just won the lottery one day and then my account got compromised or something else got injected. Somehow those things then would end up getting shipped out into kubernetes, istio, linker d, tanzu helm. All of these.

A

uh And this actually came up recently, uh the most recent 1.4.0 release of cobra had some people from kubernetes, and I was actually talking to people from six cli yesterday about this. This was huge. We noticed that there were all of these dependencies getting dragged along into kubernetes and kubecuttle. That really were completely unnecessary, so we were able to remove a ton of dependencies to sort of reduce the surface area of potential attacks of potential cves coming in from those other libraries.

A

This was really really big, but a question. I think everyone needs to ask themselves, especially you solo maintainers, is what, if I won the lottery, really ask yourself that question: would you work another day in your life? Is your project ready for something like that and it's hard it's really really hard. I don't think there is a hard solution to this at all, even on product teams. They deal with this there's silos everywhere.

A

You really have to look at it from a spectrum perspective, but I do think that there are some valid strategies. The lesson learned here is create processes to mitigate the lottery factor for solo maintainers.

A

You need to create that pathway to maintainership that pathway to the inner circle, and I'm really talking about process here like yes, you can bring people in, but there should be a process that people can start on your project to become a maintainer. Two is documentation. Documentation is really huge. You know again, I'm not the first person to talk about how important documentation is.

A

I won't be the last but documentation even of the minutia, even of the little things that you think, don't matter or aren't a big deal can become absolutely huge as soon as you're gone. Maybe you've abandoned the project. Who knows consumers? The same goes consumers of these projects. You need to get on those tracks to becoming a maintainer, to getting some kind of access right and contribute back to docs as well. You can look at the docs, you can read the docs and see where there are problems and start contributing. Those back all right.

A

Last thing I want to talk about is incentive models, and this really this really could have been its own talk in itself. uh Incentive models are systems to encourage desired behavior, uh and you can look at this from your own perspective in the way that you do work. You know if one day at your job, you just weren't getting paid anymore. I seriously doubt you'd continue working.

A

These are incentive models, but they get really really tricky and open source for me personally, it's about personal satisfaction, it's about seeing something get used at this high level and seeing something that I create or prs or issues that I'm a part of get kind of brought into the bigger good.

A

I think about this, sometimes from the perspective of game theory, game theory is sort of the study and the idea of how n different number of factors could be a lot affect a very complex system and open source. If not the kubernetes cncf ecosystem is incredibly complex, so those incentive models are really complex. You not only have individuals but also groups of individuals, small companies, huge companies, these incentives really are anything and all over the place.

A

Why is this important? Well, really, in my opinion, without incentives, no one would do anything, so you really have to look at the incentives of the important projects that you are consuming and also that you are creating.

A

Let's tell another story: uh this is about faker.js and the dark side of open source. I realized uh yesterday that I was taking a pretty big dunk on javascript love javascript, but these are very relevant stories. uh So faker.js was or is a testing library within the npm ecosystem that you can use to generate all of this testing data. It's really useful for front end when you have forms or a lot of different data with names dates all kinds of stuff.

A

It's really really useful, but one day back in january of this year, the owner of this project just just blasted it away, they removed the git, commit history. It was just one commit with basically an empty repository. uh The npm packages disappeared as well.

A

You know, and we could go into what the heck was going on here, but you know a lot of speculation popped up and one interesting thing from years back that this person had said was no more free work from merrick pay me or fork.

A

This uh people did fork it and it continues on to this day uh without the involvement of this person, but I think you know a a deep analysis of these important projects and a deep analysis of the incentives of the important people in these important projects should have led some people to this kind of conclusion that hey this is risk. This is pure risk right here, so the lesson learned is: without these incentive models. Without looking at these incentive models, everything else risks falling apart. uh You risk your collaborator community falling apart.

A

You risk your secure software supply chain falling apart everything, uh those critical people and their incentives should align so solo maintainers. uh You need to identify your own incentives. Maybe it is getting paid uh start a patreon. There are github sponsors, you need to start working towards those things.

A

It's critical that you identify your own incentives and that you start working towards those for consumers of these projects. I think this is especially key here. This is especially key. You also need to identify your own incentives and that gets very complicated.

A

You know big companies, small companies, they're all consuming these projects for different reasons, different products, all sorts of things, but you need to identify your own incentive incentives, how those align with the important people in those projects and then also identify those important players within these projects and start to identify their incentives. And you can you can start to uh see sort of the ecosystem of incentives and where you might be at risk all right.

A

I want to leave you with really one thought if you come away from this talk with one thing: it's this um any solo, maintainer dependency is a risky dependency. uh Just look at the like high level logistics of it one person, one person that is having that thing plopped into your product, plopped into your project.

A

That's super scary. It's super risky, so invest, invest, invest, invest uh you need to invest engineering resources if you are blindly consuming these projects blindly, just just pulling them down without being a part of the contributor community without looking at the important players and what their incentives are you're, not helping. You need to be a part of those communities and bring engineering resources into the inner fold so that they can start getting on the maintainer track with that. Cobra needs help.

A

So, if you're interested, I would love to chat with people after or you can find me there on uh twitter, but I'd love to take some questions as well. So thank you very, very much.

A

I think the uh there's a mic here in the center. uh If you would like to ask a question, please use the mic.

B

For the talk um one question I have is, I think, you've made a pretty good point, that this is a risk we should be aware of. But how do we best do that? So I I just, for example, go dependencies. You mostly pull from github, and you cannot just see on the on the github page that this is a risky dependency, something that's just bound by one maintainer, um I'm not not sure for security things. There are websites that uh collect security, cves, etc that you can use for to assess risk of dependencies.

B

Is there some similar mechanism for maintainable risk, or do you have suggestions how to approach that.

A

Yeah, that's a great question: there are automatic things that can start to uh pick up these things like dependabot on github. You can use the bendabot dependabot to start picking up cves within packages that those things consume and on down the line.

A

But a lot of what I ended up talking about was really people, processes and and people, and I don't think that there's automatic processes for picking up on those things. It really means that you have to be a part of those communities in some way.

A

Even if it's you know, you know, looking at the issues and seeing you know like, what's the tone of some of these people, like uh you know, anecdotally, a lot of people don't really seem very happy with me within cobra within the issues, and you can pick up on that in just looking at the issues, partly because I just don't have a lot of time to actually dedicate into the project. Somebody could look at that and say hmm it seems like there's something going on here. Maybe I need to get involved.

A

You know reach out to me on twitter and just ask like hey: is this project? Okay, yeah? So it's it's really really hard. I don't know if there's a good solution from the from the people side of it beyond getting involved in the community yeah great question.

C

Hi I had the same question about how to actually identify dependencies, so thank you for answering that and great really insightful. Thank you. Thank you. Thank you.

D

Yeah great talk, uh so do you believe that open source projects should be owned by companies or sort of organizations instead of individual maintainers.

A

That's a that's. A great question um really could have been. Probably also could have been its own talk. uh I don't know you know. Cobra is really hard because it's been around for so long at that import path, so github.com, spf, 13 cobra, has been around for almost 10 years now. So what happens when we try to move that into a github org and it destroys everybody's import path? I know, there's you know like different systems and things and we've talked about this and I don't think, there's an easy solution for that.

A

So the history of a project, I think, is really important like for cobra. You know it's been around forever, so moving it into an org would be really challenging. um I don't know yeah. I wish I had a better answer for that, but orgs are great and I think that you know orgs with good governance, like we heard in the keynote this morning, can be really critical to ensuring the success of a project.

A

I honestly should have added that, as part of my uh uh contributor community part but yeah having something in an org with good governance and good contributor guidelines and like the the getting on the maintainer track, is well defined. That can be really really powerful. Yeah.

D

Thank you, yeah.

E

So how do you deal with kind of uh being the single maintainer I mean? Doesn't that stress you out? I know it would stress me out totally like being responsible and I'm on vacation, but suddenly something comes up.

A

Yeah, it's it's not a good time.

A

uh It's you know, I do think about the weight of it and I think about the responsibility of it, uh which was you know, part of that effort to bring a few more people in and actually get them also maintain our access. But in the end today it's really it's really three of us. uh Even that is scary, and even that I would say, is a huge risk for such a big project. um You know for me personally, it comes down to uh like personal productivity things.

A

You know I time box cobra pretty hard, so I'm not gonna. I'm not gonna like dedicate much more time to it than I can like I'm not getting paid to work on cobra, uh so I can only give it so much time so that my incentive models continue to align with it like. I do get a lot of personal satisfaction from it, but I'm only going to work on it. So much as my personal satisfaction and my incentive models continue to align, um it is stressful yeah.

A

I won't deny that, and you know please uh come chat with me about getting on our maintainer track and contributing back to cobra thanks yeah, no problem.

F

Thanks john, it's a wonderful talk. um The question I had was around the the maintainers circle and how you bring people closer in that circle. Do you have sort of a litmus test on trust and when you would decide to give someone certain access rights and what sort of tears do you have? Is it just a singular tier, or do you think about multiple ways to come closer? That's.

A

A great question yeah, um I do think multiple tiers can make sense like still today, I don't actually have ownership access over cobra spf 13 is steve francis. He is the product lead of go at google, so he, I think by himself maintains uh or has the ownership access over that repository.

A

Again, I would say maybe that's a risk, but I individually would not be able to go up that level to the admin level and blow the repository off of github. That's only steven, you know he's kept this thing around. For that long, uh I do think levels makes sense uh as far as the litmus test. um It really is people that I don't know. Even that's so hard again, that's a that's a people question like how do you judge somebody's character by their pull requests their issues, your interactions on slack that can be really hard.

A

There's been hacks in the past where people have gained. You know somebody's trust, and then you know kind of work their way into the inner circle. Before you know it they're, you know pushing bad code or something so I would even say within your inner circles, there need to be systems like there shouldn't be people pushing directly to your main branch. I think that's just a good practice, but you should also have multiple people reviewing pr's, there's good, like processes around the inner circles that I think can be powerful to mitigate that uh those things.

A

But for me personally, it is sort of a gut check sort of a gut feeling. You know I I've interacted uh with a lot of people within cobra, uh one person caught my eye pretty pretty. Clearly his name is mark and he did a lot of the completion work within cobra recently. So today, within the newest releases of cobra, you can do powershell, z, shell, fish, bash completions, and it's really really good work. I don't think somebody would dedicate that much time and that much energy into this project, if they weren't at least somewhat trustworthy.

A

So that was a big litmus test for me with mark and he actually has come into the inner circle of maintainers. So it's really hard. Thank you. No problem.

G

Hey great talk, so I learned something about my code base today. It depends on cobra and uh I don't know what package brings it in, but I don't use it directly and auditing hundreds of packages that are brought in by other packages that are brought in by other packages. How do you fix that.

A

How do you fix that so you're asking? How do you fix the transitive dependencies.

G

A

G

Because that's nasty I mean the the entire community needs to do their auditing, starting with.

A

G

Or I don't know.

A

Yeah, that's an excellent question. I don't really know if that's a solved, uh I don't know if there's a really good solution. I'd love to talk to you afterwards, because even at vmware within tanzania community edition we've had problems with all these transitive dependencies coming over from, like some other package that that's importing some other thing and then you know, dependabot is alerting and we're like. Oh my gosh like what is this, we don't use this right.

A

um I don't know if it's, if there's a good solution within go itself or if there's work, that needs to be done up in the upstream within gomod, but you know, I think, the model that we saw with kubernetes coming to cobra and saying hey, we see all these dependencies coming in from viper. Can we remove these and I looked at it?

A

I said yes, it'll be a huge effort, but we can so finding where that's getting drawn in you know using gomod graph or whatever it is finding where that dependency is getting pulled in maybe going into that project saying. Why can I can we remove this? Do we need it? You know finding out what is necessary right.

G

Thank you um follow-up question. If I may uh sure I'll.

A

G

As late as you like, okay, um I didn't even know I was using cobra before today. So what about all the other hundred packages that I don't know about, that are single maintainer.

A

uh Yeah I mean even more philosophically I would say this is a very big problem, not not just cobra, not just kubernetes, but you know like there are hundreds of thousands of these throughout you know throughout the internet and the uh the just the software industry as a whole, um I don't know I wish I had a better answer beyond like hey get involved like hey. We need to be better about this. um It's really. It's really about trust, like I think about. I think about that event.

A

Stream package story all the time, because it was one person. It was one person that just said like yeah, I don't use this anymore, give it away. uh I wish there was a better system beyond like being vigilant, creating those automatic systems to maybe detect. uh When you know, there's inconsistencies in your deliverables or something weird is happening.

A

That's the best we can do from an automatic software standpoint and then yeah getting involved in important projects that live in your upstream yeah.

G

Super. Thank you very much.

H

uh Thanks for the great talk, I was going to actually ask what the gentleman earlier asked around how you would vet a new maintainer when they joined the project, um but I was also going to ask clearly coming here today as a sort of call for help in part. Have you tried other techniques to try and get more maintainers and have they been successful or what have you haven't? You tried.

A

uh Yeah yeah, you know I've tried a number of different things. um It's.

A

You know it sort of is the game of thrones of open source, sometimes where you know you don't want to step on this person's toes by doing this, and that, like, I think it would be a mistake to just go and fork cobra again. It's all the import paths, it's like how many things it touches across the cncf and kubernetes ecosystem. I mean that would just be like a herculean effort to just be like we're going to fork this into our own organization.

A

uh Without you know, first chatting with steve francis chatting with me chatting with people in kubernetes and throughout the cncf um yeah again. I wish I had a better answer, but yeah I don't know. Thank you. No problem.

I

Hi, thank you for your talk and so.

C

I

Would like to share maybe an idealist opinion, but rather than having a single maintainer, which is a single point of failure. Do you think that it is for it is possible to say to people, for example, that okay, you contributed this to my project and so for future modification of this part that you contributed.

I

You will be the maintainer of this part only, and so it will relieve you some pain, because you will still have to click the merge button for sure, but you will have someone that already contributed and that will just do the code review for you, for example, I know that some projects do this, for example in bedroot. When you add a recycle, you are sort of maintainer of the recycle, so you are not forcefully to you have not to maintain it, but it is appreciated from the main maintainer. If you do it.

I

So, what's your opinion.

A

That's, I think, that's a really interesting idea and I do think it's something that's been tried. um Unfortunately, like I go back to that chunk on the lottery factor, I mean people I mean I think you'd be pretty hard pressed to get anybody to stick around say they. You know, contributed 200 likes of code that person's gone more often than not.

A

You know and like that's a useful, maybe that's a very useful chunk of code, that's being committed, and I don't know if you would want to uh hold off on a merge just because this person's, like you know, maybe I don't have the time. Maybe I don't have the bandwidth, but this would be really useful. To me, I think, being accessible is is really important within a contributor community, so I don't think I would personally do that it.

A

um It would keep, I think, a lot of people out and in the end you would almost end up with just like an extreme silo of just a few people willing to stick around to do that. You know I'm saying um yeah, it really is sort of a like. You know you're really fishing for those people. You know the really dedicated people like I brought up mark uh mark is great, and um you know I noticed pretty pretty early on that.

A

You know he was showing up and he was uh contributing to the discussion, not only in slack but issues and prs and doing reviews. It was clear to me that this person was sticking around. um I think it'd be a lot harder to just tell somebody like. No I'm not merging this until you, you know dedicate yourself or dedicate some time or something and again incentive models like why. Why would anybody like it?

A

It gets really hard to say like no, I'm not going to merge this until you stick around people are still going to leave, even if they agree and they're going to tell you yeah sure I'll, stick around they'll stick around for a month or two, and then you know like wash away essentially right. It's it's super hard. I think that's an interesting idea and I do think it's been tried before, but I don't know if that's approach I would take okay. Thank you. Yeah, no problem.

A

All right. We're a bit over time! Thank you, everybody I'll, hang out here. Thank you.