Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2022, 11 Nov 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Lightning Talk: Paying Down Debt: Converting Kubernetes To Use Go "Workspaces" - Tim Hockin, Google

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from April 17-21, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Lightning Talk: Paying Down Debt: Converting Kubernetes To Use Go "Workspaces" - Tim Hockin, Google

Kubernetes is one of the largest and most well known systems written in the Go programming language. Kubernetes is also a fairly complex codebase, which often pushes Go to its limits. To make it work, we sometimes have had to go outside of the "normal" usage of Go's tooling and ecosystem. Anyone familiar with the project will probably groan when they heard phrases like "staging repo" or "code generator". We have accrued a pretty significant amount of technical debt over the last few years. With Go 1.18 comes a powerful new feature - "workspaces". This is what happens when the language team looks at what their users are doing and as asks "how can we make this better?". We can wipe away a lot of that debt - "just use workspaces". But, sadly, it's not as easy as it might sound. This talk will introduce listeners to the problems we have, the gross workarounds we use, and how workspaces make it all better. We'll talk a bit about the work that is being done to make "just use workspaces" a reality.

A

All right, I'm totally not going to follow this trend. I get five minutes and I have 25 slides, that's 12 seconds of slide, so stay with me. Okay!

A

uh Anybody here ever looked at the kubernetes code base. Anybody very quickly. Yes, great! Have you found these gems things like our staging repository things like a bunch of sim links in Vendor things like a whole bunch of go mod files scattered all over our code base? In fact, 30 of them? Don't worry! I will talk about all of these in a minute.

A

Kubernetes is not your usual go project. We kind of do things our own way, not always for really great reasons, but for a lot of history. So.

A

Let me tell you a story once upon a time we had a mono repo and everybody was happy the end, except for that one guy who wanted to import Kate, Style, kubernetes, sorry, you're gonna have a bad time. Kubernetes depends on approximately the entire internet.

A

uh So, as kubernetes got popular, we built a client library because it seemed reasonable that people would want to. You know, code against kubernetes, um so anybody who had to link our client Library got this so funny story. When I was prepping. This slide I wanted to get a little chuckle there and so I searched for the worst dependency graph. I could find on Google, Images and I found this one I was like cool I'm. Bringing this one into my slides and I was like what chaos is at the beginning of this. It's actually kubernetes.

A

So my point was made I think um we wanted to publish small, focused repositories that people could use in the normal go way with less dependency help, but we don't want to give up our lovely monorepo, so that explains the staging directory. So we put all of this stuff in staging and we then export it through a robot which publishes things into a separate GitHub repositories, but pre-go modules. The only way that you could trick go into doing this was with Sim links.

A

So we put a bunch of sim links in our vendor directory, which point back to our staging directory, which is that strange hell there did anybody know you can check Sim links into git. You can so we also do a a lot of code generation in kubernetes um and we have a lovely little toolkit for this. We call gengo and we use gengo. It predates Go modules, so it's written against goes older tool chain.

A

Libraries we have generated about 33 000 lines of code in the kubernetes code base the to tie it all together because it all depends on gopath and it's all pre-go modules. We have this cute little script called run in go path which sets up a totally bogus go path. Sim links all of our code into it sets things up, so we go as of like 1.8 needed, and then it would run our tool in this. Go path, tons of special case stuff in Vendor directories.

A

I threw one little example there, where we said the staging directory into vendor, so that the tools will work, and you know the truth is most of it works, but it's gross right, so Along Came go modules and all of goes tooling is broken. When multiple modules are involved, there's just nothing that you can do that will make it work. I tried, I, tried, I talked to the go team, I begged and cried none of it works, but the go team. They hurt us.

A

They are awesome people and enter, go workspaces, so they've, given us a new way of describing this crazy setup that I have arranged in kubernetes how to use multiple modules at a time, and it's not exactly built for us, but it's perfect for us. So in theory all I need is this go.work file and holy cow. It kind of works, I'm able to actually run go across multiple modules. At the same time, for example, I can run golang CI lint across the whole code base.

A

All in one go so all that goop that I described before we don't need most of it anymore. Some of it's actually broken now. So it's time to clean the house so step, one I enabled modules for our build all hell breaks loose. All sorts of things fail in reality. I'm doing this one piece at a time, but I'm fast forwarding, because I've only got 12 seconds, then I hacked up gengo to use the new packages module, which is the new way of doing things. The modules compatible package, it's a pretty major change.

A

My goal right now is to drill a hole all the way through the mountain there's going to be some debris left behind. I've got about 40 fixmes in the code base. So far, a lot of work to do still, step 3 convert all 13 of our code generation tools. Why do we have 13 tools that would take more than the time I have tldr? They all need to be touched, every single one of them and I'm only partly done with this step step. Four, what everything got: slow, really slow like 10 to 40x slow.

A

uh What's going on well, the new packages library is very different, so we've talked to the Go Team how to optimize for this stuff. But my goal is that we get to plausibly acceptable performance step. Five Purge started deleting stuff make all the references to go path, go away, hundreds and hundreds of touches step. Six lots more cleanup to do. I have a running go cleanup file, so the status is I'm currently at 83. Actually, when I wrote this, it was 83.

A

Now it's 88 commits I'm not done yet still going I'm, hoping that we'll get it into kubernetes. 127 I, pity the fool who has to review it.

A

um Go X tools, packages, I'm working with the go team to see if I can justify making it a little bit faster, maybe doing it a little bit better lack of vendoring. We got to figure that one out, that's it! Thank you.