Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2021, 29 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: A Safari of Kubernetes and its Natural Habitat - Noah Abrahams, StormForge

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

A Safari of Kubernetes and its Natural Habitat - Noah Abrahams, StormForge

Have you ever wondered what life in the data center really looks like, after the sun goes down and the people leave? Come along on a journey as we visit some of the inhabitants of the Cloud Native savanna on this trip through the ecosystems of Kubernetes and its natural predators. Attendees will learn the warning cries of an ever alert Prometheus and watch as the cluster is hunted by a gaggle of red teamers, while they all try to drink from the same data lake. This talk will be a whimsical introduction into the daily life of kubernetes and common production deployments while you listen to some very mediocre impressions of famous naturalists.

A

Ladies and gentlemen, sir david attenborough.

B

It is evening in the data center most of your I.t staff have already made a hasty retreat.

B

They are hopeful that on this evening, they will enjoy some peace and that no clarion will sound so far this evening is a quiet one back in the fields. Here we can see a wild kubernetes.

B

This kubernetes cluster is blissfully unaware. It is currently grazing, among other instances, looking for.

A

B

Delicious yaml to consume.

B

This yaml provides the kubernetes with all the nutrition it needs to perform its daily activities and it guides the herd's actions of carrying workloads.

B

But the cluster shares its environment with.

A

B

Cloud native projects and others, a pride of developers, arrives hungry for idle resources that it can use for its applications.

B

While the cluster remains unaware, the developers are quite active. They function on a different schedule.

B

Now, while the developer starts functionally blind looking for purchase in this vibrant and robust ecosystem, it is wily and creative.

B

Eventually it adapts to its environment, but it may find the footing unstable and it can be fraught with perils.

B

This pride is currently searching to sate its hunger and knows that when others are not using the cluster, many additional resources may become available to them and their applications.

B

These developers, who previously fought for easier access to the systems, bring along some yaml of their own.

B

Some of them carry it along in helm, charts only ever wanting to change one or two variables, but for others it is the ease of deployment they care about, and for them it is carried forth by this river named argo cd.

B

Argo maintains the health of this valley and ensures all the ammo is nutritious and up-to-date.

B

With argos help, they are able to consume many more resources than if they were to sow their yaml by hand.

B

But with a ravenous appetite the developer says I want it all. I want to test quickly, so I want as many resources as you can give me.

B

The reuse of a quick one-liner in bash developer makes copies of their application. Tens sometimes hundreds they then copy and paste. Within the argo configuration.

B

But what's this with the influx of all these new development, qa and pre-production deployments, a problem arises: these nodes are full and can consume no more, but what happens next is truly extraordinary.

B

To protect itself from the developers resource consumption, the kubernetes relies on one of its greatest strengths it scales.

B

In doing so, it is able to bring many more resources into the herd and respond to the developers insatiable appetite.

B

But while scaling provides more resources, it is not magic. It cannot help with everything in this case, because of the constraints upon the cluster's design specifications, there are no more internal ip addresses available for nodes.

B

The allocated pool has simply run dry.

B

The over-consumption from the developers, hunger begins to reach a boiling point and suddenly new nodes and new deployments are beginning to fail.

B

As the coming chaos, unfurls alert manager begins to take notice of these problems.

B

The group of prometheuses, known as a gang, sends out a warning cry to alert whoever is listening.

B

They shout about node failures, about there being insufficient cpu and memory to fulfill all the requests and even about indecipherable errors from the upstream cloud providers.

B

With the prometheus's cries crossing a threshold, the clarion has wrong.

B

The iit staff is spurred into a defensive formation huddling around and protecting the cluster. The pride of developers, of course, will collectively proclaim no knowledge of what has transpired this evening, though, the kit logs will tell us otherwise.

B

Meanwhile, some of the nodes may have succumbed, but through sheer numbers enough, make it through to ensure the survival of the applications as a whole.

B

The next day after meeting upon meeting quotas are being set and open policy agent or opa is being installed.

B

The previous approval process that was loosened is reinstated, so the developers can no longer check in changes to the argo configuration without oversight or at least a thumbs up.

B

They have paid for their indiscretions, but this particular problem will not arise again. The developers will have to be even more crafty to save their hunger tomorrow.

B

The previous cluster managed to keep damage to a minimum because it was protected between dex and gangway handing authentication back and forth. The prometheus alert, manager's constant eyes and later opa enforcing its rules. Controls stood guard to keep watch over the herd.

B

This, however, is nigel: nigel is a brand new on-prem cluster, the first of many just released into production on a monday evening for an adoption test.

B

The original design team, however, had flawlessly decided while building non-production clusters, but since the access keys were only given to gitlab and the administrators, they felt that they did not need to worry about additional authentication or login patterns.

B

This, as will surprise, no one was not corrected before being propagated to this newly launched production cluster.

B

An unprotected cluster, however, attracts other visitors with more nefarious intent.

B

Birds such as these will take advantage of any easy meal that they can catch.

B

As soon as the cluster was turned loose into the wild, this gaggle of red teamers, acting from a remote location, was swift and efficient.

B

They have found that the kube, config and admin keys are embedded in the image for the custom, get lab runner and it points to an api endpoint that is accessible from anywhere within the corporate network.

B

These are design decisions that never should have made it out of testing, so their attack is merciless. As with all birds of prey. The carrion, the detritus, the potential for disease to spread and infect others in the biome has now been identified and corrected.

B

If this had been a real attacker by morning, there would be no secure nodes left in this cluster.

B

No pod would be safe if the internal corporate network were compromised or the actors otherwise had access on the inside, the entire cluster would be lost.

B

Unfortunately, with no elastic cloud capacity behind it, it would have no room with which to scale. So by the time they are done, mining, bitcoins or whatever they're doing this cluster would have simply run out of resources.

B

The herd would be picked clean.

B

What's more once compromised, the flock can also be expected to drink deeply from any and all storage buckets they find behind this cluster. They would simply drink this data lake completely dry.

B

Luckily, for nigel, the red or offensive security team makes a formal report and the blue or defensive security team has now responded.

B

They had very little knowledge of the danger they were in, but have now been made aware of the terrible design decisions that were made long before these clusters were ever thought about being put into production.

B

Once made aware of the threat, they are forced to be reckoned with, they will implement strict authentication and authorization tied to the corporate ldap service.

B

They will bring in another avian challenger falco to mine, the threats and keep watch over the herd. Significant rules will be created and processes will be changed, but first they need to tend to their gitlab runners.

B

This was created as an upstream problem, which they have enshrined themselves in the registry of harbour if they do not fix the danger that was checked into that image, it will only be a matter of time before a feast like this presents itself again.

B

Instead, keys will now be encrypted and then stored as secrets for the runner to decrypt at runtime and new network segmentation will be put in place quite a lot of work for day two of production.

B

But what's this while the infosec team is implementing new segmentation and separation policies, a new vpc and new cluster have arrived meet. Bertram bertram is the latest release, even though it only has 12 months until the end of its life.

B

Many will not even consider it feasible until it is mature, but by then the growing herd will have left it to die.

B

Will this young upstart bertram take its opportunity to challenge the dominant cluster.

B

No, as part of the segmentation, this new cluster lives solely for pci compliance and has even gone through a completely separate and quite rigorous qualification process.

B

While it may drink from the same lake, and it may look similar, its paths will not cross with the other clusters.

B

Now, when the predators return, they will find this prey difficult and hardened, not exactly what they were after.

B

Despite all the new measures put in place after a laborious and disastrous day, one the herd cannot, however, think themselves completely safe. These birds of prey will continue to circle, watching from afar forever searching for their next meal.

B

By now, you have surely decided that this world is one worth exploring, but you may be wary of the threats looming around you. The world of cloud native, however, is not all predators in danger.

B

There is plenty of fun excitement and even leisure to be found here. The excitement that awaits you is just part of a much larger.

A

B

Here, though, in this unforgiving land, it can be difficult to get one's bearings, especially with interesting denizens competing for your attention. It is easy to get lost even with a map.

B

Luckily, for all of us there are guides aplenty, the cloud native locals are not just friendly, they are incredibly hospitable. They are always eager and happy to help, because it is the people that make this community so vibrant.

B

They can help you understand service meshes for example, or maybe some of the indigenous githubs platforms, or perhaps you may need help with something ubiquitously problematic, like dns.

B

It's always dns.

B

And let's not even get started on how many denizens there are, that call observability their home category.

B

Why the monitoring section alone is so densely populated. It is hard to tell those taking action from those who sit by and idly watch, but many of them can bring you peace of mind and help you sleep at night.

B

But there is even more in motion here: there is also a hierarchy among the projects vying for dominance and contributions with enough market acceptance.

B

Some of them will graduate, while others may be left in the sandbox, but perhaps you would like to join them in this sandbox. There are dozens of projects here. That would be happy for some of your attention and would just love for you to come and play with them and contribute.

B

Once you have your bearings, you will find a wondrous world stretched out before you, with projects ranging from security to database scaling cicd to network management and from cross-cloud control planes to cleaning up those cloud workloads.

B

On the human side, there are many various meetups conferences and other gatherings the denizens, sometimes congregating, simply to discuss how much they love to congregate.

B

You will always find an outstretched hand here from one of these ambassadors.

B

While the real savannah is full of endangered species and faces many crises, the cloud native ecosystem is healthy, thriving and growing.

B

Whatever part of this ecosystem, you find yourself in you can rest assured that there are people willing to help you on your journey and events to make it informative and enjoyable.

B

That's all for this evening and we hope you will join us next time as we take a trip to the clouds.

A

Thank you, sir david emperor.

A

So, yes, I opted for the cold open on that talk.

A

I thought it would be fun to just sort of go in. I had a couple people suggested and what would normally be the intro slides here we put at the end. So who am I? Why did you even come to my talk? My name is noah abrahams. I am the open source advocate at stormforge, which means I am split between open source program office and dev advocacy.

A

If you want to know more about stormforge find us at the booth. A talk is not the place for talking about that. I'm a cncf ambassador. I run the meetups out of las vegas uh and I've been doing this cloud thing for quite a while, since about 2008 and I've spent the time bouncing back and forth between low-level hands-on work being devops or sre being the guy that broke the thing, because I didn't realize that wasn't: name-spaced, whoops and being much higher level and having conversations like this to broader audiences.

A

Why did I make this absurd talk?

A

This talk started from a tweet. Actually, um it was a tweet by chris nova.

A

Who said what would your entrance music be if you were walking onto stage for a keynote at kubecon, and I have a feeling you all can guess what the music was that immediately popped into my head, because I've hummed it like 15 times through the course of this talk. um You all know the non-words.

A

Because I love nature shows- and I think getting in touch with the natural world and getting in touch with what's around you, it just helps ground you as a human being, but that evolved from what would the theme music be to? How would you build a talk around that which eventually morphed into? How would you get sir david attenborough to come?

A

Give a talk like that which the answer is you don't because he's very busy, so I had to do it for him and we did that because it's another set of analogies there's a lot of analogies, there's no lack of nautical analogies, that's for sure in the kubernetes environment uh or buildings, with concrete foundations, building up stacks and layers or shipping containers or spider webs or any number of them, and I wanted this one to be something that people could associate with uh to show you some starter patterns.

A

So now, when someone says prometheus, you don't think prometheus. That's the monitoring solution, you think prometheus! That's the yelling meerkat! um So you're welcome! That's all in the back your minds. Now uh we wanted to call out a couple patterns that were good to avoid too the flip side of. Why did I make the absurd talk is? Why did I make the talk absurd, um partly because I think I'm funny less of you do now than when you walked in that door?

A

uh Also, I really wanted to just do an impersonation on stage. So thank you for letting me get that lifelong dream out, but I did it so you'd. Remember it because we have to have those associations. We have to have something that we build upon as we fumble our way through this incredibly vast ecosystem and if you've got anything to associate it with, then it helps you learn the projects, the people, the ecosystems, helps.

A

You learn everything, so that's why I did it and also because it's been two years since we were all together in person and you all deserve some fun. I hope you all had fun.

A

A few points to remember just as you're all getting up to run out and go to lunch, which I'm sure will happen very shortly. um This is a living breathing ecosystem, the same way that the savannah is full of plants and animals.

A

This ecosystem is full of people and projects, and the projects are just more people, so always remember that. There's people under the hood for everything that you do and it's important to care for these resources.

A

One analogy that you'll see in a lot of places. People talk about pets versus cattle. That was obviously an analogy written by someone who never had to take care of cattle.

A

B

A

That some things are disposable and some things you just don't give attention to well, as I just pointed out, if there's things you don't give attention to, suddenly your entire system can be compromised, so treat things well, give them respect, um and, as I mentioned just a moment ago, that everything is built on people in both technology and in naturalism and conservationism and things teamwork is important. You have to be able to work together as a group, you have to be able to go hand in hand and sing kumbaya and go down this road.

A

Happily, together, we don't do this alone. None of us gets through this without the help of at least a couple thousand other people that are here today.

A

Cooperation is far more important than competition in this industry, because we're all relying on open source projects.

A

I also want to take the moment to point out that the work we do is tied to the physical ecosystems number that we had in a previous report is that every year data centers create over 100 million metric tons of co2, and, if you don't pay attention to the resources, if you don't pay attention to scaling, if you give your devs unfettered access to the system and allow them to spin up eight billion new instances, um it doesn't just harm your wallet. It harms the planet.

A

So keep that in mind. That happens. It's real, and you know I mean this was this was tongue-in-cheek, it was meant to be fun. It was meant to be silly, but the issues are serious, so don't forget them.

A

Apologies apologies to sir david attenborough for my terrible impersonation, I'm sure uh apologies to the national geographic society for using an orange or yellow rectangle, to public broadcasting and to viewers, like you.

A

Thank you all for coming to my talk very image, heavy. Any questions.

A

I'm going to take that as a no do we have any questions from online.

A

Thank you all for coming.