Cloud Native Computing Foundation CloudNativeSecurityCon 2023 - Seattle, 2 Feb 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Keynote: Learn by Hacking: How to Run a 2,500 Node Kubernetes CTF - Andrew Martin & Andrés Vega

Description

Don't miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from 18 - 21 April, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Keynote: Learn by Hacking: How to Run a 2,500 Node Kubernetes CTF - Andrew Martin, CEO, ControlPlane & Andrés Vega, VP of Operations, ControlPlane

TAG Security has run a CTF at Cloud Native Security events since 2020, but with a twist: instead of dastardly black hat hackers duelling for the title of Ultimate Kuberninja, we’ve focused on helping everybody to hack, teaching approachable security principles to increase the industry’s level of cloud native security expertise in novel and engaging ways. In this talk, Andrés and Andy detail their learnings, techniques, and often last-minute fixes needed to run Kubernetes CTFs with thousands of nodes, hundreds of cloud native hackers, and buckets of coffee. During these distributed orchestration challenges the events have seen servers burned, scenarios shredded, and authentication bypassed in all sorts of nefarious ways by the willing and able players of the game. In this talk we detail our experience and discuss: - How to build a tumultuous and exciting CTF challenge - Why hands-on practice is the best way to ingrain security concepts - When automating a chaotic cluster pipeline doesn't scale - Why points don’t always win prizes - And how sharing knowledge helps us grow together

A

Hello and welcome to our talk, learn by hacking, we're.

B

Infinitely grateful that our RX or the cloud native invite Enlightenment, and we extend a heartfelt thank you to you. The attendees volunteers contributors and the cloud native Computing Foundation involved in the vision and delivery of kubernetes documentation and Bug fixes do not write themselves and the incredible selfless contributions that drive open source communities have never been freely given or gracefully received.

B

Security controls are generally more difficult to get right in complex orchestrations, with the functionality that kubernetes is known for to these security teams, especially we thank you for your hard work. The tax security ctfs are a reflection of the pioneering Voyage of the good trip of kubernetes out in the choppy and dangerous free Seas of the internet.

A

We have a free book available to download written by myself and my veritable co-author Mr Michael hasenblast.

A

There may be useful tips and tricks for individuals looking to assail the cloud native infrastructure if you're intrigued enough by this talk to join us on Thursday and play our new scenarios.

B

So we're going to talk about how we run ctfs at Cloud native security con over the past years, what we did, what we did, what do we hope to achieve for you, the attendees, and we, how we maintained our fragile sanity while provisioning, thousands of notes on demand per day.

A

Over the years we have run ctfs in person virtually planning for these events began in the Ethereal and Misty Nights of 2019, as we looked to determine what it would take to put together a CTF running on kubernetes for kubernetes without having the entire infrastructure come crumbling down around us.

A

Nothing is more terrifying than building a self-hosted security game and inviting all and Sundry to have a blast at it. Our goals are to communicate how best to secure Cloud native infrastructure.

A

They are red team scenarios with full demonstrations, teardowns and remediations.

A

The best form, as Sun Tzu has told us of defense, is attack, know thyself, and we look to give something back to the community that has so generously shared. So much with us here stands the nefarious pirate Captain hash Jack, leading the charge with his eponymous bag of Bitcoin and the entitled Goose that sparked so much security enthusiasm across the last few years.

A

B

Is a learning experience for all skill levels, starting with an introductory scenario at either each event, building up in difficulty, progressing as a game that gets more and more difficult throughout the journey to Captivate and Intrigue all skill levels and help nurture and seat the interest in and around security with dedicated infrastructure. We look to emulate as much of the real world systems at some cost of complexity that we're going to dwell slightly further into the presentation.

B

The goal is not to find the Ultimate Security ninja, but the right Rising tide that will lift all boats and that players have fast feedback and support to enhance their learning. Experience.

A

But first here is a digital reconstruction by the street artist. Banksy of what a bunch of pirates would look like playing a cloud native CTF.

B

With a splash of stable diffusion for effect as well.

A

So there is no such thing as secure software as fast as we can possibly paint over the colorful selection of vulnerabilities in our infrastructure. New features are shipped software moves quickly and those new features potentially feature untested code paths and any of those on tested code paths with security side effects are by default vulnerabilities. We prioritize shipping features over vexing vulnerabilities at our Peril, of course, and this is the reality of modern software that we balance in our day-to-day every day.

B

Resemblance to the friction we see between red teams and blue teams any any resemblance in that last animation is not a coincidence. Organizations not only evolve for competitive Advantage; they must do so in order to survive because their adversaries are also evolving. So how do we represent that complex puzzle in a safe learning environment?

B

As scenario authors, we look to construct the most realistic s, scenarios that we can manage. That includes building infrastructure, diagrams and thread modeling scenarios and help us develop with greatest degree of realism and is born from our experience Consulting with regulated, Industries and startups alike. Here we can see a supply chain attack in progress.

B

Captain hash Jack has a tax systems through various Network systems in the past, including packing repositories, container Registries and the notable hypertext coffee pod control protocol, with no stone left and hindered running the platform on real infrastructure gives us the opportunity to emulate simulations. We see in real life, coffee pots and iot pots and coffee pots and iot attacks are not the exception.

A

And we are looking to educate as many people as possible as effectively as we can here. We see a pod security treasure map from the book hacking kubernetes. These scenarios are intentionally increasingly complex and difficult. This helps to crystallize players, understanding of a wide and complex security landscape and, as with all ctfs attendees and players, must enumerate the visible Horizon escalate that privilege reverse back from dead ends and intentional misdirections.

A

This pod treasure map details many of the routes that an attacker might explore within a kubernetes cluster, but with each scenario there is only one intentional route for a player to find with the notes and caveats, the players have found multiple bypasses unintended shortcuts and Wily workarounds as Noble routes to completion.

B

So how's the CTF played with a terminal IP address an optional authentication credentials for an SSH session, a kubernetes cluster or other mystery piece of infrastructure. Here we see the first view of a cluster that a player has when placing a root enabled container running in cluster.

B

They must enumerate the visible, uncover the invisible and start to explore potential routes of compromise in the cluster. This may include hijacking sessions and credentials attacking apis and data stores, routable from the starting point and escalating privilege, through any a null identity mechanisms. A cloud native system may have my personal preference is to bring a portable set of bass scripts to speed up enumeration and exploitation.

B

However, there are no rules. What tooling you can bring with you? You just need to find somewhere to execute it from over.

A

The years we have looked at a whole host of vulnerabilities and misconfigurations from container image file system, foobars and runaway runtimes to secure config, malicious mounts, unsequested Secrets, privileged pod security policies, sensitive service accounts, no firewall, networking, awful admission, control exposed, scds, terrible TLS implementations, Federation failures, chronic control, plane configurations and unlimited user exposures and there's more from Ingress to the kernel.

B

We played this remotely we're playing it in person this time around adapting to the pandemic and running from virtual events. Online Gatherings running the infrastructure over Slack peace was never an option.

A

So how did we go about this high pressure live deployment of 2 500 nodes across each day? We play the game. We start with the open source, breach simulator tool built to help control planes, clients secure their clouds.

A

One five node cluster is provisioned for each scenario and plus the ReUse is not enabled we add a double dose of supermassive provisioning engines, backed by k-native scaling to zero. These are not centralized in a unified engine.

A

This sits on the back of a pub sub queue and re-enterence state machine which tries to ensure that even half provisioned clusters are forced into life and reinvigorated when half deployed, which just means distributed systems, as we know, are hard, then the unique credentials are distributed to each player for each scenario. Cluster pairing and the games begin. Our trusty task, Masters are on hand in person and on slack to hand out hints, offer suggestions and guide people through the path of the challenges.

A

All this adds up. We spin up the Clusters as needed, and we hit the lofty Heights of two and a half thousand nodes spun up per day with 80 to 100 participants. We hope to beat this record on Thursday.

B

What has gone wrong? Would you really believe us if we set nothing? There was one time an AWS internal Ubuntu mirror went down, as you can imagine nothing completed. Provisioning simulator was just host dead in the water. Hell checks decidedly unhealthy to Showcase that typical distributed system problems when ec2 wouldn't provision VMS a very red dashboard.

B

A rising sense of panic came from it that required us reprovisioning into a different region post haste that time somebody figured out how to bypass the initial SSH tunnel sandboxing, which was also interesting securely provisioning and secure infrastructure, is hard. We don't want it too insecure.

B

When the cluster admin R backs, the rbac credentials, disappeared, gone found their way and through Bastion node, when the test Suite only covered. The happy paths also.

A

And through it all, we seem to have hit our goal. We're thankful to everybody who has played the game, given us feedback and give us the opportunity to iterate on this wonderful, thoroughly, enjoyable platform that we love to play.

B

Ctfs can often be big can feel daunting. So we hope that you focus on The, Learning Experience walk through the scenarios and if you feel like smashing machines channel that please.

A

And uh clusters and scenarios today are, of course, open source running on this Cloud native breach simulation underneath the kubernetes simulator project there is a hosted version and control plane are running a private beta. If you would like to be involved, as you can see, nothing is perfect. There's an out of bounds exception on the end of the slide. If you would like to pop some shells with us, please come and play tomorrow.