Cloud Native Computing Foundation Cloud Native Security Day EU 2021, 14 May 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Lightning Talk: Securing CI/CD Infrastructure for Tinkerbell - David McKay, Equinix Metal

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Lightning Talk: Securing CI/CD Infrastructure for Tinkerbell - David McKay, Equinix Metal

Tinkerbell, a CNCF sandbox project, has some pretty unique CI/CD needs. As a bare metal provisioning system, CI/CD involves running servers for DHCP, iPXE, virtual machines with QEMU, and a few others bits and pieces. This use-case is not natively supported by most CI/CD SaaS vendors. To tackle this, the Tinkerbell team has automated the management and provisioning of their own CI/CD runners using a collection of off-the-shelf tools. You will learn how Tinkerbell secured their unique infrastructure and how to approach securing your own CI/CD stack. We will demonstrate Tinkerbell’s provisioning tools and dive deep into how they were configured for security. The same tools are publicly available and could be used in your own CI/CD setups. You will also learn how to secure engineer access to your infrastructure without getting tied to a single cloud provider.

A

Hello, welcome to my session today today we're going to be taking a look at securing the continuous integration and delivery infrastructure for the tinker bell project. My name is david mckay and I am a senior developer advocate for equinix metal. I am also a cncf ambassador and I do quite a lot of live streaming. You can find my streams at rawcode.live.

A

My goal with streaming is to provide educational resources for all of us to learn the vast cloud native landscape together.

A

And today I want to introduce you to tinkerbell a bare metal provisioning system, open sourced by equinix metal, now a cncf sandbox project that aims to solve some challenges that have been difficult for a long time and that is commoditizing bare metal, which is no easy task. So before we dive right in. Let me just shrink myself down.

A

So tinkerbell isn't your run-of-the-mill project when we're working directly with the metal. There are a lot of things that a little bit harder than working with virtual machines. First tinkerbell has to run an n memory operating system that can handle partitioning disks, encrypting, disks, writing and installing operating systems through container-based workflows.

A

There's multiple microservices that are responsible for understanding which bare metal devices are coming online through mac address identification. There is ipexa for building the operating system and streaming the operating systems over the network and, of course, there's ip address management too. In dhcp and while you can use traditional ci systems, you're, probably you definitely have to provide your own runners.

A

So in order to build out our continuous integration and delivery system for the tinkerbell project, we need access to some metal. Personally, I work for echo next metal and economics. Metal do donate a substantial amount of infrastructure and inventory to the cloud data foundation for projects like this.

A

My go-to tool for spinning up brand new machines on any cloud provider of choice is blooming, and there are a few reasons I want to talk about polymer for this session.

A

One polymer very graciously gave us a free access to polymer cloud for the tinkerbell project. This comes with a whole bunch of benefits. From the security side, it meant that we could commoditize access through their our back system. It also meant we could take advantage of their secrets management as well.

A

Something that bulumi does really well is allow us to really adhere and adopt get ops by having everything that we need, including secrets and the repository and pushed. They are of course encrypted using the polumi cloud back end.

A

However, if you want to use pollumi- and you want to stick to the open source, you can use any cloud kms as a backend as well, and this is our actual production stack file which is open source on github.com. You can see, we have aws credentials here and we also have our equinix metal credentials.

A

And next it's not, we can't just spin up metal and magically does something right. We have to go through some professioning stage, so we need some software on the devices too. My go to tool for this is solid stack again, focusing on the security reasons of why I'm using the salt stack, there's one there's no ssh. As a transport protocol, solstack uses 0mq based messaging to pass messages which the minions are subscribing to from the master and executing those days.

A

Salt also has a concept of pillars which allows us to have secret information available on the salt master, node and selectively, distributing the keys that we want to each individual machine or minion based on a whole bunch of grains and parameters.

A

One final thing about the messaging system here is that it simplifies all of our network policies. The minions only have to be able to speak to the salt master, we're not opening up ports for the salt master, to reach all of our minions and that's a big win too, and the way that we're consuming salt stack or a provisioning salt stack on these machines is leveraging palumi's secret store.

A

Writing the secrets that we need to cloud in it and then they'll come available to salt and the secrets being stored in pillars means that we can selectively distribute them based on grain data to each of the minions. So the menus only get the secrets that we allow them to see, and it's worth pointing out. The while tinkerbell was open sourced by equinix metal and a majority of the team comes from equinix metal. That is a cncf sandbox project.

A

This means that we're using hardware not on our equinox metal accounts, but on our cncf accounts. It also means that any maintainer or contributor, regardless of where they, where they are employed, should be able to have the same amount of access. We want to protect against the bus factor, of course as well, so we need to commoditize the access to the machines.

A

And for that I'm falling back on one of my other favorite tools. Teleport teleport allows us to disable open ssh.

A

We don't need to rely on giving people access to the machines by reaching out and getting their ssh keys or scraping them from github. We don't need to add everybody to the project on equinix metal.

A

We can use teleport's ssh server, which is backed by github sso, and restrict access to these machines based on a group that we create within the tinkerbell organization on github.

A

So in order to give people access to the runners or to the salt master itself, we just have to add them to a group on github and that's pretty cool.

A

We're keeping teleport secure by only allowing private ipv4 access for other nodes to join the cluster and again, the tokens are all stored in the blooming. Store are encrypted by polumi and distributed via cloud and at two salt stack to the runners as needed.

A

So what does that all look like.

A

Okay, so first you can see all the codes to provision this infrastructure and the applications running on top of the machines at github.com tinkerbell infrastructure. We have the paloma directory, which is responsible for running the pollute up provisioning. The bare metal writing everything that we need to cloud in it to self bootstrap, the salt setup from there. Salt takes over and installs everything else that we need on itself and the runner devices using polumi's cloud. We have access to see when pollutant commands are run against a stack.

A

We can just click on tinkerbell infrastructure production.

A

We can see the outputs, you can see the configuration used, including secrets, although they are nicely sophisticated and what else is cool is that we have the activity view that shows us every time someone ran the blooming stack, so we got really good visibility and to when any of these seekers were accessed, the state was changed and nodes were spun up.

A

Now, in order to get access to the machines we can browse to teleport.tankerbill.org, there is only one option to log in and that is through github. I click the magic button and I now have access to all of the machines within the infrastructure.

A

Think what is really cool about teleport, as our ssh means, is that we can have the ability to see active sessions and in fact we can even join them if they were in progress and see what the people are typing or doing or whatever, and the sessions are also recorded. So, let's take a look at that.

A

We can jump on to our salt master.

A

And I can just run nice, simple salt command, to ensure that all of my devices are online.

A

If we pop back over here, we can click on active sessions, and we can see that I have an ssh session in progress and I have a join button which, if I type echo hello, I can see both of my terminals very, very cool.

A

Let's end, both of these sessions and refresh and that'll end in just a moment. Hopefully it shows up here.

A

And now our session is gone, we can go to our audit log. We can see that sessions were started. We can see the single sign-on from github. We can see that someone joined a session and we can see the user disconnected.

A

We can come back here and click play on our recorded session and see all of those commands that were executed. We got the cell, we have the cell test.peng, followed by our.

A

Echo thank you for watching this session. I hope you get as much value out of pollumi soul stack and teleport. As I, the tinker bell and equinix metal have a great day.