Kubernetes IoT Edge Working Group, 20 Apr 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Can You Keep a Secret? Securely Interacting with Edge Devices i... Adithya Jayachandran & Yu Jin Kim

Description

Can You Keep a Secret? Securely Interacting with Edge Devices in Kubernetes - Adithya Jayachandran & Yu Jin Kim, Microsoft

Kubernetes on the edge is becoming increasingly popular for orchestrating workloads closer to where the data is located. However, the edge usually consists of leaf devices that are too small, too old, or too locked down to run Kubernetes on their own (i.e. IP cameras, thermometers, humidity sensors), and they may operate with intermittent availabilities and downtime. On top of that, security becomes a major challenge at the edge. Each of these sensors can have different authentication mechanisms and have unique requirements for handling credentials. So how can these leaf devices be dynamically bridged to a cluster? And most importantly, how does one determine whether newly discovered devices are real or malicious, especially at scale? In this talk, Adithya and Yu Jin will go over how they enhanced Akri (a CNCF sandbox project for exposing leaf devices as resources in a cluster) to integrate with Kubernetes secrets and secrets stores allowing devices to uniquely identify themselves in a Kubernetes cluster. They will show how even newcomers to Kubernetes can quickly connect their edge devices in a secure manner and restrict access on resources based on a customizable criteria.

A

All right welcome everyone to our session about securely, interacting with Edge devices in kubernetes.

A

So my name is Eugen I'm a product manager at Microsoft, and my fellow Dev Aditya could not be here today in person, but without further Ado. Let's get into it so I'll just kind of go over the problem of iot devices at the edge and then I'll briefly explain what is awkary and how does it work and then the problem with secret management at the edge then I'll talk about our solution proposal and I'll show a quick demo for a proof of concept and then a couple more considerations with our solution.

A

So, as you may know, at the edge there's a heterogeneous ecosystem of devices, they all have different amount of computes, so you may have servers PCS as well as iot devices that can act as controllers sensors and more and this iot environment is constantly scaling up or down with devices, and they often depend on network availability as well.

A

So since these iot devices are too small, too old and too locked down to run kubernetes on themselves today, how can we manage and coordinate these from a cluster.

A

So the solution for this is awkary, which stands for a kubernetes resource interface and agree, is currently a Sandbox project and it makes connections to iot Leaf devices via their protocols. So, currently awkward supports common protocols like OPC, UA, onvif and UW, but users can also use our template for writing.

A

Custom Custom protocol handlers as well so new devices are detected automatically, which makes scaling really easy and devices that are taken off or down due to network availability will also automatically disappear, so these devices will be exposed as kubernetes resources on your cluster, just like memory or CPUs, and the augery Brokers will allow you to use the signal from these devices in your applications and workloads can be assigned to specific devices or groups of devices, even if they're attached to other nodes.

A

So this means you can get direct signal by running on the Node closest to the device which eliminates latency, and if you have multiple devices connected to the network, all the Clusters and nodes can see the awkward devices and also if a node or cluster goes down, then the others can continue to pick up the work for you and as per developers, augery makes it easy for you to deliver. Containerized workloads meant for iot devices with ease, so you don't need to code for each and every specific camera.

A

You can just write a more generic code for each type of device or groups of devices.

A

So here's a brief overview of how awkward works, there's five main components that you should know. So the first is the augury configuration, and this is a crd which tells augri what kind of device to look for, and you can also tell it to deploy a broker for those devices.

A

Then you have the discovery Handler, which uses its protocols to find the devices, and then it will inform the agent which runs on all the worker nodes and the agent connects to the cubelet. According to the kubernetes device, plugin framework to expose availability changes to the kubernetes scheduler and then the awkward instance is an instant crd created by the agent to track the availability and usage of the device.

A

And then the controller which runs on the master node in the cluster deploys the Brokers for the devices to connect to them and utilize them, and it can also handle any node of disappearances to modify the relevant instances.

A

So this is kind of like the workflow of how awkward works, so the cluster operator first applies. The augery configuration then accuracy, which is the config crd, is created and detected by the agent. Then the discovery Handler that is specified in the configuration is deployed and it will go and find those devices and then tell the agent of the discovered devices.

A

Then the agent creates the instance ERD, which is auguryi for each discovered device and the awkward controller detects the changes in the instances and schedules broker pods for those devices.

A

Then the broker is allocated and the instance is updated with the reserved slot and the broker pod begins to run and it'll establish connection with the device.

A

So there are several challenges with credential management at the edge right now. um Currently, you know, passing credentials into augery would be plain text which that might not be as secure as attackers can just easily spoof. That Secrets might also get changed and we need to be able to monitor the updates and pass in the updated credentials and devices also have heterogeneous requirements for authentication and storing secrets. So, for example, OPC UA uses certificates, whereas onvif may use connection strings like urls.

A

So our proposal is to have the awkward agent retrieve the secrets or other data and pass it to the Discovery Handler as a part of its Discovery request. And we wanted to have a native kubernetes experience and we plan to support both secrets and config maps to be able to enable the credential management.

A

So on the right you can see our example configuration yaml and we have added a new discovery field called Discovery properties which can refer to config map data, Secrets or plain text.

A

So at the top you see like our normal configuration yaml with the spec and the discovery Handler and in the details section you can do filtering like excluding certain IP or Mac addresses Etc and then in the discovery properties section where we'll be passing the secret data, it's in the form of a key value pair list, and so these get initialized as a part of the configuration by the user and these properties can apply to a single device or a group of devices so that we can provision all the cameras in a particular area the same way and then there's also an optional parameter, and this is defaulted to false.

A

So if the key doesn't exist in the secret or config map, then the configuration deployment will fail. But if it's set to true and the key doesn't exist, then the agent will just not add the entry to the list passed to the Discovery Handler and the deployment will still succeed. So that's up to the user.

A

So now, let's look at our new workflow I've highlighted the changes here so before everything it's up to the cluster operator to set up the secrets and make sure that they're all properly provisioned, then the cluster operator will apply the configuration and when the accuracy is created, the agent will pull down or update the necessary Secrets. Since it has the API server access to do that, and then it will start and the discovery Handler is deployed and then the discovery Handler gets the credentials from the agent and verifies that they can connect.

A

So the rest of the workflow is the same, and as of now, the agent can't monitor changes in the secret data after the configuration is deployed. So in the case that it has changed I'm. Currently, the operator would need to manually redeploy the configuration yaml.

A

So I'll be showing like a quick proof of concept demo of how this all works. um So let's look at the configuration yaml. So in the first half um we have the secrets. We have the demo odd secret, which is a kubernetes secret and we used string data to pass in the username and password so we'll be using an on the device today and for OPC UA.

A

You can pass inserts and keys for example, um then you can use the endpoint reference for the on the camera to get the device uid for the username and password lookup and then on the right. We have the actual auger configuration yaml, where we specify the protocol for the discovery, Handler so again we're using onvif, and then you can filter out any cameras.

A

You don't want to discover through the IP address, Mac address or URL strings, then in the discovery properties we specify the username and password keys, then we're just specifying that at the end we want our broker container to be deployed for the devices that are discovered.

A

So first I can verify that my camera is working properly by using the on this device manager on my host. So at first you can see that we can't watch the live stream because it has authentication, but when I log in using my credentials, which is admin admin in this case, I can log in and see the live stream properly.

A

And then I have a k3s cluster running on my device here now, usually we can just deploy awkward really easily through Helm charts, but for this demo, I'm running it locally. So each terminal is running the agent Discovery, Handler and controller, and then I've already applied the configuration and secrets yaml. So now, if I do Coupe control get accuracy, you can see the configuration crd and the capacity which I set to five cameras.

A

Then you can do Cube control get augury eye and you can see that the instance has been created by the agent for my Discover device and now, when I do get secrets, you can see my demo off secret and here I'm, just inserting a screenshot of the description and you can see the key pairs and then from there. If I look at the pause, you can see that my broker pod is running for that device.

A

So let's look at the logs of the broker here we can see that it's properly receiving the frames from my camera and it's accessing the rtsp stream and for the demo, we've logged The, Source URL of the stream, which it's accessing with the username and password. As you can see in the URL and then from here you could have an app something like a web application, for example, running on your cluster, and then you can have that connect to the broker so that you can use the video stream in your application.

A

So there are a couple more considerations for the solution that I want to talk about. So the first thing is obviously you want to use the best practices for kubernetes Secrets augery accesses the secrets objects via the kubernetes resource API, and it relies on the cluster owner to encrypt secret objects and arrange access permission properly to ensure that secret objects are secured and to reduce the risk of accidental exposure and then, depending on the cluster configuration.

A

You may also store sensitive data in the cloud and you may need to pull the secrets from a cloud-backed Key Management Service like Hashi Corp, Vault, AWS, Azure, key Vault or whatever it may be, and in this case you can actually use the kubernetes Secret store CSI driver, and this allows you to mount the existing Key Management Service, backed Secrets or certs as a native kubernetes secret, and then you can use them in our Discovery properties just like before, and this allows our solution to be vendor agnostic.

A

Some other things in our roadmap for secret management is improving the agent so that it can continually monitor the secret data.

A

So anytime, the data source is changed, it will reissue the discovery request anytime and then, if the mandatory properties are ever deleted from the data source, then augury agents should be able to monitor for that as well and revoke the discovery request.

A

We're also looking at how we can organize our secrets a little better within this discovery properties, section So currently in this demo I just showed us mapping the enviv credentials directly to each device ID to the credential, using the key names in the format of device. Id attended appended to either credential field category so device ID, username or password, and eventually we might want to have a list that contains an array of objects where the device ID is referred to the username and password Keys, which point to the actual kubernetes secret information.

A

Or you know you can use certs and keys as well.

A

Eventually, we also want to modify the discovery, Handler and broker to have several stages of authentication and conditional on what we pass in. So this would allow us to have separate sets of credentials each with different access roles. So, for example, a sensor read, credential can only read sensor data and a separate set of credentials would be needed to update firmware. So this has much more many more stages of security, and this is just one of many ways to do this.

A

um We showed a more kubernetes native fashion here, but there's a plethora of cncf projects that could simplify this. So, for example, Dapper has a secret management feature and they leverage this a lot. They leverage a lot of the same native components and special interest group work, so one could try to spin up a separate pod from Dapper that manages secrets and then passes them into either the discovery Handler or the agent and in general, we wanted to keep the solution as open as possible so that individual operators can customize this as needed.

A

So you can read more about our secret management proposal in the docs PR, which is linked here. You can also take a look at the openpr in our GitHub for implementation and we're always open to feedback and review so feel free to take a look and comment on the pr um we're planning to have a new release this week, so keep an eye out for that. It will include lots of bug, fixes and improvements to the uwev, Discovery, Handler and Etc.

A

You can also join our slack channel to ask questions and stay up to date with augri and our community meetings take place on the first Tuesday of every month at this time and with that I'll open it up to questions.

B

If anybody's got a question, let me bring you the mic.

B

Hi would it be possible to also integrate something like industry cameras which are usually uh used as USB cams? Sorry,.

A

What kind of cameras industry.

B

A

B

A

Yes, so I think yeah. You can use the similar like our Discovery properties, to pass in the authentication and then from there. You can just use our U Dev Discovery Handler to connect to the USB camera.

B

Anybody else got a question.

B

Well: okay, thank you. That was a great talk.

A

All right, thank you. Everyone.