Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2021, 29 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Real-Time Data Anonymization the Serverless Way - Yuval Lifshitz & Huamin Chen, Red Hat

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Real-Time Data Anonymization the Serverless Way - Yuval Lifshitz & Huamin Chen, Red Hat

How do you ensure privacy protection in the far-flung computing workloads that make up many Edge infrastructures? One way is to ensure that personal information is hidden, on the fly, without introducing lag. Seems like a tall order, but it can be done. This talk presents a Cloud Native Serverless architecture to ensure real time data anonymization, using KEDA and Rook. Specifically, we have extended Ceph to support AWS SQS compatible APIs and developed an external Scaler in KEDA to allow Serverless functions to query, pull, and anonymize objects. This architecture is lightweight, reliable, and scalable. More importantly, the queue trigger mechanism in this architecture does not require us to expose external endpoints to Serverless functions that could become additional attack surfaces. This talk will demo an open source Serverless workflow based on the above technologies. It uses object detection AI models to anonymize images that are produced by Edge workloads.

A

Welcome to our kubecon north america 2021 presentation: it is our pleasure to meet you virtually in this event. In this talk, we will walk you through our service architecture to anonymize personal sensitive data to preserve data privacy.

A

Here is the agenda for today. First, let's look at the global legislation: landscapes on data protection and privacy preservation, what's their impacts on our industry, and how should we address to these challenges?

A

Next, we present to you the solution architecture through our target's use case, that is, to anonymize personal and sensitive data from images that are in a hosted, storage environment.

A

Rest of the talk will deep dive into each of the building blocks. We will show to you how we use, adapt and fine-tune these technologies to meet our scalability and security goals.

A

Data protection and privacy is an increasingly important issue for global data controllers. Care must be taken to process exchange or store sensitive personal identifiable information and honor privacy preferences.

A

So here we have a diagram from the united nations dated year. 2020 about 66 of the countries in the world have data protection laws for reference just a couple of years before that the number is about 50, so we anticipate more and more countries in the world will have data production laws in the future.

A

The it industry addressed to such requirements by using two techniques pseudonymization and anonymization. The difference is whether the transformed data can be switched back to their original forms through another process, for example, data encryption and decryption is a form of pseudonymization once encrypted. The data can be used in computation, transmission and persistent.

A

In fact, data protection laws such as gdpr recommends pseudonymization because it protects data privacy, but data encryption and decryption often incurs computational and operational, or sometimes even financial overheads. So this is a mass solution for all use cases, in fact, in many use cases, especially machine learning, the privacy of the private personal privacy privacy information are not used at all, so they can be completely removed from the original data form, and this is the process called data on monetization.

A

According to eu gdpr, recital 26 data anonymization does not fall into the scope of the law and in the rest of the presentation, our use case is to anonymize data. This is the hosted storage environment.

A

Here is the high level overview of our solution. Architecture, as you can see here, we use mostly cncf project, namely rook and kera. Rook is the infrastructure orchestrator we use rook to create self cluster and creates a self idw bucket and also enables bucket notification mechanism.

A

So activities within buckets are pushed to the message queue endpoint we use cada for service as a service framework for our execution engine, so cata or kubernetes event, driven architecture is lightweight and the service provider oriented is very suitable for our use case here.

A

So the camera service functions use the machine learning algorithms to detect objects of mostly personal and sensitive information in within the image. As there were so many uh mechanisms around, we use a hard feature classifier from opencv and deep neural network from tensorflow with us lots of generality.

A

So the whole workflow is like this.

A

When images are uploaded to the self-igw bucket, the activities are pushed into the message: queues, the service that subscribes to the queues, rhythm events and downloads, the images from the bucket using objects, detection, algorithms, define the region of interest and anonymize the personal and sensitive the anonymized image are then replaced original images in a bucket.

A

The whole workload is running on the microchip's cluster. Microchips is our last way with implementation of open shifts and kubernetes, it's very fast to start, usually in seconds versus minutes in regular kubernetes and openshift clusters. This consumes very little. This resource ice can run on the small single node cluster, with only two cpu cores and 2gig of memory.

A

We will spend more time on these components in the deep dive sections.

B

So now that we've seen a use case for bug notification, let's look deeper into the details, so bucket notifications are built from several building blocks. The first one is a topic. The topic tells us where to send a notification.

B

A topic can aggregate different notifications from different sources to a specific endpoint.

B

The endpoint could be a kafka message, bus or more precisely, a topic inside a kafka message, bus, a topic inside a rabbitmq server, an http server and we're going to add more like this we're going to add aws sns slamda nkp10 and using some scripting and customization, you can add even a nats message broker as an endpoint for notifications.

B

The second building block is the notification itself. Identification itself is what ties together the topic with a bucket. It ties them in a way that tells us which notifications we're interested at which kind of events, creations or deletions or both, and we can also add filters that can filter out different objects from being sent or not sent. As notifications to the end point, the last building block is the event itself, and this is the information that we're sending to the topic. Based on the configuration in the notification.

B

So if you feel like setting up the system with the bug, notifications and the endpoints is complex, you're, probably right, but here we have something that should help us, and this is rook. Rook is a storage operator for kubernetes. It's a graduated cncf project that provides different storage providers for kubernetes and has both file block and object storage, we're going to focus on the object storage. Now? What do you get in? Look you get kind of tuned the price of one.

B

First of all, your storage solution is actually hosted inside kubernetes, with all the value of orchestration and managements that come with it. The second thing is that it gives you storage for your kubernetes applications.

B

Now I'm going to show how, in five simple steps using five, simple yamls, you can actually build the entire system that we described before the first thing that you have to do. You'll have to define the self object store when you define the cipher object story. Ammo, you give it a couple of parameters, saying what is the size of the replica set and how replication is done, different pools that you have and what's your object gateway once you have that set up. You need to define the storage class storage sites.

B

Are the type of storage that you have. You can have multiple storage classes in your system, but each bucket that you're using has to belong to a specific storage class storage. Trust has something like uh reclaimed policy and other parameters that distinguish the different. The way the storage should be handled by those buckets belonging to the storage class.

B

Now that we have the object, store and the storage class, we can define an object bucket claim an object. Buckling is what eventually generate those buckets.

B

When you define the object bucket claim, you can give it a couple of parameters. Very important. One is the storage class that it references, um you can have give it a name or a name generator and an extra bit of information that we can give here, and this is related to what we're trying to achieve.

B

You can define as labels that notifications that we want on this bucket, so the only extra piece of information that is needed at the bucket level to enable block notifications is to reference notification via label.

B

Labels are very useful here, like in the side note just saying that whenever you want to let's say, look for all the buckets that use specific notifications, searches are becoming much much easier and also this is not very intrusive into the actual ammo of the object bucket claim. So now that we have the object bucket claim, we actually need to define the topics and notifications.

B

You probably know everything here is declarative, so you can define the bucket before the notifications. After it doesn't matter, everything eventually will fall into place where all the information will be available to our operator. So another step that we take here is define the topic. As we said before, the topic is the endpoint. Here we define the rabbit q server as the endpoint and we're giving a couple more details of information.

B

Some of them are related to the actual topic, like the exchange we're going to use or what is the acknowledgement level that we're expecting from from them? The others are more generic like whether we're using business notifications or we can pass opic data and so on and so forth. So now that we have the topic the last bit that ties everything together is the actual notification notification.

B

Here just declares the association between the topic that we just defined and what kind of filters and events are interesting for our notifications, the association with the bucket, doesn't exist in the area of the notification. Instead, this is just here in the label below as part of the object bucket claim. You can see here that we have to reference the topic and give some extra information about what kind of objects we're interested at. We can filter on the keys of the objects or we can filter on metadata. We can filter on tags.

B

There are many many options here and we also say what kind of events we're interested at.

B

Now, let's see what it actually means to send identification to the servers function.

B

The notification based on the configuration similar to what we described before is sent outside with a ceph cluster into, in this case the kafka k native bus.

B

Now we have a service function that is subscribed to the topic that we publish to kafka and is getting identification on the other side. This is an end-to-end solution. It is reliable because sef is going to store the notification using radius until it gets the act from kafka. Kafka is going to store identification until the identification is being read by the serverless function and the solution is also scalable, because kafka is scalable. Safe is scalable and the serverless function is going to spin more functions as more messages are coming on.

B

The topic that they're interested at so this is the whole solution which is reliable and scalable.

B

So if we have a solution which is both reliable and scalable, why isn't that enough? Why do we need something else? Well, this works very well. If the processing done by the service function is simple. Sometimes this is not the case. Those servers function can have processing that takes a very long time and may fail midway.

B

They can have processing that involves multiple steps that involve multiple functions, that invoke one another and each one of them can fail in the middle or just go stale or crash or not be responsive anymore. For those complex cases, we need something else.

B

We need the ability to know that something or some piece of work is never going to be processed in the end, even if it was fetched from the queue and we need a way to make sure that this piece of work is being processed later on by some other consumer that is able to process it, and we want to do that without, ideally introducing a whole set of complexity into the system as another message broker or another system that allows that, given those requirements for those cases, we would like to introduce a native message.

B

Q api, where the actual message queue is embedded inside ceph.

B

So, let's try and summarize the two different words in which we can operate the existing push mode. The messages are sent to an external message: broker the service function. Programming model is that it's based on the queue that stores the messages scaling is based on the utilization of the serverless function.

B

The reliability of the producer, which is ceph, is the fact that we're storing everything in radius until it's being act by the message broker and the reliability of the consumer is based on whatever reliability. The message bus gives them now in the new mode in the pull mode. The notifications are stored inside raiders itself, so not an external system.

B

Whatever reads, the messages is based on an outscaling trigger which sees what is the size of the queue approximate that and spins as much consumers as needed, based on the size of the queue the reliability of the producer is just radius and the consumer's reliability is is based on the fact that we have a mechanism where, if somebody doesn't act, the the end of their processing within some given timeout, then we're making those notifications available for another consumer to consume them.

B

We've chosen to model our message view based on the aws sqs api. The reason is that it fits very well with our requirements. First of all, it has an at least once guarantee, which means that whenever a consumer reads a notification from the queue the notification is becoming invisible, so other consumers can keep on consuming other notifications.

B

This guarantees the scalability of the solution. However, as we said, processing could be complex and therefore we have a visibility timeout. So after this timeout expires, if we didn't get an ack from the consumer telling us that the processing actually ended for this notification, notification is becoming visible again. So any other consumer can pick up on that, and this allows for reliability in case of client failures.

B

We also have a retention period, which means that stale or obsolete notifications are being deleted automatically from the queue and we're making sure that they're not consumed by any client.

B

Another reason for selecting the aws sqs api is because anyone which is already a user of ceph probably have some kind of a client sdk that support all the aws api.

A

B

We are aws compliant for s3 and other things so by using the cpi, we're making sure that the solution is simpler to integrate and doesn't require any special additional sdks or clients we're using raiders for this scalability and the availability of the solution and we're based on the aws sqs, but we're not a general purpose message queue. We implemented that only for the sake of notifications.

B

However, in the future we may expose more apis to make it a more general purpose message to solution now. I know that I've said we don't want to add more complexity to the solution. This is why we want the internal message or the native message, but sometimes you would like to use an external one. This could be for the reason that this external solution is already there or could be for other reason. For example, your serverless function, or your scalar in the case of cada, already want to support a certain type of message queue.

B

You can use kafka and this would be for a scalable cloud solution. You can use the wrapped mq for a more lightweight compact solution. Both of them are coming with. Operators that makes deployment and kubernetes much much easier. Kafka has stream g and webmq. Has the built-in kubernetes operator.

A

So currently there are two cncf service frameworks, canadian and acada kubernetes events driven architecture both supports scale down to zero through auto scaling.

A

The difference between the two is that the k-native is more natural, currently and categorize more service oriented. So here we have the two diagrams from the architecture picture.

A

On the canadian side, you see as the service function defined as a creative service, which will manage two different concepts called routes on the configuration the routes will split. The traffic into different revisions and revisions will go much different, endpoint ingress, endpoints implemented in the networking stack one canadian starts it's using ucl for its is defaults, not working stack, but it is expand beyond that and supports the logical plugin model that is currently you can use steel, groove solo and choreo, and probably even more.

A

The nice thing with the canadian is that the service functions is http triggers you only to focus on your business logic, how to decode those cloud events and handle the events. We do not have to worry about how the events is sent to the service functions or the different routing is happens. In the background, on the other hand, cata is very service provider oriented it deals with the service providers such as apache kafka or some other queuing mechanisms.

A

It will use metrics as the heuristics for scale up and scale down. It's also have this con its own implementation of scalar.

A

Thus we are just as a bridge between the current detection and service function invocation in the catalog service functions. It is more complicated.

A

You have to deal with the events and you have to decode any type of um receivers, but some certain cases you have to remove it once the event is already handled, but scatter is very lightweight.

A

Its personality requires a lot of dependencies and we will see in the next slide that is very suitable for our use case as a standalone, single purpose, service deployment.

A

We choose cata for our service framework in this solution because it has the following features. First of all, the service functions to process images have to be long-standing because.

B

A

They are our image upload to the rgw buckets. The service function have to stay alive, otherwise getting up and scaling down could cause overhead, and it is not quite efficient.

A

Long-Running service functions will become an issue for auto scanners. This could mess up with all the scalars um calculation and prediction, but this is another issue for canada, because cata does not require external water scanners. Second cut is very lightweight. It does not need any uh dependent networking components. This is very suitable for edge computing or single purpose applications.

A

Third, because camera is not http triggered the service functions do not need to have external endpoints.

A

External endpoints are not always available because of the next connectivity issues or security issues, and last cannot preemptively auto-scale surface functions to miss incoming requests, so this can make the real-time processing a possibility.

A

Now, let's look at anonymization service function, the service function. First, read the message from the message: queue and find the location of the such self-igw bucket.

A

It downloads the image from the bucket and detects, if that, if there are any personal and sensitive information such as faces and license plates and blurs the region of interest using rectangular box and then replacing the original image in the bucket.

A

As you know, there are many object: detection algorithms around such as har cascade classifier and different learning neural networks. All these algorithms have similar features. They both detect well-defined features as different scales and they match up to the predefined objects.

A

The half feature- cascade classifier, is very fast, but can only do one detection at a time it can only detect one class. If you have multiple class, then you have to have multiple classifiers, and this is obviously not quite convenient.

A

Dna models, on the other hand, can detect multiple classes, but they are more complicated and often require high performance hardwares like gpu and isaac, so without loss of a generality.

A

This solution uses the harm feature cascade classifier from opencv to detect faces because faces, have well-defined landscape and also use the pre-strained tensorflow model for license plates destruction because license phrases have different forms and not our forms are the same. So uh deep learning models is the better choice in this case and the last.

A

Let's look at the cluster orchestration system called microshift. So microchips is a lightweight implementation of our openshifts and kubernetes. It is optimized for large computing use case that has small factory devices, with resource constraints or for environments that only serve single purpose workloads as this one.

A

It provides a minimal yet customizable, openshift experience when it gets booted up. It's only startup very minimal openshift components, but you can add more and customize it with ease.

A

It is a single binary with both the data in the control plane, so you can deploy it either as rpm or as a container. It's runs on linux, mac, os or even windows. Currently, it supports both amd 64 and r64.

A

Thus, the community have already derived the binaries to deploy on risk, 5 and power systems.