Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2021, 14 May 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: CloudEvents - Looking Beyond Event Delivery - Remi Cattiau, Nuxeo

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

CloudEvents - Looking Beyond Event Delivery - Remi Cattiau, Nuxeo

While CloudEvents helps in the delivery of events from a producer to a destination, when considering the entire lifecycle of an event there's a lot more to consider. For example, how does someone discover which events can be generated and by which producer? What transport mechanisms for event delivery are available? How does someone subscribe to receive events? Today there's no consistent mechanism to answer these questions. This session will explore how the CloudEvents project is working to solve these issues by providing a quick overview of the new specifications being developed - and, of course, give an update on the CloudEvents specification itself.

A

Hi welcome everybody to cloud events looking beyond event delivery, I'm remi caccio and I work for nixo, where I'm responsible for the security and compliance program like every security person, I like to collect events from all systems, github for commits single sign-ons vpns and all those events are always from different formats and different methods to retrieve them.

A

So, while working on this project, I was searching for a group who normalized events- and I found one here in cncf with the cloud events groups and even better- they were all nice and welcoming.

A

So what is cloud events? Let's have a closer look, as you will find in our repository. There is a nice quote in the redmi events are everywhere. However, even producers tend to describe events differently, so cloud events define a common set of metadata for events and where to find those metadata inside the messages.

A

It also defines different type of transport like http or kafka. So you can basically compare cloud events to high p, while the tcp and udp is for us, the transport like http or kafka.

A

In this session we will see cloud events and how it looks in detail what the community provides to you and a little bit of history on the project. Then we will focus on interoperability with the discovery, the subscription api and the schema registry.

A

So, let's see what does it looks like here on the left. We have the htcp structured event with everything in blue is basically the context and the metadata defined by cloud events.

A

So we see the spec version as 1.0 as we don't put the patch version in this attribute the type of events the source, the id which are the the one in bold, are the four required attributes and then a data content, type and finally, inside data you'll find the correct events sent by the system with all its specific attributes.

A

As we manage different transports you, we can also have a look on http, binaries transport, where you will see that all the metadata are basically transferred inside the header of the http request and the payload contains only the content of the event.

A

So, let's have a look on the metadata. Four attributes are required and four attributes are optional in the four required attributes we have the id, which is a unique id per source. The source is a reference to the system. It can have one or several producers behind a source, but if there is several producer as the source plus id must be unique, that means they need to have a way to ensure the id is unique between them.

A

The spark version without the patch version, the type of the event are the four required attributes. The optional attributes are the data content type, which is basically the mind type of the data.

A

The data schema, which define the structure of the content, the subject where you will find whatever you want to put as the even producer and the time, let's have a look on what it can look like if it was a github event.

A

So in our case the spec version will be 1.0. The type if we create a branch will be com.github.create.branch.

A

The source will be the url of our repository. In our case cloud event, spec the id is a unique id generated by github.

A

In this case, it's just a normal uuid and the data content type will be application json. The subject is the commit reference and finally, in data you will find all the attributes that you will have found in any github.

A

Events, so what's in the box, what is provided to you by the community? The sdks include all serializer initializer of the cloud event and at least the http transport.

A

There is several languages: c-sharp go java, javascript, php, python, ruby rust, and we also manage several transports like http, mqp, mqtt kafka, nuts, and you will find all the definitions like avro definitions and others. Inside our repository a little bit of history, the project started in february 2018 and the first version in april, the v 0.1 in may. We got accepted as a cncf sandbox project in december. The v02 was released, followed in 2019 by the v03 in june and, finally, the 1.0 in october 2019.

A

Since then, we worked on the interoperability specifications, including discovery subscription and schema registry.

A

So the interoperability is basically how to consume the events. We already define what the event looks like and what the contents, but we still need to discover the events that are emitted by our systems.

A

We need a way to be able to subscribe to those events and then, ideally to have one place where to publish the schema that defines our events. So what is the discovery? Api?

A

The discovery api allows you to understand, who produced the event of interest and what type of events they produce also what different subscription they do and all and how you will be able to subscribe to those events.

A

So you can dynamically query the producer and the source to understand the events emitted. So why? Because there was a lack of standardization on that there is like webhooks and other, but each provider is basically implementing its own methodology, so you always find yourself digging into their documentation and for each new system you have to dig into the documentation the discovery api of cloud events aim to solve this by having one way of questioning the different events and get the result and how to subscribe to them.

A

So no more documentation browsing where you spend hours and hours, because each system are clearly different.

A

The deliverable as the specification group are the discovery api specification, which is the open api and the http json mapping to understand how to subscribe.

A

As we discussed in the discovery api, the logical following is the subscription api. Once you discover which events you want to subscribe, you want, you need to be able to subscribe to it.

A

So the subscription api is basically an open api definition on how to manage subscription to events in this subscription object. As we see on the right side, you can define the different types that you want to subscribe to, and you can also filter through different dialects that are defining the norms.

A

Then the sink is basically where you want the events to be pushed and with which protocol and the settings that goes along with it. It basically enables the automation of the subscription so combine with the discovery api. You can completely automate the discovery and subscription to an event and have more automated systems.

A

Finally, the schema registry, so, as we see here in a sample of an event, there is a data schema. The data schema basically defines the structure of what's inside the data, but that means we will probably need to be able to publish schemas in different formats and different versions, so the schema registry, basically aimed to define how you consume and publish schema publicly or privately through a schema registry.

A

Finally, let's go back to the interoperability with what we saw. We can definitely have different type of topologies, one pretty simple. Basically, the consumer is sitting next to the producer and the producer expose both discovery and subscription api and then the consumer just use those apis and retrieve the cloud events through the normal network.

A

Then there is the aggregator one, so we can have a system that will aggregate different cloud events from different producer and will expose the same discovery, api and subscription api. So, as a consumer, you can just go through your aggregator and retrieve any events known by this aggregator.

A

That's a simple use case of an aggregator. You can still sit on all the same network. The more complex one is the aggregator chain. So in that case we can have a aggregator two, who basically know two producers and expose a subscription and a discovery. Api. The aggregator one can just use the aggregator too, and even do some change in the transport going from http to kafka and also expose one subscription and one discovery api.

A

So the consumer can talk to the aggregator 1 and see everything that is below from the aggregator one, and it can be even nicer if we consider that the aggregator 2 is in fact in company b and the producer 1 and 2 as well, while in the company a we have one aggregator that link automatically all the events from the company b.

A

So you can expose to your internal consumer of company a some events from the company b without them even having to know about how the company b is managed or even the connection between aggregator 1 and aggregator 2..

A

So this is a little bit more complex, but I think it's a good proof of the power behind cloud events and normalization as it has a high potential to simplify communication between systems and companies with a common way of subscribing to events.

A

So this is it for our global overview of what is cloud events. We meet every first day at 6, 00 pm european time, 12 pm new york time and 9 a.m. California, time I'm in california so on the 9am is important. For me, the you can find more information on our website and the spec repository is uh with the link below don't hesitate at all to join. Everybody is really nice and welcoming so I've been with the group for a year and I really enjoy all the talks and the knowledge that this whole group share.

A

So thank you very much for having me listening until the end and don't hesitate to ask any question. Thank you.