Cloud Native Computing Foundation KnativeCon NA 2022, 28 Oct 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Knative Eventing Installed Then What Next? - Aleksander Slominski & Lionel Villard, IBM

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Knative Eventing Installed Then What Next? - Aleksander Slominski & Lionel Villard, IBM

You have successfully followed documentation and installed Knative Eventing. Now you have channels, subscriptions, sources, sinks, brokers, triggers. How to make sense out of it? We are going to show what are use cases and what are the main differences between different components of Knative Eventing such as channels and brokers. We are going to show what is happening when we run an example event-driven serverless application. We will use the hands-on approach by comparing self-hosted Apache Kafka (Apache Strimzi) and SaaS versions (IBM Event Streams and Red Hat OpenShift Streams for Apache Kafka). We are also going to show what to do when you run into problems, how to troubleshoot and debug Knative Eventing. After this talk you should have a better understanding of how Knative Eventing works and what problems it was designed to solve.

A

Good, so we do not plan it, but we are really following up from the two previous talks. So I want to encourage everybody who was like confused or want to learn more about K native to use Quick Start.

A

It's really works great and we saw kind of next steps after you use the quick start for Eventing, and hopefully we can answer some of those questions that were just mentioned that people were asking. Obviously we will not answer all of them, we'll just cover a few, but hopefully things like scaling and cada, and things like that will become more obvious foreign okay. So you can think about our talk.

A

We only have 20 minutes 25 minutes, so we will not give you really detailed knowledge, but we want to provide kind of like a map how to get to your destination. So after you installed quick start for Eventing, you have something running. What exactly that you have running and where do you want to take it from there so think about it? This is like you know, looking for the Nirvana of your production, Eventing Cloud native serverless, anything that you want to run workloads.

A

Obviously there are some problems and we want to show to you what kind of problems you may run into, and so also what we think are current best practices and where is it going in future next, one? Okay! So when we talk about uh K native and serverless, everybody kind of I think expect that it will be a scalable, easy to use and so on, but with kubernetes you have all those building blocks and then you are supposed to assemble what you want kind of the previous talks before also we're talking about it.

A

So how do you assemble those things together? So if you look on it on extremely high level, when you start with k native Eventing and k-native serving, you just need to know this. So that's a good news. That's like serverless Simplicity in one slide, just you know, put your service put some Source sync, some routing for your events, mission accomplished next, one: okay, and then you run into this like UCO, that actually, when you installed a k native Eventing, this is just for kinetic Eventing and you start looking on documentation.

A

Oh there are so many other things that are out there. How do you put them together and which one are important to what you are doing or that destination I was talking at the beginning? Okay, next one, so you can start by thinking about what you get with the quick start. So with the quick start, you get a running system, it is providing you few event, sources and very simple broker and channel.

A

So you can essentially do a little bit of like hello world, uh send event print something and apply tutorial commands, and you are happy next one now. What we are going today to show to you is that this is very good, quick start, but you should start considering limitations of that quick start and in particular you should think about it.

A

What is that Eventing system going to do so? Quick start actually because of the Simplicity, it's not using any messaging Eventing provider. So all your events that you are sending for hello world is very good. They are fast, they are in memory, but you definitely want to put some actual real messaging system and in this demo we will show today it is using Kafka everything supports other systems.

A

We do not have time to actually show all of them, but this is the first concern you should have when you are doing Eventing is that we all started with quick start. You want actually something that is working for you and then I put here at the bottom. As soon as you do it. You start thinking about serverless, which I mentioned at the beginning. Yes simple, but it also should be Auto, scaling and so on. So here is the cada HPA and some other things, and then you kind of start building on top of it.

A

So you would like to have not only like Kafka Source but other sources in your system. We will show in this demo what is a kind of two types of urban sources you need to know about, and GitHub sources the example of it and then. Finally, um if you're running the system, as we know always something goes wrong, so how you should be thinking about how to configure it for high availability, how to deal with things that are not working like set your default retries.

A

What to happen, if you actually cannot send Evans to your consumers and so on? Okay, so we are almost there. We will be showing you very soon uh that destination, but we did not talk about some other things uh that you probably need to know.

A

We will just mention them very high level: go go to the next one, so you can see that there are kind of logical groups you can see in k-native Eventing, starting with that initial picture, where you had source and sync, but you have all those also channels and Brokers and we'll talk about more why there are two different kind of abstractions that seems to overlap in functionality.

A

Then you have those underlying very important. Those.

B

A

Systems at the bottom, what I was saying that you really need to have them so kinetic Eventing does not come with those systems out of the box. You probably already are using Kafka or other Eventing systems and Canadian venting will integrate with them, and then you have all those kind of in the middle boxes that are providing the very important services like that question that was before, like what is scada. We will go into that.

A

Okay, so the way to address it is not that we have all those boxes but think about how do they come together or how do they connect? This is very much like a kubernetes approach. You have the building blocks, but you are supposed to build build things by connecting them together.

A

Okay, so the very simple pattern you have Source, you send it to a Channel or broker and you're receiving sync. The layer in the middle is doing routing then, as soon as you start doing. Something like that. You realize yes, but who is doing Auto scaling for my uh Eventing systems so for the type of urban that are using HTTP, we can use built-in K native serving it provides the scaling for HTTP workloads so like GitHub or pixels, or things like that. You can use out of the box by thinking about it.

A

They provide the auto scaling for you now. uh There is another type of everything that is not HTTP, everything that is maybe creating persistent connections to underlying messaging providers.

A

That's a problem so what it means that we do do not see exactly how those providers are sending evidence and when to do scaling, you need additional component and that's where the cada or something like that comes into picture. Essentially, it is monitoring the underlying Eventing provider and then doing scaling decisions for you, okay and then. Finally, this is this is just to give you one more mental model that pretty much everything in Eventing falls into that picture here.

A

So you have something that is receiving things over http, storing them into persistent Eventing system and then something else that is taking it out of the Eventing system and then is sending them to whoever wants to consume. That is like what that's what is broker. That is what is channel sources just one part of the picture and so on, and then we will talk a little bit in the demo.

A

You really want to know about that letter, sync or kind of how to provide reliability for your system and then, when you are using the data, sync works the same way for Brokers and channels: okay, okay, so almost there. So one very important thing is the high availability. Again, we will not have enough time to talk about it right now. However, there is another talk at 4 pm which talks about high availability.

A

We worked on that so, but the high level idea is this: the only way to provide that high availability is to run multiple copies of your underlying Eventing infrastructure, so you create the different availability Zone with kubernetes and then run those pieces of Eventing in those different availability zones, and we provide all that is necessary, for example, for Kafka to automatically recover. If one availability zone go down very good okay, so um we will go fast. This is what you get when you install Quick Start.

A

This is just a listing. You can look on the later on slides. This is just what is actually K native Eventing. So if somebody asks you what is K native Eventing well, that's it I mean this literally. What is running in your cluster when you install quick, start 4K native Eventing, some controllers that are doing some things when you tell them to do next one- and this is what you get when you actually install something more here.

A

It is running a cada and running half castles and providing some necessary things like you can see that Kafka source is running, multiple dispatchers and they actually can be configured to run in multiple availability zones. And then there is something hiding on the in the other nations, which is scada. So even though you're really running K native I, don't think you added and installed other things, and we do not show here. Where is the Kafka running?

A

Okay and then this is just the description. What are those pieces and I think we are about to get to the the biggest question that we get and we find that people get very confused about K native Eventing, so why there are channels and Brokers or what is the difference between those two? So you can think about it that they do on High level similar functionality. But it is all about those little details about how you look on them and how you are processing your events. So like is it Q?

A

Is it pumps up system and so on? So let's go to the next one? Okay, so here's the little bit like I will not go into details. I think we want to run the demo so and again, if you have any questions, just go to our slack in K native, ask those questions, and- or if you are here, you know, ask us or anybody else who is around, but you can think that there are some questions.

A

You will be asking yourself when you are looking on channels and Brokers and how you want to use them, and we also look a little bit on in dimmer on that and then finally, we are showing demo. There are steps in the demo or questions that the demo tries to answer again. We cannot cover all the questions and things in 10 minutes, but you can look that we will kind of show the best practices and patterns that we think are common.

A

If you are trying to run creative Eventing in production, some things will not be covered, but that just ask us for future. So, okay, so super high level in the demo. You take all those patterns. We were showing you kind of push them together. What happens next? One, yes, so we will. We will see that you know the things again map to those building blocks we were showing before and then you have to install them to support those underlying patterns next one and for demo purposes we actually simplified it.

A

So we will be showing like that letter uh thing. We will be showing Apache Kafka, GitHub stores and so on, slack Channel, and then we will be showing some uh very few building blocks together. Okay, so let's do the demo.

C

So I would do a little bit more than just being the kicker.

C

I'm gonna have some Logistics here: the the mic and type on the keyboard. Let's see how it works. Okay,.

A

I can provide you.

C

Yeah, it's working, okay, that's a bit bigger than what I had before all right, so I'm going to show basically a couple of very simple patterns: Eventing patterns, something that should be fairly straightforward for most of you I hope.

C

So it's we're going to I'm going to show this through, like a very simple application that we developed just for this event here um and so starting with I'm, going to present this by using the first pattern, which is the Direct Delivery, which actually doesn't use Eventing, but that's something or part of Eventing and just the source and and the things, uh but without any uh persistent you know layer into it. So the Direct Delivery pattern looks like this.

C

uh You have for in this application, so you have as a source a GitHub Source connected to your GitHub repository then send event via the GitHub adapter to a stack adapter, which is in this case my application or our application. And then this application is forwarding event to uh to, after some transformation to uh to a slack channel right, so to show that it's working, hopefully right, there's a couple of things that needs to happen. I know order for this work, so I ran this locally on my machines.

C

I can't I use Quick Start a great quick start with additional things. So if I create a new issue on top here is where I can create new issues. That's the the Gita, repo and I. Think I have a script here, so that's typing on the keyboard and the mic.

C

I don't know one second.

C

Foreign, something.

C

So I submit this issue.

C

And, uh on the right hand, side on top, we see that there is like a new slack measure that has been published with uh you know. The message help me right, thank you, Alec and so that's very simple, extremely simple. uh It's very easy to use lightweight very few components in the in the picture. The prime is. uh We can see if you're familiar with this, why there is no guarantee of ordering right. The second thing is, there is no persistence and, as Alex said this something always can go wrong right.

C

So, if slack is having an outage for a long time, then you're going to do the event right or token expire, or you know the strike. You suddenly receive a lot of messages from GitHub. Then there's too many request for the application to handle and return the 429 boom that doesn't work anymore. We lose even uh so. What do we do right? First, let's solve the ordering issue.

C

Ordering issue can be solved by using a channel and an order Channel, and in this case we are going to use Kafka as a backend, because Kafka preserve has the you know preserved order. If you use like a topic, was one partition, you know, then you can have this fifo kind of uh type of delivery.

C

um All right, so how does how does that work? Okay, so in my uh small slack, adapter here application, I'm building some um uh delay right or like errors. uh In this case, it's going to be.

B

C

So if I do time of time sensitive this one, let's say uh if I paste this message as a comment in the comments and then up so I will post this command. This command is going to trigger delay in the application in the slack application of three seconds.

C

All right and then I will paste this one. So in the, um if you look here, I'm still showing the uh on the on the right hand, side, so I still show the thread of the Direct Delivery right. So it's still running in the background so, and we see that in this case the messages arrived in out of order right, no worries before sorry of the delay.

C

So when you're having a conversation, it's not great right, so I have another slack Channel receiving the event coming from this other pattern, right from the the Kafka Channel, and in this case, if I open the thread, we see that the messages are in order right once again.

C

C

All right, thank you, um so how much time do I have still do you know plenty of time all right? Wonderful, all right! So it's very good. So in this case, just by doing this, uh we solved the ordering issue that we were having um uh also because Kafka has a very long queue. It's not like an in-memory queue and we also solve the too many requests issue that we were having one of the common is like well now we need to bring into the picture a lot more of Eventing right.

C

It's uh uh Kafka Channel, like also a real Kafka right, uh Apache Kafka cluster into the picture, um and another coin is like well, it doesn't solve all the issues. We still have issues that can be lost so how we solve that right.

C

So the third pattern here, it's more like a best practice, also kind of thing right is to add some retry in the the first thing is to do is to add retries in the in the in the delivery of event for the in the Kafka channel right. So in this case, um uh I added like a retry of five, uh so try five time uh using the linear, backup policy and wait one second before retrying right now, I need my two hands.

C

uh So here we see if I type, this special command that will trigger the slack adapter with trigger like an error which is not permanent.

C

um It's not a permanent error, so we trigger like a 10 minutes. Thank you. It will fail like for three times and then after a while, the message will be posted. So I received the the message here right. This is Ken channel. So that's the Channel with the delivery guarantee right. I received the message: if I go back to the K and direct slack Channel I open the thread and I see that nothing is happening here right because I don't again receive the the message, because the GitHub adapter does not retry um all right.

C

C

So that's good! We solved these less errors. Right so I mean we'll cover a bit more uh in the in the type of error that we can handle by using uh retries.

C

uh It's good for like more transient errors like slag, is having like a blimp right, it's down for like 10 seconds, so something like this, but that doesn't work really well with the the poison pill kind of messages where you know it's sending a a message to slack adapter and but uh maybe the slack adapter is trying to check for your picture on GitHub and you delete your account and it doesn't work, and it fails something like this something a bit uh stupid in this case.

C

So what you want to do, and also it's important to not also block the uh the the delivery of event so many times you want to have like a short retry period uh and then have the event sent to a delivery uh dead letter sink right to keep the flow going, um so so for the for the channel. So then, you can also configure the letter sync that we call a thing because it can be anything it doesn't have to be. um You know it can be really anything right to think.

C

So let me show you this in the case how it works um so I have this comment special comment that will trigger a special I mean like a permanent error in the slack adapter I think um so, which means that the event is never going to be delivered in this in this channel, but instead at some point it should appear in this in this channel. If we need to wait like I, think about three seconds, I think you know something like this, um you will receive it.

C

I, don't know yeah one minutes ago, so we saw the the message appearing in the uh in the slack KN DLS channel, uh so in production. Obviously, you don't want to send your event to a slack channel right, so you want to send the so. The DLS is more like it's something that is uh highly available and and persistent and uh uh and uh to make sure that you know you don't lose it right.

C

So if you do all these, the chances that you're going to lose events are pretty slim in that case and always going to be somewhere arriving somewhere all right. Finally, uh the last pattern- let's say as I like say as Alex said uh many times: we we get this very often. The question is: when do you pick broker? This is a channel and we see that one is a big difference between Channel and broker. One thing is the ordering right uh you don't have any guarantee of ordering in when you use broker.

C

For some I mean there is like a nesterx in this one, but that's okay, um so and also you want to use a broker when you have like multiple sources, and you want to do some filtering on the on the content of of these sources. So in this very small demo here so I extended the the this GitHub to select application with multiple a little bit more sources, sending the event to a broker by by Kafka.

C

um And then having planned this event to some sort of aggregator, where in this case, I don't really care about ordering it's an aggregation, so I can forward this to I can use a broker in this case and then send the the event to Ken broker um so I have also, maybe so I have like this special channel for that select Channel, um so I can create, like let's say a new, and we see we saw already I think we didn't see that maybe this one is broken. We see if I create a new issue.

C

And I call it like hello, very simple, so that's uh sending this um I'm here right, I send the M on my demo. 22 GitHub send an event to this GitHub adapter forwarding to broker, which then do some sort of filtering and uninteresting in event type issues and coming from the demo 22 repo, oh since the audio works, but the demos doesn't seem to be working in this case.

C

That's all right, um all right, so it doesn't make sense to go and and do the filtering on the other side, because it's not working uh and then that's.

C

um So if you go to the pro and cons, uh it's like uh as I said right. So in this case uh it's it's simpler in some sense that you only need like one broker, usually inside your domain right versus like having multiple channels.

C

uh You have like a building support for filtering and uh but, as I said before also um there is no like any ordering guarantees and I think we can go back to the presentation right. So one.

A

Thing I think we should mention.

A

One thing we should mention um that all that demo code is available on GitHub, so you should be able to reproduce that demo and even the trust part that did not work. You should be able to make it work and, if not come back to us with questions next tonight on. That, essentially, is what we want you to get out of. This short talk that K native is all those building blocks, but it should not be like too much or too many or feeling overwhelmed with complexity. They were all put together to achieve some goals.

A

We hope now. It is a bit easier to understand how to put those blocks together. We did not go through all of them, but you find all of it in documentation. Also at the stock at 4 pm, we will be talking little more about high availability, Auto scaling and cada, so you can see some other blocks how they come together and then this is work in progress.

A

So, as other speakers were saying, join us in k-native slack, but also if something is not working, bring those issues and then maybe help us to fix it or make it better. I think that that's about it I think we have two minutes for questions Maybe. So thank you.

A

D

So that's great folks, thank you very much for that. But I was wondering. Is there a future where we are looking at simplifying the choices for the folks who are implementing this? Because this is great amazing but I see I, see that it could be complex for certain use cases.

A

So the question is about complexity, or why, fundamentally, we have so many pieces, they are needed, but also we are going to simplify them. So one piece that I was showing actually here will no longer be there soon, which is some like bridge. That is doing Auto scaling and we want also to make it so out of the box. You could get all of it running and you really do not need to put or figure out how to connect all those building blocks together.

A

Then, if you want you can you know put things if you want.

E

E

So I have a question: can all my needs be served by broker and not use Channel at all.

A

In Terror, yes, it depends what exactly you want to do. I think there are some use cases where that ordering and tight control over horses and how events are protesting. Queue is very, very useful, but.

C

Yes, short answer is yes, yeah and also there is for compliance reason. Sometimes you want to really uh isolate some of the events from other events right. You don't want to have like a big bucket, but you want to say okay, this one for compliance reason has to be in this uh in this, you know together not mixed with other events right this kind of things and ordering we talked about it. Yeah.

E

Thank you very.

B

A

All right there's one more.

E

Yeah yeah, that was a great presentation. Thank you guys, I was just gonna ask. Does it support other git providers besides GitHub, or is it mainly focus on get up for now.

A

So we are, the connective, is essentially API and then there are different implementations for the API, so the different providers are supported. We have many Eventing sources and we also hope to have many more in future, so that hopefully it will be very easy for people to add and maintain those.

B

There are about 40 of those pre-canned sources available on the k-native documentation, site and we'll be doing a contribfest on Wednesday to polish. Some of those up.

E

B

Kind of following up on that is there going to be like a a provider interface. That's going to be standardized eventually.

A

You're asking how to add new providers.

D

B

Yeah is there plans for a standard interface? We.

A

We see it ourselves a little bit, but because we are like collaborating Kafka that would be very good to you know, reuse parts of implementations between different providers and that I think over time, when we have more, that would be very natural to have those well-tested building blocks like I was showing that dispatcher and receiver pattern.

A

That provides like 80 percent of your implementation for you, when you want to implement broker or Channel and or Source in k-native Eventing.

E

Thank you, Leonel and Alexander.

B

A

C