Knative Demos, 20 Apr 2020

Previous Meeting

⏯

youtube image

►

From YouTube: Knative eventing blockchain demo

Description

This demo, produced by software engineer @ IBM Johanna Saladas, walks you through how to use Knative eventing feature.

It highlights the following:
- What you can do with the basic Eventing components; broker and trigger with a simple producer and consumer;
- Streaming events in real time;
- In-stream transformation;
- Push-based front-ends

Check out Jo's repo to reproduce the demo in your local environment: https://github.com/josiemundi/knative-eventing-blockchain-demo

A

Hi-Yah- and this is a re-recording of the demo that I gave in the first K Native community Meetup, then we had some screen sharing issues so I'm just recording a nice fresh one for everyone that they can follow along without being three slides behind I'm gonna show a really simple, K native eventing demo that you can follow along with just get to know how can a native inventing works and try out some reasonably cool stuff?

A

It's basically gonna be a simple Bitcoin event source that sends messages to okay native broker, and then we've got an event display service that consumers. These events and they're displayed in real time to a UI they're. All update I should highlight the basic Canadian venting component, so we've got a brokers and triggers a couple of producers, and consumers will be streaming events in real time doing a little bit of in-stream transformation and we'll have some push-based front-end.

A

So I used the blockchain, the Bitcoin data, because it's a real rich data stream and it's free I think Twitter is a good streaming data source as well, but it's quite overused and also I. Think for companies like finance or insurance, the Bitcoin message is or something like fin hub where you've got. Financial data is probably a bit more relevant.

A

You've got a really good number of fields that you can play with in the data. So in each message, there's a range of different fields: I'll, try and add an image actually to the repo, but just some ideas like from the data science side, you could try predicting transaction fees, you could classify transaction on the fly and send alerts so, for example, later in part, 2 of this demo will classify transactions based on the value size and you could send I can an alert like a push based alert to say: hey.

A

This transaction is really big. You need to take a look there. You can also do some really cool visualization stuff with this data, so I've per link below I think it's actually for a product which I'm not endorsing, but the visualization is really cool, which you could also try.

A

So, to run this demo, you need a few things installed, which you can look at the K native Docs to get instructions for how to do that.

A

All of the code for this demo can be found in github at the following link. So the top one is the readme and files for deploying this. This demo and the other repo is the just the event source for the Bitcoin messages, and we've also got a blog where I've got a couple of K native tutorials, there's stuff, mainly on there to do with event-driven, architectures and big data.

A

So um one of the things that I really wanted to look at was. Why would someone even do this I think you know there are other ways to do. What I'm about to show you and I think I wanted to highlight the following advantages. One is that K Native is good if you want to do push based messaging, we've got a true decoupling of producers and consumers, which again I've written more in depth about on my blog per take you on that point.

A

We can apply a business logic well data's in motion, and this is good. This is really handy for the day's streams we're working with real time event streams, especially for things like fraud, detection, where we need to make really fast decisions as the transactions happening and also, if things really good for things like ml ops and when you get into architectures, like the rendezvous architecture, which I'd like to work to build out a bit further using Ken a to preventing time which I haven't done yet.

A

So this is what we're going to deploy for part one. Basically, we've got a WebSocket, which is the blockchain info web socket and it's streaming messages our event source is reading the messages, turning them into new cloud events and then sending them on to a broker. And then we've got an event display which you subscribe to these events for your our trigger and it's updating a UI front-end in real time.

A

So I'm going to switch over to terminal window so that I can run through the first part with you so in the repo, which is what I've got open at the moment. There's a few folders in there the sauce, blockchain event display UI and classifier transaction size are the folders that you need. If you want to build the images for those services yourself, if you just want to run the demo, you can just go into the Yama folder and in there there's a number of deployments that you need to get this running.

A

The first one we're gonna do is we're going to apply this name space channel that creates a namespace called k-, eventing WebSocket sauce, and it enables K native eventing in that namespace. So we create a broker at that point.

A

If you want to change the namespace name, that's fine you'll just need to change with the other yeah more files because they have their namespace house hard-coded in are then gonna apply the deployment file. So this is our sauce. This is the one that takes the messages from the WebSocket and sinks them into the broker and.

A

Then we're going to apply our trigger and this basically subscribes anything that comes from the blockchain sauce and it's gonna say: please subscribe our UI front end to it, so our event display service and then we're going to deploy the event display service, which is in zero, five zero and zero six zero, so kubernetes deployment. So we've does two files associated with this one.

A

And now, if we head over to.

A

Localhost 3 1 2 3 4, which is the port that this is running on. We've got our transaction and you can see the transactions are coming through in real time, and this front end is updating.

A

So that's part one I'm going to just delete the last deployment because I'm running on the desktop and it doesn't like if I try to run too much traffic, but there is no real reason why you couldn't run this one together, just so that this does not crash out I'm gonna just leave these ones.

A

Ok, so can I go back to my presentation now so I've put some useful commands that you might want to use while you're doing this, but they're all in the readme as well.

A

Like I said earlier, you could swap out using the Bitcoin source to be a different website. It's so you could use fin hub, as I mentioned before, I'd like to add an example first swapping out in the future. So hopefully that will come soon before I move on to part two I just wanted to run through what the WebSocket source application is doing so for the Bitcoin source.

A

So first so first we import the eventing and WebSocket client. So that's the cloud events and gorilla WebSocket packages.

A

We define the WebSocket source URL. So that's our blockchain info pay address that we use we're gonna read in the sink, so the sink in this case, so the sink is. Where do we want these cloud events to go, and in this case we want them to go to the broker and we push this in from the llamo that we deploy so in the llamo. We just specify our sink address and the way we get the broker. Url is why doing this cube? Ctl get broker command and you'll see.

A

It will return you URL, and that is the the URL of your broker that you need to link to.

A

Next. In the code we passed the config Mpho variables. We set up a cloud event sink and we set up a cloud event: client using HTTP transport and sync.

A

Then we connect to the specified WebSocket URL, so that's our source that we specified at the top and then we need to send this opt unconfirmed, sub message to the WebSocket to initiate the transaction string and then our messages will stop coming through. Then we read each transaction message from the WebSocket and we're going to create a new cloud event for each message that we get and when we create this new cloud event we're going to set some of the field so we're going to set the type to WebSocket event.

A

We're gonna set the source to be the source that we specified at the top, which was our blockchain dot info WebSocket. And the message is the message that we pass.

A

Okay, so for part, two um I wanted to build out the first bit there just to show something a bit more like what you might do for a real business scenario. So we've got the messages going to our event, display, which is the top part, and then I wanted to address a really simple classifier. That might seem like something that maybe you might do as part of a data science or analytics workflow.

A

So we've got a really simple classification service, which is our classifier, which is going to be subscribed to the blockchain events coming through it receives them, and then it classifies them as small or large. It's really simple for now, but it's just to illustrate the concept.

A

It then creates a reply, then, with the classified size which it sends back out into the que native event in ecosystem and then another service, which is our test display service, subscribes to our classification reply message and it's just going to log it so just log it out so I'm gonna go back to terminal.

A

And then in the so in the llamo, folder you'll see there's another folder called classification, so we're gonna go there and in here there's some more some more yeah milles. So the first one that we're gonna apply is the trigger, and this is the first trigger. So this is the trigger that says anything that comes from blockchain WebSocket events is going to go to my new classifier service, which is not running yet and then we're gonna apply the classifier service.

A

And then I'm gonna apply me the second trigger, which is going to say.

A

That's going to be our test. Displays subscribe to our classified messages, so the ones that are sent back out.

A

And then, if we go back up just one level, we're gonna just apply this zero three zero.

A

Test display service. So now we look at what we've got so.

A

We've got classifier service, our test display service and then we've got these triggers set up and the subscriber your eyes are what we want them to be. So now, if we look at the logs for this service for the test display service, which is only subscribing to our size, classified messages, we should just get those coming through.

A

We can see here. The messages are coming through and for each message that it receives is classifying it as large or small.

A

Okay, so that's it and obviously you can make it much more complicated. So if you look in the in the folder, for if you look in the classifier transaction size, folder you'll see the code for the classifier and you'll see that it's really not very complicated, but you could make it as complicated as you want.

A

So I was part of that process. There's a few things that I needed to do so I needed to marshal the message that comes through and all of that is in the was in the green person. You can see how, when, if that was done, and the struct to marshal it is all there.

A

So I said that in the future near future, I'd like to look at more complex ml patterns- and one of them in particular is the rendezvous architecture which I haven't I haven't built this yet. But this is like what I'd like to do next I'd also like to add some other models. So maybe one of these models might be like my go classifier, but I also want to have some that are made me written in Python.

A

There repose still like a work in progress but I hope to add some more stuff very soon.

A

So event is really cool, but it can be quite tricky to get your head around and sometimes it's hard to find real examples of what you want to do. So these are sources of great information that I found really helpful.

A

I think particularly this medium article I went back to you many times, just try and figure out how to do some of this stuff, so that was really good, but also you know all of these and many more that I, probably can't even remember I looked at I mentioned this on the call, but I'll just mention it again that the areas that I think could be added to by the community are I thought the cloud became cloud events package dots were well there, weren't really any and there weren't many use cases.

A

It would be nice to see some more use cases for that. I think also just having some realistic business use cases and patterns, 4k native eventing that speak to maybe a higher level of the business would be really interesting.

A

Thanks for listening, I welcome, any feedback or suggestions are really new to all of this event-driven and cloud native stuff, so I'm always open to ideas and there's definitely room for improvement, so please go play with the demo. Let me know what you think and let me know if you do something really cool with it and that's all. Thank you.