Knative Demos, 25 Feb 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Knative Demo: Transforming CloudEvents with Bumblebee

Description

On February 24, 2021 the Knative community hosted a meetup featuring a demo, "Transforming CloudEvents with Bumblebee" presented by Sebastien Goasguen, Co-founder of TriggerMesh, who introduced Bumblebee which allows people to define cloudevents transformation declaratively. Combined with knative primitives it helps build event flows that integrate separate services.

A

Okay, so sebastian co-founder of trigger mesh uh so bumblebee the transformer all right. So that's how we called the the transformation engine. uh The author actually of bumblebee is a developer in our in our team. Who's been with trigonometry since the beginning he's contributed to k native timor is out in kyrgyzstan, uh he's the main author, but because of uh you know time zone differences. He cannot present so I'll I'll present bumblebee bumblebee is open source. You can get it uh trigger, slash bumblebee.

A

uh Potentially we could give it to the sandbox uh written in. Go uh you deploy it with ko. It's pretty simple, uh simple use case: again, you have a source cloud event. Type fool goes to a broker. uh You wanna transform that cloud event. So what you do is that you end up with two triggers: there is a trigger that gets the type foo you send it to a transformation which is an addressable declarative. Api defines how you're going to modify the the attributes and the payload gets back to the broker.

A

Second trigger uses the new type to actually send to the uh the end target um things that we faced. I mean I'm sure. Maybe some of you faced it quite a lot. Is that if you, if you're, not careful with your types, you know you can get tons of event loops. So you know definitely you need to be extremely careful.

A

The object looks like this kind transformation, and we have you know a first in the spec. We have a first section which is context, and that has to deal with the cloud event. uh You know I want to say attribute so id source type things like this.

A

You can actually store the you know the context of an incoming cloud event in variables, but you can also add right or you know, modify so if you want to modify a cloud event type uh you you do context operation, add and then the key will be type and doing this you switch the the type of the cloud event uh on the fly.

A

If you want to modify payload, then you define uh in the spec. There is a section data and then you can add a key. So you add a key foo bar in this case. You set a value. You can shift the key from full to wolf or you can delete a key right. So very you know basic transformation, all declarative.

A

uh You know quite quite useful, um so demo uh demo so demo a little bit of advertising for the google cloud uh cloud. Shell. Can you see this? Yes? Okay? So it's uh it's a gke cluster. I installed uh kennedy from scratch. uh You know latest release and then I deployed I deployed bumblebee.

A

With uh ko, so if you look at the tree of bumblebee, you know it's pretty straightforward and when you deploy it with ko, then you end up with uh you know: basic transform, controller running and transformation, namespace right and you're you're good to go. So what I did is I I launched uh sockeye the new version.

A

Thank you, scott. So.

A

ah And I lost network. No, can you still hear me.

B

We can hear you.

A

Okay, so what's going on with my sockeye.

C

D

A

Right, okay, so that's the demogod.

A

Xip server cannot be found.

E

uh Try http not https.

F

The error is named not resolved, which suggests that xip.io has decided to take a little break.

A

F

You can try switching to nip.io. I've heard that maybe that's slightly more reliable, but it seems like these things are. um Apparently you know you get what you pay for slightly less.

A

Yeah yeah, okay, so fine, let me show you the uh so we're gonna go in the in the config. You know: basics basic samples. You you in the bumblebee repo you'll, find the basic sample with the the diagram that I showed. What I was going to show. You is a basic. uh You know a pink source, all right, pink source that has you know first uh and last name, alice smith right.

A

So we can, uh you know we can apply the we can apply the source right and maybe, while we do this, so we can apply the source. Of course we have. uh You know a broker right, that's running uh perfect and then, if we look at uh the triggers, so I have two triggers uh the basic ping goes to uh the transformation. So you see how you know the transformation can be set here as a you know, subscriber here in the in the trigger.

A

So it goes to the uh you know: the transformation service, if you wish and then the the response comes back to the to the broker, and then you know what happens that you need uh the second trigger to. Actually, uh you know send it to uh to sockeye right. I mean here, it's commented so that I'm gonna get all all events. uh You know all events display in sokai uh simultaneously, uh don't think I, okay, so they're still in place, and now, if we look at the the transformation object right.

A

So the first transformation that we can do you know is simply hey. I want to to modify the cloud event type of the ping and I switch to trigger mesh transformation ping source right. So it's not the k native ping anymore. We just switch the cloud event type and then we add, you know a new key in the payload, so you know middle the key will be middle and the value will be maria all right. So if you do this, then you're modifying your thing. So we just apply.

A

All right, so we apply the transformation. It's a crd. You get the idea so get transformation. You know we see the object, it has. You know an address here, let's see if our sockeye has started. Oh.

B

A

A little bit of luck from the demo, god right. So what should we see on the sockeye on the sockeye? We're going to see the two events, because I didn't, I didn't remove the the ping ones so, every minute we're going to see the the original ping event right and we're going to see that type and we also should see a transform event with a new key middle with the value, maria and a new type right.

A

So if we wait one minute, you can still hear me.

B

C

A

There you go demo, gods are with me, okay, so can I zoom that so perfect? You, we see the first event: dev canative sources, ping. You know that's the original original payload alice smith, and now we see the second event. That's been transformed, key maria and a new, a new type, all right, so events being transformed on the on the on the fly.

A

uh Let's look at a second transformation uh this one again, you know the so we switched the type. uh There are different things you could store. You know the source, you could store the id which could be very useful and so on. You know uh here. Second transformation we're shifting a key, so the key, which is first in the payload, becomes k native right. So it's going to be k native. You know alice right, so we apply.

A

We apply that one and I'm sure that you know by now. You have you know you, you got. You got the idea right, so uh you know we need to wait another minute and we're gonna see that uh what's that transformation that I said so, the the key first is going to shift to k native right. So we're going to see k native alice and we're going to still see last smith middle.

A

Mario, I should have put less than a minute. I don't know if we can do less than a minute in the pink source. Can we?

A

Oh? No, it's a crown tab.

E

Well, while we're waiting, you can show off the sockeye filter feature if you click on the upper left, oh the new version, and then you can add in the attribute your filtering on is maybe type includes trigger mesh, and then you should see only the transformed events in sockeye.

A

Thank you see, I didn't even know and.

E

Then push add filters and then this will dynamically update. The display that you see, and so the idea here is that you could kind of play with what a trigger might do in production on a stream of events.

A

Yeah cool, so now you see and we got our event with our shift. So now we shift it from first alice to canadian alice right. Is it big enough? I mean it's pretty straightforward, so I guess I guess you all. You all understand what what this is doing right and and it just works.

A

So that's when a delete. You know, of course we can. We can delete keys, so you know apply uh I'll, do the delete and then one last one which is, which is nice with the actual cloud event context.

A

So this one we're going to delete we're going to delete the key last right. So what we should see is kennedy alice, middle maria and last smith. Should this should disappear.

A

And this is useful to us, because we've we've developed a trigger mesh, a set of event syncs, we actually call them targets and our event. Syncs actually accept a particular. uh You know event types right, so you know if, for example, we get github and we want to talk to twilio the github events they come in as github knativ and.

E

A

We need to send them to our target, which takes in you know twilio.triggermesh, so we need to transform the events and uh and set the type in addition to the payload, of course,.

A

There you go, and now you see we have deleted the key all right, so the last one I wanna I wanna show you is uh is one which uh actually here uh we can store what an attribute of an incoming cloud event.

A

So here you see that in the section context we are doing a store operation and we are going to store the value of the the cloud event source right uh in a key here within the context of that of that manifest we're going to set it as dollar source right and then I further uh down here in the data section, I now add a new key foo bar and I set that new foo bar key to the value of that. You know incoming cloud event source right, so this is just showing that you can.

A

You can store incoming attributes, uh which can be, which can be handy, so apply dash, f, uh trans, three.

A

There you go so now. What we should see in the next event is that we should see that we're going to have an additional key, which is going to be fubar, and it's going to have the value of the actual cloud event. Source makes sense. So you can, you know, mix.

A

A

So I actually don't know, can we can we put less than a minute in the in the crown tab.

F

I think you need to use the heartbeat source for that. Oh okay, I was just looking it up.

A

Okay, so, meanwhile, you know the repo trigger mesh bumblebee open source uh apache software license. Oh no, okay, we put them under licenses. Okay, why is it under licenses? Is because when we, when we try to get things certified with red hat, we need to put a hula and a license under licenses directory, but anyway it's apache software license. We have a release process, uh so this gets tagged. uh You know automatically when we release new software.

A

There you go so we have our bot cyber wood ford, it's not the it's, not the connective.

A

You know actions.

A

Yeah and deploy with ko so a simple ko apply, should do it for you, you'll get the crd the controller, and then you can start. You know doing your your transformation.

A

If we go back to our soca here, you go so the last. You see that we now just added a key foo bar and it's set to the value of the you know the cloud event source right. So you know all of this. Hopefully what you're seeing is you know it's, uh it's quite straightforward, but it's actually. You know, I find it extremely neat, actually clever and super clean.

A

uh You know basically manipulating cloud events on the fly with a declarative api. All right.

A

And that's it. Thank you.

B

Wonderful, thank you so much for presenting that. uh There are a ton of comments here on the chat box and given how far we're into the meetup, I'm not gonna, go into breakout rooms, uh so please feel free to uh mute, starting with jax, who has his hundreds.

D

It's sebastian, I have one question, is it? Is it limited to cloud events.

A

B

Reason: okay, so.

A

D

A

Yeah, it could be anything.

D

Yeah, I I smell future controllers being written. This way is what I was thinking. A lot of controllers just take one bit of json move.

A

D

Around and spread out another piece of json.

A

Yeah, so we and- and you know I did it with- uh I did it with uh a broker and so on. But of course you could just curl straight into the the endpoint of the of the service right.

A

You know it's pretty and it could be yeah any any json.

A

Thanks so should I go back, should I go through all the questions or you all got your questions answered.

B

uh Maybe we can call them uh one by one.

A

Shift a question from alex shift: uh yeah shift replaces modifies the key. It keeps the value, but it shifts the key to something else.

A

Yeah right now you know operations. We could totally see adding additional operations, but right now, that's you know. Those are the operations that we needed, so you know add especially add, delete and and use the uh use the the cloud event uh context.

A

Thank you. V lay yeah. I didn't do anything. Actually, it's all timor uh yeah. I have the filter on yeah. Don't do event loops, be careful with your event. Types yeah! Okay, that's it! Thank you all very much.

F

I think there were two more things. There was one comment about testing um and whether there, whether you had any patterns for you, know, as you start to build more complex transformations, making sure they work right.

A

uh Honestly, I don't think we have, I don't think we have uh testing for the actual.

A

You know I'm totally going to punt on that one. uh So we've worked, I mean on testing, so some of you know uh some of the folks in the in the team, so uh antoine antoine coten works for for trigger mesh. We've done a lot of work internally on using uh ginkgo ginkgo for end-to-end testing, so we have end-to-end testing for our sources and targets and also some of those integrations right, uh but we don't have yet end-to-end testing purely for bumblebee. I don't think we do.

F

um I actually meant something different. I met consumers of bumblebee who are writing starting to write code effectively in these yaml files. Right may want to test that their code, that is the bumblebee transformations, actually does what they want.

B

I think that was alex's point. Okay,.

F

And yeah, so that may be something that um there may be. You know: hey open.

A

Source contribution.

A

um No, but it's qui, it's quite fun, you know it's it's a little thing, that's that's quite powerful and uh yeah I mean when uh you know we we discussed this internally and then timor implemented it and you know we think it's it's super clean and you know very useful.

F

I, like it, um oh the one other thing I was going to point out was: I noticed your releases um have um the binaries, but if you use co-resolve, you can get a yaml, that's all of the config, bundled into one chunk um with references to like gcr.io. If that's, where you're, storing your containers, yeah.

A

E

A

F

We use for k-native releases.

A

F

um I don't know if that's a thing you want to do, but I figured I'd mention that no we should we should. We should make.

A

F

A cube control apply some url from github.

A

Yeah yeah yeah totally totally fair. We should do it. That's that's! uh That's a new mission. uh I actually ran into a ko okay problem uh when I was preparing for the demo. So so yes, we should add the manifest to the release.

B

It looks like carlos has some comments. Do you want to unmute something about metrics.

C

No yeah, I don't know if you have time, but um do you have a metrics endpoint or you push metrics about getting stats about transformation that didn't work errors or how many shapes or operations you're doing.

A

No, I don't think we're, um I don't think we're exposing a matrix uh endpoint on that one.

C

To get the health of the controller does, does it scale this horizontally, stateless, I'm guessing.

A

Good question: I don't even know how it's is it wow. This is just using a kenyan service under the hood.

A

The controller.

A

Hold on I'm looking.

C

See mariah what'd you do when you would mute me. No, no.

A

Yeah, no, the transformations are actually pure deployments right so that we don't have. We don't have due to scaling out of the box uh any.

C

Reason why they need to be deployed instead of okay native, so.

A

C

A

I'll I'll have to check with timor actually, because we should we should be hold on. Let me check no, no, no, my bad, my bad yeah, the transformation or creative services. Yeah, of course,.

A

Yeah yeah, so the transformation they themselves have the the auto scaling of k native um out of.

C

The box, so it's a matter for the matrix thing, then maybe it's a matter of adding a open telemetry and do the pushing yeah of the metrics.

A

I'll put I'll put an issue.

F

Actually, if it's a k-native service, you may be able to get some of them just out of the box with the services yeah. Yes, response code and latency and stuff like that,.

C

Yeah kind of like mesh mesh type of metrics, not business inside.

A

Yeah, so you know, we think we think it's pretty cool. Definitely you know there is potential for more work and.

C

That was another question. Leonardo is asking, I think the same question I asked uh before in the chat about multi-tenancy and he thought about what the tenancy of that of that controller, maybe isolating that that store or that state.

A

Or uh so we haven't, we haven't dealt with multi-tenancy on the tr on the transformation on bumblebee uh right now. Actually uh antoine has has moved all our sources to being multi-tenant. So if you look at all the trigger mesh aws sources, they are now all multi-tenant, uh but we haven't done it for uh we haven't. We haven't looked at it for uh bumblebee yeah.

F

If I'm understanding correctly, the store is in the context of a single event, it's not applied to things in the future right.

A

No, no there's no persistency in that one. Yeah.

F

Yeah, so this is stateless.

F

Shouldn't be an issue.

A

Yeah, I'm not even just I'm.

B

A

Sure, I'm not even sure what multi-tenant transformation I mean.

F

You could use a single pod and handle.

C

F

Namespaces uh but since you're using a creative service, um you know you'd get small incremental savings. Not you know like oh hey. We have these pods running all the time and most of the.

C

C

F

Now it may be that ibm, you know running an enormous cloud actually cares about saving those. uh You know those pennies from each namespace, but.

F