Kyma Core Special Interest Group Community, 18 Mar 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Core SIG off-schedule meeting about AsyncAPI in Kyma and Knative needs

Description

This was a meeting between Kyma's Core SIG leaders, AsyncAPI founder, and Knative representants working on concept of events registry for Knative eventing.

Meeting covered:
- Explanation how AsyncAPI 1.0 is at the moment consumed in Kyma
- How we plan to change use AsyncAPI 2.0 in Kyma with our Headless CMS and Asset Store components
- What are Knative needs, how we address them at the moment, how AsyncAPI could help

A

Okay, whatever let me show my screen so the purpose of this meeting is to give you an overview of how, in Chima we are um yeah using a Singh API in in all areas of Chima, but the most important part is the even registry, of course, that you also are thinking about in key native eventing.

A

So maybe a little bit of background, if you don't know it kima what it is just in two sentences, so kima, it's a it's a young person's project based on core energies that we, where we focus mainly on extensibility of your legacy, monolith old, huge applications. Name, it see wish so. Basically the use cases that there you have a ya old application that you want to extend in cloud native way, but you can't really easily do it.

A

Although there are many interesting presentations about how to go from one oddity into micro-services, so one of the functionalities where we needed as an api was the way of kind of documenting what functionality is exposed by this external service, because external service can expose api and also can send events into Keima.

A

So a single API, first of all, is used in Kumasi yeah kind of API reference Docs right. So we we simply in the catalog view whenever you have all the services coming from any service broker, but also including our broker for those applications.

A

So in because we want to have really nice deaf experience, we always in the context of the of the user whenever he has to take some service and use it create instance. You can see the docs rendered out of the asking API spec delivered by the application. So simply when you register application, you can also send the spec for us, so we render it, but then the the most interesting part for you is the the part about the registry. So, okay, you get the spec as a developer.

A

Everything is here, but at the end, you'd like to have a different content for you. So in kima you can extend your applications, for example, using the server less and and whenever you write some lambda code, you need to define when it's triggered. By what event is it triggered? So we have this functionality in the UI that you yeah Tuesday even trigger for a lambda, and here is something that I refer to essay even registry.

A

So the in a given night space, the developer gets a list of all the events that he is allowed to use and said: I say trigger for the lambda and the the payload of the event whenever it comes to him eyes always delivered to to the lambda for the use. So you get least for now with version one of us in API and yeah the current version of kima. We support simple, a use case that we just returned the event name and also the version, because in our case we have the.

A

We have a use case that the application in the real time can provide a new version of a of the same event and send it to us. So you need to have versioning per per message per event. So that's for now what is being returned in a even catalogue for a given context for a given namespace.

B

A

And it's of course needed those data, but that because I'm Indian in this case in the UI, you need those details to create a subscription to subscribe. Your application tree to the event.

A

So about subscription not sure if I should talk more about it, I don't think it's important for this discussion, but yet we just have in came our custom resource where you can subscribe your application to a given event and yeah. You need those details that we have in our catalog and the most important I think for you is how we want to change it in the future. So not sure if it's big enough, can you see it in this diagram clearly or should I make it bigger? First.

C

It's okay! For me, it's.

A

Very cool, okay cool, so here those two huge boxes, that's that schema and so I will not talk about how we do the persistence of the spec now at the moment, because we are now factoring that so how that works. Oh, you have a external application, it's outside c'mon and it registered service. So it's it tells us because again and the exposure of the oddity functionalities of the application we we based it on open service broker, API aspect, so we have our broker and so, of course, except of all the details needed for for provisioning.

A

We also made it possible that when you register the application, you provide the and URL to the spec and we have a controller for our Docs topics. Er and we we call this functionality a headless CMS domain in in c'mon, and we also have very generic solution for staring storing assets like specs or mark down documentation or actually whatever you want, and it's we call it assets store this component, so how it's done so we we provide a docks topic, CR and I'm.

A

Gonna quickly show you what it is, but I probably have to change this screen. You can you probably still see the browser right.

C

Yeah just see the browser: okay.

A

Let me switch to the comment line.

A

A

Okay, you see the comment line. Right, looks good, okay, cool.

A

So I'm gonna show you now how this Docs topic looks like.

A

I need to take this particles.

A

A

So it's basically a resource where you yeah, you can provide some inspection yeah. This play name description and we yeah for our need for cumin eat. We solution not only for asking VP I suspect, but it's for any other spec and also marked on documentation. So you can specify different sources, and in this case you can see an example of yeah I. Think API example and you can provide a link where you store the specification so yeah pointing to the file in the internet, and then we once you create such a CR, we create e.

A

We create the asset that is part of our asset store to fetch the URL. So you can then later do something like an after creating dogs CR.

A

You can get all the assets for this.

A

So it's a a resource responsible for fetching and it also has a functionality. If you want to do something with fetched file before it's uploaded to our storage and then it's it start in the mini, which is the s3 solution.

A

So you get in the status of the CRA a base URL to the bucket, where the site is located and, of course, the and the name of the file on there which it's created. So it's. If it comes to the details of the definition, let me know again sure diagram.

A

Okay, so we create the asset and in simple case, we yeah. If we put it into me no, but we use minion, recommend me only for for development yeah, because we yeah we in our case, we always recommend to the customers to reuse, managed services as as much as possible.

A

So, instead of really maintaining some I setup of me know like using ruqaiya or something like this, we simply say: okay on production, you have to upgrade min your chart to use the Gateway, so the files really are saved in GCS aw, aw Sh, your blog, whatever you want because mean your supports. This mean your gateway mode and then what we will do with version Ising API to zero.

A

We need to introduce something that we don't have so, of course, validation of the asking API. Then we will have to for the customers instead of telling them to modify the spec on their own. We will have some service that can do some basic transformation from one with version, one to version two, and we will have to provide this kind of registry parser, because for now what you could see in the UI that I can get this list yeah for first version, we used only our special layer for the UI to do that.

A

So we have a basically in case of kima in console. We have graph QL that kind, so we simply have a graph fuel on a on a graphical back and fungal. We have a functionality that is part in the this pagan giving this this registry, but we want to, of course, get rid of it because it's useless for command line users, so assets store that we created it, provides and gives you a flexibility of defining a web hooks services that, before the spec is uploaded Tomino, you can mute, hate it or validate the file.

A

So that's how we will do it, so we will basically have one file at one service that is serving any purpose for the asking API and it will have an endpoint to which you send this back and we will generate a new file called register, I. Think API registry, for example. That's not defined. That's still in progress for discussions, so we'll generate this new file and and create it in the scope of this asset. So it's going to be uploaded to Kamino. So in return at least that's the plan. For now.

A

D

A

C

In back in your example of making a simple lambda for a selected event, using a little JavaScript lock, what what comes back, describing the shape of the event that you've selected does the UI help you understand what what you're going to receive.

A

Yeah, so I can switch. um Maybe first let me share, maybe the desktop in general, because it's not efficient how I switch.

B

Its its, we only have support for that. Basically, these fields of the subscription that Lucas showed this solution. C are the schema itself is just right now, just documented. This object that the alum that will receive.

A

Writing off so quickly, I'm, hopefully, that it's gonna work but I can show you the graphical layer.

A

A

Wonder if I'm using the right, yeah.

A

Then just it for some time.

A

Now you can okay, so you can see. We basically have a graphical query that were except of some details related in this case. For specifically for kima, we really return very minimalistic information that is needed, so something that I've said.

A

We just need for us, it's an even type and the version and the description is just for the UI sake and in future we would like, of course, to have not only the description, all all the nice things that we can later use in different cases like the example payload, the spec and everything we actually have at the end in the part of us in API.

A

So the the registry will be much more much much richer. Let's say one.

B

Thing I heard I'm not sure how is that going? But you remember there was a idea about providing a tests function, so you can click on the on the on the londo that tests the dry run or something like that, and then you get a UI where you can send a custom event out of the UI and then test inside your lambda, what the oldest, so that we'll probably need to understand the scheme of the even that will be sent like a like a dry invoke or something yes,.

A

Yes, and by using on this example that we would return from the registry, you could of course yeah.

A

They cannot sample, object and interact in this test, and there are even discussions that if we have it already, if we're gonna have it already in the in the in the registry, we can even generate some objects so for out of completing day in the UI, because it's no yeah, you have to really take a look into the spec to know what's in the payload at the end it okay, so we are in general, it's just sum up, so we have a say, storage in directory at the end, it's it's me know for the whole processing.

A

We have an asset store, which is super generic I mean, and it it's not very much. Let's say dependent and kima is not very much dependent on it in the way or even maybe the other words around, so I think API I think story's not so dependent on kima. So, even if you would consider that that would be a also good solution.

A

So you don't have to reinvent the storage on your own, that we could extract it from kima, so use it in him as a dependency, but maybe have it separately, so it can be used somewhere else.

A

But the headless timers, it's it's useful for you, I'm, not sure what accounts you have others and adjust the events registry can.

C

You see more of the async API and how you're using that is from from what I understand it's it's more about, describing the shape of the objects and their destination inside the context of that service, like coupling events to queues, but in a system that doesn't have queues exactly I'm wondering how you're you're able to link what async API is doing. They're telling you about a bet service to how you're actually going to use it internally. We're like where you route requests.

A

So naturally, and got your question correctly so, except of the just the this yeah describing its assay dogs.

A

So how we, how we know all of that so when, whenever we register but I'm, not sure if it's actually something useful for like you- but yeah. So whenever we register the application.

A

Its again, it registers a set of service classes, phasing on the on the open, our open service broker, API and the, and we were using in kima the kubernetes service catalog. So it didn't connected application can say: okay, we have some commerce ATP. I we have some whatever or the management API, but we also have some events so.

E

Directors, god I think you were mean that the the events were tied to a topic in this case right and what happens if you don't have a topic like right. Is it exactly.

C

E

C

So Canada this plane with the idea of the data Lake where events just go into a pool and then they get queried out. So we don't. We don't have a topic per se. We have a collection of data streams, yeah.

E

So some person to what we did is that we.

E

Serialized topic says us URLs, like they look at like URLs, so in version two, you have this slash a slash. This last topic will be like the root topic, like no topic like like a box where you throw everything right and and everything can go there and you can publish or subscribe up to this last topic, and that will mean like no topic at all. That's the example, for instance, as web sockets in web sockets. You have you, have some software like socket IO, that simulates that there are topics or channels but I?

E

Do they don't exist, they're, just different web sockets connections being web sockets, you don't have the concept of topic or of a channel or whatever right. So it's the same thing. So that's how we provide support for web sockets, and it will be the same for that case. So.

C

I guess that brings up another question if, if the service providing you, these events understands how it's going to how it wants to namespace the events, but the integration software as it gets into my cluster, chooses something different. huh Is it expected that each hop in that connection rewrites the async API.

E

C

It could so, let's say I write an adapter that takes something that is strongly cute and bridges it into a names like a single namespace event, area and changes its so you know, can ativ is gonna, be using only cloud events and no matter what the external thing is producing they're gonna get bridged in as a cloud event, and then in this way we can do this filtering and selecting if it's not on a particular channel or queue or or whatever, so we're trying to abstract the details of how that thing becomes asynchronous from like I guess.

C

What we're trying to do with the registry is understand where the event came from and what its gonna look like when it gets to you, but not necessarily how how you find it internally, because that those details are meant to be slightly opaque to the user. Okay,.

E

So you mean yeah, and your concern is about the hops that can be the in the way like transformation, that the event can suffer in along the way since it's publish until it's received- and this is quite right- yeah.

C

I'm I'm concerned that if there's data encoded in the async API that understands how that those particular happen, I might have to rewrite the document for every for every perspective, through an integration layer.

E

C

I, don't maybe I just need some more experience to know, but yeah.

E

So I mean I had these kids before and what I told people is. It is to have an async API file per layer. So if you grab a message, you transform it and publish again in a different channel or in the same channel. If you, if you're gonna, do it have an icing, KPI percent risk or pair per layer, and then in this case, what you can do in actually in version two, what you would do you will be able to do is to use overlays overlays functionality.

E

I, don't know if you know ramble Ramla specification, so overlays in over listening an async API system as overlays in pyramidal. So you will be able to have anything KPI file that you apply it's another layer and another layer in top, and you will be able to change some things of this document or add things or remove things usually remove things. It's not very common, but but usually most of the times is changing and adding stuff.

E

So for this layer you might want to change the schema of the message because you're gonna transform it right, so you might want to add some fields or change the name of the field because it has changed whatever right, and so you didn't have to rewrite the whole document. Again, you will have the original async EPA document, then in every pub right you have an overlay another like document that will just understand what it would need to change to make the resulting message.

E

Look like like it will be, like you know, alright, if I, if I'm explaining correctly, but so.

C

The very top layer does async API have the concept of like the minimal viable specification like something that's just a little past wet like a like a JSON.

C

Sorry, a JSON spec like I Jason specs, not quite enough I need, like you know, a so open, API write as specifies schema for specific objects, but it's coupled to usually like an endpoint or a service, and so that's not gonna work for me and async ap I'm, like sounds intriguing.

C

If I could do something that decouples it from the topic and it's it's more of a way to do semantic, understanding of versioned event types and they're, there schema that's going to come to you.

E

C

Yeah, it is a non solved problem. Okay,.

E

So because we are not entering that layer of the how the data to look like and and we leave it- the Senate like an OPEC object, and it can be anything for us for our point of view and that's why that that's that's the reason. I was asking about cloud events, because one of the things that I want to do actually is to saw an example of how to use cloud events with async API to both combine right because.

C

Events is supposed to provide a well known envelope, so you can do like first past parsing of an object and potentially help route that message to its destination. You know based on internal configuration, but cloud events are also thinks of the the payload as a bake. Okay,.

E

So so, yes, your concern is more about the type of events or the message or whatever right and not so much you. So basically you want the list of events right. That's what you want well.

C

I actually do want the so. Okay, so like my ultimate dream, would be to enable something in K native that allows you to register some adapter. That provides some something that describes what its events are going to be like at this registry idea, and then you, you can similarly click a button and it returns you back like the fully populated go struct or whatever language you choose of the this is this is the function you should implement to receive the event you've just like did because it's on your cluster or whatever case you.

E

Need the the event and also that some kind of reference to the service publicist I, don't.

C

Even I don't need to know what the service made that event I just need to know what the shape of the event will be, and a bunch of like linking back to like how to look up what it does and what it means.

E

Topics, no producers exactly.

C

That that is what I'm looking for.

E

That's if I'm not mistaken that, because that's what Lucas team is already doing is you can extract those messages from all days and keep gate opens right? Will it be possible yeah.

C

I think we just need to agree on the the the schema of the payload, but what chemo is doing is very interesting and and then plus the scheme of the payload. And then you get something really interesting where you can actually generate code that handles the function.

E

You need a specific user schema for your pillow inside K native yeah.

C

E

So we can create that we can create a schema and another schema for K native, and then we plug it into into I mean actually I'm gonna explain briefly. So in version two, we decouple a CD API from JSON schema by that's. Why I was saying that we we don't care. What's inside the message you can define it there. If you want I mean you have to actually make sure that if you want, but we don't impose any in a specific language right, we don't spell imposed, is a schema.

E

And that's why we're gonna start supporting something? We call schema parsers and that would be some scheme. A parser for JSON schema, defined messages brought above schema parsers, so you can define with protobuf or flatbuffers as well for gaming, it's very interesting and and with a ver as well. So, and that's why I was thinking saying before you can use it cloud events schema parser right like and then what we can do is we can have a que native schema parse it. If you want that, we can agree on a specific schema format right well,.

C

So cloud events is more of a transport abstraction, so a cloud events can come on any sort of cueing or request based transport. The you can support and it's it's metadata to understand what what's coming to you and Canada doesn't actually have any opinions about. Well it it does have opinions, but we don't not necessarily making the content of the event we're enabling, like we're. Basically middleware.

E

Like using cloud events, this is peg not necessarily as an antelope, so it doesn't have to be in the message in the payload of the message, it's more like, like an async API, it's separated. You use both layers to validate the other right. One.

B

E

I did the other, but it's not gonna, be included on the other. On the final payload, it's.

B

E

Yeah so yeah we can I think we, then we we have to agree on schema a certain schema for K native I. Think that's the only missing part here and- and that will be also interest- is for for keema I'm sure, because if we can make all these things understand each other right. That would be amazing. Yeah.

C

That's that's go cool demos and the discussion that okay, thanks.

E

C

Gonna still hang up like stock around your channel and see what we can do. I, don't know if you know maybe kima doesn't really want to write their own registry and they want to leverage other things. Is that true or no.

A

And yes, we'll definitely there's an option to be something than will. First, our first rule, we have don't reinvent the wheel, yeah.

C

We're eager learn from what you've you've learned doing this this task. So that's I, think that's why this conversation is interesting.

B

You know if there is any any update on the registry regarding after the decision was taken because last week I was not that much available and I. Remember last thing: it was like a tie there right so probably I'm guessing you guys are discussing it locally or prototyping, something or something on. We.

C

Have some ideas actually not sure it's been working.

D

C

D

B

D

Ahead, yeah I've been kind of starting some prototyping, some stuff, the things that came out out of the UX comparison document yeah, nothing defined, I kind of studied, a separate dock that I share internally and once I, let's say, get the thumbs up: I'll, probably post it into a group, so that I can get some comments and stuff yeah.

C

It's a pretty complex problem that we have to think where we want to draw the line, because in cloud events the source like the actual resource emitted, the event can change for every message for every unique endpoint. That's actually emitting something. So if, if the registry able to tie back event type to actual resource, maybe even in the temporal basis like every event, that's ever happened in last week, unique source. That registry can grow very large, so yeah.

C

We in the background, where we're also working with cloud events specification to decouple the the the unique identifier for the resource, submitting the event versus their the service that holds them. So, for example, if you're, if you're listening to a bucket- and you there's like a object, change notification at the moment, our interpretation of the spec would be that that resource is the thing emitting the event because it changed and then you can go back and find it. But wouldn't it be nice if, instead of the source, we're like the the ultimate thing.

C

That's emitting that event is a 2-part right like the bucket and the resource, and then we could have the registry know about the buckets, but not necessarily the individual resources, because it's not that interesting. So we're trying to split that in cloud events it'll make our life for cloud events and K native a little easier for registries.

F

B

Yeah yeah cool, um actually I'll, please post that whenever it's ready.

E

So we can have a look, you can keep here as well and we might be able to provide better help them just just for the record. I've never used.

E

All the way and some of the things you say it's like what so if you can write down some of the these things and so I can you know I can have an idea on what are the needs for the registry I'll be happy to work on that further. It.

C

Sounds great and one.

F

F

Think, that's! That's! Alright! That's all I got that's big awesome.

A

So good, we think I guess we're on all in contact anyway, right tractor, so any objections, if I put it on on our my channel I mean c'mon YouTube channel, so we can share it with anyone who's interested internally. That's.

C

A

Thank you thanks guys, cool.

B