AsyncAPI AsyncAPI Conf 2022, 3 Nov 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: AsyncAPI as a Driver of the SDLC - Jesse Menning, Solace

Description

AsyncAPI Conference 2022 - Day 1
3rd November 2022

Event-driven applications benefit from the power and flexibility of asynchronous interactions. But they remain applications, and ideally use the same processes and tooling used by their more staid synchronous counterparts. But what's the best way to deploy async applications? Provision their infrastructure? Test their functionality? Rather than bespoke property files and custom solutions, enterprises are integrating existing SDLC tooling like git, Ansible, Postman and Jenkins with AsyncAPI, with exciting results.

A

Hey everyone- this is Jesse Manning I'm excited to hear to be here to talk to you about async API and this role that we see in driving the sdlc life cycle, the software development life cycle. So it's interesting cruising around uh on the internet.

A

Looking for things uh related to this talk, and we see a lot of things about life cycles uh and one of my favorite things I found actually was from Ken Lane uh Ken Lane uh he's worked for Postman and does a lot of thinking about uh API life cycle more more generally and sort of evangelizes. The use of uh of apis and I found this uh really cool uh diagram here that includes the postman Spaceman. That's why there's a couple of astronauts uh interacting there? You can see that there's two interacting life cycles.

A

This is for for an API. uh You can start. You can usually visualize things that started on the left. You need a producer in order to have a consumer uh and he sort of steps you through the different stages of a API and then once the API is distributed over on the right side.

A

There is a consumer life cycle and, of course, Ken does a great job of explaining things uh and walking through how things give are given birth to and also go through a life cycle and then eventually, presumably die after their after they're no longer needed, and it's interesting we're seeing uh talking with with people who are interested in event-based uh apis and that there's this big desire uh to bring these kind of life cycles into event driven uh worlds as well, using primarily actually async, kpi and even more so is combining those two worlds of sync apis and async.

A

Apis I can't tell you the number of people I've talked with that want a single. You know: One-Stop shopping for uh apis, whether they're synchronous asynchronous whatever they are. They want to have one one place where a developer can go in and uh and pick something that makes sense for their for their particular use case uh and and get access to that information very very easily.

A

uh So it's an exciting time, as we all know, for uh event, driven architecture more generally and event, driven apis as well, but thinking through it, uh it's interesting that there seems to be an interrelated life cycle here as well uh and they'll Focus here on the on the producer life cycle. uh There's certainly a correlate over to the consumer as well, but we'll just you know, for a scope, focus on the producer life cycle and the that is as long as we are advertising these uh these apis, these event apis.

A

um We need to actually have the ability to process them. In other words, if we can think of the the API and a classical sense, uh it's the interface, but we still need a uh an implementation to make these things real and those implementations actually have their own separate and distinct uh life cycle as well, which is as important as our our API life cycle. You know if we don't have the the meat and the sandwich.

A

The sandwich is just kind of bread and and sits there a little bit so and obviously you know these event-driven applications if you're uh uh any size, Enterprise You're, Gonna Want, to have multiple environments and test these things and integration test them uh and have a Sandbox environment where your developers can can give a wild and explore ideas, and so you're going to need to have this life cycle exist in multiple different places and have it be repeatable as you uh as you so desire.

A

So I was thinking more about that and sort of the implications and challenges of that and that's sort of the the basis of this talk. How do we interrelate async API the producer, API API cycle and then also sort of the the event driven application implementation cycle as well?

A

So just give me a high level overview. What are the goals here? Well talk briefly about async uh sdlc without async bi. Certainly a lot of people try to do that. uh Talk a little bit about the challenges in uh in that regard. Talk about why async API helps solve the challenges associated with with sdlcs and finally, you know room for growth, of course, challenges that I see in terms of the async pass spec and using it for for sdlc. So plunging right in.

A

Let's talk just a little bit briefly about what uh nasync sdlc looks like without async API, uh because for a long time we've had async architectures and we haven't had async API. So what does that? Look like I think? The underlying challenge is that there is typically some infrastructure, that's associated with async architectures and certainly brokered architectures, which is kind of the world. I live in. You know, as you're moving things through environments.

A

We have to make sure that things are in lockstep things like cues and topics, endpoints and security, and you know just there's a lot of configuration. That's typically involved with moving an async uh uh application through environments and that's a great thing. I mean I've got all the infrastructure serves a purpose and I think a really great purpose, and since you're listening to basic API conference, I assume I.

A

Don't have to tell you all about that, but the the decoupling is fantastic, but it's also creating this sort of shared touch point between a whole bunch of different uh different applications that don't necessarily know about each other, that's decoupling! So it becomes a bit more of a challenge in terms of moving things through an sdlc.

A

How do people solve it? Well, there's the classic we're going to roll our own approach, and certainly I've worked for a lot of people who have tried that you know there's some like custom, proprietary, Java or Java.

A

You know Json format that people use to describe things or the properties of the applications that moves through environments- and you know, there's custom scripting that processes the processes loads a lot of times that that revolves around the sort of raw object, definition, definition, language of the particular broker that you're dealing with and that that's a a viable option, but I would argue that that's actually kind of a constraining uh option as well.

A

So let's move on how could async API help standardize? Sdlc uh I aggressively have stolen, slides from Fran, uh because I think a is a lot prettier slides than I typically would typically produce and I think also that it shows that a lot of the concepts that are Central to async API, that Fran eloquently talks about on a daily basis, can be applicable not only to the interfaces of applications but used for the implementations as well. So here you go.

A

Here's a friend slide on the purpose of uh of async API and you can say event driven by microservices and channel things, apis streaming apis and pretty much anything based on messages and I'll sort of glow those things out there, those micro services and pretty much any system based on messages. You know, once we move beyond the pure API use of async API, we very quickly move into a realm where we're introducing implementation.

A

And here's another slide that I stole from Fran. It says you know, the specification is protocol, agnostic, which I think is fantastic. uh It avoids vendor lock lock-in makes it you know, sort of generalizes things, big points there and as Francis Fran Mendez uh protocol information is important for implementation details, but it shouldn't be a blocker for defining the interface.

A

Absolutely don't worry about until later, but the correlate is eventually you're going to have to have a protocol or nothing is going to work, you're going to be shouting into the void and that's that's from guessing money. So how do we bridge that Gap in the async API? Well, thankfully, we have a fantastic solution for it. We are protocol agnostic, but we also have binders that allow us to Define things in an open standard, best, practice kind of way and I forget I.

A

Think it's it's amazing because of the open source nature of things you get the best of Knowledge from an IBM expert, ibmq expert about like this is what matters when you're, defining the IBM mq and a Kafka expert I, don't know as much about Kafka as I should so. Like I get to rely on people who care deeply about Kafka and its bindings to figure out like these are the important things that we should be thinking about.

A

When we're configuring Kafka an amqp and that's you know the list goes on and on and that that binding can, you know, configure because we're at different levels we can configure sort of Q level things it can configure server level, uh it's pretty fantastic, just to have that access to that knowledge.

A

And the the spec serves the the purpose pretty well I mean in its original and the current uh conception. The spec represents a file per application uh and so yeah this can be the app.

A

You know how you interact with the application, but it also can really lead you on the right, correct, correct path for implementation, or at least give you a head start for it and give you the associated infrastructure, and you know we have a an info object that can give you a little bit more detail about the implementation and also provides versiony information, which is a pretty key as we sort of move into a multi-environment uh situation, the the versioning above and beyond what you know, GitHub uh Branch, would tell you a semantic version, becomes really really key.

A

So we're looking at some really good good, uh good Baseline here for uh things that could drive your sdlc, then I think the final component of it is the tooling I mean we talk all the time about the the great nature of the community here and I. Think that's uh that's uh evidence of it. You know, rather than having to create the wheel over and over again at every different company or every different initiative.

A

We have tooling that you know, creates things automatically and uh I think a fantastic example of that is code generation and to sort of break up me talk Talking for forever. I'll show you a brief little demo of uh something that we did. That saw us that leverages not only the open source code generation, but you know async API produces a uh or hosts actually a endpoint. That will create a a code package for you so that in just a little bit.

A

So what's the conception here, the conception is that you know- and this is a very, very generic sdlc and some very specific uh different logos here, but you can Envision. This could be. You know your own custom sdlc, uh and it could also be your own custom logos in terms of the tools you use, but using async API each step along and Away here to to interact um with the life cycle and rather than having a proprietary file format that you use to sort of understand where your application is and what resources it needs using.

A

Async API to you, know one file to one file to rule them all in the words of uh of Lord, of the Rings uh use, use the async API. Instead.

A

So, as I alluded to earlier, I'll take a brief, uh interlude here uh and run a demo of IntelliJ cogeneration. uh It's powered by the open source Community under the covers calls the the async API API, which I didn't even know, existed, and then my colleague find it and we're like this is exactly what we need. It's unbelievable and then we found it through slack. So a nice little example of community I will say that the way that we find the async API is through a solid product called event portal.

A

You'll notice that in the in a demo, but like very well, could be just you know you you pick up the async API file off of a hard drive, so just know that, but I think the the result is pretty cool.

A

Async API, as you might guess, lets you define async interfaces. You can think of it as an open API for the event-based world. It's a relatively new standard, but it's gaining unpopularity. You can create async apis in development, tooling, or products like asapio can Auto generate them, regardless of where the async API comes from. The async API importer can suck them into event portal and make them findable and usable by other developers. So here's a simple example of an async API file.

A

You can see it defines the application itself and also the events that come in and out of it now we'll head over to the Importer. It's a CLI tool that makes installation pretty easy, so I'll go ahead and install it.

A

I should say that's because it's a command line tool, we'll run it as a command line, but it can also be used by additional tools like GitHub actions or custom scraps, so we'll go ahead and initiate the command line here.

A

Okay, now we've got a report for our import here and we'll move over into the event portal and see what we've got on the other side.

A

Refresh and then portal organizes applications by domain and we've specified the new new domain for this called Acme retail, Automation and click into it, and the retail customer mobile application that we described using the async API is there for us to use clicking in. We can actually see the various different events that it subscribes to you. If we dig further we'll actually see the schema, that's being used.

A

Now, let's update the async API and see that we can actually import uh new versions of the async API. If we want to, let's say that we really want to have the product ID included in the uh included in the async API description and that helps us be able to Route Things based on the product. So we'll add in the product to the async API, save it and then we'll run the Importer again.

A

So now that the Importer has completed we'll, go back to event, portal go back to our domain and take a look at the the application that we had to find there.

A

You see now that we've got a a new version that reflects our updated topic strings for our event and we're ready to go.

A

So, uh coming back from that demo, certainly it's not all rainbows I think it would be appropriate because we're always a growing community community to talk about the challenges associated with this particular approach.

A

So uh I think it's interesting that you know talking with Fran and Lucas and rest of the guys about uh about the growth towards uh version three of the spec, uh as we start componentizing things and promoting reuse, which I'm a big fan of. uh We also run into the problem of shared resources.

A

You know both at runtime now and at More, Design time uh and so I think that our Approach at Solace at least, is that, rather than interacting directly with a pure async API, uh but for the role of the async API catalog will come more into play, in other words, rather than somehow trying to figure out how things interact using flat files having something more generic that allows those objects to relate in a more Dynamic kind of way is going to become more and more crucial.

A

That's it! You know, Solace is on that. uh Dave uh has a fantastic open source product called uh I believe it's called event. Catalog IBM I know is looking into it. Sort of solutions like this as well uh I think it's really intriguing space to think about, um and then that, in turn also provides a really nice place to link back into that event.

A

Api lifecycle that we talked about uh to to uh to begin with, in other words the async API catalog, creates a you know a little bit of a buffer mechanism by Between the the raw async API that you know, don't necessarily want all exposed out to your to your API consumers. uh It also provides a level level of government governance. There.

A

uh In a very specific example of that is you know we have the the ability to link in uh server information in the in the async API.

A

um That's certainly something you would want to have abstracted away from your if you're having external people interact with your your application, but perhaps even the internal people, um and the reason you would want to do that is, as we all know, sort of the the physical mapping of of servers changes all the time uh again going back to to France or conception of late binding. You know if we're late binding to a protocol, we're certainly one of Light by late, bind to a uh a particular server physical server implementation.

A

One of the challenges is we've sort of have been thinking through our own uh Journey at Solace. Is that, rather than having a physical server definition, it would be awful nice to have a logical server definition. In other words, this uh this application gets deployed to the red broker group.

A

um In contrast to this, you know other application gets deployed to Yellow, but in Dev rad broker group is, you know, a single broker, because we don't want to spend a lot of money, but in production, that's you know a highly available cluster that you know it will not be taken down, regardless of whatever happens.

A

You know highly resilient uh production broker and being able to map that not in the async API itself, you might have a little layer of abstraction, but something that you're using uh behind the scenes, perhaps to deploy your architecture, something like terraform, something like ansible, where there's a mapping between these logical Concepts, logical broker groups spread, for example, and actual physical inventory of like okay. This is the the actual uh broker that thing slant on so with that said, uh just another brief demo about uh deploying out infrastructure.

A

Is it in this case we uh have a ansible plugin. Actually that takes an async API and deploys that physical infrastructure doing a crosswalk. uh You know we sort of have hacked together our own little logical, uh logical, crosswalk, uh logical into physical and then deploy out in this case, uh Solace infrastructure.

A

uh Obviously we're interacting with the basalts binding you could. Probably you can easily expand that out upon into mqtt or other other protocols as well. So take a deep, uh a brief pause here uh to run that demo and we'll uh we'll see you when we get back your event. Driven applications need to have things like cues topics and subscriptions to work. It would obviously be better if you generated those things automatically and even better still. If that auto generation used an open standard, that's what ansible plug-in for event portal.

A

Does it uses async API bindings, which are an open standard, so no weird proprietary stuff to generate objects for a particular application. So we'll start off by checking out an async API file, and you can see up here that we've got a number of different server definitions as we scroll down. You can also see that we have a Q binding, or rather a technology binding for Solace, there's technology bindings defined within async API for Solace for Kafka any number of different brokers.

A

So we'll use this to drive our uh our creation process, so we'll use Postman to kick Jenkins off, which will in turn call call ansible, which is a pretty common pattern, but just know that ansible has a rest interface called ansible Tower. So it's just a rust call. The rest call includes a path to an async API file which again contains a binding to either Kafka or Solace, and it also tells it what environment to deploy in so just to prove that there's no funny business going on here, we'll navigate over to the GUI.

A

You can see that there's no no cues right now, but we'll then we'll look back over at Jenkins and see we just are finishing up with ansible Tower and we'll move over into the GUI and see what we have here. Now we have the the queue here.

A

So I should say that underneath the cover uh ansible hosts playbooks, which you know they're- sort of they're they're programming language, which can create underlying infrastructures for a wide variety of brokers, um it also hosts something called the inventory which can map between a logical label like stores or Warehouse, to the actual Brokers for an environment. So you can use the same async API file across multiple different environments. So, in conclusion, uh first of all, thank you.

A

uh It was it's an interesting topic for me to think about, and this is definitely a prompt to think more deeply about it uh and thank you actually to the uh the wire Community.

A

um Certainly in our efforts uh in my efforts being able to leverage the great work of the open source, Community has been really really helpful and hopefully you know, and things like the co-generation uh we've been able to to donate things back as well uh and I. Finally, this is, you know a work in progress. uh It's an intriguing topic for me, like I, said, but that's really welcome feedback and and better understanding how these things work in your particular environments. So thank you and have a great day.