AsyncAPI AsyncAPI Conf 2022, 3 Nov 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Patterns for Event-Driven APIs - Erik Wilde, Axway

Description

AsyncAPI Conference 2022 - Day 1
3rd November 2022

Event-Driven APIs and Event-Driven Architecture are a popular topic these days. But what exactly does it mean and how do various patterns of event-driven designs fit into the bigger architectural context? In this presentation we will go through a series of patterns for event-driven APIs. We start with the simple idea of individual APIs being designed and implemented as event-driven APIs. Me then move on to API landscapes supporting the pattern by providing infrastructure elements. We discuss the idea of the entire landscape being exclusively event-driven. We then move on to discuss specific patterns such as CQRS and event sourcing. The goal of this presentation is to allow attendees to walk away with a better overview of the space in which events are used today, and how they factor in the realities of existing approaches and landscapes.

A

Welcome everybody, and thanks for joining my presentation today at the async API conference and I'm really happy to be here. It's called patterns for event, driven apis and the way I came up with that topic was to look at how people are using apis where they come from and maybe different ways and how you can look at events and how you want to use them in your apis and in your overall API landscape before I get started. Just a very brief introduction about myself and what I do so.

A

My name is Eric will I work for x-way, so we are a company producing API management software for mostly large organizations who are interested in managing their API landscapes.

A

One thing that I want to pitch very briefly is when Corona happened, I pivoted a little bit to also producing videos, because I was doing a lot of conference presentations and presentations with clients and I am keeping that up. So in case you're interested in API content check out. My channel you'll find me on YouTube, just search for my name: I, publish interviews with people and and sometimes just about topics that I find interesting and I. Do all of this.

A

As part of my work in the X-ray Catalyst team or a small team, and our goal is to help organizations to use apis in a better way and oftentimes. That means to talk to them about what do you even want to do with the apis right, so not so much to dive into the details of in the good old rest days right, which HTTP method are you using, which maybe is an interesting discussion to have but oftentimes it's more about?

A

What are you even trying to achieve, and is that good direction to go in, and that's also the motivation for my presentation today? So one of the things you could ask yourself is a little bit and I'm showing you here, the the hype curves that I've shown I'm sure you've all seen right. Have we reached Peak event? Events have become something that is being talked about a lot and that's good, and um it's something. If you just look at the API space, then you could say it's just another kind of hype curve.

A

That's happening right, so we have soap, which maybe was the in the initial wave of API popularity. Then rest came along and said: oh soap is not good. You should all use rest, then, for a little while we had graphql came along and everybody said: oh graphql, it's the new thing like it'll everything's, better in graphql, and now we kind of have the same thing, maybe for events and I'm really not trying to diss any of these things.

A

I think these are all good contributions to how to design apis and how to make sure that you get the most out of your apis and the consumers of apis have something useful to work with. So I think the more interesting question really is not so much have we reached Peak event, but have we reached Peak API right? Is that something that happens at some point where overall, the popularity of the API space will will be reduced and I think that's definitely not the case, so I think the API Community as a whole.

A

As well as those individual, let's say, sub communities like event space and graphql space right I mean we're all participating in a environment that I say is still very robustly growing, and you can kind of look for any study that you're interested in I just picked one more or less at random, after Google Googling with the API Market going, and you will always see that the predictions are the API Market is growing at a very robust and, um let's say long-term projected pace, and the reasons for that I think are fairly obvious right.

A

One is that more and more things are getting digitalized, so we just have more and more things that become available in a digital way and I.

A

Think we also see that more and more organizations are understanding that when they use apis, they can start doing things that they couldn't do before, so they either can be more competitive or they can just at least not be outperformed by the competition and therefore, in my mind, just looking at the API space, we definitely still have a a good path ahead of us, regardless of maybe what the next hype may look like, but one of the things that I always encounter- and this is a little bit a motivation for my presentation today- is there- is sometimes this idea of just by using apis or maybe just by using events right magically great things will happen, and that's not the case and I have those two lovely people here that um I'm I'm watching the comments.

A

So anybody who recognizes those people in the movie this is from please let me know you may know it it's a silly movie, but it's two people who are not overly bright. Very nice though, and um that's just the example. Sometimes what I see in organizations that they say?

A

Oh, when we use apis wonderful things will happen right and that's not necessarily the case, so you have to make sure that you actually use the right API so, for example, that you use apis that actually drive your business goals and not just some random apis, some very technical API.

A

So so there is a I think, a need for us also in the API Community, to better understand what people really want to do, and one of the things that I really liked that in my recent videos was with Stephen bungay, so Stephen bungay is a Management, Consultant and I loved his way of putting how to make apis or anything else.

A

For that matter, more successful and I'm sure you will never forget this once you hear that, so what he says is you always have to ask the Spice Girls question and the Spice Girls question is tell me what you want. Tell me what you really really want and I think this is really important to keep in mind, because in the end it may be slightly controversial claim you could make is to say nobody really wants apis right. Apis are just a technical thing: nobody really wants apis.

A

What people want are to get things done with apis and apis help them, but we also have to make sure that we understand what people really want to get done with apis instead of just shouting at them, you should use rest, you should use graphql, you should use events right and and I think this is always a very good thing to keep in mind that apis are just something to get stuff done and I think in the event space.

A

There is a similar thing that you can get different things done with events, and this is what I want to do today is just to talk you through different patterns of how you can use events and to always be mindful that, if you're using events, if you're using environments where people say maybe we should do event driven things right, always also take a step back and ask why, like? Why would that be a good solution?

A

Maybe why would that be a better solution than the one we have or the one we, alternatively, consider and just think through? Is that really where we want to go and the reason why this is an important question is that events cover a very, very wide spectrum right. There are very, let's say, simple and I would say, ephemeral events where you could say these events are really not all that important, so to speak.

A

So I spend a lot of time in the internet of things and web of things, space right and there are things such as sensor, readings right. So, for example, if your temperature sensor, then it may emit regular readings like every minute. Something like this.

A

It is interesting, but it also really is not such a bad thing if you miss those readings right. So these would be events where you say: Well yeah, if I listen to that sensor and I get the latest reading. That's fine. I know what the room temperature is, but if I miss that reading, not such a big deal right so so that would be not so important event.

A

Other events may be different, so, for example, there may be events where already you would want to have some historical data. For example, if there is a sensor that measures when heating is going on and off, maybe that's something where you really want to know even looking back right, when did the heating system turn on when it didn't turn off so that you can maybe monitor energy consumption or whatever it is that you want to do, and then there are even other events that are extremely critical right.

A

So, for example, if somebody withdraws money from an account, that's a really important event right. If you forget about that event, um the whole accounting system won't really work anymore. So so this big spectrum of events, I think, is important to keep in mind that if we talk about events, that is just something that okay, it's an event that happens, but these events can have very different relevance for a business and I.

A

Think that's important to keep in mind and what I want to do today is just walk you through those different patterns of things you can build and different approaches that people are taking just to give you a little bit better idea of what are we even doing here and maybe not so much just about one API, but also what are we doing for the API landscape? What is our approach to using events, and that is something that may be interesting. So, let's look at this spectrum of patterns for event apis.

A

If we start with a very basic idea, then we can start with saying at the very basic level. An event based API is just an API that reverses the control flow of our typical request, driven apis.

A

Instead of waiting for a request coming in from a client that API will emit those events, there is a way of how you can listen to those, and then you get notified of the things that happen. That's kind of the very fundamental and basic definition of what an event API looks like, and it's relatively limited because you have to be there listening. Why? Those APN API events happen, but it's useful, and there are some protocols and I'll just give you some examples um that kind of originate from that space.

A

They also do more things both of the examples that I'm giving you so here we'll be talking about. Amqp I'm also be talking about xnpp uh mqtt, sorry didn't pick xnpp, um but the main thing that I want to talk about here is that these are kind of event. Delivery protocols right so amqp is a protocol that, if you as a client, Implement amqp, it's a way of how events will be delivered to you.

A

Amqp originates in the financial sector, so it was something where organizations wanted to have a protocol. If you have trading data or you have a stock, how do you call these stock.

A

Quote coming in at a regular rate right, these may be information that you wanted to work with an amqp specifically try to make that very efficient because, as we all know, for financial information, sometimes the efficiency of these delivery mechanisms is very important because it's time sensitive information. So you want don't want to get delayed notifications, and it's just one example of a protocol that in that space and that example, came from a specific area, the financial sector.

A

um But in the end it's a protocol, that's designed to deliver events to Consumers. It also has in there this ability of something that um calls about exchanges. So it actually has more um complexity in it. But we'll talk about um that a little bit later. So, let's ignore that for now. The second protocol that I want to mention here is mqtt.

A

You could say it's similar to amqp in some sense and this basic idea of it's a protocol that was designed to deliver certain messages to from producers to Consumers like amqp. It also has an idea in there of kind of message. Exchanges. Let's also ignore that, for now the difference here is just that.

A

Mqtt comes from a very different background, so it has been designed as a protocol for very low footprint devices, so something like those temperature sensors I was talking about, and the idea here, then, is that you can deliver events through that protocol as well.

A

But if we look at those two protocols right they're, just at the level we looked at them for now, they're just protocols that deliver an event from a producer to a consumer. Now this is a little bit tricky, sometimes because when the consumer is not around, you may miss events, and when there are many many consumers, maybe the producer gets overwhelmed by having to service all these consumers. So a pattern that you see a lot that is used in event.

A

Architectures is to avoid this direct connection from the event producer to the event consumer and instead have some kind of event cue in there right. In that event, Q then, is a piece of software. You can call it a message exchange or a message broker or an event queue.

A

There are many names how you can use that, um but at this level, when we still just look at one producer and kind of one consumer or a set of consumers connected to that one consumer, it's mostly all about offloading a load from the client, and it's also making sure that if that is something that you want to support in that scenario, for example, even Q will store events and will allow clients to catch up reading events, even though they may already have passed in terms of when they actually happen at the producer side, but they can still consume them because they are still present in the queue right.

A

So we have a little bit of buffering in here and, and one thing you see in API environments and API Landscapes is that this is not something that every API producer should do individually. So this is something that you should support by, for example, providing some infrastructure that API developers who are emitting events can then use to emit those events to that queue instead of having consumers directly connecting to them.

A

So if you're looking for this kind of pattern, then it already makes sense in API, Landscapes and API environments. To say this is a supportive thing, so maybe, for example, we'll give you something like rabbit and queue so rabbit and queue which actually is based on a and QP would be a piece of software you can put in the middle. There use it as such an event queue and make it easier for the developers of event apis to make their events easier to read.

A

It could still be something where it's just a one-to-end relationship, so you're, saying I'm, just a producer I want to have any number of producers, uh consumers what's an effective way of achieving that, and you can say well, you can just put in a revenue queue in there and um if that is something that you want to do, then maybe providing that as supporting building block in your API landscape would be a good idea. So this already kind of gets you a little bit further along those patterns to a stage where your API delivery.

A

Your event, delivery of your API becomes more reliable.

A

Foreign, then hear is not just doing this, but kind of going the next step and saying we have many apis that emit events, and we actually make that multiplicity of apis and events all available through one shared event broker, and in that scenario Now you kind of decouple the event production from the event consumption quite a bit better or more, let's not say better more, and that is something that you see a fair bit right. So this is also something you can do with with Revit mq or it's something you can do with Kafka.

A

It's something you can do with many other pieces of infrastructure you can put in in there, and in that case you might have something that a lot of people would say an event driven architecture, saying that this becomes an important part of your overall landscape design. Right, so you could say now: I'm not just giving this individual piece to Developers for the individual apis and really saying hey our API landscape uses events as a first class citizen. It's an important thing for us to manage as part of our API platform.

A

And now what happens is that producers and consumers have an end-to-end relationship and what becomes more important and that's I, think the architecturally important part here is. You now have to think about how do I even know which events can happen on that broker right, because all the events that can happen on that broker?

A

They are the the um the combined set of all events of all the individual apis, and that can be quite many, and each of these apis can emit multiple events and and then the question becomes, how do I even find out where to find those things, and then you can go into places like API marketplaces. Maybe you have an event marketplace where you can find out which events are being emitted? That becomes something where you have to think about that, like at the organizational level.

A

Another thing you might have is that you also might think about which apis do I use internally and which do I use externally. So, for example, if you would say, I am having an event-based architecture internally, because that's what I want to do.

A

Is that still something that you want to support for externally facing apis, so for apis that you design for partners or even for the public, and in many cases you will see that those apis are using different mechanisms, different protocols, so they may use things such as web sockets or server send events, and in that case you really get into this area of thinking about. Do we have guidelines that talk about this and say for internal events? We always use Kafka or revident queue for external events.

A

We always use web sockets or something else right, and then for both of these things. Ideally, you would have infrastructure in place that makes it easy for developers to develop apis for that specific target audience. So, for example, if you have event driven apis that go to Partners or the public, you would have some websocket infrastructure and there is an I would say an emerging emerging set of products there. That will help you to build up a websocket infrastructure and there can be other things. So.

A

Another thing that you may have heard about as part of your API guidelines could be to say: we support a pattern that at least we sometimes use which is called C chorus. So cqrs means that you use you design an API by having different API variant or whatever you would call them for commands and queries. So cqrs stands for command and career responsibility. Segregation, it's kind of a little complicated thing there, but it again is something that maybe in your API landscape, you would say this is how we design apis right.

A

So, for example, for commands for making things happen in an API, we actually use a rest style API and for events for things that happen within the API and get um get message to Consumers. We use rabbit, M, Cube or Kafka, or something else right, and if you do that, then again, that is something that your guidelines should clearly stay and then there should be infrastructure that helps with building that, so that you can actually make this picture happen across all of your APL landscape.

A

And um the last level I would say of how far you can go with API with event apis or with events. Let's say this is something that you may also have heard about it's event, sourcing, so event sourcing. Is this idea that instead of treating events just as these more or less ephemeral things, you build your architecture really that they become the source of Truth, so that your system is designed in a way that you don't keep state in the database?

A

You really keep state by logging all events, and then your application state is the result of all these events right. This is, for example, how traditional banking systems work. There is no database that has your account balance in it or, let's say that's, not the source of Truth. The source of Truth is all the interactions you had with your bank account and whatever the end result of all these interactions is that's your account balance so event sourcing. Is this interesting idea of saying I'm going all in on events?

A

Events are the only thing that really matters for defining how my system actually behaves and how my system state is defined and I. Think it's an interesting approach, but the one thing that I would like to point out is that I think this is a good approach for certain sub systems.

A

So, for example, if you would build the accounting system of a bank, maybe you're going in that direction and say we're using event sourcing as as our system design, what it typically would probably not be a good choice for building the whole Bank right, because you still have a lot of systems in the bank that are around I.

A

Don't know, loyalty, program, customer information mortgages this and this and that right, so a lot of other subsystems that make up the bank so event sourcing, even though it's interesting I would say, is mostly confined to certain subsystems to certain spaces within organizations. So it is a choice that you can make, but it is one that typically would not be the the approach that you use for your overall API landscape foreign.

A

Patterns for um event-driven apis, what I wanted to do is just give you this idea that events can be used in very different ways. So you may be familiar with this idea of Maslow's hammer and saying that. Well, if the only tool you have is a hammer, every problem looks like a nail and and I I really love this picture, but it's I think it's also an important concept to keep in mind that also, in the event space like in the general API space.

A

There are many different problems you can solve with events and they have wildly different properties so always think about. Why am I doing this? How does this translate into the best way? How I should do this? Maybe just design a simple single API? This way maybe put a component in between. Maybe you really have like this event, driven architecture or go all in with event sourcing, but really think through that, because, in the end, what you're trying to do is Spice Girls questions right? Is you shouldn't try to just use the apis?

A

You should try to solve problems and the API that you're using and the event pattern that you're using should just be a good fit for the problem. You're trying to solve so don't be too fundamentalistic about that. Try to be aware of your doing, try to take step back and look at, am I, doing the right thing and then I think you'll be on a good trajectory with your events and with that I'm done. Thank you very much.

A

If you would like to look at the slides, I, just tweeted the link to the slides in my Twitter feed, so you can find it there. It's also here on the slide. um The slides are online, so they are web-based. You can just look at them in your browser. If you want to find out more about more what I'm doing you can find me on Twitter, Youtube and Linkedin, and um that's it for today. Thank you very much.