AsyncAPI AsyncAPI Talks, 17 Sep 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Going AsyncAPI: The Good, The Bad, and The Awesome

Description

Ben Gamble, Ably

In this talk, I’ll explore the good, bad, and awesome aspects of building Async API into our open data hub. As advocates of open source tools, it is our mission to simplify the collection and distribution of streaming data by taking care of everything under the hood, including business-to-business exchange of data and “last mile” delivery to end consumers.

Beginning with a discussion on open API, I’ll walk you through our deliberations, and why we chose AsyncAPI, how it helped us and what it cost. I’ll tell you how we improved our tools to make use of AsyncAPI specs; how we managed the gaps in the specification; along with the benefits.

AsyncAPI spun out of open API with a goal to solve some of its shortcomings. The initiative set out to standardize asynchronous and event driven APIs across the industry. With the proliferation of IoT devices and the connectivity promised by 5G, having standard ways to connect has become more important than ever.

AsyncAPI has been added to every product we host on our open platform. Why? Because we believe AsyncAPI is a good standard for open event based data/APIs, and we want to support a proper way to carry out code generation and validation - with specifications that make sense.

Thousands of software engineers around the world have provided code, documentation, tests, or other improvements to open source projects. With the help of initiatives like AsyncAPI we want to help people liberate their data by tackling the common challenges they face when trying to distribute it.

A

Hi everyone hi, my name, is ben gamble and will continue to look on going async api.

A

The good, the bad and the awesome- that's somewhat singers jenny through our implementation, in all the sync api, on our open day, to help.

A

So, just starting off with a bit about me.

A

Search software time head of the operations at.

A

A real time r. a. real time, pups of unspoken.

A

Until person, who makes everyone read the phoenix projects by like cool things and I've very rarely, possibly p. I. by which I don't stop poking.

A

Out and gently, which initially is like an open api, this makes it really awesome.

A

So what is ably expose- and why am I- why am I talking about a list so ably- is a globally distributed data streaming network? We do pops up.

A

We do lots of lots of cool stuff, but that's a wall of text, so yeah pops up and you know we allow people to publish.

A

And subscribe and bidirectional channels from almost every device you can imagine to almost any other device. You can imagine.

A

We are really fast, which is possibly our biggest selling point.

A

Which is that, like we have low round trip times via all.

A

The usual class around can, you can imagine through another bank, planes of various cloud service runs.

A

And we really do a lot to try and maintain the reliability and ordering and availability through. All of this.

A

We can do some rather fun things like you know the usual five plus nines of reliability.

A

Of delivery, but things up, but eight nines of reliability when it comes to data integrity.

A

And all this is basically based around the idea of having a series of globally distributed costas in.

A

Multiple different data centers, and on top of that doing the whole age thing. We have a two hundred prisons points and we always try to deliver everything.

A

We can get through the network and kind of like this was then that's just who we are.

A

And waverly is answered thanks for the question is, like you know these managing these api's hot, because as a world out here at the school, this conference is about api's and all the hard to manage, and the answer is yes. Yes, they are because we won't be able to use them, and you know kong, eight of us nordic.

A

And swagger have invested a ton of money to make these things better and they make a little money, making them better for good reason. It's really really hard to manage these things.

A

Right and you know it's so important- we actually got an initiative out of it. Openapi.

A

Has continued to flourish of the is and has really driven people to create the right sort of data. People can read that can generate documentation for itself and is incredible into all the tools.

A

To allow you to actually generate code and start using api's, faster and better, so the real question is: what happens when you need.

A

More than just rest, the rest is great. It stateless. It's.

A

Very much driven by you know wanting a data you push, people.

A

And you get what you need, but when you wanna go more than that, we have a problem.

A

If you want to support real time data on a support day to say.

A

Synchronous, you need more than just like rest api, and this is where it gets a bit complicated.

A

Because we enter the fragmentation of protocols, we have everything from devices that stream on roll udp, two.

A

Things just use rest themselves and everything in between all the various tcp based protocols.

A

Like mqtt, the h. g. stand, other standards.

A

Like sse and some other things like websockets and it gets complicated.

A

Because they're all a lot of languages- so you know openapi is brilliant because it of course supports all the languages you want to use rest requests on.

A

But when you have multiple particles and multiple languages, you start multiplying up the problem.

A

And particularly from where I sit, the problem comes from the following thing: we will help, we wanted data to be open and we want to kind of bright for this next wave off the event driven ecosystem.

A

And for us it was basically we realized that we needed to make sure the data was available. The people see what happened and also it because it should be free, so deeply held is a literal collection off.

A

Of streams of open source data, which we provide free of charge,.

A

To anyone who wants to use them get an account subscribe and you'll get a feed of data, and it's all well and good, but you know the model of its arc. We have.

A

Fifteen data sources, each with a number of channels and channels are not like us to topics in able, even after the so I'll use the two terms.

A

Interchangeably if you're familiar with pops of standards are so each of these things has a variety.

A

Of channels and all of which you know some of them going.

A

To the hundreds, the one there from london to schedule has electric.

A

Channel hurt you've station.

A

This means that, as you, try and actually a new right all these channels to work out what.

A

Is actually that you have a serious problem because it's a lot of work and it gets very difficult to manage.

A

So the question then, is like you know: it worked why we can't that is open free and you know we wanna make sure this data can get to the people.

A

Because it's not really relevant if it's all behind closed doors, hit the pay for everything. That's not what we want to do this. We wanted to make sure that, like you, know the bills, everyone and, of course it's a lot of articles. We have our own personal.

A

Quote, which is effectively in the real time world.

A

A websocket based vertical but we'll says what a ctv we have allow for us to see and then to t., t. or is a subset of it, and we support nine languages across our platform with.

A

Our suitcase, but you start multiplying these things together,.

A

Well, you know fifteen sources an average of.

A

Ten also giles purcell's nine languages.

A

Times four plus particles, you start getting into some really fun geometrical progression scaling, and this is where we kind of have a problem.

A

Because, without doing something.

A

You know what can we do to manage it, and this is where we kind of had to stop thinking.

A

Like what else is that now, the thing is right.

A

As with everything, you're, never alone,.

A

Everyone else has had a similar problem. The full everyone who has presented a few has a public facing api.

A

At some point as thought, what can we do to manage it? An open api is a very common way.

A

To deal with the state's arm, was it perfect beginning not? But it's getting to the point where it's just you, because it's that's brilliant, because it means there's now a standard which people can understand.

A

So what happens when you want to do things, and not just rest and yeah? A heady brew, crazy, came to the rescue.

A

So, based out of ideas that hitch.

A

Combined with thoughts from like the opening night, the pid stiff and also like way, rommel's tying to really sought specifying.

A

Proper machine, readable specifications.

A

How came async api- and it is quite the brew of interesting.

A

Things I call the brew in this ticket.

A

Second sense, because it is a genuine mixing of things.

A

It's the idea, saying we must be able to have more than one.

A

Thing in more than one way from more than one provider, none of the mock. None of these things really stand all good enough as a stand alone.

A

So you end up with what can only be described as a lot of different things under the async api bana.

A

These are everything from what you would expect from.

A

A documentation, generators to code generators to verify is.

A

And other things, and of course it comes back to a machine, readable.

A

A

You know this allows for yeah a pretty dry martini of code machines, love it and they will use it any day of the week.

A

But the really valuable thing is: it allows the consistency.

A

The one big thing about async.

A

Code amply with message based systems is.

A

You don't necessarily know what the data is.

A

B. overtime, because you you don't know what it is until it receives it, and it can be different. Every time messages can contain, by definition, variable data.

A

And what you can do with async api.

A

Is you can actually specify the structure of this down to the typing? Not all types of supported.

A

Logically, but you know everything from indigestion, e. nam.

A

To strain to float are actually supported and that full verifiable type it is up to the generation how this is actually used.

A

What you do when something's not conform to specification, but it does allow you to test it, and this is where we get to the kind of critical value.

A

Proposition for us was we needed to be able to both.

A

Our gift somebody away to be data.

A

And given the way to know the danger is correct,.

A

To verify its correct and discuss various things.

A

But also because we want to shed any day so we can get hold off. If you give us open data, we will take.

A

You to share it and we want you to be able to give us a specification.

A

For that data, so from what we want to build a verify, the dangers good before we stream it and specs allow us to do this.

A

So then we get the thing of like what are we actually trying to do so we are. We need to build generations so easy pi r. as it currently stands, has a series generations.

A

Full the known open the company known.

A

Open async protocols- these would be things like anti t., t. s. s. c. r.. The problem, of course, is that we have our own protocol.

A

Which is being built after much research in much time.

A

And a lot of money invested, try to make something which is ridiculously reliable problem is so big complex. It allows.

A

For a very wide variety of things, including the ability to rewind through time to get messages alien, you connected instead maintaining synchronous t. to maine and presence on other such advanced features. You'd expect.

A

From a fully fledged pop up pops up type system.

A

The problem with that is most like mqtt doesn't maintain any of these things. Most. These.

A

Open standards don't have a concept of presence.

A

Which basically means that, as you, we try and went through the specs, we found that, to be honest after the clicks backs a lot from you.

A

Aren't because it gives you too many options in some respects, all these options are very valuable, but we came up with things which are. Generators are a bit.

A

Lackluster, what the actual specs supports.

A

Next thing we try to do, which was in some respects, even more fun was actually generate. Specs.

A

Because if you have a machine, readable spec is machine readable, I can generate code logically, you can, you know, be data and generate spec. So we tried this once again.

A

It actually worked, which surprised us this is the actual specifications are quite rigorous and you can literally yeah listen to a data streams along.

A

For a while and gain a pretty reasonable coverage that helped.

A

The problem is, there is no guarantee of completeness.

A

Because all the way, both our network.

A

And m. t. t. t. work in this case.

A

Are you have no way ahead of time to know what channels are going to be sent along unless you have pre defined that free to find from the source.

A

If the not predefined from the source, you can't guarantee that you have all the information.

A

Not to say we're, not gonna, try and continue down this road, it's just that they will be known in complete one day. Maybe we'll have the ability for all effectively expect to be transmitted on the wires well,.

A

And that can be then be reading for our system. At least it works pretty well, but it's a little bit off kilter off the way. The actual kind.

A

Of general specifications work, and it relies on the fact that our sdks can do quite a lot of magic.

A

Behind the scenes, so when we kind of go from there is.

A

Like you know, how about all the people specs.

A

So we have an open hovers previously mentioned and we are really keen to. Ghana finds much open source data.

A

As we can, as we can get a bit on that.

A

We have the data sources you saw earlier, they range from rest api's. We behind the scenes, fall to mqtt sources to some other things, and general thing is that we've basically spent awhile looking around for other people's specifications and then have them ingested in and have those products, because we want to as much you can do this, because we think it's useful what we want and we want to be open.

A

But what we ran into two new kind of big problems.

A

The first one is up while const, and the second one is to limit as wild cards is a p. t. t.'s way of subscribing to multiple.

A

Channels of topics, at the same time, you can subscribe to all devices under a certain name.

A

At the same time, and get all those channels being received.

A

A

and what that is, that is really powerful.

A

It's not something we in our network directly support.

A

Our ingressive by defaults it all out and means all things get broadcasting correctly and it handles quite nicely. But by default we also directly absorb mqtt.

A

Thank you, don't go fired the aggressor. It means that.

A

We have the problem with channing spaces, don't match them, as I. dilemmas don't match, and this is actually goes a bit deeper because cross all the different ways. You can send message based.

A

Data there isn't a standard delimiter, there isn't a kind of lingua franka. That says this is the delimiter or channel or namespace all.

A

The limits of of a topic that doesn't actually exist, there is no one specification, it says this is a delimiter.

A

And then define it in five different particles.

A

That doesn't exist anyway. Instead, every single real time particle, I've seen.

A

Does something slightly different, nearly always there within a certain kind of a ballpark?

A

But you know we support rec exxon out channels.

A

Which means you can send a red accent: it'll get the right excellent channel things was back.

A

The problem is some of those records. Things are actually the limit, as in other s. vacations,.

A

So we have this kind of problem with that, because we can't directly convert to a.

A

True, the generic there's, no g. tt three equivalents.

A

Of real time topic management, so we have the fact that that doesn't exist.

A

So we have some interesting ways to work around. It.

A

But they're not perfect and the last thing, which is in fact cited sensors because they think api's new.

A

I'm young they're all actually not many specs out there.

A

I will be proved wrong by the way and it is any any kind of like open source data which has a spec attached.

A

And we missed it, just send it to me.

A

We will cherish immediately as fast possible, be bought these things and we will host up.

A

But we found a few are, but they were they had the usual issues. All the early steps in this direction often incomplete.

A

Not the most best use cases don't allow us to really demonstrate the power of the actual specifications self. I'm.

A

Handed out what has what has been done as far as the next real question, so we've learned a lot and you know we.

A

Had to dig a lot yet this is a young project and it has been built remark of his b. delivery. Marble.

A

Rega- and there is a lot of tools for it and the with the learning was required- was pretty vigorous.

A

He sat out with a high minded expectations all of.

A

Which I bunch things and not much work, but we were pleasantly surprised by the amount of things that.

A

Actually worked, I'm like reading.

A

For us back and just generating our own coat work surprisingly well, generating as a c. code from specs we've written, our own systems worked.

A

Perfectly first time the point to where we were all surprised.

A

Which is wonderful because it works in all the different languages.

A

We wanted to work in.

A

So there was, we went through it'll, be kind of you know.

A

Worked out we had to build so we build generators for the ab stk.

A

Because it is quite a powerful, as he can do, a lot of magic to make it a bit better.

A

But we we saw going further because we can start to do things like defining the way mission books based on the channels.

A

Classified inspects, and so, as we kind of went forward as we build a lot tools and we aren't releasing these on- I get help in the next few weeks out, because right now they are, some of them are a little bit too alpha to be tried to be had a treat our fight so they're in a state of current testing hopping.

A

And the idea is to release some of the next few weeks are fully open, of course, on.

A

The basis of the license we can anyone to use in any circumstance.

A

And also like, we wrote a lot of specs.

A

And I do mean a lot of. We have written a spec for every single product which have you have on the hog.

A

That thought there are various levels of readiness so as we found that something didn't did not work.

A

We actually upgraded our systems behind the scenes. We actually now you specs behind the scenes for a lot of our actual immigration, because it allows us to type check when we get sick.

A

Natalie status was a perfect every time, so we now use the fact that we have you used async api jenner, we've written a sync api.

A

Spec, because some of the data sources, we then use that spec to verify the data we get from the souls.

A

And it's actually improves the quality of our actual product.

A

Now it's interesting because, like we've now put up at least a few specs on hub has said.

A

As time goes on, we'll be all the our products will have a spec.

A

And hopefully, even the ones which are added, which don't come.

A

With specs will have generated spec for that project is still in flight.

A

But we hope to be reporting back on that's as soon as possible and.

A

What gets really interesting, then, is the fact that.

A

We can try and do is effectively rita source.

A

Generators back from it who's. This backup allow someone else. Just ingest updates, also press the button and the tree on the website be receiving.

A

It this is the stream which at no point was not machine. Read we eighteen people will accept the point now where.

A

Things without a single line of code can indeed do all the magic.

A

So the final thing, if you things for sure you some stuff about this- is the shots.

A

So how's it all cool things. It's all about standing on.

A

The shoulders of giants- and here is mine- tractors, compatible team awesome.

A

A friend mendez big shout out to you for actually sharing the whole the sync api efforts.

A

For taking the risk to go out and build it and for being a wonderful results,.

A

And genuinely being really supportive and then tongue camp who's been my wonderful engineer.

A

He's been driving a lot of this forwards and putting up my madness and ideas about how to how we should try.

A

And do it he's been the mastermind behind our open source hub.

A

For since its inception- and he has done a wonderful job, making sure that we actually have some specs for you show each day, so I'm still hopeful screen now actually take you straight to our website. So you can see at least one of the specs.

A

He said so this is a. This is a.

A

Real time, city transit data from somewhere in connecticut, it's a dcfs back, which is a a number of missiles, specific cation for travel later are. These are the various languages we support.

A

On uresti case, but also by.

A

At t. v. a c. n. n. e. t. t..

A

But the point is that you know what no one wants type code that I have to want, that lovely dry.

A

Martini effect, so here is the async api spec.

A

Here's all the definitions for all the various points are you can subscribe using q. m. q. p.

A

Rast, thank you t. t., secure or not.

A

M. and that's the thing like, we can specify the channels you specify a rate strings.

A

And currently, if anyone generally use you have to control, the kind of message is marissa.

A

Come out because these types are hard enough to allow backed.

A

And that's something that has been missing from from both our own systems and every other real time messaging system. I've used.

A

In the past, outside of something no horribly hard like post protests, is that.

A

You can't type check stuff, you con beach hundred percent sure.

A

That what you said was what you expect and receive. Was it exactly what you expected, and this gives us a way to do it and he's quite wonderful spectacles long, but machine readable means machine, readable.

A

So you don't really need to read this. You could documentation field agenda.

A

And everything else will just work.

A

So please do what you can do this. I would love to see some demos. We have a series of our own demos based on this data.

A

Which I liked running on not books, maps and showing the kind of usual fish tank effects, but would love to see what you come up with anything which is cool, will cost host for you to see it and run it to show.

A

This is the world conceit and stagers, of course, reviews. So, please sign up and download and just plug in and see what else.

A

Tell us now I have some time for questions.

A

Sorry, the screws will sometimes going through some questions.

A

Here, yes, risk impossible. I was that is indeed my thinking as well. Patrick.

A

Someone actually beat me to pasting the, u. r. l.

A

the the mqtt specs. I will go to that. Now.

A

Sorry, I hopes through bits.

A

And I will zoom in attached.

A

So you can just specify out.

A

Which they are, we only support saudi arabia version mqtt.

A

Which is our because it's just how our dental work spot.

A

It is complete to receive a little bit different to send.

A

And you know like this is indeed fully open on the code behind the list will be open.

A

Soon so the in the in just for the hub, the suspect spectrum right once actually is complete will be open. The wait to do.

A

Type checking in the end just as old it will be up as well, and also our generators firing vertical.

A

And anything we we have done to the current charges will also be open cut. Most of the cool thing. Is it just really books.

A

As you might think it does, I'm.

A

Just kind of scroll through a fella kind of the fun things here, aw lake.

A

Because the channels all topic definitions are included, you don't need to wait for the inevitable fact of that channel might not exist ahead of time. Now. We think you.

A

Have it with mqtt that's less of an issue, because, while concentrations but with something like s. c. vi, don't know in advance you're not listening to it and that's not useful.

A

That means that you are going to miss stuff without knowing sophistication ahead of time and writing out the few potentially thousand potential channels it could exist, for this is just too difficult so in a combination of up some magic behind the scenes. Lately.

A

Will will slow did you bill for you down to one connection, but also even in the wild? It's really really powerful. No, what.

A

Should and shouldn't be that.

A

So just go back into the seven.

A

Any further questions.

A

If not, then you have any kind of cooler than they do. We think we shouldn't we should share to the world any cool specs. You think we should try reading.

A

Generates more coastal and any cool ideas? What you think we should we should do with open data. Please tell us: we really want to make this thing go everywhere.

A

Because it is free, it is old like you do, deserve the count. Yes, but it is a free account. Are we don't take any details from me to do that, and will we need from all we need? Is people to use this data.

A

Will keep supporting it.

A

Okay: well, if there are no more questions I'll give it.

A

Another minute.

A

A

well, thank you very much for listening. I hope you have a wonderful day and please send us a new puzzles that you have.

A

But what she misses most in school part of.

A

Right, the working in the casket.

A

We actually have a casket supporter ready on scenes. If you want to use it talk to us we'll show you a some catchy stream straight to it. Arm.

A

Signs go far down the shaft. Yes, I think that is everything.

A