►
From YouTube: An Auto-Documenting Event-Driven Architecture Using Async API and Inngest - Dan Farrelly, Inngest
Description
AsyncAPI Conference 2022 - Day 2
4th November 2022
The Async API spec is absolutely fantastic, but what if you could generate this automatically from the data already flowing through your application? If only the data flowing through your event stream could be captured and have event schema automatically generated, versioned, and recorded in a database. This talk walks you through what we’ve done with Inngest to implement this into our system and how we built a tool to generate Async API specs automatically. From inferring event schema, creating an versioned event schema registry and generating Async API specs from this data. With this auto-generated Async API spec one can then take advantage of all of the wonderful tools in the Async API ecosystem.
A
Hi
there,
my
name
is
Dan
and
today,
I
have
the
pleasure
of
sharing
with
you
a
little
bit
about
an
auto
documenting
event-driven
architecture
using
async,
API
and
ingest
here's
a
short
overview
of
what
we'll
cover
in
the
next
20
minutes
or
so.
First,
we'll
briefly
cover
why
event
schemas
are
important,
then
we'll
discuss
why
one
would
want
to
automatically
generate
event.
A
Schemas
I'll
then
dive
into
exactly
how
we
do
this
at
ingest
and,
lastly,
I'll
finish
with
how
you
can
do
this
yourself,
all
right,
so
first
I'll
share
a
little
bit
about
the
person.
That's
speaking
to
you
right
now
again,
my
name
is
Dan
Farrelly
I'm,
a
Founder
at
ingest.com
and
I'm,
the
former
CTO
buffer
I
love
chatting
about
Benjamin
architecture
and
home
restoration.
A
I
just
recently
renovated
100
year
old
home
during
the
pandemic.
So
you
know
hit
me
up.
If
that's
your
thing,
if
adventure
of
an
architecture
is
your
thing
which
I'm
guessing
it
is,
if
you
want
to
reach
out
to
me
after
this
talk,
shoot
me
an
email
at
Dan,
ingest.com,
that's
two
ends
and
ingest.
I
n
n.
Okay,
that's
very
important!
A
So
I'd
like
to
take
this
opportunity
to
explain
what
ingest
actually
is
so
ingest
is
an
event
driven
platform
that
makes
it
easy
for
developers
to
build
test
and
deploy
serverless
functions
that
are
triggered
by
events
without
having
to
worry
about
infrastructure,
cues,
stateful
services
and
to
think
about
it.
The
key
components
of
ingest
are
a
serverless
event
stream
and
a
job
scheduler
that
calls
your
functions
on
demand
as
events
are
received.
So
with
ingest
you
kind
of
get
everything
you
need
for
an
adventure
of
an
architecture
in
a
single
platform
from
schema.
A
Registries
logging,
metrics,
sdks,
tooling
Etc.
So
that's
not
exactly
what
we're
talking
about
today,
but
let's
dive
in
and
go
back
to
our
agenda
and
start
with.
Why
build
event
driven
now,
I,
don't
believe
that
I
need
to
convince
this
audience,
but
with
the
focus
of
this
talk,
I
wanted
to
share
my
opinion.
You
know,
I,
think
that
event
driven
architecture
can
be
beautifully
simple.
A
What
is
the
most
important
part
of
an
event
driven
architecture
I'll,
give
you
a
little
here.
This
is
not
a
trick
question.
Okay,
it's
the
most
important
parts
of
events.
It's
events
right.
This
should
have
been
pretty
obvious,
but
I
think
it's
worth
a
mention,
as
we
often
get
carried
away,
sometimes
as
engineers
in
the
details
of
the
architecture
itself
and
lose
focus
of
the
core
component.
A
A
So,
given
that
you're
tuning
into
the
async
API
conference
today,
I'm
going
to
assume
that
you
know
what
async
API
is
and
I
think
it's
not
too
controversial
to
say
that
the
core
of
async
API
is
the
spec
itself
right.
You
need
to
create
a
spec
for
it
to
be
anything
right
without
the
spec,
it
doesn't
work
right.
So
how
do
you
create
those
specs
I
think
the
first
option
that
most
people
start
off
with
which
is
fantastic
is
to
write
it
manually
right.
You
want
to
describe
your
system,
you
want
to
be
intentional.
A
You
want
to
think
about
what
events
you're
sending
you
can
collaborate
on
this
with
your
team.
Maybe
you
end
up
using
something
like
git
to
collaborate
on
your
system.
Spec
right,
you
debate,
Fields
And,
how
it's
structured
and
consistency
naming
casing
all
that
right
formats
that
you're
going
to
use
and
I
wanted
to
highlight
today
another
option
which
is
to
automatically
parse
the
events
flowing
through
your
system
in
real
time
and
then
generate
the
documentation
itself.
A
So
it's
kind
of
the
reverse
and
I'm
sure.
Maybe
some
of
you
don't
like
this,
if
you're
very
adamant
about
the
first
one
and
there's
pros
and
cons
to
each
approach,
but
I
think
that
auto
automatic
generation
is
underrated
and
I
as
I
wanted
to
talk
about
this
today,
to
highlight
a
little
bit
more
and
say
what
are
some
cool
things
that
you
can
do
to
generate
these
async
API
specs
and
what
you
might
be
able
to
do
in
your
own
system
so
going
from
there?
A
Now
we
get
into
the
details
of
how
we
do
this
at
a
jest
right,
early
on
and
just
chose
to
use
Json
as
our
event
format
for
our
API
since
Json
is
ubiquitous,
and
it
requires
no
tooling
compared
to
other
formats
like
Avro
or
protobroth,
so
nothing
against
those
formats.
Just
we
wanted
to
go
simple,
something
that
you
actually
didn't
need
a
library
or
anything
to
get
going
with.
Each
user
sends
their
event
payload
in
Json
to
the
ingest
API,
using
a
unique
key.
A
So
the
system
enables
each
client
or
service
or
microservices
architecture
to
specify
what
this
version
of
the
event
that
they're
sending
is
you
and
we
have
a
standardized
V
field
to
specify
the
version,
which
is
just
a
string,
and
you
can
put
anything
you
want
in
it.
So
every
time
that
V
value
changes,
we
expect
that
it's
a
new
version,
so
every
single
new
version
is
parsed
generated
and
then
stored
in
this
meta
format
and
The
Meta
format
that
we
use
is
called
Q.
A
You
may
or
may
not
have
come
across
Q
yet,
but
we
think
it's
a
pretty
phenomenal
tool.
Q
is
a
data
validation
format
which
can
parse
validate
and
generate
Json
yaml
and
open
API
compatible
Json
schema.
So
we
chose
this
format
as
it
enables
interoperability
across
many
different
formats,
as
I
mentioned,
especially
Json
schema.
A
A
So
you
can
see
an
example,
Q
definition
here
in
this
slide
on
the
right
in
our
standard
of
event,
payload
format,
and
it
has
our
V
version
field,
as
you
can
see,
which
tells
our
system
that
the
version
has
been
changed
and
it
should
be
parsed
again
and
now
that
we
have
a
format
that
can
generate
Json
schema
right.
We
can
generate
async,
API
specs
right
because
that's
what
it's
based
on.
So
this
is
where
it
starts
to
get
interesting.
A
Now,
I've
described
to
you
in
Broad
Strokes
how
our
system
works,
but
I
did
want
to
share
this
high
level,
generally
simplified
architecture,
diagram
of
how
ingest
Works
in
this
fashion,
including
our
schema,
validation,
parsing
and
the
red
string.
So
as
I
shared
a
couple
slides
ago,
clients
send
events
to
the
ingest
event
API
and
we
immediately
publish
the
event
to
a
pub
sub
topic.
Implementation
doesn't
matter
in
this
just
a
pop
sub
system,
so
those
events
are
then
consumed
by
an
event
recorder,
which
then
in
turn
passes
the
event
to
the
schema
validation
Library.
A
A
A
So
as
our
system
works
is
we
have
serverless
functions
that
them
are
declare
what
events
they
want
to
trigger
them?
It
could
be
one
of
them.
It
could
be
multiple
events,
but
that's
how
our
system
works.
So
when
the
event
comes
through,
the
scheduler
goes
and
finds
those
corresponding
serverless
functions
to
execute,
and
then
it
invokes
them.
So
this
isn't
just
in
a
nutshell:
there's
schema
parsing
validation
registry,
all
out
of
the
box
right.
So
the
end
product
is
our
event
schema
registry.
A
This
is
a
given
example
of
an
event
called
user.logged.in
right,
something
simple,
a
simple
concept,
and
this
shows
the
different
formats
of
the
event
schema,
including
Q,
Json,
schema
typescript
and
a
simple
table
view
with
all
the
fields.
So
this
is
built
into
our
platforms
for
every
single
event.
A
You
get
the
history
and
you
can
see
what
functions
were
called
or
are
associated
with
that
given
event,
and
then
you
get
every
single
version
of
the
schema,
so
you
can
go
through
when
it
was
first
seen
what
you
call
the
first
version
and
what
the
format
is.
So
that's
really
useful
of
knowing
about.
When
did
things
change?
When
did
things
update
and
then
all
these
types
are
generated
and
we
can
use
these
meta
formats
to
now
generate
even
things
like
typescript
to
be
used
in
our
typescript
SDK,
which
is
exciting?
A
What
I
think
is
really
cool
about
this.
Is
you
know
all
this
data,
of
course,
is
coming
from
somewhere,
so
it's
coming
from
our
API,
which
now
means
that
it's
portal
right
this
view
that
you're
seeing
right
here
is
only
showing
the
latest
version
of
this
given
event,
but
at
the
higher
levels
you
can
click
through
every
single
version.
So
we
think
this
is
pretty
useful
and
our
users
get
a
lot
of
use
out
of
it
and
we
use
it
internally
and
love.
It
I
think
the
end
result
is
a
fantastic
match.
A
So
let's
see
how
the
ingest
system
comes
together
with
async
API,
so
each
user
workspace,
which
is
separated
into
production
and
test
environments,
has
its
own
registry,
its
own,
separate
it
registry
and
each
event
requires
a
name.
So
these
are
great
things,
as
you
know,
the
async
API
spec
now
together.
This
now
enables
us
to
fetch
Json
schema
from
the
ingest
API
and
use
that
schema
to
easily
build
our
async
API
spec.
We
have
Json
schema
you've,
seen
where
this
is
going
to
open
API
compatible
to
simplify
this
part.
A
We
wrapped
all
this
logic
into
a
simple
npm
package,
so
this
is
just
something
small
that
you
can
use
very
quickly
to
get
going.
So
why
translate
this
to
async
API?
That's
just
a
little
pause
question.
Before
we
go
further.
We
already
have
q,
we
have
Json
schema,
we
have
typescript,
we
have
that
table
View
and
why
translate
it
I
think.
Of
course
the
key
benefit
is
to
allow
users
to
tap
into
the
async
API
ecosystem
I'm
sure.
A
There
are
many
other
talks
today
and
across
this
conference
that
are
going
to
be
talking
about
various
parts
of
the
ecosystem,
including
tools
like
code
generation
and
documentation
and
sharing
system,
architecture
and
whatnot.
That's
the
value
in
embracing
this
format,
so
so
one
feature
that
we're
also
looking
to
support
in
the
future,
which
is
kind
of
the
reverse
of
what
I'm
talking
about
today
is
where
you
can
take
that
async
API
spec
and
then
push
it
using
it
as
a
format
to
manage
your
skimmer
registry.
A
So
you
can
update
your
versions
and
you
can
push
it
as
a
kind
of
a
management
format.
I
think
this
is
something
that
some
tools
are
doing
well
and
but
there
are
still
a
lot
of
gaps
with
managing
event,
schema
and
enforcing
it
and
whatnot.
So
we
think
that
as
I'm
sure,
a
lot
of
you
all
today
think
that
async
API
should
be
the
format
to
be
able
to
standardize
around
these
systems.
We'd
love
to
really
lean
into
that
further
and
extend
this.
So
would
love
feedback
from
you
all.
A
A
Otherwise,
if
you're
not
it's
just
a
little
package
manager
for
JavaScript
ecosystem,
so
you
can
grab
the
ingest
COI
quickly,
log
in
to
your
account,
where
you've
been
sending
all
your
events
and
then
you
can
use
npx,
which
is
a
little
npm
extension
command
which
allows
you
to
install
and
run
something
very
quickly,
and
you
can
just
run
npx
at
ingest,
async,
API,
and
what
that
does.
Is
it
uses
your
login,
your
current
session
on
your
machine
and
it
grabs
all
of
the
Json
schema
from
our
API
and
then
it
iterates
through
all.
A
There
gets
the
latest
version
only
currently,
and
then
it
generates
the
entire
spec,
so
it
generates
it
in
yaml
and
formats
it.
So
it's
almost
instant
because
it's
already
been
generated
right.
We
already
have
the
Json
schema.
We
can
iterate
through
every
single
version,
so
we
can
look
at
latest.
We
can
look
at
anything.
A
It
can
be
pretty
Dynamic
and
pretty
flexible
and
I'd
love
to
hear,
like
I,
said
more
ideas
than
this,
because
it
could
be
really
cool
generate
this
really
quickly
and
you
just
need
this
one
command
and
you
can
on
the
Fly
generated
so
really
in
theory.
Anyone
on
team
could
log
in
and
pull
the
spec
automatically
from
this
event.
Schema
registry.
That's
it!
Now,
with
your
generated
spec,
you
can
use
all
of
the
great
Tooling
in
the
async
API
ecosystem,
to
generate
code
documentation
and
share
across
the
entire
organization.
A
So
I
wanted
to
share
this
screenshot
of
just
this
is
my
ingest
demo
account,
and
so
this
is
all
like
a
bunch
of
test
events
that
I
have
in
my
mini
Venture
and
architecture
system
and
it's
using
async
API
studio,
and
what
I
think
you
can
see
here.
Is
that
these,
because
they're
all
published
events,
you
can
see
every
single
event
that
you
have
automatically
generated
in
the
format
that
maybe
many
of
you
are
used
to
or
want
to
or
currently
learning
about
and
wanting
to
embrace.
A
So
this
is
just
one
example
at
the
top,
where
it
shows
you
that
there's
this
account
created
event.
It
shows
you
the
nested
formats
and
of
course,
this
event
is
not
super
interesting
in
terms
of
the
schema
of
it
itself,
but
yeah
I
think
it's
really
cool
and
you
can
then
tweak
it
edit
it
or
generate
other
things
generate
your
own
documentation.
A
There's
a
lot
that
we
can
do
here
and
I'm
excited
to
start
to
embrace
the
async
API
spec
and
to
use
that
as
a
format
that
maybe
some
folks
can
use
and
jump
in
to
ingest
or
pull
out
and
have
some
familiarity
as
I
think
we.
We
hope
that
it
grows
and
becomes
an
even
more
popular
format
that
everyone
can
be
used
to
and
be
comfortable
with.
It
allows
people
to
enter
and
exit
systems
and
pick
up
things
and
understand
the
system
much
easier.
A
A
So
thanks
for
listening,
I
really
appreciate
it.
I'd
love
to
hear
some
questions
and
again
you
can
hit
me
up
at
DJ
thoroughly
on
Twitter
to
send
me
any
ideas
or
feedback
file.
Bug
reports
I'll,
send
you
the
GitHub
repo
whatnot.
You
can
find
it
at
the
ingest
again,
that's
I-n-n-g-e-s-t
on
GitHub.
You
can
find
our
library
there
and
you
can
find
it
on
npm
and
email
me
at
Dan,
ingest.com
anytime.
Well,
thanks
for
listening.