►
From YouTube: Building Event Streaming Applications with Serverless Kafka - Ricardo Ferreira, ConfluentSDCC
Description
Building Event Streaming Applications with Serverless Kafka - Ricardo Ferreira, ConfluentSDCC
Apache Kafka is a well-known technology from the on-premises world, that is gradually being adopted in the cloud. However, because it has persistent capabilities (just like databases) architectures built around serverless are hard to implement. This will present an interesting approach to address this concern, where developers interested in serverless could still leverage Kafka without giving up on the pay-as-you-go and scale-as-much-as-you-need principles. A demo using a speaker-enabled device and functions will be used to clarify the concepts.
A
A
This
talk
is
gonna,
be
about
Kafka
and
implementing
things
using
Kafka,
so
for
those
of
you
that
are
not
sure
if
you're
in
the
right
room,
so
that's
gonna
be
the
title
of
this
presentation.
First
things.
First
Who
am
I
right,
so
my
name
is
Ricardo
right.
Don't
bother
to
try
to
pronounce
my
last
name
because
I'm
from
Brazil
originally,
so
that's
why
they
have
the
accent
that
I
have,
but
Ricardo
would
be
more
than
fine.
A
A
For
those
of
you
that
doesn't
know
what
company
is
it's
a
company
that
basically
develops
Kafka
on
your
community,
so
we
possess
maybe
more
85%
of
the
commits
on
the
projects
do
open
source
project
but,
more
importantly,
we've
build
an
event
streaming
platform
out
of
it
for
that's
what
we
do
and
I'm
gonna
be
showing
this
a
little
bit
today.
I
work
for
the
developer
relations
team,
so
basically
my
job
is
to
create
awareness
and
keeping
things
relatively
simple
for
developers
to
work
with
Kafka
and
the
things
that
conflict
develops.
A
Right
I've
made
a
very
good
effort
to
make
my
handles
kind
of
easy
and
prevenient
and
all
the
social
network.
So
if
you
want
to
follow
me
on
Twitter
or
if
you
want
to
send
me
an
email
you,
this
is
Ricardo
at
confident
at
I/o.
So
it's
kind
of
very
easy
and
by
the
way
this
presentation
is
already
on
my
blog.
So
don't
don't
bother
in
taking
screenshots
for
things.
Everything
is
already
there.
Just
go
to
that
website.
Slash
presentations
is
gonna,
be
there
all
right.
A
We
have
two
basic
objectives
for
this
presentation,
guys
and
as
my
job
as
a
developer
advocate
we're
spending
a
lot
of
time
today,
like
educating
developers
about
what
is
that
thing
over
there
right,
because
many
developers
know
Kafka
I
sure
about
that
many
developers,
love
Kafka
because
of
the
scalability,
the
the
simplicity
of
the
AP,
is
but
event
streaming.
It's
something
relatively
new
and
that's
pretty
much
the
face
that
people
does
when
we
start
thinking
about
potential
streaming
right
and
if
event
streaming
is
already
a
very
complex
topic.
A
If
you
start
thinking
about
how
to
implement
event
streaming
using
server,
less
principles,
that
is
the
same
face
that
people
does
right,
so
a
little
bit
more
harder
right.
So
what
we
are
going
to
be
showing
today
is
both
of
things
all
right,
so
I'm
going
to
be
trying
to
explain
what
event
streaming
is,
which
has
to
do
with
the
fact
that
Kafka,
it
is
not
a
messaging
technology
all
right.
A
So
it's
about
paying
the
technology
with
others
paralyzed
the
way
I'm
going
to
be
presenting
this
today
is
going
to
be
a
challenge,
because
I've
brought
implementation
for
you
about
an
event
streaming
application,
which
is
essentially
going
to
be
the
pac-man
game,
all
right,
I'm
going
to
deploy
from
scratch
the
game.
That's
going
to
be
the
challenge
because,
as
I
deploy,
I
will
start
to
explain
things
and
my
whole
code
is
implemented
using
terraform.
A
So
the
terraform
I've
tested
a
couple
of
times
here
this
morning,
and
it
takes
about
four
four
to
five
minutes
to
create
all
the
resources
and
the
cloud
provider
that
I
need
API
gateways,
lambda
functions
and
everything
that
I
need
all
right.
So
meanwhile,
I'm
going
to
first
create
a
cluster
using
the
implementation
that
we
call
service,
Kafka
right
and
I'm,
going
to
start
the
deployment
of
the
night
terraform
code,
which,
as
I
said
it's
gonna,
take
about
five
minutes
and
meanwhile
we
are
going
to
actually
put
in
details
this
architecture
here.
A
This
is
what
I
call
a
conceptual
right.
The
conceptual
view
is,
you
guys
are
going
to
play.
Pac-Man
literally
right,
get
your
phones
ready.
Okay,
each
game
is
going
to
emit
events,
and
those
events
is
going
to
be
processed
in
new
real-time
and
we're
going
to
come
up
with
a
scoreboard
for
check
whoever
is
rocking
on
pac-man.
So
that's
the
conceptual
view
all
right
pragmatically
speaking.
A
What
we
are
going
to
do
is
basically
have
in
this
application,
deploying
a
crowd
provider
right,
you're,
going
to
access
user
phone
or
laptops
I
would
highly
encourage
you
to
play
the
pac-man
with
your
laptops
because
you
can
use
the
arrow
key
arrow
keys
on
your
keyboard
is
make
things
easier
or
just
because
I
so
can
pac-man
away
with
the
phone.
So
maybe
that's
the
explanation.
So
the
game
is
going
to
emit
events
to
an
API.
A
That's
going
to
be
exposing
that
API
is
job
is
basically
write,
those
events
in
to
Kafka
into
a
topic-
and
there
will
be
this-
let's
call
for
now
thing:
that's
going
to
process
those
events
or
in
other
words,
do
it
event
stream
processing
itself
and
then
the
result
of
that
is
going
to
be
reading
into
another
Kafka
topic
that
an
application
will
display
very
simple
all
right.
So
that's
gonna
be
the
game.
A
Everything
is
on
the
github
right
I'm,
going
to
provide
a
link
of
this.
So
there's
instructions
how
to
deploy
by
yourself,
but
keep
in
mind
that
the
penny
of
the
region
that
you
select
the
region
might
not
have
some
of
the
resources
that
we're
need
for
the
plant.
So,
for
example,
you
can
select
your
pickup
or
a
region
for
an
either
blast.
It
doesn't
support
it
di
gateway,
for
example.
I.
A
Don't
know
if
that's
true
or
not,
but
just
an
example,
so
just
keep
that
in
mind,
because
all
the
terraform
code
relies
on
the
region
that
you
select
here,
okay,
so
that
kind
of
a
glue
everything
together,
alright
I'm
gonna
watch
it
that's
why
we
call
servlets,
because
the
closest
is
ready
to
go
alright,
so
I'm
going
to
generate
some.
The
terraform
code
expects
you
to
provide
some
variables,
so
you
see,
I
have
implementations
for
AWS
Azure
and
GC
p.
Alright,
however,
so
those
are
the
variables
that
I
need
the
ADA
blast
implementation.
A
That's
the
only
one,
I've
refactored
the
code
should
be
a
hundred
percent
server
less.
So
everything
is
literally.
The
architecture
is
service,
the
azure
implementation
and
the
GCP
implementation
that
there
is
a
to
do
on
my
list
to
refactor
the
code,
because
right
now
they
spin
up
a
bunch
of
compute
instances
and
block
storage,
and
this
is
not
service
right,
so
I
still
have
to
refract
for
everything.
That's
so
the
only
one
that
I
have
a
refactor.
So
far,
it's
a
de
Bresse!
A
That's
why
I've
chose
AWS
in
DM
in
the
cluster,
so
I'm
going
to
select
here
the
endpoint
of
my
for
those
of
you
that
know
Kafka.
This
is
the
bootstrap
server,
so
that's
the
first
endpoint
that
your
producers
and
consumers
need
to
touch
with
and
there's
a
magic
that
Kafka
does
this.
It
discovered
the
rest
of
the
I
have
to
generate
an
API
key
I
forgot
that
sorry
API
key
in
secrets.
How
do
you
authenticate
yourself?
So
this
is
the
API
key.
A
Okay-
and
this
is
the
secret
which,
if
there
are
some
security
experts
in
the
room,
please
don't
get
scared
to
what
I'm
doing
here,
because
now
I
literally
showing
the
secret
and
a
screen
right.
So
people
that
are
from
security
backgrounds,
they
kind
of
a
gets
our
dude.
You
should
not
be
doing
this
I'm
going
to
retire
those
credentials
after
this
demo,
so
bear
with
me
with
this
and
I
need
to
generate
an
API
key
secret
for
schema
registry
as
well
all
right.
A
What
I'm
doing
here,
basically
I've
I've
created
in
terraform
I,
have
to
create
a
concept
of
variables
right.
But
if
you
are
a
Java
developer
group
developer
go
developer,
closure
developer!
You
have
examples
of
the
properties
that
you
have
to
set
for
your
programs
to
start
communicating
with
this
service
Kafka.
So
that's
how
you
would
do
in
your
programs,
but
because
I'm
deploying
things
using
terraform
I
had
to
came
up
with
variables,
so
we
are
ready
to
go.
A
My
inner
Blas
API
key
in
secret,
are
here
yeah,
I'm,
gonna,
close
it
because
the
being
recorded
right.
So
you
just
recorded
my
API
key
and
secret
for
the
aww,
as
I'm
gonna
have
to
retire
those
after
this
demo.
So
let
me
run
a
terraform
plan
here,
so
we
can
check
if
everything
is
ready
to
go:
okay,
okay,
okay,
okay,
okay
and
then,
as
I
told
you
right,
we're
ready
to
go
I'm
going
to
start
the
deployment
of
the
application
pac-man
and
we
can
walk
through
the
code
itself
all
to
approve.
A
Here
we
go
okay,
just
started
it
might
take
about
five
minutes.
Meanwhile,
we
can
start
discussing
the
implementation
bill.
All
right.
We
did
this.
So
as
I
told
you,
the
presentation
is
on
my
blog.
So
that's
why
you
can
grab
the
link
of
the
YouTube
implementation
if
you
want
to
try
by
yourself
just
clone
this
wrapper
right
here
and
that's
the
folder
that
has
the
code,
because
in
this
wrapper
right
here,
there's
a
bunch
of
demos
that
our
developer
relations,
team
kind
of
a
build
so
there's
a
very
other
interesting
ones
as
well.
A
The
terraform
code
that
I've
just
started
is
creating
this
right.
Let
me
walk
you
through
and
basically
what's
being
created
right
and
then
we
can
start
looking
the
code
itself.
Okay,
so
remember
that
I
told
you
that
there
will
be
there's
application
running
on
your
phone.
So
basically,
what
I
did
is
the
application.
It's
I'm,
creating
a
bucket
on
s3
and
the
pac-man
game.
That
you're
going
to
play
is
a
bunch
of
HTML
ljs
and
CSS
files,
they're
terraform
uploads
they're.
To
that
bucket
point
number
one.
A
A
Alright,
that's
kind
of
expected
I
mean
if
you
have
heard
about
Kafka
streams,
that's
more
respected
right,
so
Kafka
streams,
it's
a
Java
and
Scala
library,
so
you
can
develop
Java
applications
or
scale
applications
and
express
your
event
stream
processing
using
those
programming
languages
right
very
cool.
Very
awesome
works
fine,
a
bunch
of
applications.
Reason
using
this
framework.
However,
we've
recognized
in
kind
of
a
very
interesting
point
in
the
communities
some
developers
may
be
the
Java
one
developers.
A
They
do
not
want
to
write
Java
code
to
express
their
event
stream
processing
application
because
it
takes
times
you
need
to
redeploy
the
there's.
There's
a
concern
about
the
the
footprint
of
a
JVM
and
all
of
this,
so
Kay
sequel
is
an
implementation
on
top
of
Kafka
streams.
So
we
didn't
reinvent
the
wheel
right.
Basically,
it's
a
JVM
in
the
end
of
the
day
basis
that
promise'
jvm
to
run
Kafka
streams,
and
it
does
one
thing
only
which
is
do
events
during
processing
right.
A
The
technical
reason
that
I
like
to
explain
the
difference
between
Casey
Kuehn
Kafka
streams
is,
if
you
use
Kafka
streams,
you
get
actually
embed
your
event
stream
processing
code
into
your
JVM
of
your
application.
So
now
your
application
is
using
CPU
cycles
for
doing
whatever
your
application.
Does
your
micro,
Service,
Desk
and
string
processing
right
if
you're
going
to
do
this
you'd
have
to
take
care
about
garbage
collection
right
so
a
way
to
solve
that
problem
is
to
offload
all
this
VIN
string
processing
into
as
dedicated
specific
jvn
that
we
call
key
sequel.
A
Now
your
application
can
do
whatever
your
application
does
and
basically
consume
and
produce
the
events
that's
being
kind
of
a
process
by
KC.
That's
the
technical
reason
why
we
created
key
sequel
right
and
currently
my
terraform
code.
By
the
way,
let
me
check
the
stairs
of
it
still
deploying
we're
almost
there.
A
It's
basically
creating
the
API
Gately
KC
Co
is
currently
running
on
top
of
ECS
elastic
cloud
service
and,
more
importantly,
is
running
on
top
of
our
gate,
which
is
basically
a
server
less
way
to
deploy
containers
and
in
a
tab,
less
prickle
everything
that
case
sequel
is
going
to
produce
as
output.
We
call
those
things
right.
It's
going
to
be
written
into
a
destination
topic
that
I
strategically
call
scoreboard
right
and
I've
wrote
a
grow,
client
application.
That's
going
to
subscribe
to
that
topic
and
basically
display
the
content
right.
It's
gonna
dump,
whatever
it
isn't.
A
Scoreboard
is
going
to
dump
in
the
output
all
right.
So
basically,
this
is
the
architecture
that
we're
talking
about
right
over
here.
Okay,
very
good,
very
good
timing,
so
our
terraform
code
is
done
so
I'm
gonna
ask
you
a
favor.
This
is
basically
the
link
of
the
pac-man
game.
I'm
not
going
to
ask
you
to
type
all
of
this.
It
would
be
really
rude
of
my
part,
so
I'm
going
to
create
a
peer
code.
A
A
A
When
you
see
the
application
you're
going
to
be
asked
it
to
provide
your
name
right,
try
not
to
use
more
than
18
or
20
characters
more
forward,
be
able
to
display
all
of
it
over
here.
Alright
and
then,
when
the
game
starts,
you're
going
to
see
an
animation
of
the
pac-man
game
with
all
the
ghosts
is
going
to
have
nicknames
like
the
controller,
the
the
partition
and
the
offset.
All
of
this
just
click
it
with
your
finger
and
the
game
is
going
to
start.
A
Let
me
see
if
you
have
topics
someone
is
already
playing
because
we
have
topics.
So
if
we
print
this
topic
user
game,
if
someone
never
saw
a
stream
in
their
life,
this
is
a
stream.
A
stream
is
a
continuous
unbounded
set
of
events
that
keep
happening
all
the
time
and
they
never
finish
all
right.
There's
no
finish
in
streams.
A
That's
basically
the
fundamental
difference
between
your
program
for
oriented
to
databases
and
your
program
oriented
to
streams
because
databases,
you
can
is
acute
point
in
time,
queries
to
retrieve
whatever
data
is
being
stored
on
the
database
and
you
can
start
acting
upon
those
data.
But
streams
are
continuously
happening
right,
so
if
instrument
applications
is
the
ability
and
the
principle
of
start
processing
those
events,
while
those
events
are
in
motion,
all
right,
you
don't
need.
In
other
words,
you
don't
need
to
expect.
A
You
don't
need
to
expects
that
the
data
is
storage,
so
you
can
start
working
with
it
all
right.
So
now,
let's
go
to.
Let's
start
talking
business
I'm
gonna!
Stop
here
your
events,
you
can
keep
playing.
Actually.
Can
you
stop
playing
just
for
a
second,
because
if
you
keep
playing
you're
not
going
to
pay
attention
to
the
actual
code
right
that
does
the
key
sequel
processing,
so
you
can
die.
A
Okay,
so
in
the
code
there
is
this
folder
called
pipeline
and
pipeline
basically
creates
a
set
of
streams
right
and
on
top
of
those
streams,
it
creates
tables
right
and
there
will
be
a
final
table
called
scoreboard
yeah.
For
those
of
you
that
never
seen
this
Kafka
supports,
streams
and
tables
right,
so
tables
are
basically
pointing
time
snapshot
of
a
stream
like
a
stream
that
happened
for
the
last
hour.
So
you
can
flush
that
into
a
table
right
and
make
the
table
available
for
external
consumption
right.
A
That's
the
best
way
actually
to
provide
data
access,
because
otherwise,
if
you
expose
your
streams,
what's
going
to
be
what's
going
to
happen
with
the
streams,
data
will
keep
coming
in
all
the
time.
It
never
stops
right.
So
what
I'm
going
to
do
is
let
me
grab
the
endpoint
of
key
sequel
that
has
been
exposed
by
four
gates,
I'm
going
to
enter
the
folder
that
has
the
pipeline.
A
A
So
now
you
can
see
that
data
actually
has
been
aggregated
right.
So
there
are
two
types
of
events
that
are
coming
out
of
your
phones
right,
the
first
one
I
call
user
game
user
game
is
that
stream
that
I've
just
show
you
it
contains
your
scoring
score
current
level
and
the
number
of
lives
you
have
right.
That's
one
string!
Okay!
Every
time
you
game
over
on
pac-man
is
going
to
emit
another
event
that
I
call
user
losses.
Okay.
A
So
what
that,
basically,
what
that
pipeline
code
is
doing
is
the
user
game
is
aggregating,
calculating
the
sum
of
the
highest
score,
calculating
the
sum
of
the
highest
level
calculating
the
sum
of
your
lives?
And
this
is
going
to
be
one
table
all
right.
There
will
be
another
table
created
with
the
number
of
losses,
which
is
the
number
of
time
you
game
over
okay
and
I'm,
going
I'm
doing
a
join
between
these
two
tables.
Let
me
stop
the
square
over
here
for
a
second,
so
you
can
see
the
code
you
see
here.
A
This
is
essentially
you
what
you
would
do
with
a
typical
batch
ETL
program
right,
which
is
tracks
data
for
your
transaction
databases
and
push
data
to
your
analytics
database.
So
you
can
start
processing
data,
but
that
is
happening
as
events
coming
in.
So
the
principle
is
the
same.
But
what
needs
to
happen
is
that
there
is
no
all
right.
We
need
to
wait
until
this
code
runs
at
midnight
out
of
the
business
hours,
write
that
code
continuously
execute
and
now
to
see
whoever
is
rockin
on
pac-man.
A
Remember,
I
told
you
the
go
program,
so
the
go
program
is
going
to
be
on
this
scoreboard.
So
this
is
the
go
program.
Basically,
what
this
code
does
I
create
a
cache
for
the
statistics
and
I
might
create
a
unique
consumer
group.
Consumer
group
is
a
Kafka
building
block
and
then,
let's
create
a
new
consumer
and
I
am
subscribing
to
this
scoreboard
topic,
okay,
which
is
happens
to
be
the
table
that
we've
created
right.
So
let
me
ask
you
this
code
and
then
CD
score
bird.
Let
me
do
this
go
ran,
run
scoreboard.
A
A
The
biggest
difference
between
this
table
and
a
table
that
you
were
seen
in
the
database
is
back.
This
table
here
is
live
right.
It
keeps
being
updated
all
the
time.
So
whoever
is
interested
on
that
information
is
not
going
to
be.
The
famous
gonna
have
to
do.
The
famous
okay
I
have
out
of
data
data,
so
I
have
to
read
again
for
the
sirs
and
bring
it
to
my
application
right.
You
can
subscribe
to
a
live
feed
all
right
that
type
of
application,
guys
I'm,
sorry
I,
should
not
use
the
term
guys.
A
That's
very
inappropriate
because
we
have.
Ladies
in
a
row,
the
usage
of
event
streaming
applications
allows
you
to
have
this
type
of
applications
where
data
is
alive.
However,
what
is
the
biggest
challenge
when
you
are
dealing
with
a
server
last
type
of
architecture,
because
everything
here
requires
to
create
state
all
right?
That's
the
challenge,
all
right!
So
the
challenge
here,
for
example,
when
we
do
a
summarization,
a
sum
right
highest
score
to
calculate
this
column
over
here,
for
example,
we
have
to
do
a
summarization
in
a
user
basis.
A
A
So
when
you
start
thinking
in
server
less
end-to-end
service
applications,
you
have
to
think
that
some
of
the
layers
you
have
to
come
up
with
strategies
to
start
that
data
reliably
and
you
know
I
think
that's
what
probably
the
previous
speaker
was
trying
to
explain
when
he
would
gave
the
example
about
the
the
cache
the
memory
state
and
some
of
you
kind
of
asked.
Ok
but
I
have
to
use
a
cache,
and
he
explains
that
was
performing
reasons,
but
some
applications
needs
to
start
State
alright.
A
You
can't
stop
playing
or
there's
any
one
more
plate
who
is
danger,
danger,
deep,
ok,
no!
You
have
to
you
have
to
shut
it
out
for
him
you're
good.
Now,
because
every
time
I
try,
this
I
never
go
beyond
like
15,000.
That's
so
30
in
your
phone
is
quite
an
accomplishment,
so
I
have
to
give
it
to
you.
I
think
we
are
on
time,
I
think
we're.
We
have
5
minutes.
Let's
use
this
5
minutes
for
Q&A
all
right
I.
This
presentation,
the
whole
purpose
was
basically
showing
you
a
typical
event.
Streaming.
B
A
C
One
of
them
is
take
a
Kafka
message,
turn
it
into
HTTP
and
just
throw
it
over
HTTP
to
some
function
and
the
other
one
is
what
like
kada
and
azure
functions
does:
where
will
actually
pull
from
Kafka
directly
so
that
you
get
like
ordering
and
all
that
other
stuff
I,
don't
know
if
you
have
a
thought
on
in
terms
of
like.
Do
you
think
Kafka
can
play
in
both
of
those
or
do
you
have
an
opinion
on
where
one
might
fit
or
the
other
one
might
not.
A
Think
I
think
using
the
azure,
durable
functions
that
you
mentioned
I
think
that's
a
very
common
pattern
that
we've
seen
over
there,
because
it's
all
about
being
able
to
start
the
state
somewhere
and
as
your
durable
functions.
A
very
very
interesting
I
was
on
a
couple
weeks
ago
and
Microsoft
ignite
and
I've
talked
with
a
lot
of
developers
that
we're
doing
exactly
this,
not
necessarily
using
the
several
scoff
implementation,
but
we've
even
hub
so
yeah.
That's
I!
Think
it's
a
pretty
nice
job,
yeah.