►
Description
In this Kongcast episode, Danica Fine, Senior Developer Advocate at Confluent, talks about her hardware projects with Kafka and Python as well about fully serverless cloud-native data pipelines with Confluent Cloud and ksqlDB.
Subscribe to get email alerts for the latest new episodes and see which guests are coming on the show next: https://bit.ly/3zQV71e
▬▬▬▬▬▬ TIMECODES ▬▬▬▬▬▬
TDB
▬▬▬▬▬▬ ABOUT KONGCAST ▬▬▬▬▬▬
Hosted by @Viktor Gamov and Kaitlyn Barnard, we interview software developers and technology leaders at the top of their game every other week. We’ll also give you the tools, tactics and strategies you need to take your distributed architectures to the next level. Kongcast goes beyond the buzzwords and dissects real-life applications and success stories so that you can tackle your biggest connectivity challenges.
▬▬▬▬▬▬ KONG COMMUNITY ▬▬▬▬▬▬
⭐ https://github.com/Kong/kong
🐦 https://twitter.com/thekonginc
🌎 https://konghq.com/community/
🍻 https://www.meetup.com/pro/kong
❓ ️https://discuss.konghq.com/
💯 https://konghq.com/kong-champions/
#StreamProcessing #Serverless #Kafka #Kongcast
A
A
B
Okay,
so
it
is
called
practical
pipelines.
Practical
cloud
pipelines
is
built
entirely
within
the
confluent
cloud
and
it's
a
hardware
hack.
So
I
have
a
lot
of
house
plants
and
I
wanted
to
build
something
to
monitor
my
plants
and
see
if
I
can
do
it
completely
with
confluent
conflict
cloud
and
yeah.
So
it
was
a
really
cool
project.
It
was
my
first
hardware
project
and
it
works.
I
get
notifications
on
my
phone
now
and
my
plants
need
to
be
watered,
so
I
was
excited
to
come
on.
B
B
I
had
to
build
a
kafka
from
source,
it
wasn't
really
working
right
or
stuff
like
I
had
to
get
liberty,
kafka
and
start
from
that
and
build
everything
up,
and
then
I
had
the
the
confluent
kafka
python
library
as
well,
but
yeah,
so
that
was
just
the
basic
side
of
the
raspberry
pi
and
then
I
got
these
these,
like
capacitive,
I
squared
c
moisture
sensors,
which
are
pretty
cool,
had
to
do
a
lot
of
soldering
nice.
I
I
really
like
doing
that.
It's.
A
B
And
that
was
the
angle
that
I
was
kind
of
going
into
this.
You
know
I'm
I'm
newly
a
developer
advocate
right.
Prior
to
this
role.
I
was
building.
Thank
you.
It's
a
great
role
to
be
in,
but
prior
to
this
role,
you
know
I
was
building
kafka
streaming
pipelines,
but
that's
easier
when
you
do
it
for
work
right,
you're
kind
of
told
what
to
build.
You
know
the
overall
goal,
but
I
feel
like
when
I
was
learning
kafka
I
wanted
to.
B
I
was
looking
for,
like
real
examples
online
to
learn
how
to
do
it
right
and
exactly
there's
that
plug.
But
the
thing
that
I
really
like
is
having
a
practical
use
case.
Like
you
see
a
lot
of
a
lot
of
pipelines
online,
a
lot
of
examples
and
they're
like
they're,
very
toy
use
cases-
and
it's
like
oh,
is
that
something
that,
like
does
that
motivate
me
to
learn
this
technology
right.
So
I
figured
that
like
doing
something
hardware
based
and
you
know
getting
getting
your
hands
dirty.
B
A
So
question
to
you,
so
you
said
you're
going
to
be
using
you.
You
used
liberty,
kafka,
which
is
the
implementation
of
kafkan
c
that
will
be
used
by
your
python
script.
Why
why
you
opt
out
from
the
concept
of
like
a
rest
proxy
like
why
you
need
to
like
deal
with
the
native
client
versus
just
using
something
like
event
gateway.
B
Yeah,
so
I
I
wanted
to
go
back
to
sort
of
basics
with
the
like
leaning
into
the
raspberry
pi
sort
of
things,
and
I
feel
like
a
lot
of
these
hardware
projects.
It's
when
you
feed
information
into
kafka,
like
that,
you
just
you
know,
use
the
use,
a
basic
kafka
producer
and
a
lot
of
the
examples
I'd
seen
online
were
leveraging
as
well
like.
B
A
But
you
need
to
it's
much
easier
to
you,
bring
the
you
know
your
events
into
kafka
through
some
sort
of
like
a
rest
api.
So
that's
that's
something
that
I
started
seeing
a
lot
and
I
started
having
some
conversation.
How
like
people
can
use
quant?
I
don't
want
to
like
plug
it
here,
but
like
we
also
have
like
pretty
cool
integration
with
kafka
and.
A
B
B
B
A
B
A
This
information
to
to
whatever
consumer
would
be
interested
in
aggregating
information,
especially
in
the
serverless
world
yeah
you
know
function
or
like
a
stateless.
It's
easy
because
there's
no
state
don't
need
to
store
it
like
how.
How
would
you
recommend
for
people
to
deal
with
this
type
of
stuff
when
they're
building
stateful
serverless
applications.
B
Yeah,
that's
a
great
question,
so
my
answer
to
you
as
if
we're
saying
completely
cloud
native,
it
is
k,
sqldb
run
through
confluent
cloud.
So
casey
db
is
the
sql
wrapper
on
kafka
streams,
and
you
know
once
you
have
a
confluent
cloud
account
you
can
utilize
kscodb
completely
within
the
cloud.
You
don't
have
to
run
anything
else
outside
of
it,
which
is
wonderful,
but
it
is
great
at
stateful
operations,
because
that's
what
kafka
streams
is
great
at
right.
B
That
is
what
we
as
that's,
how
we
conduct
stateful
transformations
in
kafka
right
so
casego
db
is
the
answer
to
that.
But,
as
you
said,
like
there's
a
lot
of
other
products
that
handle
the
stateless
aspect
of
that,
and
I
think
what
a
really
cool
demo
was
like
you
know:
kcodb
excels
at
the
stateful
processing.
A
So
something
like
also
involves
some
sort
of
like
connectors
that
can
like
integrate
with
some
external
systems.
I
know
that
you
have
a
background
on
building
like
pipelines
like
this.
You
spoke
about
this.
Your
previous
experience
before
you
joined
conflict
in
the
accounting
assignment.
A
lot
like
tell
us
a
little
bit
about
the
the
connectors
architecture.
What
kind
of
role
they
play
in
this
type
of
the
use
cases
like
when
you're
building
like
serverless
applications,
definitely.
B
Exactly
yes,
so
there
are
tons
of
connectors
that
are
fully
managed
available
in
confluent
cloud.
So
you
again,
you
can
completely
stay
pretty
serverless.
In
that
case
right,
you
don't
have
to
spin
up
those
connectors
on
your
machines.
They
run
completely
through
the
cloud
and
then
you
can
bring
in
that
external
data
into
kafka.
So
those
sit
you
know
pretty
well
in
this
demo
at
the
beginning
of
the
pipeline
and
then
at
the
end,
you're
free
to
you
know,
use
another
connector
in
my
and
the
demo
that
I
just
talked
about
today.
B
A
I
think
so
is
this
the
things
that
we're
talking
about
available
through
some
sort
of
like
api
that
allows
us
to
to
to
bring
the
automation
to
default,
and
you
know,
implement
things
around
the
very
popular
concept
of
the
githubs
like
when
you
have
all
this
deployment
and
all
this
rollouts
and
all
these
things
triggered
by
some
sort
of
approval
of
the
pr
of
certain
developers
who
you
know
put
the
things
in
places.
Do
some
changes
in
configuration?
Is
it
still?
B
Absolutely
it
wouldn't
be,
it
would
be
great
if
you
couldn't
do
that
right.
Having
it
you
know,
being
able
to
automate
it
and
having
it
be.
Scriptable
is,
is
really
important,
so
there
are
rest
apis
to
handle
that
there's
the
confluence
cli.
So
you
can,
you
know,
script
it
and
bash
old
school
style
if
you
really
want
to,
but
the
rest
api
is,
is
pretty
powerful
and
then
just
from
the
get
up
side
of
things,
I
know
where
we
just
had
the
newly
released
terraform
provider
for
confluence.
That's
nice.
B
B
Yeah,
I
really
want
to
link
to
that,
because
that's
what
everyone's
excited
about
right-
I
just
I
just
gave
a
talk
in
in
melbourne
at
a
devops
conference
about
being
able
to
script
your
your
hadr
strategies
right
completely
through
confluent
and
well.
I
didn't
have
the
terraform
provider
available
and
you
know
being
able
to
do
that,
but
you
can
still
use
the
cli
and
also
the
rest
api
to
completely
automate
these
things,
which
is
pretty
cool
yeah.
A
A
We
are
I'm
asking
my
guests
to
demo
some
of
the
things.
That
is
something
that
we
decide
to
do
when
we
start
this
contest
that
wouldn't
be
cool
if
we
were
not
only
talking
with
our
guests,
but
also
ask
them
to
show
the
technology
that
will
be
passionate
and
today
we're
writing
this
in
conference.
So
we're
not
going
to
do
in
this
life,
we're
going
to
be
using
magic
editing
and
we
will
bring
the
demo
that
danica
will
show
and
technical
what
you
will
be
showing
to
our
alice
anderson.
B
Yeah,
no,
I
think
the
the
coolest
thing
that
I
could
demo
right
now
is
that
one
that
I
was
talking
about
that
demo,
that
bill
b
jack
had
put
together
on
yeah
on
on
serverless
stream
processing
right.
It's
really
cool
because,
like
I
said
it,
these
are.
These
are
buzzwords
and
it's
I
think
it's
really
nice
to
see
something
tangible
to
show
a
real
functioning
pipeline
on
that
so
yeah.
He
has
the
demo
available
on
on
github
right
now.
Do.
B
All
right
so
in
this
demo
we're
going
to
be
showcasing
two
serverless
technologies
we'll
be
using
aws
lambda,
together
with
kafka
and
kc
db.
So,
as
we
touched
on
earlier,
kscodb
is
pretty
great
at
stateful
and
stateless
processing,
but
where
it
really
excels,
is
in
the
stateful
processing
realm.
It
easily
allows
you
to
set
up
stream
processing
applications
that
aggregate
window
and
transform
your
data
using
simple
sql
syntax.
B
On
the
other
hand,
we
have
aws
lambda,
that's
perfect
for
stateless
processing,
so
using
the
two
in
conjunction
should
offer
you
the
best
of
both
worlds
and
that's
what
we're
going
to
show
you
in
this
demo
today.
So
we're
going
to
start
off
by
forking
the
confluent
cloud
serverless
integration
repository.
B
B
B
B
Then,
from
the
project
root
we're
going
to
clean
and
build
a
project,
and
once
that
is
done,
we
can
run
a
second
script,
and
this
is
going
to
kick
off.
The
creation
of
the
aws
lambda
you'll
be
asked
to
confirm
whether
or
not
you
want
to
create
these
resources.
And
then
it's
going
to
set
up
an
aws
secrets
manager
with
the
information
to
connect
to
confluent
cloud
you're,
going
to
get
an
aws
role
with
the
permissions
needed
for
the
lambda
to
actually
execute,
and
then
it's
going
to
set
up
the
aws
instance.
B
So
we're
going
to
take
a
quick
look
at
the
actual
lambda
itself.
So
we've
created
a
class
stock
record
handler
that
implements
the
request
handler
and
this
request
handler
will
receive
batches
of,
I
think,
by
default.
100
records
via
the
handle
request,
method
and
it's
within
this
method,
specifically
within
the
payload,
parameter,
that's
being
fed
into
this
method.
We
can
access
the
actual
records.
The
the
kafka
records
that
we're
using
as
triggers
to
the
lambda
and
this
records
data
set,
contains
a
mapping
of
topic.
Partitions
mapped
to
a
list
of
kafka
records.
B
B
All
right,
then,
back
in
the
terminal,
we
can
actually
check
once
this
lambda
has
been
executed
to
see
that
the
lambda
is
successfully
receiving
these
events
and
processing
these
logs
look
good.
So
with
that
we
can
go
back
into
conflict
cloud
and
we're
going
to
use
this
pretty
cool
tool
for
stream
lineage
to
actually
track
the
flow
of
data
through
this
pipeline
that
we've
created
okay.
B
So
from
this
tool,
we
can
see
that
we
have
our
data
generators,
writing
data
both
into
the
stocks
topic
and
that
user
topic-
and
you
see
the
k,
sql
db
application,
that's
actually
conducting
the
join
of
these
two
data
sets
together
and
feeding
that
into
another
topic,
which
is
then
being
consumed
by
our
aws
lambda
instance.
We
have
on
the
right
there.
B
All
right,
so
the
pipeline
is
we're
creating
data
we're
joining
the
data,
we're
letting
it
flow
into
the
lambda.
But
that's
not
it.
So.
We've
been
doing
a
lot
of
stateless
processing
here,
but
I
said
that
ques
is
really
good
at
stateful
processing,
so
we're
going
to
run
a
third
and
final
script
to
execute
some
additional
k,
sql
db
statements
that
are
going
to
kick
off
some
more
stateful
analysis
of
the
data,
that's
flowing
back
into
kafka
from
the
lambda.
B
B
And
then,
once
we've
executed
those
queries,
we
see
a
second
stream
within
our
stream
lydian
tool,
and
you
can
see
here
that
we
have
our
lambda
producer.
Writing
that
data
to
a
kafka
topic
and
from
there
we're
splitting
those
out
into
separate
topics
and
then
conducting
a
windowed
analysis.
On
top
of
that.
So
I
know
that
that
was
a
relatively
simple
demo,
but
I
really
do
think
that
it
gives
you
a
good
jumping
off
point
the
next
time
you
want
to
leverage
serverless
technologies
in
an
upcoming
data
pipeline.
A
And
now
we're
back
from
this
live
demo
danica.
Thank
you
so
much
for
for
showing
this
to
us
yeah.
I
think
we
can
do
like
a
few
announcements.
I
will
try
to
release
this
before,
so
the
people
would
be
excited
to
know
where
they
will
see
you
next
yeah
and
what's
as
a
developer,
we
get
we
finally
start.
The
world
start
open
up
and
we're
getting
back
to
meeting
developers
where
they
are.
B
B
A
You're
so
good,
and
I'm
excited
that
you
found
the
time
and
you're
scheduled
to
be
together
with
me
on
the
qualcast
yeah
it's.
It
was
really
great
to
catch
up
and
learning
some
of
the
cool
things
that
you
do
in
wish.
You
good
luck
and
but
usually
asking
some
questions
about
like
unrelated
to
technology
at
the
end
yeah.
What's
the
any
books
that
you
recently
read
that
you
can
recommend
to
our
list?
Oh.
B
My
goodness,
so
recently,
that's
a
great
question.
That's
such
a
good
question.
I
recently
got
back
into
reading
fiction.
I
don't
usually
read
fiction
books.
I
try
to
just
stick
with
nonfiction,
so
I
feel
like
I
learned
something,
but
the
it's
probably
one
of
the
best
pieces
of
fiction.
I've
read
in
probably
my
entire
life,
a
gentleman
in
moscow
by
immortals
yeah.