►
From YouTube: Data Processing at Scale with Knative and Benthos - Mihai Todor & Murugappan Sevugan Chetty, Box
Description
Data Processing at Scale with Knative and Benthos - Mihai Todor & Murugappan Sevugan Chetty, Box
Knative serving provides push-based autoscaling (scale based on rps/concurrency) leaving a requirement on a "component" to push these requests. This works well for real-time http/grpc requests. How about event processing and batch processing? For event processing, we could leverage webhooks or knative eventing (different types of sources, brokers, etc). Challenge lies in processing batch data from databases, CSV files, etc. These are common enterprise use cases. To attain auto-scaling for each batch usecase a bespoke component needs to be developed and this is where Benthos shines. Benthos is a stateless data streaming engine that implements transaction-based resiliency with backpressure. When connecting to at-least-once sources and sinks, it's able to guarantee at-least-once delivery without needing to persist messages during transit. Data transformations can be expressed using a DSL. It's a safe, fast, and powerful way to perform document mapping within Benthos. In this session, Mihai and Murugappan will demo how to leverage the best of Knative and Benthos to process data at scale.
A
B
B
Going
to
be
talking
about
my
favorite
feature
in
canada,
which
is
the
k
native
port,
auto
scaling
yeah,
the
other
favorite
feature
of
main
is
the
developer
experience
that
you
get
out
of
k
native
what
I
mean
by
developer.
Experience
is
like
you,
create
one
creative
search
and
that
creates
all
the
deployment
k
native
sorry,
kubernetes
artifacts
for
you,
apart
from
that,
you
also
get
auto
tls
and
the
custom
domain
things
like
that.
So
about
k,
native,
auto
scaling.
B
How
does
canada
provide
this
auto
scaling
for
you?
There
are
two
metrics
that
it
could
use.
One
is
concurrency
number
of
concurrent
request.
Your
application
receives
or
number
of
requests
per.
Second,
the
default
one
is
concurrency
and
the
default
value
is
100..
For
example,
you
create
a
canadian
service
without
changing
any
default
values,
and
you
send
in
like
150
request
your
application.
B
You
should
have
two
parts
of
location
running,
and
this
could
be
changed
based
on
your
needs.
It
could
be
either
concurrency
or
request
per
second,
and
you
can
change
this
value
around
and
if
you
see
who
is
sending
these
requests
to
you,
it
could
be
end
users
like
millions
of
end
users
accessing
our
application,
or
it
could
be
an
application
calling
your
apis
or
it
could
be.
You
could
be
leveraging
k
native
event
and
which
could
source
data
from
multiple
event
sources
and
send
it
out
to
your
application.
B
B
B
Yes,
you
could,
but
you
have
to
develop
a
component
which
is
like
for
each
use
case,
which
takes
the
data
from
pulls
it
from
our
database
and
send
it
over
to
the
k
native
service
and
since
it
pulls
the
data,
we
call
it
like
full-based,
auto
scaling,
and
what
I
mean
by
like
a
special
component
or
a
bespoke
component
is
like
suppose.
If
you
want
to
source
data
from
a
csv,
you
need
to
develop
a
component
which
can
source
it
from
csv.
B
B
Wouldn't
it
be
nice
if
there
was
a
component
which
could
integrate
with
multiple
data
sources
and
it's
just
configurable,
and
it
could
also
integrate
with
multiple
things.
We
don't
even
need
multiple
things.
We
just
need,
like
http
grpcc,
to
send
the
data
over
and
it
is
scalable
based
on
the
load
and
is
it
is
resilient
to
failures.
If
there's
any
failure
during
the
whole
processing,
it's
able
to
catch
up
and
last,
but
not
the
least
observable,
we
should
be
able
to
observe
the
whole
process
to
talk
about
this
component.
A
A
It
supports
many
sources
and
many
things
like
real
time,
batch
file,
storage,
etc,
and
you
have
a
configuration
for
it
and
in
the
configuration
you
specify
your
sources,
your
syncs
and
then
various
operations
that
have
to
happen
on
the
messages
as
they
are
in
flight
from
a
source
to
a
sync
and
what
kind
of
operations
do
I
mean?
Well,
you
can
do
transformations
like
imagine
a
schema
migration
for
various
messages
filtering
like.
If
you
want
to
drop
some
messages,
you
don't
care
about.
You
can
just
do
that
or
hydration.
A
A
Well,
as
you
probably
guessed
by
now,
it's
written
in,
go
there's
the
go
go
for
here,
it's
performant
and
simple,
so
you
know
very
good
at
this
kind
of
small
workflows
where
you
launch
it
as
a
lambda,
and
it
does
the
thing
it
supports
yaml
and
qe
configs.
Well,
q
is
kind
of
a
recent
addition
which
is
used
to
drag
the
yaml
config
for
bentos,
it's
stateless,
so
it
doesn't
store
any
state
locally
about
the
messages
that
are
inside
in
flight
in
ventus.
A
You
can,
however,
configure
a
cache
for
it.
So,
for
example,
if
you
want
to
have
something
like
an
improcess
cache
or
external
cache,
you
can
do
that
and
it's
extendable.
So,
as
I
said
it's
written
in
go,
you
can
import
it
as
a
library,
create
a
custom
bentos
binary
with
your
own
additions
and
have
that
as
a
custom,
functionality.
A
So,
let's
combine
them,
we
can
take
ventus,
we
can
take
k
native
serving
put
them
together
and
see
how
this
thing
works
and
how
you
can
get
k
native
to
the
total
scaling
for
it
and
do
whatever
you
want
for
your
pool
based
auto
scaling
model.
That
moore
was
introducing
back
then
back
in
the
video
and
for
the
demo
that
I'm
presenting
here,
I'm
going
to
have
a
local
kubernetes
cluster,
which
is
going
to
be
kind
basically,
and
it's
going
to
run
k
native
serving
and
canadian
serving
is
going
to
auto
scale.
A
In
my
case,
I
just
have
a
postgres
locally
running
and
there's
a
source
and
sync
table
and
bentos
is
going
to
read
data
from
the
source
batch
load
it
and
then
it's
going
to
construct
this
batches
that
it's
going
to
send
for
processing
for
the
sentimental
isis
app
and
then
it's
going
to
batch
store
in
the
sync,
the
data
that
it
gets
and
having
said
that,
I'm
going
to
go
into
the
demo
and
please
bear
with
me
because
this
is
going
to
be
kind
of
complex.
And
I
hope
I
it
won't
blow
up
on
me.
A
A
A
A
A
A
So
we
created
the
config
for
our
benthos
vader,
vader,
being
the
sentiment,
analysis,
library
that
I'm
using
and
now
I'm
launching
benthos
vader.
So
this
is
running
that
keynative,
crd
and
starting
it
up,
and
we
can
see
here
the
first
bentos
benthos
vader
pod
coming
up.
It
takes
like
20
seconds
to
start
up,
so
I'm
just
waiting
for
it
and
the
meantime.
A
We
have
this
curl
command
here,
pre-prepared.
Okay,
you
see
it
came
up,
so
it's
running
it's
healthy
and
if
I
run
my
curl
command
oops,
you
can
see
here
all
the
way
at
the
bottom
that
it
did
produce
this
output.
So
I
said:
hey,
you
know
I
love
benthos
and
it
computed
the
sentiment
for
this
and
it
came
up
with
a
bunch
of
numbers
that
I'm
not
going
to
try
to
explain
detail.
A
A
A
A
A
A
A
A
A
A
So
yeah,
that
kind
of
shows
you
what
I
wanted
to
demo
and
if
I
go
back
to
my
slides,
I
just
want
to
say
in
summary,
k
native
supports
push
based,
auto
scaling,
but
for
pool
based,
auto
scaling.
We
require
bespoke
components
and
bentos
is
one
way
of
doing
this.
It
provides
all
this
functionality
for
free,
it's
configurable
and
it
lets
us
use
a
lot
of
components
and
we
can
write
a
bunch
of
bentos
dsl
to
transform
our
messages
and
it's
like
a
kubernetes
native
application
that
can
be
run
supports
all
the
probes.