►
Description
This webinar is hosted by the Lenses.io engineering team – the ones behind bringing DataOps to Apache Kafka. Here they talk about how to build a framework for Kafka streaming applications on Kubernetes. These deployed applications follow 12 factor app principles that prioritize security, resilience and portability across different Kafka & Kubernetes environments. You’ll learn: + The 12 factor app principles + Introduction to Lenses.io Streaming SQL Engine for Apache Kafka + Some of the challenges in managing Kafka streaming applications on Kubernetes + How you can deploy streaming applications on Kubernetes following the 12 factor pattern.
Presenters:
Andrew Stevenson, Chief Technology Officer & Co-Founder @Lenses.io
Francisco Perez, Senior Backend Engineer @Lenses.io
A
Okay,
we
are
going
to
get
started
hello
and
happy
friday
to
everyone.
Thank
you
so
much
for
joining
us
today
for
today's
cncf
webinar
12
factor
streaming
data
apps
on
kubernetes.
I
am
jerry
fallon
and
I
will
be
moderating
today's
webinar.
We
would
like
to
welcome
our
presenters
today,
andrew
stevenson,
chief
technology
officer
and
co-founder
at
lenses.io
and
francisco
perez
senior
back
and
engineer
at
lenses.io.
A
We
just
have
a
few
housekeeping
items
before
we
get
started
during
the
webinar.
You
are
not
able
to
talk
as
an
attendee.
There
is
a
q,
a
box
at
the
bottom
of
your
screen,
so
please
feel
free
to
drop
your
questions
in
there
and
we'll
get
to
as
many
as
we
can.
At
the
end.
This
is
an
official
webinar
of
the
cncf
and
as
such
as
subjects
of
the
cncf
code
of
conduct.
So
please
do
not
add
anything
to
the
chatter
questions
that
are
in
violation
of
the
code
of
conduct.
A
B
B
So
am
I
I'm
andrew
stevenson
just
said:
I'm
the
cto
of
lenses
on
the
left,
looking
there
with
two
glasses,
not
just
one
pair
of
glasses,
and
we
also
have
fran
perez
who's.
A
the
back
end
engineer,
who's
going
to
walk
you
through
the
how
we
actually
implemented
lenses
and
its
integration
with
kubernetes.
B
So,
first,
the
agenda
right,
I'm
going
to
give
a
brief,
a
very,
very
brief
overview
of
lenses
and
a
high
level
overview
of
what
we
mean
by
a
data
ops,
streaming
platform.
I'll
then
follow
that
up
by
a
quick
demo,
just
to
show
us
deploying
an
application
from
lenses
in
side
of
kubernetes
and
I'll
hand
it
over
then
to
fran
who
will
walk
through
how
we
actually
built
the
framework
that
we
use
to
deploy
data
intensive
applications.
B
B
This
could
be
apache
kafka,
it
could
be
apache
pulsar,
it
could
be
kubernetes
right,
but
all
these
things
are
distributed
and
complex
right.
So
what
we're
trying
to
do
at
lenses
io
is
actually
to
reduce
that
pain
and
the
cost
and
the
complexity
of
it,
of
modern,
using
this
technology
to
actually
build
up
a
modern
data
streaming
platform.
B
So
these
are
some
of
the
examples
of
what
we
do
in
the
customs
we
have
right.
We
have
babylon
health,
so
babylon
health
is
a
unicorn
startup
in
the
uk,
where
they're.
Basically,
their
goal
is
to
provide
affordable
healthcare
for
everyone
on
the
planet
using
ai
as
well,
so
they
they
use
lenses
and
with
their
in
their
data
rops
organization,
to
actually
focus
on
that
noble
deed
right,
there's,
nothing
more
important
than
your
health,
so
they're,
using
lenses
in
combination
with
kafka
and
kubernetes
to
provide
that
solution
around
the
world.
B
We
also
have
in
an
analytical
space
for
texas,
so
basically
they're
tracking,
seabourn
freight,
again
all
the
kafka
and
the
using
lenses
to
actually
get
visibility
into
the
application
flows,
and
we
also
have
the
adrenal
group,
who
were
a
customer
and
transaction
payment
system
in
the
uk
as
well,
and
they
all
want
to
go
real
time
and
they're
all
looking
towards
this
open
source
technology.
That
is
hard
and
painful
to
operate
in
some
cases.
B
So
we
do
have
a
commercial
offering
offering.
That
is
lenses
where
we
integrate
with
with
kafka.
These
are
some
of
our
other
clients
that
we
have
here
who
we're
allowed
to
speak
about,
but
we
also
have
an
open
source
contribution
as
well.
So
we
have
a
number
of
open
source
tooling
around
mostly
kafka
as
well,
so
we
have
over
a
million
downloads
of
our
fast
data,
dev
docker.
B
If
you
want
to
try
lenses
right,
you
can
call
inside
or
start,
and
you
can
get
the
the
docker
and
everything
is
set
up
for
you.
You
get
the
brokers
in
there,
schemer
registries
lenses
other
bits
and
pieces,
so
you
can
be
productive
and
really
get
into
building
data
intensive
applications
or
you
can
head
on
over
to
our
github
site
and
play
around
with
our
code,
and
if
you
want
to
contribute
things
like
connectors,
awesome,
we'll
always
welcome
pull
requests.
B
So
what
about
what
we're
talking
about?
When
I
mean
a
data
platform
at
a
very,
very
high
level
right
you've
got
a
source
system,
it
could
be
your
microservices
deployed
inside
of
kubernetes.
It
could
be
a
massive
oracle
database,
it
could
be
a
mainframe
and
you
want
to
get
it
somewhere.
So
you
can
build
a
data
product,
so
you,
but
what
you
need
is
a
buffer
in
between
which
is
where,
in
this
case,
apache
kafka
comes
in.
B
So
it's
a
buffer
that
allows
you
to
transport
data
around
to
your
target
system.
For
whatever
use
case,
you
want
it's
actually,
basically
central
nervous
system,
but
then
you
also
have
to
actually
deploy
apps
right.
You've
got
to
deploy
your
intellectual
property,
the
data
landscape
on
top
to
actually
make
use
of
it.
Otherwise
it's
just
infrastructure
that
is
burning
dollars.
B
B
Topics
are
then
split
into
partitions
and
distributed
across
the
cluster
and
as
a
producer,
you
produce
data
into
a
topic
and
each
message
has
an
offset
and
then,
as
a
consumer,
you
can
form
consumer
groups
and
you
can
actually
then
just
consume
the
latest
records
or
you
I
can
actually
say.
I
want
to
consume
all
the
records
from
this
offset
in
this
particular
partition
as
well.
So
that's
a
very,
very
high
level
overview
of
kafka.
I'm
not
trying
to
explain
to
you
in
depth
what
kafka
is
just
to
give
a
brief
brief
primer
on
it.
B
B
So
that's
what
I
call
the
tech
intensity
part,
but
to
actually
have
a
successful
data
platform
and
to
scale
that
data
platform
out
to
get
more
tenants
onto
it.
So
you
can
build
data
products.
You
need
to
have
a
layer
on
top,
and
this
is
what
lens
is
actually
then
providing.
So
we
have
self-service
so
we
can
administer
all
the
data
sets
inside
the
underlying
system,
such
as
kafka
or
elasticsearch.
B
We
have
the
discovery
of
data,
it's
very
important
that
you
get
the
people
with
the
business,
knowledge,
access
and
visibility
to
the
data.
Let's
take
covid
right,
I'm
not
a
doctor,
I'm
a
technologist,
and
if
I
can't
get
visibility
to
the
data
to
the
right
people,
then
we
don't
win.
We
don't
take
advantage
of
all
this
great
open
source
technology.
That's
out
there.
B
You
also
need
a
way
to
process
it.
Yes,
I've
got
kafka
where
my
application's
going
to
go
right,
so
that's
where
kubernetes
will
come
in,
but
then
how
do
I
deploy
the
app
and
how
do
I
deploy
the
app
and
monitor
it
and
be
compliant
as
well
right?
I
have
to
have
security
and
role
based
governance
around
all.
B
So
if
we're
not
going
to
talk
about
deploying
kafka
here,
because
we're
not
talking
about
how
do
I
deploy
kafka
inside
of
kubernetes?
That's
not
what
we
want
to
focus
on.
What
are
we
actually
deploying
right?
So
we
have
things
like
the
data
entities,
so
creation
of
topics,
for
example
data
policies.
Maybe
I
need
to
mask
that
data,
so
I'm
in
compliance
with
gdpr
right.
I
also
need
my
alerting
rules
as
well,
but
there's
the
applications
and
that's
what
we're
focusing
on
here
and
that's
what
fran
is
going
to
go
into
in
details?
B
How
do
we
deploy
that
app,
which
is
your
intellectual
property?
That's
the
bit!
That's
driving
the
business
value,
that's
your
data
product!
How
do
you
deploy
that?
Doesn't
matter
about
the
infrastructure?
I
might
want
to
swap
out
kafka
for
apache
pulse
all
right,
but
my
data
application
data
is
the
protagonist.
It
remains
constant.
B
So
this
is
how
we
do
it
at
a
high
level
with
lenses
right.
You
know
this
is
a
great
thing
that
kubernetes
is
actually
giving
it.
If
we
have
a
docker
and
we
have
a
config
right
and
we
have
apis,
we
have
a
ui.
We
can
also
do
this
by
githubs
as
well,
and
lenses
will
deploy
that
application
and
monitor
it
for
you
now.
B
This
is
more
focus.
This
talk
towards
kubernetes,
but
we
also
have
to
deploy
to
other
frameworks.
For
example,
we
have
a
lot
of
clients
that
are
on
premise
and
then
don't
have
kubernetes
yet
so
we
allow
people
to
deploy
to
kafka
connect.
What's
important
here
as
well
is
the
notion
that
we
abstract
away
connections
because
deploying
applications
is
just
one
part
of
lenses
where
you
also
have
the
ability
to
fire
sql
statements
at
lenses,
so
you
can
debug
and
get
that
visibility
into
the
systems,
so
connections
are
managed
centrally
and
governed
as
well.
B
So
this
is
a
more
concrete
example.
This
is
just
basically
what
I'm
going
to
show
you
right
in
in
the
demo
now
of
lenses
deploying
our
own
sql
processor.
So
in
this
case
you
know
we
can
join,
we
can
filter,
we
can
aggregate
and
we
deploy
and
monitor
them
outside
of
lenses
directly
into
kubernetes
as
well.
B
So
what
I
will
do
is
now
I
will
just
go
to
the
demo,
so
let
me
just
change
my
browser
tab.
So
here
it
is.
This.
Is
this
lenses?
Hopefully
you
can
see
it
right.
I
think
the
lenses
is
just
a
set
of
apis.
The
ui
is
one
of
the
clients.
We
can
automate
all
this
in
in
githubs.
As
well,
if
you
want
to,
but
what
I
want
to
build
is
actually
my
application
landscape.
This
is
my
application
landscape.
This
is
important
if
it's
on
kafka,
if
it's
on
pulsar,
is
that
really
the
interesting
part?
B
If
this
is
running
in
kubernetes?
Is
it
interesting
to
me
as
a
let's
say,
a
non-technologist
where
it's
running
not
really?
This
is
my
intellectual
property.
This
is
the
bit
that
is
generating
value.
So
what
we
have
here
is
different
processes
that
are
deployed,
but
what
I'm
going
to
do,
one
now
is
quick
demo.
Is
I'm
just
going
to?
Let
me
just
refresh
the
browser,
because
my
internet's
a
little
bit
flaky
today,
sql
processes
I'm
going
to
deploy
a
new
one.
B
I
want
to
give
it
a
name,
let's
say
andrew
demo,
and
I
want
here's,
my
sql,
that
I
have
right
I'm
going
to
deploy
into
this
kubernetes
cluster.
Let's
pick
a
namespace,
let's
put
it
in
default
and
I
create
processor
and
it's
now
being
created.
I
just
need
to
start
it
start
the
sql
processor
and
we
deployed
our
real
time
sql
processing
inside
of
kubernetes.
B
You
can
see
here's
the
pod
name
right
and
that's
it,
so
I
can
do
this
actually
as
a
non-technical
person,
and
I
don't
actually
need
to
know
anything
about
kubernetes,
and
this
is
what
we're
trying
to
do
as
well.
You
know,
for
example,
we
could
scale
it
up
as
well.
Let's
say
we
have
a
spike.
We
could
scale
it
up
and
now
we're
telling
it
to
scale
out
this
this
application.
So
that's
it
right.
I've
gone
from
having
nothing
to
within
five
seconds,
deploying
an
app
inside
of
kubernetes
all
config
driven.
B
You
can
do
this
all
by
the
api.
You
can
put
it
in
version
control.
You
can
have
the
lens
of
cli
actually
apply
your
desired
state.
That's
also
githubs
as
well,
but,
more
importantly,
it's
all
bound
by
the
raw
based
security
that
we
have
inside
of
lenses,
and
this
is
building
that
data
intensity.
How
do
you
scale
out
your
platform
as
well?
I
couldn't,
for
example,
deploy
this.
If
I
didn't
have
the
right,
I'm
logged
in
as
admin
right
as
well,
so
I
can
do
everything,
but
I
also
couldn't
deploy
sql
processor.
B
That
is
maybe
touching.
Data
sets.
Maybe
a
topic
in
kafka
that
I'm
not
allowed
to
see
because
at
the
end
of
the
day,
it's
all
about
building
this
topology,
and
at
that
point
that's
my
bit
done.
So
it's
quite
quick
and
I
will
hand
over
to
fran
to
take
you
through
the
actual
how
we
built
this
application.
There's
my
application.
I
just
deployed
that
andrew
demo
right
run
over
to
you.
Thank
you,
andrew.
C
Yeah,
can
you
see
the
right,
I
hope
so
so
yeah?
When
we
are
talking,
we
are
working
with
streaming
data.
There
are
a
unique
set
of
new
challenges
that
need
that
are
coming
into
the
plate.
We
have
an
unbounded
set
of
data,
we
need
to
manage.
We
have
many
different
sources.
We
need
to
interact
with,
but
more.
C
We,
our
security
model,
complicates
a
lot
and
we
need
to
put
governance
over
the
access
to
our
data.
It's
very
important
to
know
who
who
is
accessing
our
data
and,
in
the
end,
that's
that's
very,
very
important.
The
main
difference
between
real-time
applications
and,
let's
say
regular
applications
and
microservices-
is
that
when
we
are
talking
about
microservices
or
regular
applications,
everything
is
everything
is
self-contained.
We
we
have
our
mysql
instance
deployed
along
our
application.
C
We
can
access
it
most,
probably
from
localhost,
but
when
we
are
talking
about
an
streaming
application,
we
need
to
interact
with
third-party
services
and
we
need
to
interact
with
us
with
with
the
real
world.
Our
data
is
out
there.
We
need
to
bring
it
in
so
to
put
one
more
level
of
complexity
in
in
the
in
the
game.
We
also,
we
are
talking
about
cloud
native
applications.
B
C
C
The
when
we
started
to
talk
about
this
new
component
deployer
component
is
that
we
need
to
think
on
the
platform
as
a
new
service.
It's
not
that
we
are
going
to
be
full
control
over
our
platform.
This
is
what
the
platform
as
a
service
is
talking
about.
We
think
on
this
environment,
where,
where
our
application
is
going
to
be
running
as
an
external
service,
and
we
cannot
rely
on
having
specific
files
in
that
or
having
specific
services
running
into
that
environment,
and
this
consensus
goes
beyond
kubernetes.
C
B
A
C
A
C
The
problem
into
four
different
entities-
secrets,
provider,
connections,
applications
and
deployment
targets.
We
are
gonna
we're
going
to
disguise
each
of
them
in
in
detail
in
the
next
slides.
But
before
that
I
want
to
introduce
one
more
concept
before,
besides
these
four
entities,
unless
the
the
templates,
the
templating
system,
we.
C
When
we
were
starting
to
design
the
solution
that
there
were
some
common
concerns
around
those
those
four
entities
and
in
the
additionally
in
the
end,
at
the
end
of
the
day,
we
realized
that
they
were
basically
the
same
thing.
They
were
basically
descriptors
but
with
different
meanings,
and
in
addition
to
this
fact
or
to
this
situation,
we
were
also
influenced
by
other
technologies
like
hell.
C
That
uses
already
issues
in
templates,
but
we
decided
to
be
the
round
templating
system,
basically
or
most
importantly,
because
we
as
lenses
are
us,
are
also
using
users
of
of
those
entities
and
we
wanted
to
to
bring
governance
and
control
over
them
to
the
lenses
users.
So
the
laser
user
could
control
those
entities
and
bring
security
and
governance
on
them.
C
So
we
ended
up
having
three
different
template
categories:
yeah,
I'm
saying
three
different
things
categories
in
seat
of
four,
because
at
this
moment
we
are
not
modeling
secrets
provided
with
templates.
We
are
considering
considering
moving
them,
but
that's
not
that's
not
where
we
are
at
this
moment.
So
we
what
we
currently
have
are
templates
for
connections
for
applications
and
for
deployments.
C
So
having
these
three
categories,
we
can
now
model
a
connection
as
a
with
a
template.
We
can
model
an
application
with
a
template,
but
most
important
is
that
we
can
model
an
application
with
a
template
that
has
an
explicit
reference
to
another
template
which
is
in
this
case
could
be
a
connection,
and
in
that
way
we
can
apply
governance
and
control
over
the
connection
that
those
applications
can
use,
and
we
are
also
as
a
one
more
detail
is
that
we
are
also
using
these
connection
entities
to
build
our
our
data
catalog.
C
C
As
I
said,
let's
start
by
the
secrets
provided,
but
I'm
starting
from
this.
Basically,
the
security
was
from
the
very
beginning
one
of
our
top
priorities,
if
not
the
very
first
one,
and
we
were
wondering
where
am
I
going
to
store
my
sensitive
data?
Why
not
allow
cloud
security
providers
to
store
my
data?
C
I
should
give
also
hashicorp
keyboard
if
our
clients
are
already
in
azure,
why
wouldn't
they
be
interested
in
installing
their
sensitive
information
there?
So
we
ended
up
deciding
that
we
wanted
to
to
allow
this,
and
ideally
the
the
way
would
be
to
that.
We
would
inject
reference
references
to
these
secrets
and
those
references
will
be
resolved
by
the
application
at
runtime
to
create
the
real
connection.
C
About
the
resolving
these
secrets
properties
at
some
time,
we
we
implemented
the
reference
that
referendum
has
custom
protocol
containing
information
around
the
secrets
provided
managing
that
that's
a
specific
secret,
more
information
around
the
template
key
and
some
other
information
like
the
data
type.
If
it
was
mounted
or.
A
B
C
Stuff,
it's
true
that
we
still
need
to
create
a
trusted
relationship
between
the
application
and
the
secrets
provided,
so
it
can.
The
application
can
fetch
those
details,
but
the
good
thing
about
this
model
is
that
either,
if
we
inject
credentials
specific
credentials
to
the
secret
provider
or
we
rely
on
a
service
account,
we
will
be
able
to
reject
access
to
any
secrets
referring
to
to
any
of
those
services,
just
just
by
restricting
a
single
user
account.
C
C
C
B
C
C
Protocol
you,
you
will
be
using
so
yeah,
that's
why
we
are
treating
that
that
way
and,
as
we
said
before,
we
are
modeling
these
four
entities
of
these
three
entities
with
with
templates
and
one
curiosity,
is
that
these
connections
were
the
first
entity
we
modeled
with
with
templates,
but
we
can
now
have
since
it's
a
category
itself.
We
can
have
multiple
templates
for
connections.
C
C
In
that
same
diagram,
we
can
see
that
we
are
the
government's
ledger.
We
were
mentioning
before,
where
we
are
playing
security
and
control
over
those
entities
and,
basically
that's
why
one
of
the
aspects
that
the
templates
are
are
helping
us
to
to
be
able
to
restrict
access
to
a
full
or
the
whole
service,
just
by
restricting
the
access
to
any
instance
that
was
created
from
from
a
specific
template.
C
Let's
see
how
how
connections
and
and
secrets
provider
are
are
linked,
and
basically
here
we
have
the
connection
templates
saying
that
for
this
connections
type
of
connection,
I
need
to
provide
a
good
sub
server.
I
need
to
provide
a
protocol
or
give
us
all
the
properties
I
need
to
create
a
connection
to
kafka,
but
this
just
a
set
of
them
is
an
example,
but.
C
A
C
This
example,
we
have
two
different
cigarettes,
provided
we
have
lesson
lenses
itself
and
we
also
have
azure
keyboard
so
for
this
example,
we
have
a
kafka
instance
of
the
kafka
for
our
kafka
connection,
template
for
development.
The
name
is
kafka
dev,
and
we
are
saying
that
all
the
properties
are
stored
in
lenses.
So
what
we
are
doing
is
we
are
creating.
C
C
We
we
are
only
referring
to
what
the
application
itself
needs
to
to
run.
We
split
the
split
out
the
application
concern
from
the
deployment
consent,
because
there
are
some
when
we
are
talking
about
the
application.
We
are
not
caring
about
what
the
the
the
underlying
platform
needs
to
run.
Your
application,
we
only
care
about
which
application.
What
are
the
input
parameters
for
your
application
and.
C
A
C
Matter
we
have
splitted
the
or
we
have
make
a
difference
between
in
the
application
templates
between
direct
values
and
reference
values.
These
reference
values
we
were
talking
before
allows
us
to
create
to
have
references
to
to
act
to
other,
to
other
templates,
so
having
our
application
manifest
when
we
are
describing
an
application.
Sorry,
the
kafka
connection.
We
are
going
to
say
that
that
specific
property
is
not
going
to
be
provided,
but
by
the
user
itself
directly
when
creating
the
application,
but
that
is
going
to
come
from
an
instance
of
a
connection
of
a
different
template.
C
B
C
In
this
example,
we
have
what
we
already
saw
before
the
connections
instances
and
where
we
have
our
properties
stored,
and
here
we
have
the
kafka
connection
template
these
instances
have
been
created
from
it.
Then
we
have
the
new
element,
which
is
the
application
template,
and
the
application
templates
again
is
exactly
exactly
the
same
than
the
kafka
connection.
C
So
once
we
have
this,
and
we
want
to
create
an
instance
of
the
application
template,
the
only
thing
we
need
to
provide
is
the
sql,
which
is
a
direct,
let's
say,
a
direct
property
and
the
name
of
the
kafka
connection.
That
is
going
to
the
cafe
connection
instance.
That
is
going
to
provide
the
references
to
all
these
links.
We
did
here
so
the
important
thing
around
this
is
that,
just
by
changing
this
kafka
connection
reference
here,
we
are
going
to
be
able
to
very
easily
change
the
source
of
the
data
we
are
reading
from.
C
So
we
are
going
to
be
able
to
promote
applications
from
one
environment
to
another
and
bringing
information
from
one
place
to
a
to
another.
Here.
This
is
how
it
works.
In
the
end,
once
we
are
creating
the
application,
what
we
are
sending,
what
we
are
really
registering,
let's
say
when
we
are
creating
the
instance
of
the
application,
is
that
we
are
fulfilling
all
the
properties
that
our
template
application
is
defining,
but
the
values
are
coming
as
a
reference
to
the
underlying
system
where
that
value
is
stored.
C
A
C
Connect
and
bringing
the
final
overview
mixing.
It's
not.
There
are
a
couple
of
small
diagrams,
but
let's
say
that
here
is
how
we
are
cooking
everything
the
four
different
entities
at
the
same
time,
and
what
we
are
doing
bringing
in
the
common
template
is
that
basically,
we
are
building
a
funeral
or
digital
template
compounding
component
from
two
different
templates,
the
deployment
and
application.
C
A
C
C
So
as
soon
as
this
step
is
done,
we
have
the
certainty
that
that
is
a
deployable
unit
that
we
mean
we
will
be
able
to
deploy
at
any
time
to
to
kubernetes
what
if
we
were
nervous,
is
available,
and
then
once
the
application
is
here
is
when
we
are
going
to
be
is
when
the
application
is
going
to
to
really
get
the
real
details.
The
actual
details
that
it's
going
to
use
to
to
open
the
connection
between
the
application
and
the
third-party
service.
C
But
it's,
it
is
k2
to
any
type
of
application.
We
just
need
to
support
new
sorts
of
applications.
The
only
thing
we
need
to
do
is
create
a
new
template,
a
custom
template
for
up
for
our
application,
and
then,
if
we
want
to
connect
to
a
set
to
a
service
that
we
don't
already
have
a
template,
we
can
create
a
connection
template
for
it
as
well.
So
we
are
able
to
create
applications.
We
are
able
to
create
connections.
We
are
able
to
model
any
possible
application,
not
only
the
sql
processors
and
yeah.
C
The
last
piece
I
want
to
mention
is
that
we
are
not
only
focusing
on
the
right
side
of
the
spectrum.
We
are
also
taking
care
of
the
application
when
once
the
application
is
running
in
the
underlying
system,
so
it's
not
only
about
deploying
it's
also
about
knowing
and
monitoring
what's
happening
there.
C
So
we
have
built
a
set
of
watches
for
each
of
these
underbody
and
the
landing
platforms
that
we
support,
and
then
we
translate
the
the
information
we
get
from
from
them
into
our
own
events,
api
and
then
we
are
able
to
manage,
handle
those
events
and
create
different
views.
Materialization
start
metrics
and
so
on,
and
just
these
are
two
very
brief
themes
from
from
our
internal
documentation.
C
This
is
the
the
general
or
the
global
overview,
how
we
are
linking
the
deployment
side
with
the
watchers
or
the
creed
side
and
how
we
are
then
creating
an
output
channel
running
a
stream
that
contains
events
that
we
can
use
to
to
to
materialize,
different
views
or
to
start
metrics.
As
we
already
said,.
C
Just
an
another
pin,
this
is
a
very
an
under
a
specific
implementation
detail
on
kubernetes
how
we
are
doing
this
injection
and
this
resolution
of
the
of
the
secrets
properties
at
runtime.
Basically,
we
are
deploying
the
processor.
We
are
creating
an
image
container.
That
is
the
one
connecting
to
the
third,
the
secrets
provider,
it's
resolving
the
secrets
and
it's
creating
a
a
file,
a
shell
in
a
shared
location
between
the
init
container
and
the
main
container.
That
will
be
running
the
application,
but
everything
is
inside
inside
the
same
pod.
C
So
it's
ephemeral
in
the
end
and
won't
be
persisted
anywhere.
That's
all
I
have
andrew
anything.
You
want
to
sorry.
Well,
I
have
one
one
more
site,
sorry
I
I
forgot
about
this
one,
but
basically
that's
the
the
summary.
What
we
are
beginning
with
all
these
stuff
we
have
been
talking
about
is
that
we
are
enabling
detox.
We
are
able
to
to
move
applications
around
different
environments,
and
especially,
we
are.
We
are
bringing
in
governance
and
security
control
over
the
the
locations.
B
B
Kubernetes
is
great,
but
you
know
in
five
years
time
I'm
sure
there'll
be
kubernetes,
mark
two
right
and
a
different
version
of
it
and,
as
I
said
at
the
start,
the
application
landscape
is
the
important
bit
yeah.
So
if
you
want
to
try
lenses,
you
can
go
to
lenses,
dot,
io,
start
and
download
the
box
and
we're
always
interested
to
hear
your
feedback.
A
A
A
B
No,
so
we
we
don't
so
lenses
will
run
across
any
cafe.
It
can
be
any
distribution
of
kafka
on
any
cloud
with
any
kubernetes
cluster
as
well
openshift,
for
example,
or
a
aks
or
eks
the
sql
engine
we
have
is
our
own
sql
engine
right.
You
know,
there's
some
advantage
to
that.
We
have
the
control,
but,
like
you
say
you
know,
maybe
you
decide
the
apache
kafka
isn't
for
you
and
you
want
to
go
to
pulsar
or
you
want
to
go
to
ready
streams.
For
example,
we
can.
A
B
Yes,
so
the
sql
processor
basically
takes
the
the
sequel,
and
at
the
moment
it
boils
down
to
a
k,
string
map.
So
k
stream
is
a
kafka.
Apache
kafka
connect
api
for
doing
the
joins
and
aggregates.
You
can
write
the
java
code
if
you
want
to,
but
our
own
sql
engine
takes
that
and
basically
translates
it
down
to
a
case
frame
application
that
we
then
deploy.
We
can
deploy
it
to
kubernetes
we
can
deploy
inside
of
kafka
connect.
B
We
can
run
it
actually
inside
of
lenses
as
well,
but
that's
only
one
part
of
the
sql
engine
and
you
know
we
want
to
swap
kafka
out
for
other
streaming
technology,
but
we
also
have
our
own
sql
engine
that
allows
you.
We
call
it
snapshot
engine
where
you
can
take
a
relational
database
style
query
on
top
of
kafka
itself
and
also
on
all
the
systems
we
connect
to
as
part
of
date,
catalog
for
elasticsearch,
for
example.
B
Yes,
so
lenses
has
a
rule-based
security
model.
So
when
you
log
into
lenses-
and
you
can
log
in
via
ldap
active
directory-
for
example,
a
single
sign-on
you're
into
a
group-
and
then
those
groups
have
virtual
name
spaces
in
inside
them
and
a
set
of
permissions
on
top
of
that.
So
we're
able,
for
example,
to
provide
virtual
multi-tenancy
on
top
of
kafka
and
that's
every
api
will
honor
that
as
well.
B
So
you
can
have
example,
create
topic
permission
or
view
data
permission,
for
example
as
well,
and
then,
when
you're
deploying
an
application,
we
respect
those
role-based
rules.
So
I
can't
deploy
an
application.
If
I
don't
have
the
deploy,
sql
processor
room-
and
I
can't
create
a
processor
on
a
topic
that
I'm
not
allowed
to
see
as
well
and
everything
then
is
auditing.
And
it's.
We
also
have
another
thing
in
here
because
of
data
policies
that
we
can
actually
mask
the
data
from
with
sql
engine
as
well.
B
B
B
So
we
also
have
the
ability
to
see
the
data
in
elasticsearch,
not
only
the
schemas
of
the
indexes,
but
we
can
also
fire
the
sql
as
well.
So
you
can
query
bound
by
the
same
rule-based
security
and
governance
we
have
with
the
auditing
and
the
data
policies
as
well.
We're
extending
that
out,
for
example,
we'll
add
postgres
support
soon.
B
A
B
Yes,
so
everything
in
lenses
is
an
api
and
we
have
the
lenses
cli.
So
that's
for
more
of
a
traditional
ci
cd
process,
but
we
are
working
on
a
githubs
operator
in
fact,
a
few
weeks
ago,
and
even
at
the
kafka
summit,
we
did
a
demonstration
of
this
where
there's
effectively
a
lenses
operator,
monitoring,
git
and
we'll
speak
to
lenses
and
have
applied
the
desired
state
as
well,
because
we
will
have
all
the
knowledge
and
the
control
inside
of
lenses
to
make
sure
that
we're
applying
the
governance
as
well.
So
that
is
coming.
A
Excellent,
just
a
quick
reminder
to
everybody:
if
you
have
questions,
please
leave
them
in
the
q,
a
box
at
the
bottom
of
your
screen
and
not
the
chat
section
just
so
that
this
way
I'll
make
sure
that
I'll
see
it
when
we're
doing
the
questions.
This
next
question
here
does
lenses
monitor
the
real-time
applications
after
deploying
them.
B
B
B
I'm
not
sure
what
that
question
is,
but
what
we
are
doing
is
we're
expanding
the
data
catalog.
Maybe
I
think
it's
around
the
data
catalog
to
actually
have
the
visibility
into
different
systems
such
as
postgre,
but
also
maybe
it's
moving
beyond
kafka,
which
is
certainly
something
that
we're
doing
as
well.
For
example,
red
streams,
there's
apache
pulsar
as
well,
and
we
certainly
want
to
bring
the
same
data
intensity
that
we
did
with
the
kafka
ecosystem
onto
those
other
systems
as
well.
B
Yeah,
so
it's
any
any
kafka.
Obviously
we
want
you
to
be
on
one
of
the
more
recent
versions,
but
it
could
be
msk.
It
could
be
azure,
hd
insight,
it
could
be
kafka
from
ivan
instra
cluster
or
your
own
on-premise
apache
kafka.
It
doesn't
matter
to
us
and
it's
the
same,
the
same
with
kubernetes
as
well,
and
if
you
want
to
you
know,
try
us
out
in
the
aws
marketplace
or
the
hd
insight
marketplace,
we're
in
there
there's
cloud
formation
templates
as
well.
B
At
the
the
current
version,
you
can't
you
can
deploy
it
outside
and
you
can
use
our
sdks
or
rest
endpoints
and
register
it
with
lenses,
and
we
will
monitor
it
as
well.
But
this
framework
that
fran
has
put
in
place
will
allow
us
to
deploy
anything
effectively.
That's
in
a
in
a
docker,
certainly
when
it
comes
to
kubernetes
or,
for
example,
maybe
nomad
or
azure
container
instances
of
fargate
right
lenses
could,
in
theory,
deploy
itself.
So
in
this
floor,
you
write
your
own
app
register.
B
The
template
with
lenses
builds
the
app
catalog
and
with
combination
of
the
data
catalog,
then
we
can
deploy
the
relevant
apps
to
do
what
you
want
to
do.
What's
your
business
objective
move
data
from
a
to
b:
that's
normally,
it's
not
standing
up
applications,
so
it
allows
us
to
actually
do
this
going
forward,
but
you
can
register
your
own
apps
now
for
texas,
for
example
the
example
of
a
customer.
A
B
So
you
can
actually
we're
working
on
the
git
ops
operator
for
lenses.
Now
that
operator
could
run
inside
of
kubernetes
or
outside
of
kubernetes.
What
it's
doing
is
it's
telling
lenses
to
deploy
an
application
and
lenses
may
be
configured
to
deploy
into
kubernetes,
so
it
may
be
configured
to
deploy
into
kafka
connect
it.
It
makes
no
difference,
so
it
will
work
across
them
both.
This
is
why
we
have
the
separate
templates
one
for
deployment
as
well.
So
kubernetes
is
a
deployment.
B
Template
kafka
connect,
it's
a
deployment
temporary
and
we
can
choose
now,
which
is
more
scalable,
kafka,
connect
or
kubernetes.
Well,
it's
kubernetes
right
kafka
connect
was
doesn't
really
handle
multi-tenancy
that
well
by
using
kafka,
connect
and
kubernetes.
We
can
create
a
effectively
catholic
connect
instance
cluster
per
instance,
and
get
a
lot
more
multi-tenancy
in
there,
for
example,
that
if
you
add
another
connector,
you
don't
maybe
disrupt
the
flow
of
another
instance.
That's
deployed
there.
B
So
we
see
this
quite
a
lot
that
with
connect,
you
have
to
have
end
up
with
a
lot
of
clusters
to
manage
and
kubernetes
is
a
better
way
to
scale
it
out.
But
we
provide
all
the
governance
around
that,
so
whether
it's
kafka
connects
or
kubernetes
is
your
deployment
targets
right.
You
can't
deploy
unless
you
have
the
correct
permission
models
as
well,
and
we
order
all
that.
A
B
B
B
So
I
was
just
gonna
just
finish
off
really
on
that.
You
know
so
you
you
can
deploy
lenses
anywhere,
so
you
can
deploy
in
the
cloud
you
can
deploy
on
an
ec2
instant.
You
can
deploy
lenses
inside
of
kubernetes,
so
we
have
helm
jobs
for
that
as
well
or
you
can
deploy
on-prem
as
well.
That's
no
problem,
and
we
also
have
a
portal
as
well,
which
gives
you
an
aggregated
view
of
all
your
different
lenses,
instances
that
are
deployed.
A
Just
another
heads
up
everyone:
if
you
have
any
questions,
please
drop
them
into
the
q
a
box
just
so
we
can
keep
all
the
questions
in
there
separate
from
the
chat.
That
would
be
great.
Does
the
application
deployment
work
only
with
java
or
or
scala
applications?
Would
it
work
with
net
apps,
for
example,.
B
So
if
you're
deploying
to
kafkaconnect
it
would
have
to
be
your
jvm
app,
because
kafkaconnect
is
jvm.
If
you're
deploying
to
kubernetes
well,
we've
abstracted
away,
you
know,
you've
put
an
app
inside
a
docker,
it's
docker,
plus
config,
and
then
lenses
will
be
able
to
deploy
and
manage
that
bound
by
the
security
governance
as
front
explained.
So,
yes,
you
could
write
a
python
app.
You
could
write
a
dot
net
app
register
the
template
with
lenses.
It's
part
of
the
app
catalog.
A
A
All
right:
well,
no
one
has
anything
else
they
would
like
to
ask.
I
guess
we
will
wrap
it
up.
I
want
to
thank
our
presenters,
andrew
and
fran,
for
a
wonderful
presentation
today
and
for
a
great
q
and
a
facilitation,
and
I
would
like
to
thank
everyone
for
joining
us
today.
As
I
said
before,
the
recording
and
slides
will
be
posted
later
today
to
the
cncf
webinar
page
at
cncf,
dot,
io
webinars,
and
that
being
said,
we
will
wrap
it
up.
Thank
you.