►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Managing Apache Flink on Kubernetes - FlinkK8sOperator - Anand Swaminathan, Lyft
We have designed and built an open-source Kubernetes native operator that manages the complete lifecycle of Apache Flink applications from creation to execution. FlinkK8sOperator (https://github.com/lyft/flinkk8soperator) leverages Kubernetes CustomResourceDefinition to enable native management of Flink applications on Kubernetes. In this session, I will be presenting some of the unique challenges of running a complex, stateful application on Kubernetes, and the lessons we have learnt. I will also be providing an overview of how flink operator abstracts out the complexity of hosting, configuring, managing and operating 1000s of Flink clusters from application developers, and concluding with a demo.
https://sched.co/UabA
A
Hello:
everyone,
my
name,
is
Anand
swaminathan
I
woke
on
a
team
called
as
flight
at
left
flight,
a
cloud
native
machine
learning
and
batch
processing
system
as
a
part
of
flight.
We
were
exposed
to
kubernetes
and
operators.
Fling
kubernetes
operator
was
built
as
a
collaboration
between
flight
and
the
streaming
platform.
A
So
for
today's
agenda
we
are
going
to
look
at
the
flink
operator
how
it
has
helped
us
run
applications
on
kubernetes,
but
before
just
looking
into
the
solution,
I
want
to
show
you
how
streaming
is
very
critical
to
lifts
business
and
some
of
the
problems
that
we
faced
with
a
legacy
deployment
on
the
non
kubernetes
world.
Then
I
will
also
show
you
how
kubernetes
has
solved
these
problems.
We
will
end
a
talk
with
the
demo
and
also
leave
some
time
for
questions.
So
lift
business
involves
providing
transportation
as
a
service.
Everything
is
real
time.
A
You
open
the
app.
You
can
request
bike
scooters
cars.
You
can
also
look
at
trans
information
in
real
time,
so
what
this
makes
is
that
this
makes
streaming
one
of
the
most
critical
components
for
our
business.
If
you
look
at
this
picture,
you
can
actually
see
so
many
components
that
has
streaming
involved.
For
example,
it
goes
from
pricing
to
eta2
notifications,
to
coupons
to
messaging
and
so
on.
I
just
want
to
give
you
a
very
explain
a
particular
use
case
that
will
be
familiar.
A
So
if
you
open
the
lift
app,
you
can
see
the
nearby
cars
and
the
way
we
do
it
is
every
car
once
a
driver
opens
the
app
and
joins
live.
We
start
sending
events
from
the
app
towards
the
system,
and
this
is
where
the
ingestion
into
a
streaming
platform
begins
and
from
there
we
help.
You
show
the
real-time
location
of
the
driver,
the
closest
estimate
of
the
time
the
driver
would
take
to
arrive
you
and
once
you
enter
the
destination,
you
will
also
see
the
time
it
takes
to
reach
your
destination.
A
Now,
just
start
to
maintain
high
levels
of
service.
We
have
pricing
involved,
a
real-time
pricing
involved
and
this
is
directly
dependent
on
the
amount
of
drivers
that
are
near
you
and
so
on.
So
both
pricing
and
eta
is
one
of
the
most
components
of
her
business
and
they
are
powered
by
the
streaming
platform.
A
So,
as
I
had
shown
you
the
use
cases,
it
makes
sure
that
and
I
want
to
convey
that
how
streaming
compute
framework
is
very
important
if
we
rewind
the
clock
two
years
back,
we
just
are
streaming
involved,
just
kinases
and
AWS
kinases,
client
library.
So
the
way
we
did
it
was
view
build
applications
by
writing
it
on
Python
and
we
used
a
doubles
KCl
with
Multi
multi
Python
multi-line
daemon
to
build
the
application,
but
the
problem
was
KCl
was
not
very
powerful.
A
If
you
wanted
to
do
something
like
aggregation
or
batching
or
grouping,
you
had
to
build
all
these
complex
systems
on
top
of
KCl
and
for
one
example,
is
that
the
way
we
did
this
was
we
did
it
was?
We
would
read
the
data
from
kinases
using
a
CC
CL,
you
would
write
to
Redis
and
then
we
will
have
a
separate
feat
of
workers,
sharded
by
know
certain
regions
or
a
mate
hashed
who
would
read
from
these
Redis
and
then
process
further.
A
So,
as
you
can
see,
like
first
doing
a
simple
group
by
operation,
we
had
to
build
these
complex
systems
with
you
know
having
multiple
systems
in
wall,
and
this
was
really
affecting
our
reliability.
So
two
years
back,
we
introduced
flink
into
into
our
ecosystem,
and
it
has
done
wonders.
Flink
is
excellent
because
it
has
comes
up
with
a
rich
feature
support.
It
has
amazing,
functional
api
support
for
writing,
simple
analytical
applications,
and
you
can
simply
do
like
group
by
map
order
by
very
easily
in
just
using
simple
functions.
A
It
has
amazing
sequel
support
which
we
critically
needed.
I
said
I
already
mentioned:
almost
all
our
applications
were
written
on
Python,
so
we
did
not
want
to
rewrite
the
application.
So
beam
support
was
very
important
to
us,
so
we
would
instead
just
write
beam
pipelines
which
would
just
convert
then
convert
into
a
fling
job
and
then
run
run
it
pretty
easily.
So
these
kind
of
features
made
filling
pretty
attractive,
but
it
came
with
the
downside.
The
downside
is
that
Flynn
Flinx
deployment
model
is
a
bit
complex,
unlike
KCl,
which
is
like
stateless.
A
You
can
deploy
it
like
any
of
the
stateless
service
and
you'll
automatically
a
charge
to
your
workers.
Flink
had
a
topology,
and
that
makes
the
deployment
model
complex.
I
would
talk
about
more
about
it
in
the
next
few
slides.
So
how
do
we
run
flink
at
lift?
We
have
beans
which
right
flink,
application
directly
on
Java,
like
ETA,
and
these
Java
applications
in
turn,
convert
or
fling
job
and
run.
There
are
also
people
who
write
beam
pipelines
in
Python.
A
These
beam
pipelines
gets
converted
to
fling
job,
basing
a
fling
runner
and
they
also
they're
being
used
by
the
pricing
team
at
left
and
on
top
we
have
a
team
called
as
drift.
Their
use
case
is
that
they
take
sequel
queries
from
research,
scientists
and
data
engineers,
who
we
then
convert
these
sequel
queries
into
a
corresponding
fling
job,
and
you
know,
make
sure
their
processing
is
pretty
simple,
as
you
can
see
that
all
of
this
is
powered
by
the
streaming
platform
under
needs.
A
A
Had
mentioned
to
you
about
the
complexity
in
the
deployment
model
of
link,
so
this
is
the
flink
architecture.
Flink
has
a
job
manager
and
a
task
manager.
The
role
of
the
job
manager
is
to
act
as
a
coordinator,
and
the
rule
of
the
task
manager
is
to
is
to
actually
do
the
work.
The
way
you
do
it
is
you
provide
the
job.
The
job
managers,
the
job
man
that
does
the
work
of
coordinating
the
execution
by
converting
the
job
graphs
into
an
execution
graph.
It
handles
coordinating
checkpointing
recovery
of
task
managers
and
so
on.
A
The
task
managers
in
turn
have
these
tasks
slots,
which
perform
a
unit
of
work.
So
if
you
want
to
deploy
flink
application,
you
have
to
build
this
sort
of
topology
of
having
a
job
manager
and
a
task
manager
and
basically
making
sure
that
you
know
the
task
manager
can
talk
to
the
job,
medicine
and
so
on.
So
I
will
show
you
how
we
deployed
flink
in
a
non
cognitive
world.
First,
so
--lifts
deployment
was
through
saltstack
saltstack
is
the
configuration
management
system
that
we
used
to
deploy
all
of
the
stateless
services?
A
This
could
be
like
any
sense,
like
you
know,
authentication,
service
or
a
right
service,
which
is
non
flink.
Even
all
stateless
services
was
deployed
using
Saul's
tag
where
we
would
directly
say
the
resources
that
we
want
and
those
would
end
up
being
created
in
AWS.
So
what
we
did
was
we
wrote
a
wrapper
or
a
layer
in
between
the
support,
flink
application
so
and
how
it
appeared
is
that
if
you
write
a
flink
application,
we
would
create
three
things.
A
One
is
you
will
create
an
auto
scaling
group
in
AWS
for
the
job
manager
of
size
one
then
we
would
create
an
auto
scaling
group
for
your
task
manager
for
the
size
that
you
that
you
need,
and
then
we
will
also
create
an
elastic
load
balancer,
which
helps
the
task
manager,
discover
the
job
manager
so
which
means
only
if
the
job
answer
goes
down.
You
need
an
a
static
IP
for
all
the
task
manager
to
talk
to
the
job
manager.
A
If
you
want
to
deploy
a
new
code,
what
we
end
up
doing
is
you
have
a
bunch
of
scripts
where
we
make
sure
that
we
say
point
the
current
job
and
start
the
new
jobs
from
the
same
from
this
previous
save
point
on
the
new
code.
So
all
of
this
was
pretty
manual.
It
was
highly
error-prone
and
that
sort
of
things
what
we
wanted
to
fix
when
you,
when
you
build
the
flink
operator,
to
move
to
kubernetes.
A
So,
just
talking
more
about
the
problems
that
we
face
in
our
vc
deployment,
as
I
had
mentioned,
every
single
fling
job
requires
you
to
spin
up
a
filling
cluster
witches
to
auto
scaling
groups
and
an
elastic
load
balancer,
but
on
top
of
it,
if
you
had
multiple
versions
of
the
flinging
job,
for
example,
one
for
staging
and
one
for
production,
you
would
now
have
two
groups.
Have
you
know
two
job,
two
auto-scaling
groups
each
and
an
elastic
load
balancer?
A
So
naturally,
you
can
see
that
you
can
infinitely
create
or
a
scaling
group
there's
a
clear
limit,
and
what
we
ended
up
doing
is
that
each
team
ended
up
creating
just
one
cluster
and
started
running
multiple
jobs
inside
the
same
cluster.
Now
you
might
think
that
this
kind
of
works,
because
you
know
the
flink-
does-
can
run
multiple
jobs
and
they
all
run
in
the
corresponding
task
managers.
But
the
problem
is
that
you
are,
you
might
end
up
running
like
you
know,
one
high
priority
job,
which
is
very
critical
to
your
business.
A
One
low
priority
job,
which
is
not
so
critical,
which
can
have
a
downtime
and
your
low
priority
job.
Might
end
up
affecting
your
high
priority
job,
there
could
be
a
memory
leak
in
one
job
which
would
end
up
killing
the
entire
task
manager,
and
that
would
end
up
affecting
all
the
other
jobs
in
the
cluster.
The
second
biggest
problem
that
we
faced
was
since
we
are
running
on
AWS
anytime,
a
node
goes
down.
A
if
you
want
to
automatically
scale
and
if
you
want
to
run,
if
you
want
to
add
new
task
manager,
that
also
takes
more
than
10
minutes
for
us
to
scale
which
might
not
be
acceptable.
When
you
know
the
traffic
is
suddenly
spiky
and
you
want
to
scale
the
other
biggest
biggest
problem
was
I
had
mentioned
the
manual
error-prone,
a
manual
process
involved
in
deployment
where
we
had
to
go
to
the
job
manager,
run
the
script,
to
switch
jobs
and
so
on.
So
this
process
was
slick
error-prone.
A
Our
configuration
files
were
all
over
the
place
and
also
some
other
things
like
you
know.
We
want
to
do
an
automated
rollback
like,
for
example,
let's
say:
I
deploy
a
bad
code
and
the
code
is
not
running.
I
want
something
to
happen
so
that
my
old
code
comes
back
and
the
so
is
just
not
down
so
all
kinds
of
stuff
like
these
we're
not
supported
in
our
legacy
system.
A
So
let's
go
look
at
a
solution.
The
solution
is
fling
kubernetes
operator,
but
before
you
look
at
the
flame
kubernetes
operator,
if
you
want
to
use,
if
you
want
to
run
flink
application
on
kubernetes
using
the
fling
kubernetes
operator,
the
way
to
do
it
is
by
creating
the
flink
application
customer
resource,
so
the
flink
application
custom
resource
is
the
specification
that
you
provide
to
run
your
filling
job
so
constraint
that
we
have
is
each
flink
application.
Custom
resource
will
create
one
fling
cluster
and
we'll
only
run
one
fling
job
in
it.
A
So
if
you
want
to
see
a
run,
ten
fling
jobs,
the
way
you
do
it
is
by
creating
ten
flink
application.
Customer
resources
and
the
flink
operator
would,
from
there
go
on
and
do
all
the
heavy
lifting
for
you.
There
is
one
particular
specific
field
in
the
specification
that
I
want
to
highlight.
That
is
the
image.
So,
when
you
create
a
specification
or
a
spec
in
the
flink
application
customer
resource,
the
image
that
we
expect
has
to
be
runnable.
A
What
I
mean
by
that
is,
we
should
be
able
to
start
the
job
manager
and
the
task
manager
from
the
image
since
fling
kubernetes
operator
is
open
source.
We
also
have
a
bunch
of
examples
inside
our
repositories
for
to
help
you
build
these
runnable
images,
but
that
is
one
of
the
most
critical
component
that
you
have
to
do
in
order
to
use
flink
application
on
kubernetes.
A
So
while
creating
the
flink
application
customer
resource,
you
would
provide
a
specification,
and
that
becomes
the
desired
state
that
you
expect
the
cluster
to
be
and
what
flink
operator
does
is
once
from
the
resource.
It
will
start
populating
the
status
field.
The
status
field
is
actually
the
current
state
of
the
cluster
and
I
would
talk
more
about
how
the
status
is
filled
in
the
next
slides.
A
So
when
it
comes
to
the
architecture
of
filling
kubernetes
operator,
this
diagram
might
look
very
similar
to
a
general
kubernetes
diagram.
There
is
a
control
plane
and
there
is
an
execution
plane.
What
flink
operated
appears
is
as
an
extension
to
the
kubernetes
control
plane.
So
let
me
take
the
deployment
exam.
A
Let's
say
if
you
were
to
create
a
deployment
and
kubernetes
the
way
you
do
it
is,
you
would
pass
in
the
deployment
object
to
the
API
server
and
the
deployment
object
is
stored
in
that
CD
and
once
the
object
gets
stored
in
HCD,
your
deployment,
controller
picks
up
the
object
and
we'll
create
corresponding
replica
sets.
The
same
happens
in
in
Frink
world
as
well.
So
the
moment
you
create
the
flink
application
custom
resource,
the
flink
operator,
fetches
the
resource
from
its
CD
using
the
api
server
and
it
will
create
the
corresponding
fling
cluster.
A
So
what
is
a
fling
cluster
in
the
kubernetes
world?
The
flink
russian
kubernetes
world
consists
of
the
following.
It
has
a
job
manager
deployment.
It
has
a
task
manager
deployment,
a
job
manager
service
which
the
task
manager
can
use
to
talk
to
the
job
manager
and
also
an
ingress
for
the
flinky
wide.
So
you
can
you
want
to
be
able
to
access
the
UI
from
your
laptop
and
so
on,
so
for
you
to
make
any
requests
to
the
job
manager
from
extend
from
outside
the
kubernetes
cluster.
A
We
have
created
an
ingress
for
you,
so
this
group
of
job
manager,
task
1,
is
the
deployment
the
job
manager
service
and
the
ingress
together
is
what
we
call
as
a
flink
cluster
in
kubernetes.
Now
let
us
see
how
the
flink
operator
does
all
of
this,
but
before
that,
what
is
the
fundamental
concept
that
is
behind
this
operator?
So
I
want
to
talk
about
control
loop,
because
this
is
the
fundamental
building
block
of
any
controller.
A
This
could
be
a
deployment,
control
and
controller
or
a
replication
controller,
and
this
fundamental
building
block
is
the
same
for
flink
operator.
The
way
it
works
is
the
controller
looks
at
the
current
state
of
the
world
and
from
the
spec.
It
creates
a
desired
state
of
the
world,
so
what
it
then
does
is
it
does
a
diff
between
the
current
state
of
the
world
and
the
desired
state
of
the
and
takes
an
action
to
move
the
current
state
of
the
world
to
the
desired
state.
A
So,
and
the
core
of
the
flink
operator
is
a
state
machine.
When
you
the
way
this
operator
works,
it
transitions
the
custom
resource
from
one
state
to
another.
So
when
you
first
create
the
flink
application
custom
resource,
what
happens
is
you
are
actually
in
the
new
state?
Now
there
is
no
cluster
for
you,
so
what
the
operator
does
it
first
moves
the
custom
resource
to
the
create
clusters
step
where
the
job
manager,
the
task
managers,
are
all
created
once
the
resources
are
created.
The
flink
operator
then
goes
to
wait
for
the
all.
A
The
parts
to
come
up
so
this
a
time
duration
between
when
you
create
all
the
resource
to
when
all
the
path
pods
start
coming
up
and
once
all
the
pods
are
up
the
operator
realizes
that
hey
now
it
can
go
and
start
a
job
in
the
job
manager,
and
it
does
this.
Does
it
by
you
know
transitioning
to
the
submit
job
state
in
the
submit
job
state?
It
starts
a
job
in
the
job
manager
and
if
everything
goes
well,
the
application
goes
to
running.
A
So
the
running
here
is
the
desired
state
that
we
want
the
custom
resource
to
be
in.
There
is
one
more
path
that
I
want
to
explain
before
we
go
to
a
demo.
I
will
be
showing
all
of
this
in
the
demo
as
well.
So
let's
say
you
have
a
fling
cluster
that
is
up
and
running,
and
now
you
make
a
code
change
in
Java
and
then
you
build
an
image.
You
want
to
dip
your
code
now
to
a
running
cluster.
So
the
way
you
do
it
is
just
by
simply
updating
the
flink
application
custom
resource.
A
So
once
you
update
the
resource,
the
operator
now
sees
that
hey
my
desired
state,
that
is
a
spec,
does
not
match
with
the
current
cluster
that
I
have
created.
So
what
happens?
Is
it
changes
the
application
resource
from
running
to
updating?
Now,
once
it
moves
to
updating
what
the
operated
does
is
it
brings
up
a
second
cluster
for
the
new
spec
that
that
was
just
updated
so
which
means
that
now
in
the
updating
state
you
have
two
clusters
running
one
cluster
is
the
one
that
is
actually
doing
the
work.
A
The
second
cluster
is
just
coming
up
once
the
second
cluster
is
up
successfully,
which
means,
if
there's
a
bug
and
the
second
cluster
does
not
come
up.
There
is
no
impact,
your
application
at
all,
so
once
the
second
cluster
is
up
and
running,
we
cancel
the
job
in
the
first
cluster
by
creating
at
the
save
point
and
then
start
the
job
completely
in
the
second
cluster,
from
the
save
point.
A
So
as
you
can
clearly
see
that,
if
something
is
going
wrong
in
your
code,
is
the
second
person
not
coming
up
or
if,
let's
say
you
know,
once
a
cancels
job
and
I
start
the
job
in
the
second
cluster.
If
the
job
submission
is
failing,
we
automatically
roll
back
to
the
first
cluster.
We
do
not
delete
the
first
cluster
until
the
job
is
running
successfully
in
the
second
cluster,
so
this
is
roughly
the
working
of
a
state
machine.
I
want
to
just
show
you
a
demo
of
everything
that
I
explained
so
far.
A
So
for
the
demo,
this
is
the
blink
application.
Custom
resource
that
I
would
be
creating.
Here
is
the
image
all
of
this
is
available
and
filling
kubernetes
operator
open
source
repo.
So
you
can
all
try
it
out.
You
know
whenever,
so
this
image
is
runnable,
as
in,
like
you
know,
from
this
image,
I
can
start
the
job
manager
and
the
task
manager.
There
are
a
few
other
fields
like
you
know:
job
manager,
config
task
manager
config,
since
these
are
just
not
indicating
that
how
many
CPU
that
I
want
for
the
job
manager.
A
A
A
So
I
am
now
creating
the
custom
resource
blink
application,
so
that
I
just
showed
you
and
let's
see,
what's
happening
behind
the
scenes
so
currently
what's
happening.
Is
that
the
operator?
Let
me
just
show
you
slowly
so
the
moment
I
created
the
custom
resource
the
operator
picked
it
up.
It
realized
that
the
current
faces
a
new
face
and
it
ended
up
creating
the
cluster,
and
once
the
cluster
was
successfully
created,
what
happened
was
the
operated
transition,
the
custom
resource
to
a
cluster
starting
state?
A
What
this
means
that,
as
I
had
explained
in
the
cluster
starting
state,
we
are
just
waiting
for
all
the
pods
to
come
up
once
all
the
pods
are
up.
The
operator
then
goes
to
submit
the
job,
as
you
can
clearly
see
here
that,
once
all
the
pods
are
up
for
the
cluster,
the
operator
moves
to
a
submitting
job
State
and
it
submits
the
job
in
the
job
manager
that
was
just
created
and
from
there
you
can
see
that
once
the
job
was
started,
it
moved
to
a
running
State.
A
The
second
thing
is,
even
though
the
cluster
is
currently
running,
the
operator
is
continuously
monitoring
the
existing
cluster.
So,
let's
see
if
the
task
manager
is
down
of
there
is
any
issues
with
the
with
the
filling
cluster.
The
operator
continuously
monitors
the
existing
cluster
and
updates
the
status
field
of
the
flink
application
resource.
So
let
me
just
show
you
what
I
exactly
mean.
A
So,
as
you
can
clearly
see
that
for
every
application
that
you
create,
we
we
pose
the
corresponding
filling
cluster
and
notify
the
status
of
the
job.
How
many
check
points
number
of
healthy
tasks,
measure
and
a
bunch
of
things
that
we
monitor
from
the
existing
cluster
and
we
store
it
in
the
status
field
of
your
application,
which
means
that,
if
you
want
to
monitor
your
application,
you
can
literally
just
use,
read
from
the
status
and
see
what's
exactly
happening,
and
on
top
of
it,
we
also
made
matrix
for
all
your
failing
applications.
A
A
Now
this
is
one
more
thing
that
I
wanted
to
show
you,
and
that
is
what
will
happen
if
I
go
and
update
my
application
resource.
So
for
that,
let
me
just
start
the
threads.
These
are
just
pulling
and
the
resources
the
operator
is
still
running
so
now.
What
I'm
doing
is
I
am
going
and
updating
the
docker
image
part
of
it.
So
my
Amal.
A
A
A
We
have
started
the
job
in
the
new
cluster
and
once
the
job
is
started
in
the
new
cluster,
you
can
see
that
the
existing
cluster
is
automatically
terminated.
So
the
first
question
that
was
running
the
job
is
now
terminated
because
we
were
able
to
successfully
launch
the
job
in
the
second
cluster.
So
this
is
an
intimate
example
of
how
fling
kubernetes
operator
was
able
to.
You
know.
Take
all
the
updates
from
your
application
resource
and
translate
that
to
a
corresponding
cluster.
A
So
what
is
the
impact
of
all
of
this?
The
the
moment
we
moved
from
our
legacy
system
to
running
flink
application
on
kubernetes.
As
you
can
clearly
see,
we
have
one
fling
job
for
purfling
cluster.
There
is
no
sort
of
interference,
all
high
priority
jobs
run
separately,
all
low
priority
job
run
separately,
scaling
the
superfast
and
anytime.
A
pod
goes
down
or
goes
down.
A
The
third
was,
the
configuration
became
much
easier,
anything
that
we
want
to
deploy.
We
had
to
create
a
corresponding
flink
application
resource,
so
we
wrote
wrappers
on
top
of
our
existing
flink
application
resource
to
set
these
default
values
for
task
manager
for
the
number
of
task
slots
for
task
manager.
So
any
user
who
wants
to
run
a
fling
job
just
had
to
write
like
three
to
four
lines
of
code
and
he
would
have
the
flink
application
running
in
kubernetes.
A
Lastly,
we
saw
a
huge
cost
saving
because
of
all
of
this
I
said
mentioned
in
the
previous
world,
we
had
one
fling
cluster.
That
was
highly
provision
because
we
did
not
know
how
many
jobs
you
would
end
up
running
and
we
did
not
want
to
roll
the
cluster
frequently.
So
we
created
like
these
big,
these
big
machines
or
ec2
machines,
to
run
these
monsters
filling
clusters,
but
in
the
New
World
each
fling
cluster
is
just
running
one
job.
A
So,
since
you
know
the
job
that
you're
going
to
run,
you
can
provision
the
resources
that
you
exactly
need
and
this
helped
us
save
a
lot
of
cost.
So
what
is
the
current
state
of
the
project?
So
the
all
of
this
is
open
source.
Any
of
you
guys
can
just
go
to
github.
Fling
kubernetes
operator
spat
the
operator
in
your
in
your
in
your
for
your
business
or
your
use
cases
immediately.
We
call
the
project
status
as
beta,
because
it
helps
us
do
some
backward
incompatible
change
in
case.
A
A
A
Lastly,
my
team
at
flight
is
giving
more
talks
tomorrow.
There
is
one
tomorrow
morning
at
10:55
and
one
at
5:20
in
the
evening.
If
you
are
interested
in
running
like
large
scale,
machine
learning,
jobs
or
large-scale
batch
processing,
job
I
would
highly
recommend
you
guys
to
come
check
it
out.
Also
flight
is
also
open
sourced
we
open-source
flight
a
month
back.
So
it's
also
something
that
you
can
immediately
adopt
and
start
using.
A
A
B
A
A
A
So
this
is
a
fling
application
that
is
currently
running
in
order
for
kubernetes
cluster,
and
you
can
actually
see
the
top
part
of
the
URL.
So
what
we
have
done
it
is
when
we
create
and
ingress,
we
specify
the
application
name
in
the
host
path,
so
which
means
that
each
host
path
will
no
point
of
corresponding
ingress,
which
will
then
route
the
request
to
the
corresponding
job
manager.
So
that's
what
we
have
done
and
also
its
considerable.
So
what
we
have
done
it
is.
A
C
A
That's
a
great
question
so
inflamed
what
happens?
Is
you
have
a
fling
cluster
running?
There's
like
a
bunch
of
checkpoints.
Each
task
manager
creates
a
one
checkpoint
of
the
of
the
entire
graph.
So
what
happens
is
that
when
we
want
to
change
code
from
one
cluster
to
other,
the
very
transition
is
by
creating
this
safe
points.
A
point
is
a
concept
in
fling.
We
store
all
our
save
points
in
s3
and
the
path
of
the
save
point
is
stored
in
the
flink
custom
resource.
So,
for
example,.
A
So
you
can
actually
see
that
okay,
we
should
have
done
it
during
the
yeah
right
here.
So
if
you
look
at
the
restored
path
so
for
my
example,
I
am
using
a
local
store,
but
what
we
do
is
we
store
all
the
save
points
in
s3
and
the
path
of
the
s3
would
be
available
here,
which
means
that
when
we
start
the
new
filling
cluster
we
say:
hey
start
the
new
fling
cluster
from
this
particular
save
point
more
questions.
A
So
we
started
off
with
using
operator
SDK
to
bring
build
the
fling
kubernetes
operator,
but
we
wanted
to
do
some
lower
level
things
as
well,
so
we
ended
up
directly
using
controller
runtime,
so
controller
runtime
is
also
open.
Source
is
something
that
we
used
directly
for
building
and
running
google
dis
operator.
What
I
mean
is
yep.
A
This
is
what
I
mean,
so
we
use
this
particular
package,
the
coolest
controller
runtime,
to
build
Plinko
business
operator.
We
use
the
operator
SDK
before
we
move
to
this,
but
since
you
want
to
support
like
multi
namespace,
we
want
the
operator
to
read
only
from
specific
namespaces.
So
for
building
like
these
sort
of
like
complex
use
cases,
we
had
to
use
controller
runtime
directly.
C
A
A
very
good
question
as
a
part
of
this
project:
it's
not,
but
that
is
something
that
we
have
been
talking
for
a
while,
since
that,
now
you
have
all
the
jobs
running
separately,
you
want
a
UI
which
would
show
the
aggregate
of
all
the
jobs
not
directly,
but
one
thing
that
we
are
trying
to
do
solve
that
is
using
integrations
with
flight,
so
what
we
will
be
so
think
of
it.
This
way,
apart
from
flink,
we
also
run
spark
and
sparkles.
A
It
is
similarly,
so
you
want
to
just
have
an
overview
of
not
just
fling
just
but
spark
jobs
or
any
other
batch
processing
jobs.
So
that's
running
in
the
cluster.
So
as
a
part
of
this
work,
it's
not,
but
we
are
trying
to
solve
some
of
that
using
our
integration
with
flight,
another
part
of
it
so
yeah
yeah.
C
A
A
But
if
you
were
to
split
it
up
into
separate
jobs,
you
can
just
tune
the
exact
amount
of
resources
that
you
would
end
up.
Creating
the
only
overhead
on
top
of
that
is
that
we
are
running
multiple
job
managers,
so,
for
example,
each
instead
of
having
one
job
manager
I'd
be
running
two
job
managers
purfling
cluster.
So
it's
like
times
two,
but
if
you
actually
see
that
the
amount
of
CPU
and
memory
for
a
job
manager
is
actually
not
that
high,
so
we
run
it
with.
B
A
D
Yeah,
so
when
you
were
talking
about
your
like,
like
it's
the
infrastructure,
you
had
like
your
auto
scaling
capabilities
took
a
long
time
when
you
switched
over
to
this
kubernetes
based
deployment
model.
Did
you
allow
for
like
horizontal
node
scaling,
or
did
you
say?
Okay
now,
if
you're
starting
to
you
know,
get
to
limited
your
clusters
capacity,
you
add
a
node
or
like
what.
How
did
you
guys
deal
with
that
and
then,
if
it
was
automated
like?
How
did
that
reduce
your
note
scaling
time
with
that?
That's.
A
A
very
good
question
so
currently
there
is
no
trigger
based
auto
scaling
in
plink.
So
what
I
mean
by
that
is:
let's
say
that
you
know
you
have
a
rough
filling
cluster
that
is
currently
running
and
suddenly
say
at
five
pm
that
traffic
is
higher,
and
you
see
that
this
is
like
a
curve
of
increase
in
CP
and
memory.
A
We
don't
automatically
have
like
triggers
to
say
that
hey
when
the
CPU
is
more
than
70%
automatically
add
new
task
managers,
so
we
we
do
it
at
lyft.
Is
that
by
updating
the
parallelism
of
the
customer
resource?
So
if
you
actually
look
at
the
spec
right,
we
have
a
field
called
as
parallelism.
That
is
the
spot
that
needs
to
be
updated.
A
So
today
what
we
have
been
doing
is
we
have
a
script
where
in
ignore
we
update
the
parallelism,
so
essentially
what
parallelism
does
is
based
on
the
parallelism
and
the
number
of
tasks
slot
manager
decides
the
number
of
task
managers
so
by
just
updating
the
parallelism.
We
go
through
this
path
of
you
know,
adding
more
pods.
The
difference
is
that
the
previous
Willie
had
to
add
machines.
A
Now
we
just
have
to
add
new
purchases,
which
was
the
huge
improvement,
but
to
answer
the
question
trigger
based,
auto
scaling
is
what
we
are
planning
to
build
in
the
operator
itself,
so
that
you
know
you
can
have
certain
triggers
there,
which
would
make
the
fling
cluster
scale
automatically,
but
I
believe
flink
is
also
doing
work
in
this
regard
to
automatically
scale
horizontally.
But
it's
also
in
the
roadmap
of
filling
kubernetes
operator.
Yeah.
D
A
That's
very
good
question,
so
the
cost.
Let
me
state
this,
so
we
have
like
more
than
100
applications
and
that
we
are
running
in
the
non-communist
world.
We
have
migrated
roughly
like
30
percent,
mainly
the
ones
that
are
roll
loader
labs.
For
example,
drift
runs,
40,
flink
jobs
in
a
single
single
single
fling
cluster,
and
it
will
go
down
always
like
every
two
weeks.
So
when
we
migrated
40
percent,
we
saw
roughly
20
to
30
percent
reduction
in
cost.
A
But
again
this
number
depends
on
how
well
you
tuned
the
job
manager
in
the
task,
whereas
a
conflict.
In
our
case,
what
happened?
Will
it
just
came
like
five
different
teams
and
they
did
a
good
job
of
setting
the
right
values.
But
if
you
end
up
setting
like
extremely
high
job
manager,
what
task
manager
resources
you
might
not
end
up
seeing
the
Cosman
fit,
but
it
comes
down
to
how
you
accurately
you
set
the
resource
for
your
job
yeah.
D
A
Yeah
yeah,
that's
true,
actually
one
more
thing:
on
the
cost
side,
our
sentiment
there
is,
we
have
an
another
way
of
doing
cost.
The
way
we
do
it
is
we
have
is
something
we
have
a
service
called
as
infraspinatus
another
coupe
con
talk
on
it
as
well.
That
guy
would
take
metrics
from
all
the
pods,
and
you
know,
drill
down
by
namespace
and
give
us
a
view
for
the
cost.