►
From YouTube: Running Large-Scale Stateful Workloads On Kubernetes at Lyft - Surinder Singh & Anmol Khurana, Lyft
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Running Large-Scale Stateful Workloads On Kubernetes at Lyft - Surinder Singh & Anmol Khurana, Lyft
Along with core services, K8s at Lyft also forms the base to run a large variety of data processing stateful data processing jobs which includes Spark, Flink and other jobs via various ML and Data processing pipelines. At Lyft, K8s has become the driver for the majority of our data processing needs running 10s of thousands of concurrent jobs. Operating the platform at this scale presents an unique set of challenges which get more complex with highly variable load pattern. In this talk, the speakers will share their journey through some of these challenges and learnings. - Potential pitfalls of running stateful jobs on K8s. - Knobs/tweaks to optimize K8s for stateful jobs. - Running k8s in a cloud environment. - Building a fault-tolerant self-healing system with multiple K8s clusters underneath. Talk will also focus on optimizations done to support the widely used workloads at Lyft.
https://sched.co/Uacl
A
My
name
is
Linda
Singh
with
my
colleague
here
and
mol.
We're
gonna
talk
about
how
we
run
stateful
workloads
at
scale
on
Kuban.
It
is
at
lyft
and,
let's
see
so
rough
agenda
of
the
talk
is
going
to
be,
will
first
define
what
those
stateful
workloads
are
we'll
give
some
examples
we'll
go
through
some
use
cases
at
left
and
then
we're
going
to
talk
about
our
platform
flight,
so
flight
is
our
in-house
build
platform
where
we're
on
most
of
the
lifts
straight
for
workloads.
We
do
have
a
full
presentation
on
flight
at
5:20
p.m.
A
I
will
request
everyone
to
go
attend
that,
but
for
the
sake
of
this
conversation,
we'll
go
over
the
high-level
we'll
make
sure
that
everyone
has
the
context
for
the
rest
of
the
talk
and
then
we'll
move
on
to
what
were
the
platform
goals.
When
we
were
defining
when
we
were
designing
that
platform
for
those
stateful
workloads,
and
then
we
will
move
on
to
what
were
the
pitfalls
and
learnings.
We
had
while
running
this
platform
on
human.
It
is
for
almost
a
year
now
and
what
did
we
do
for
the
issues
we
encountered
we'll
go
over?
A
A
Another
example
is
a
very
standard
detailed
or
where
there's
a
raw
data,
you
load
that
data
you
extract
load
and
then
I
put
it
into
a
form
that
can
be
consumed
by
other
users
at
the
company
and
then
yet
another
one
is
data,
things
pipelines.
We
have
the
raw
data
we
extract
features
and
then
data
scientists
come
and
work
on
models
once
they
have
a
good
model.
A
They
push
it
to
prod
for
influencing
and
some
examples
from
lift,
as
you
guys
know
that
by
the
virtue
of
what
we
do,
optimization
is
an
important
problem
for
us
and
to
do
good
at
it.
We
have
machine
learning
models
everywhere
for
pricing,
for
ETA,
for
locations
for
Maps
data
and
these
models.
Some
of
them
are
really
complex
and
then
the
platform
was
designed
so
that
we
can
onboard
all
of
these.
At
the
same
time.
So
most
of
these
models
run
on
flight.
There
are
ETL
jobs
running
on
flight.
Then
there
are
data,
backup
jobs.
A
What
what
they
do
is
they
do
a
periodic
backup
of
the
data,
a
logical
replication
of
the
data,
with
some
check
pointing
in
between
they
also
run
in
flight
today
and
then
in
the
end,
we
also
have
some
like
giant
simulation
framework.
Why
do
we
have
that?
As
since
I
said,
we
are
not
of
the
data
science
pipelines
in
the
company
and
we
cannot
always
run
them
in
prod
to
see
how
good
they
are
doing.
A
So
we
have
very
sophisticated
simulation
system,
but
sophistication
means
complexity,
and
then
we
actually
have
a
challenging
journey
to
support
these
simulation
workloads
on
flight.
So
today
they
run
on
flight
like
every
day
tons
of
them.
Ok,
moving
on
what
is
flight,
so
flight
is
a
platform
which
was
designed
to
orchestrate
machine
learning
and
data
workflows.
So
this
was
the
primary
goal,
but
secondary
goal
was
also
ensure
that
there
is
good
collaboration,
reuse
and
performing
those
ml
operations
which
in
general,
has
to
do
every
day.
We
wanted
to
make
that
super
easy
for
the
engineers.
A
Let
me
go
over
some
of
the
cool
features,
so
I
just
came
through
the
these
features
and
then
we're
going
to
talk
about
these
in
the
later
slides.
First,
one
of
our
server
list
we
wanted
to
be
server,
less
user
should
not
be
worried
about
what's
behind
the
scene,
multi-tenant
and
shareable
operational
excellence
was
a
requirement
for
us
because
being
a
very
generic
platform,
we
want
to
make
sure
that
people
running
on
a
platform.
They
can
run
their
things.
They
can
look
into
things.
They
can
debug,
diagnose
things
whenever
they
need.
Extensibility
was
important.
A
This
was
our
learning
from
a
previous
incarnation
of
flight,
where
we
notice
that
anytime,
we
have
to
onward
a
new
use
case.
We
wanted
to
do
something
special,
so
we
wanted
to
build
a
platform
where
extensibility
is
a
cold
feature
of
it
and
the
last,
but
not
the
least,
is
the
interaction
modes.
Today
we
have
G
RPC
rest
endpoint.
We
also
have
a
UI
for
manual
interaction
with
the
system,
so
some
more
concepts
tasks
is
the
atomic
unit
of
a
flight
workflow.
A
Think
of
it
like
a
single
container
toss,
which
could
be
a
Java
or
a
Python
program,
or
it
could
be
a
multi
container
job,
think
of
it
less
bunk
job
when
I
say
a
task
as
a
atomic
unit,
which
means
we
cannot
decompose
that
further.
So
you
use
those
tasks,
you
wrap
them
with.
What
are
the
resources?
You
need?
What
is
the
relationship
with
the
rest
of
the
nodes?
And
that
becomes
a
note
for
you
and
you
take
those
bunch
of
nodes.
Do
you
define
a
graph?
A
It's
called
a
work
flow
in
our
system,
and
so
workflow
is
set
of
nodes.
Some
dependencies
between
them
and
both
tasks
and
workflows
can
have
inputs
as
well
as
outputs.
Okay.
This
is
a
pictorial
depiction
of
our
graph
looks
like
on
the
left.
You
see
a
input
node,
which
is
the
input
to
the
workflow.
Then
there
arrows
going
towards
right,
which
means
that
those
nodes
on
the
right
they
need
those
inputs
and
then,
wherever
you
see
an
arrow
between
two
nodes,
which
means
there
is
a
dependency
between
those
nodes.
A
This
is
a
UI
but
primitive
form.
We
are
working
on
it
and
then
there
are
links
to
logs
and
other
things
on
the
UI.
Okay,
let's
dive
a
little
deeper.
So
this
is
the
block
diagram
of
a
flight
system.
On
the
left,
you
see
as
a
user
print
component.
There
is
a
CLI.
There
is
a
flight
client.
There
is
an
SDK
to
interact
with
the
system
in
the
middle.
There
is
a
flight
admin
that
is
a
control
plane.
A
Even
though,
in
this
diagram
it
seems
like
it's
part
of
the
execution
cluster,
but
it
is
not
so
we
made
a
conscious
choice
to
move
our
control
plane
out
of
the
execution
plane
on
the
right.
You
see,
are
the
components
of
our
execution
plane
where
there
is
a
flight,
propeller
and
then
the
bunch
of
other
operators
and
services.
Flight
propeller
is
the
brain
or
hard
whatever
you
want
to
call
it
or
the
end
of
the
system
where
what
it
does
is
it
goes
through.
A
A
So
we
looked
at
the
overall
picture.
There
are
many
components
in
system:
I
don't
want
to
overwhelm
you
guys
with
too
much
details,
but
for
the
rest
of
the
talk,
the
thing
which
is
really
important
for
us
to
understand
this
flight
propeller,
because
that
is
the
guy
who
has
a
single
cluster
view.
So
it's
running
inside
a
cluster.
It
is
the
guy
interacting
with
the
rest
of
the
Kuban.
It
is
components
and
what
it
does
it.
It
looks
through
all
the
works
within
the
system.
It
has
a
it's
a
standard
cuban
area
so
operator.
A
There
is
a
control
loop
where
it
goes
through.
All
this
workflow
symmetry
to
the
system
looks
at
the
current
state
parses
the
graph
traverses
the
graph
gets
to
the
node,
which
is
in
progress,
updates
the
status
and
moves
on
to
the
next
one.
So
it
keeps
doing
that
in
a
loop
till
we
have
the
workflow
in
the
completion
state
and
it
uses
HCD
at
the
States
or
because
we
wanted
something
consistent
over
there
and
since
you
can
understand,
this
is
part
of
the
critical
path,
because
all
the
workflows
are
going
through
this
guide.
A
Okay,
and
this
is
the
input
to
the
system.
So
when
I
say,
flight
propeller
works
on
the
workflows,
workflows
are
represented
at
the
customer
resource.
So
if
you
look
at
the
customer
resource
here,
there
are
nodes,
nodes
have
names,
they
are
kind
and
there
are
inputs
to
the
node,
and
then
there
are
bindings
bindings
is
a
way
where
you
kind
of
say
this
is
my
relationship
of
this
node
with
the
other
node,
and
then
there
are
inputs.
If
you
just
look
at
the
input
there,
is
you
see
a
structure
over
there?
A
The
second
part
of
the
cid
is
the
status
status
is
not
given
by
the
user
but
updated
by
a
flight
propeller.
As
we
progress
with
the
workflow.
We
keep
updating
the
status
with
more
information.
When
we
have
it
available,
we
update
the
node
status.
When
we
have
it.
We
write
output
directories
where
we
are
pushing
the
output
and
any
other
information
we
have
so
one
thing:
I'll
call
it
so
output
I
say
we
have
it
available
because
we
actually
ensure
that
when
the
task
runs
completely,
then
only
we
make
the
output
available.
A
This
was
needed
to
avoid
partial
failures
and
we
are
open
source
since
last
month
we
have
a
good
integration
with
a
bunch
of
open
source
technologies.
We
have
a
spark
Jupiter
notebook,
hi
q-ball,
and
then
we
also
today
run
on
AWS
and
Google.
There
are
more
things
coming
so
next
is
platform
goals.
I'll
ask
unroll
to
walk
you
through
platform
goals
and
some
of
the
other
things
Thank.
B
B
Jobs
usually
present
a
very
different
set
of
load
characteristics
compared
to
long
running
services,
in
the
sense
that
the
load
is
usually
very
bursty,
as
can
be
seen
on
the
chart
on
the
right
at
live
for
stateful
workloads,
we
run,
or
we
start
thousands
of
containers
per
minute
as
well
as
we
run
tens
of
millions
of
containers
per
month.
So
the
next
goal
we
wanted
to
talk
about
is
multi-tenancy
again
at
left
using
flight.
We
have
hundreds
of
users
using
our
system
or
platform
concurrently
running
tasks
on
them
across
multiple
teams.
B
So
you
want
to
make
sure
that
there's
isolation
and
fairness
in
the
system.
What
that
means
is
you
want
to
make
sure
that
one
user
of
the
system
is
not
able
to
impact
other
users
running
tasks
or
workflows?
At
the
same
time,
you
also
want
to
make
sure
that
one
users
or
one
user
or
a
set
of
users
or
a
team,
can
only
use
a
small
slice
of
overall.
You
know:
resources
or
combinational
power
of
the
platform,
a
couple
of
other
goals.
We
want
to
just
walk
you
guys
through
our
performance
and
extensibility.
B
B
This
can
be
the
initial
setup
time
where
the
user,
actually,
when
it
when
he
or
she
launches
a
workflow
the
time
it
takes
from
that
that
chap
to
the
time
the
tasks
are
actually
running
in
the
system,
as
well
as
the
transition
time
between
different
states
of
the
workflow
when
it's
getting
driven
by
flight
propeller
extensibility.
We
also
want
to
make
sure
that
the
platform
is
easily
extensible
and
allows
users
to
add
support
for
new
task
types.
B
Okay,
so
the
last
set
of
goals.
So
you
want
to
just
quickly
walk
you
guys
through
our
observability
and
visibility.
What
it
means
is
we
want
to
make
sure
users
or
teams
running
on
flight
have
insight
into
how
many
workflows
they
are
running,
how
many
tasks
they
are
running,
but
not
only
that
we
also
want
to
make
sure
they
have
insight
into
the
resource
utilization
for
those
workers
or
those
stars.
B
For
example,
if
you
as
a
user,
are
running
a
spark
job
which
asks
for
hundred
executors
with
50
gigs
of
memory
each,
but
you
are
actually
only
consuming
five
gigs
or
just
using
10%
of
the
memory
you
requested,
for
we
want
to
make
sure
you
have
you
get
that
insight
into
that.
This
sort
of
also
stems
into
the
next
next
goal,
which
is
the
infrastructure
cost
optimization.
B
So
once
you
have
insight
into
how
much
resources
you
are
actually
using,
you
can
actually
go
and
optimize
for
infrastructure
cost,
but
besides
that,
even
at
the
system
level
or
the
platform
level
we
want,
we
are
looking
at
a
couple
of
things
to
optimize.
The
overall
cost,
like
you
know,
exploring
low
cost
spot
instances
as
well
as
trying
to
optimize
based
on
various
utilization
patterns.
B
Okay,
so
those
were
the
goals
we
had
in
mind
while
designing
flight.
So
now
that
we
have
flight
running
in
production
for
close
to
a
year-
and
we
have
a
large
number
of
lifts,
you
know
you
stateful
workloads
running
by
a
flight.
Let
us
talk
about
some
of
the
issues
we
ran
into
and
in
some
of
the
learnings
we
had
based
on
in
the
last
year.
B
Okay,
so
the
first
few
learnings
are
around
scale
while
when
we
actually
ran
flight
on
single
humanity's
cluster.
So
the
very
first
issue
when
we
ran
flight
and
started
onboarding
new
users,
is
that
we
started
seeing
high
latency
for
API
server.
Api
server
calls.
What
happened
is
whenever
the
aggregate
count
of
Cuban
IDs
objects
on
the
cluster
goes
beyond
a
certain
threshold.
B
Besides
this,
we
also
started
seeing
that
when
we
are
running
thousands
of
workflows,
concurrently
CUBAN,
it
is,
was
really
garbage
collecting,
completed
pods
really
aggressively
and
what
happened.
What
this
leads
to?
What
this
causes
is
flight
propeller,
which
is
sort
of
the
brain
and
looks
at
the
status
of
these
parts
it
has
spawned
and
to
do
basically
to
drive
the
workflow
from
one
state
to
the
next.
B
So
a
fixie
is
that
hey
I
spawned
these
five
part
for
this
particular
task
in
a
workflow,
and
these
five
parts
are
done
its
able
to
mark
the
task
as
done,
and
you
know,
spawn
new
tasks
or
any
dependent
tasks
in
the
workflow.
But
when
Kuban
it
is
when
at
scale
Kuban,
it
is
really
aggressively
garbage
collect
so
deletes
these
completed
parts
or
these
pod
resources.
Flight
propeller
is
not
able
to
consume
that
that
these
parts
are
completed
and
the
work
flows
and
ended
up
being
failing,
even
though
the
parts
might
have
the
successfully
completed.
B
So
these
were
the
two
issues
we
saw
at
the
cluster
level.
We
also
saw
an
interesting
issue
at
particulate
level,
which
I
want
to
share
with
you
guys.
So
we
when
we
started
we,
we
were
using
large
machine
instance
types.
We
were
using
I,
think
machine
instances
with
half
a
terabyte
of
memory
and
I
think
around
60
or
70
CPUs,
and
some
of
our
workloads
or
user
workloads
were
really
small.
They
were
requesting
pods
with
just
half
CPU
resource.
B
So
these
were
sort
of
the
three
crystal
level
issues
we
ran
into.
So
how
did
we
solve
them
so
just
to
limit
the
oral
aggregate
count
of
community
resources
or
objects
in
the
cluster?
We
started
using
resource
quotas
to
limit
the
number
of
pods
or
number
of
CRTs
missing
in
a
single
cluster.
Besides
that,
we
also
started
aggressively
garbage
collecting
our
own
computed
workflows.
So
these
workflows,
which
have
completed
there,
are
no
more
state
relations
for
them
and
they
can
be
safely
deleted.
B
So
we
started
aggressively
deleting
those
workflows
as
well
as
any
ports,
any
spark
CRTs
or
any
other
dependent
resources
that
particular
workflow
owns.
Finally,
to
solve
the
cube,
let
level
issue
we
had.
We
started
using
heterogeneous
machine
pools
where
some
of
the
instance,
some
of
the
instances
we
chose
were
smaller
ones.
Some
of
the
instances
were
larger
ones
and
overall
we
we
were
under
that
pod
limit,
like
what
limit
of
100
parts
per
per
node,
which
we
which
we
saw
that
works
fine
without
any
stability
issues.
B
Okay,
so
those
were
the
cluster
level
issues
and
learnings.
So
let's
talk
about
some
of
the
state
management
level
learnings
we
had
while
running
flight
at
scale
for
these
stateful
workloads.
So
as
Sauron
the
talked
about
flight
propeller
is
an
cuban.
It
is
operator
which
implements
a
control
loop
and
is
responsible
for
driving
each
workflow
from
one
state
to
the
next,
and
you
mentioned
to
completion.
So
what
happened
when
we
ran
sort
of
a
load
test
from
the
very
initial
v1
version
of
flight
propeller
for
the
scale
we
wanted
to
support.
B
We
started
seeing
that
this
processing
time
the
propeller
takes
for
each
workflow
ceoddi,
was
really
high
and
was
beyond
the
sort
of
I
suppose
we
wanted
to
expose
your
users.
So
so,
when
we
investigated
this,
we
saw
that
this
was
largely
because
of
the
eight
seed
actually
write
x
for
the
work
flow
CRD
and
those
write
x
actually
write
hands
were
really
high.
So
how
did
we
go
about
solving
them?
One
of
the
ways
we
actually
solved.
B
It
is
by
implementing
a
version
cache
where
we
will
write
the
we
will
store
the
last
written
version
to
a
CD
in
a
separate
cache,
and
if
we
are
getting
an
older
version
for
that
CRD
from
Informer,
we
will
just
discard
it
thereby
saving
time
on
processing
time
as
well.
As
you
know,
saving
time
on
trying
to
write
to
HDD,
we
also
started
using
updating
the
CR
D
using
the
sub
resource
update
API,
which
recently
went
live
in
one
of
the
recent
Cuba
Nerdist
versions.
B
So
this
basically
for
some
of
our
workflow
CRTs,
the
spec
part,
can
be
really
huge.
We
saw
that
some
of
our
users
were
writing
really
complex
workflows,
which
had
like
hundreds
of
tasks
and
the
spec
in
some
of
these
cases
can
go
into
mb/s.
So
by
just
using
by
this
updating,
that's
the
status
part
of
it.
We
could
say
writing
that
spec
to
Cuban
it
is
blue
HDD.
Finally,
we
are
also
exploring
trying
to
offload
that
spec
altogether,
especially
when
the
spec
is
in
MVS.
B
We
can,
if
you
can
offload
respect
to
something
like
an
external
external
blob
store
like
s3
and
just
show
the
reference
in
a
treaty
that
will
actually
also
help
us
improving,
improve
that
CD
write
times
as
well
as
reduce
the
overall
size
of
data
stored
energy,
D,
ok,
so
multi-tenancy,
so
multi-tenancy
was
one
of
our
another
core
goals
for
the
platform,
isolation
and
fairness.
We
want
to
make
sure
that
the
system
is
isolated
and
there's
fairness
across
users.
So
how
do
we
achieve
that
again?
Partly
by
using
cuban?
B
It
is
resource
coders,
so
we
enforce
coders
on
aggregate
count
of
cpus
or
memory
is
a
user
or
a
set
of
users
within
a
team
can
consume
in
the
cluster,
but
some
of
our
task
types
we
support
our
external.
They
are
not
cuban,
it
is
native.
So
for
those
types
I
we
sort
of
maintain
our
own
in-memory
local
resource
resource
store.
We
will
maintain
quotas
for
those
for
each
namespace
for
those
task
types
so
that
there's
fairness
for
those
task
type
as
well.
B
Another
issue
we
ran
into
when
we
started
using
humanities
resource
quotas
is
that
we
started
seeing
that
the
api
server
latency
is
were
again
very
high,
especially
for
the
create
pod
api.
For
some
of
the
cases
we
saw
that
p90
heating
is
largest,
you
know
going
into
minutes.
We
investigated
this
and
we
what
we
found
out
the
create
pod
api
call
actually
invokes
all
the
web
hooks.
You
have
configured
in
the
cluster
and
then
goes
and
invokes
the
resource
quota
at
Mission
Control.
B
So
this
call
is
really
expensive,
especially
if
you
have
a
sizable
number
of
web
ebooks
configured
in
the
cluster,
which
was
the
case
for
us,
and
this
call
was
actually
taking
really
really
taking
a
very
high
latency.
So
we
were
partly
to
blame,
for
this
is
because
we
were
sort
of
even
if
a
namespace
had
headaches,
resource
coders
and
had
like,
let's
say
hundreds
of
workflows
in
pending,
we
would
still
pro
flight
propeller,
which
is
responsible
for
driving
that
workflow
to
the
next
shade.
B
This
chill
will
still
try
to
process
each
single
workflow
in
that
namespace
and
try
to
create
the
pods
it
can
create
in
the
inter
cluster.
So
for
these
workflows,
which
what
hitting
resource
code
has
we
were
still
trying
to
issue
create
pod
API
calls
to
to
the
API
server
every
single
time
we
were
processing
those
workflows,
so
we
are
not
backing
off
correctly
in
this
cases.
So
one
of
the
solutions
we
are
actually
implementing
right
now
is
to
maintain
sort
of
a.
It
is
a
quota
cash
for
each
namespace.
B
B
B
So
the
way
we
achieve
this
is
by
using
a
separate
set
of
reserved
nodes
with
chains
and
toleration
x'
and
then
using
node
affinity
to
place
these
system
pods
like
flight
propeller
on
to
those
specific
reserve
notes,
so
flight,
propeller
and
other
admin
component
or
other
flight
components
which
which
are
a
single
power
which
run
as
a
single
pod
and
drive
all
the
workflows
in
the
system.
You
also
want
to
make
sure
that
this
fairness
across
users
for
those
components.
B
Performance,
so
performance
was
another
one
of
our
core
goals.
One
of
the
ways
we
achieve
performance
or
improve
performance
of
the
system
is
by
providing
a
feature
set
called
discoverable
tasks
so
but
discoverable
tasks
too,
is
it,
will
just
skip
expensive
tasks,
executions
and
to
use
cash
results.
B
If
the
task
logic
and
inputs
haven't
changed,
what
that
means
is,
if
you
have
run
the
same
task
in
the
past,
with
the
same
task
logic
and
the
same
task
inputs,
we
will
just
cache
those
outputs
generated
at
that
particular
instance
in
the
in
the
past,
and
if
you're
running
the
same
task
again
now
we
will
just
skip
that
task,
revision
completely
and
just
reuse
those
outputs
generated
in
the
past.
This,
so
not
only
helps
with
performance
is
in,
but
this
also
helps
with
you
know,
skipping
expensive
computation
and
improves
cost.
B
Some
of
the
other
ways
we
improve
performance
of
the
system
is
by
using
node
affinity
for
especially
for
multi
container
data
intensive
tasks
like
SPARC.
We
spawn
like
hundreds
or
like
a
large
number
of
executors
and
a
driver
and
then
there's
a
data
shuffle
between
those
executors
and
driver,
so
placing
them
on
the
same
node
over
a
very
small
set
of
nodes
helps
with
the
oral
performance
of
the
task.
We
are
also
looking
into
implementing
a
write
through
cache
for
workflow
CR
DS.
This
basically
helps
us
reduce
the
HDD
load
for
gets
an
update.
A
So
the
natural
choice
was
to
move
to
cluster
autoscaler,
which
we
did
and
which
saw
a
decent
saving
which
is
almost
double
and
what
it
does.
It
say
it
has
a
greedy
approach
of
putting
in
the
pods
on
the
nodes
which
are
already
in
the
system
running
some
part,
but
still
have
margin
to
accommodate
another
another
pod.
This
is
this
ran
fine,
so
we
were
able
to
like
increase
utilization
from
twenty
to
forty
percent,
but
still
is
not
good
enough.
A
So
the
next
step
was
that,
okay,
even
though
autoscaler
is
doing
good,
it's
just
that
schedule
itself
is
not
good
enough.
So
we
started
looking
at
the
cube
bat
scheduler
and
we
ran
it
on
a
load
test,
but
we
still
have
not
kind
of
using
it
in
prod,
but
I'll
share
the
basic
idea
behind
that.
The
basic
idea
is
that,
since
we
are
running
bats
kind
of
a
load,
there
are
always
temporary
bursts.
A
We
don't
want
to
scale
up
and
scale
down
all
the
time,
because
whenever
you
scale
up,
there
is
a
trigger
which
will
scale
up.
There
will
be
a
time
for
which
the
machines
will
be
sitting
idle
before
they
can
go
down.
Another
is
that
whenever
you're
scaling
up
some
time
to
use
on-demand
instances
and
those
are
expensive,
so
we
wanted
to
kind
of
smooth
an
out
those
temporary
births.
So
what
we
want
is
we
want,
instead
of
scheduler
like
autoscaler,
getting
more
machines
when
there
is
a
pending
pods
in
the
system.
A
We
want
the
other
way
round.
We
want
a
cute
state
in
the
system,
which
means
that
when
there
is
a
queue
in
the
system
and
the
queue
is
building
more
than
what
we
anticipated,
then
only
we
want
to
scale
up
like
if
there
is
less
a
queue
of
10
which
we
completely
okay.
With
we
know
it's
a
temporary
burst
will
go
away.
We
don't
want
to
scale
up.
So
that's
something
we
are
trying
right
now.
We
are
also
defining
quality
of
service
classes.
A
So
in
our
system
today
there
are
critical
workloads,
which
means
when
they
come
in.
They
cannot
wait
and
what
we
have
done
there
is.
We
have
over
provision
capacity
for
them.
So
when
they
come
in,
we
don't
put
them
in
a
queue.
We
just
directly
schedule
them.
This
works
fine,
it's
just
that,
not
everything
in
political
in
the
system.
What
do
we
do
about?
Non-Critical
work,
you
it's!
The
workloads
which
come
up
which
come
with
non
stick
deadlines.
They
can
wait
in
the
queue.
So
what
we
do
over
there
is
the
same
thing.
A
A
So
in
v-0,
what
we
are
doing
is
we
are
putting
a
separate
AZ
cluster
for
spot
instances,
which
means
that
again,
the
non-critical
workloads,
which
are
which
can
wait,
will
put
them
in
a
queue
and
whenever
we
see
that
there
are
spot
instances
available
at
a
cost
which
we
are
ok
with,
will
as
many
as
we
need
and
then
we'll
flush
of
the
queue
and
more
touched
on
discoverable
TAS.
Discoverable
task
is
a
way
where
we
try
to
identify
a
work
by
the
task,
logic
and
inputs.
A
So
what
we
do
here
is
if
we
know
that
this
work
has
been
done
in
the
past.
We
just
reuse
the
results
from
the
task.
This
reduces
the
number
of
duplicate
works.
We
do
in
the
cluster
user.
Visibility
is
important
because
we
always
believe
that
if
we
give
enough
information
to
the
user
so
that
they
can
look
into
what
they
allocating
how
much
they
are
using,
they
will
have
been
sent
him
to
go
and
reduce
and
optimize
the
cost
for
themselves.
So
this
is
what
we
do.
A
A
There
is
a
UI
which
is
powered
by
that
there
is
decay
and
then
another
thing
we
did
was
we
actually
identified
a
workflow
as
a
type
like,
as
in
like
there's
the
name
to
overflow.
When
you
write
writing
to
the
workflow,
there
are
revisions,
you
can
look
at
the
historical
information
of
a
workflow,
you
can
look
at
when
things
went
south.
You
can
improve
on
things.
So
that
is
everything
that
everything
is
powered
by
our
control.
Plane.
Matrix
was
another
important
thing
for
observability.
A
Today
we
have
two
things
in
place:
one
is
the
system
matrix
your
resource,
you
say
the
neutralization
matrix
coming
via
Prometheus
no
signal
effects.
Then
we
also
have
a
stats
D
set
up
in
the
cluster
and
that
is
for
user
metrics.
So
so,
whenever
we
are
starting
on
container,
we
are
giving
them
a
hook
into
a
stats
the
daemon
set.
So
there
is
the
stats
EDM
and
said:
Rob
stats,
tea
service,
so
all
the
user
metrics
they
are
labeled
correctly
and
then
they
flow
via
stats
to
dois
wavefront
cluster.
A
So
so
far
we
have
talked
about
what
were
the
issues?
What
would
optimizations
we
have
done
for
a
single
cluster,
but
again,
single
cluster
was
not
enough
for
us
because
of
the
scale
which
we
run
there.
We
need
more
than
that,
so
we
started
with
a
multi
cluster
setup,
so
flight
admin,
which
was
never
part
of
execution
cluster,
now,
can
talk
to
multiple
execution
clusters.
So
when
you
start
up
a
new
cluster,
you
just
tell
flight
admin
about.
This
is
my
new
cluster
and
then
you
also
set
up
some
load
balancing
policy
load.
A
Balancing
policy
would
be
I
want
to
distribute
based
on
these
weights
or-
let's
say
I,
want
to
say
this
workflow
needs
to
be
run
on
a
cluster
which
had
a
certain
label.
You
can
do
all
of
that
through
a
load
balancing
policy.
This
works
fine.
This
gives
us
a
very
fault-tolerant
system
where,
when
we
see
things
are
going
bad,
we
just
remove
that
cluster
out
of
the
whole
system.
We
are
in
fact
trying
to
build
a
pipe
feedback
back
to
the
flight
control
plane
where
it
auto
discovers
that
the
cluster
is
not
doing
good.
A
It
will
stop
giving
work
to
that
cluster
and
move
on
to
other
clusters.
This
was
all
good.
Actually
by
doing
this
for
scale,
we
also
got
something
interesting
out
of
it,
which
was
we
now
we
are
able
to
push
incremental
updates
whenever
there
is
a
risky
change
in
that
cluster,
rather
than
pushing
it
all
that
closer.
We
push
it
to
a
small
cluster,
we
ramp
up
the
traffic
and
then
when
we
are
confident
we
just
take
it
out
to
those
clusters.
We
also
see
I
get
another
advantage
with
multi
cluster
setup.
This
is
like
some.
A
There
are
some
users
who
want
performance,
but
at
the
same
time
some
security
additional
security.
Let's
say
there
is
a
user
who
doesn't
want
to
pull
that
image
all
the
time
they
want
to
use
the
cache
container
image,
which
means
there
is
a
security
problem,
but
what
we
can
do
over
here
is.
We
can
ask
them
to
have
a
separate
cluster
and
then
flight
controller
flight
admin
will
handle
the
redirection
of
the
traffic
to
that
cluster.
A
C
E
A
F
A
We
have
a
thing
of
like
so
today
in
the
system,
if
you
specify
a
time
out-
and
you
say
I
want
the
job
to
be
running
for
two
days.
We
are
okay
with
it,
but
we
generally
encourage
people
to
write
small
tasks
so
that
if
they
die,
they
don't
lose
that
work,
but
we
do
have
those
ones
running
to
it.
Good.
G
A
We
have
seen
two
things
late.
One
is
that
if
we
just
increment
increase
a
right,
so
hcd
hcd
the
slows
down,
so
that's
something
which
we
ain't
any
known
thing.
Another
is
the
overall
size
stored
in
HDD.
When
we
see
number
of
objects,
a
cluster
they
going
up
or
our
own
flight
workflow
sizes
grow
up.
We
see
a
problem
over
there,
so
we
see
two
side
problem
in
the
request.
Count
goes
up
and
I
mean
overall
storage
goes
up,
so
we
are
trying
to
kind
of
reduce
on
both.
A
D
I
like
something
to
that,
so
at
some
point
there
is
an
abusive
head
CD
and
at
some
point
it's
like
the
right
level
of
SLA
that
you
want
format,
CD
I
think
we.
We
still
think
that
we
are
at
the
abuse
category
and
we
want
to
bring
it
to
the
right
and
I
think
there
will
be
certain
things
that
we
will
push
forward
to.
Actually
you
love
good.