►
From YouTube: Encoding 250,000 Songs a Day with batch/v1 Jobs - Leigh Capili & John Slivka, Beatport
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Encoding 250,000 Songs a Day with batch/v1 Jobs - Leigh Capili & John Slivka, Beatport
Tasked with rebuilding the way we deliver music to DJ's, the Beatport Infrastructure team set out to use Kubernetes to construct scalable compute for executing batch and on-demand encoding workloads in order to level-up our customer's capabilities for playing and mixing dance music. What would follow is a 5-month journey of building clusters, thrashing with software dependencies, and trudging through erratic performance and scalability issues with the kubernetes API. How did we decide to use kubernetes? Was it easy to prototype? Is etcd capable of sustainably servicing 10,000 Jobs an hour? How many Pods can the kubernetes API store? How do you monitor and manage Job failures? We'll walk you through our lessons learned and talk about our most exciting moments and deflating realizations. Join us we re-tell the story of delivering a correct system to production :)
To learn more: https://sched.co/GrYq
A
Thanks
for
joining
us
today,
a
John
and
I
are
here
from
the
port
in
Denver
Denver.
So
that's
beet
port,
not
B
port,
and
here
we
are
to
talk
about
batch
encoding.
Audio
workloads
on
kubernetes
had
some
interesting
use
cases
this
year.
So
my
name
is
Lee
Kapila
I'm,
an
infrastructure
engineer
with
B
port
we're
on
the
same
team
together,
and
you
made
me
a
see
me
around
I
helped
participate
with
sig
cluster
lifecycle.
You
know
collaborating
on
projects
such
as
coop
ADM,
trying
to
move
the
upstream
of
kubernetes
forward.
A
B
I'm
John
silifke
I'm,
also
an
infrastructure
engineer
at
Beatport.
I've
worked
for
some
much
larger
companies
in
the
past
doing
things
in
the
observability
space
with
open
source
tools.
I've
also
worked
as
a
software
engineer
for
object,
storage
as
well
and
I
might
get
of
information
at
you
here
as
well.
If
you
want
to
get
in
touch
with
me
afterwards,.
A
B
So
quick
show
of
hands
with
how
many
you
guys
this
an
electronic
music,
okay,
cool
cool
and
how
many
of
you
people
still
download
your
music
and
keep
it
on
your
computer,
yeah,
yeah?
Okay,
so
that's
pretty
much
our
market
we're
in
a
niche
space
nowadays
with
Spotify
and
SoundCloud
and
everybody
else,
but
it's
still
a
good
market
for
us
and
our
customers
are
mostly
DJs
at
this
point.
They
still
like
to
download
their
music
and
curate.
It
them
themselves.
B
So
what
is
kind
of
the
story
about
how
we
got
here
as
it
relates
to
DJ's?
We
started
out
in
a
very
physical
medium
of
having
analog
signals
on
vinyl
platters
DJing
at
that
time
was
literally
getting
together
in
mixing
these
two
records
like
this
and
cueing
it
up
on
the
headphones,
and
that
was
a
big
deal
and
then
we
finally
got
into
the
digital
space
with
CDs.
This
was
a
little
bit
more
of
a
compact
format.
We
were
able
to
carry
things
around
a
lot
easier.
B
A
A
You
know
some
of
you
maybe
have
been
to
Burning
Man
and
you
know
there's
a
lot
of
creative
sharing
of
audio
happening
there.
How
does
Beatport
take
our
legitimate
business
into
the
cloud,
and
what
does
it
mean
to
use
the
cloud
to
provide
new
tools
and
new
platforms,
new
products
and
new
ways
for
people
to
more
instantly
and
more
excessively
DJ
in
creative
manners,
with
less
of
the
arduous
workflow
of
like
custom
like
file
naming
and
playlist
curation
and
stuff
right
like?
How
do
we
make
the
experience
more
like
this
century?
A
And
that's
the
question
that
we
asked
yourself
and
that
delivered
some
interesting
business
goals?
There's
some
interesting
business
problems.
We
decided
on
a
couple
of
product
directions,
some
really
exciting
stuff
coming
down
the
pipeline
for
DJ's.
It's
really
the
next
pivot
in
the
format
of
how
we
perform
and
play
music.
But
we
have
like
eight
five,
eight
point:
five
million
tracks
in
our
catalog
right
now,
and
these
new
products
require
all
of
our
raw
assets
to
be
encoded
in
creative
ways.
That
can
be
served
in
that
instantaneous
format
and
we're
not
gonna.
Do
that.
B
So
we're
gonna
be
overloading
a
couple
terms
here,
so
I
just
want
to
make
sure
that
everybody
is
using
the
same
language.
So
I've
outlined
some
some
definitions
here,
one
is
master
when
we
talk
about
masters,
we're
typically
talking
about
our
authoritative
copy
of
the
PCM
data.
The
raw
audio
data,
a
derivative
asset
for
our
purposes,
is
going
to
be
anything
that
we
derive
from
that
master.
The
back
catalog
is
basically
the
entire
of
our
collection
that
petabyte
of
storage
that
Lee's
talking
about.
We
kind
of
we
track
all
of
our
our
releases.
B
Our
releases
are
a
collection
of
all
the
tracks
that
we
have
in
our
catalog.
So
these
are
just
logical
groupings
of
those
things,
and
this
is
we're-
probably
gonna
mess
this
up,
but
we
do
it
all
the
time,
but
for
our
purposes
a
task
is,
is
any
any
set
of
work
that
we
put
into
a
pod
will
probably
end
up
using
the
term
job
at
some
point
as
well:
we'll
try
to
differentiate
from
kubernetes
jobs
from
from
tasks
here.
B
B
A
lot
of
our
assets
were
still
in
this
data
center,
so
we
needed
a
design,
a
system
that
was
portable,
that
we
could
kind
of
prototype
out
in
the
cloud
which
we
ended
up
doing,
but
ultimately
run
that
first
iteration
in
our
data
center
in
a
cost-effective
way
and
then
be
able
to
later
port
that
out
to
the
cloud
without
any
additional
work
being
done.
So
some
of
our
other
constraints
were
supporting
these
multiple
modes
of
storage.
We
have
some
Google
Cloud
Storage
that
we've
been
using
more
recently.
B
We
wanted
to
minimize
our
round
trips
across
the
network
from
the
data
center
to
our
points
of
presence
in
Google
Cloud,
and
we
needed
to
make
sure
the
system
was
observable.
This
is
not
as
well
documented
or
written
about
kind
of
how
you
observe
ephemeral,
workloads
and
keep
track
of
those
and
be
able
to
see
things
like
failures
and
successes
and
make
sure
that
your
your
system
is
correct
and
it's
doing
what
you
wanted
to
do.
B
A
A
It's
very
important
for
a
DJ
to
be
able
to
identify
where
the
drop
is
in
the
track.
Say
right
like
that,
and
then
also
like,
say
ffmpeg.
You
know
it's
open
source
component
and
we're
able
to
take
advantage
of
to
you
know,
do
correct
transformations
of
PCM
data
into
the
encoded
formats
that
we
need,
such
as
AAC,
and
so
you
can
see
right
here
that
we
have
like
a
very
simple
dag.
A
It
doesn't
change
really
much
at
all,
but
you
can
see
there
that
we
have
these
additional
volume
amounts
and
it's
important
if
you've
never
worked
with
the
NIC
containers
before
to
make
note
that
the
list
of
init
containers
in
the
pod
actually
runs
in
serial,
and
this
is
something
that
you
can
do
to
keep
your
workloads
consistent.
You
can
say:
hey
like
run
an
authentication
pod
first
and
then
take
that
mutate,
something
in
a
volume
and
then
run
another
container
on
the
contents
of
that
volume.
A
Right,
maybe
fetch
the
authentication
token
for
like
getting
something
out
of
storage
here.
You
can
also
see
that
the
read
only
masters
are
only
at
guessable
to
the
init
containers
and
the
necessary
ones
for
pulling
that
asset
and
that
none
of
our
vendor
binaries
actually
have
read
access
to
our
entire
catalog,
which
is
a
very
important
security
like
it's
a
lease
privilege
model
and
the
pods
API
allows
us
to
implement
that
same
thing.
A
The
waveform
container
in
the
containers
array,
when
it's
running,
alongside
with
all
of
its
friends
the
mountain
named
space
inside
of
that
container,
is
different
than
the
mountain
namespace
of
the
mp3
encoding
container
and
the
AAC
encoding
container
is
a
least
privileged
of
volume.
Mount
model
same
thing
with
you
know
if
you
wanted
to
use
Google
service
accounts
in
order
to
enforce
privileges
with
with
the
container
or
sorry
with
the
Google
storage
buckets,
and
so
we
had
those
things
swappable
with
a
feature
flag
and
some
some
pod
features.
B
So
you
know,
all
of
this
stuff
is
very
simple:
we're
using
well-tested
software,
we're
not
going
out
of
our
way
to
write,
really
complicated,
distributed
systems
and
code.
We
have
stuff
that
works
right.
These
are
things
that
we
could
run
our
laptop.
We
just
want
to
be
able
to
do
it
at
scale
and
utilize,
our
entire
data
center
and
use
that
compute
without
writing
a
ton
of
code
that
we
have
to
test
right.
We
have
a
business
goal
that
we
need
to
deliver
on
and
not
a
lot
of
time.
B
So
what
we
decided
to
do
is
introduce
a
queue
into
our
system
for
publishing
these
jobs.
This
is
kind
of
important
because
we
didn't
want
to
rely
entirely
on
the
grantees
api
to
store
this
state
right
like
we
don't
want
to
have
a
single
point
of
failure
and
lose
eight
and
a
half
million
jobs
that
we
need
to
go
and
query
our
database
for
again.
B
So
that
was
a
conscious
decision
that
we
made
so
then
the
question
we
have
to
ask
ourselves
is:
how
are
we
going
to
take
a
simple
message
right
like
what
is
the
asset
we're
working
with?
What
do
we
want
to
derive
from
that
asset,
and
how
can
we
get
that
into
language
that
kubernetes
understands
and
allow
kubernetes
to
handle
the
scheduling
of
those
workloads?
So
we
came
up
with
this
thing
that
we
wrote,
which
is
a
Python
kubernetes
client
that
is
running
as
a
daemon.
It's
reading
off
this
queue.
B
B
You
can
also
see
here
were
communicating
with
an
external
API.
That's
storing
some
of
the
metadata
that
we
compute.
We
we
do
things
like
checksums
and
metadata.
The
BPM
then
key
that
Lee
was
talking
about.
So
what?
What
do
these
messages
look
like?
What
is
what
is
the
minimum
amount
of
information?
We
need
to
give
our
system
together
to
do
work,
so
we
have
a
request
ID
up
here
as
well,
which
is
for
tracing,
and
then
we
have
a
release
ID,
which
is
our
collection
of
tracks
that
this
specific
asset
might
be
associated
with.
B
We
have
an
asset
ID,
which
we
use
to
communicate
the
unique
identifiers
to
the
API.
We
have
an
asset
gooood
which
we
actually
used
in
our
storage
layout.
So
this
is
important
as
well
and
then
give
me
a
list
of
things.
I
need
to
produce
what
are
the
different
types
of
derivative
assets
that
we
want
to
be
able
to
plug
into
this
system
and.
A
A
Those
messages
into
compute,
that's
specified
in
kubernetes
kubernetes
does
the
rest.
Now
these
were
our
hardware
assets.
It's
a
like.
We've
got
like
basically
one
rack
of
servers
in
our
data
center
they're
like
from
2012
2006.
Let's
show
that
says
the
del
PE
29
50s,
those
things
are
tanks
and,
if
you've
ever
racked
the
hardware,
you
can
understand
why
we
might
just
have
a
bunch
of
old
servers
hanging
around
it's,
because
these
things
were
decommissioned
from
production
for
good
reasons
like
bad
RAM
slots.
Failing
dracs
spooky
stuff,
it
didn't
want,
run
a
workload
properly.
A
You
know
it
just
like
things
that
you
don't
really
want
to
run
in
production
and
we
thought
well.
Could
we
cluster
these
into
like
a
kubernetes
cluster
I
mean
like
these
things.
Do
not
have
you
know
like
any
business
running
kubernetes.
We
decided
to
install
send
to
us
on
all
of
these
things
and
start
installing
kubernetes
start
installing
kubernetes.
This
is
a
big
deal.
You
see.
This
is
a
bare-metal
installation
that
requires
network
topology
updates
actually
like
sending
people
to
the
data
center
to
like
cut
cables,
plug-in
mix,
rack
things
replace
RAM
slots.
A
This
is
a
serious
undertaking
and
like
when
we
decided
to
prototype
the
system
when
we
have
targeted
into
running
this
workload.
This
cluster
didn't
exist.
But
if
you
remember
from
our
business
constraints,
we
wanted
to
build
on
kubernetes.
So
that
we
would
have
portability
and
the
cool
thing
is
that
we
can
go
straight
to
GK
and
we
could
say:
hey
like
can
I,
please
have
a
13
node
cluster
like
can
I.
Please
start
developing.
A
My
software,
like
can
I
see
if
this
is
going
to
be
a
reasonable
way
to
achieve
our
business
goal
and
John
and
I
alone
were
able
to
in
a
week
provision
the
infrastructure
provision,
the
H,
a
revenue
and
Q
cluster
using
helm
and
write
a
simple
Python.
You
know
program
to
start
consuming
the
messages
and
start
minting
things
into
the
kubernetes
api,
and
we
were
able
to
model
that
the
workload
was
actually
going
to
schedule
and
meter
performance
needs,
and
this
was
a
week.
A
Walls
of
JSON
I
would
not
recommend
writing
in
Python
and
we're
actually
going
to
demo
a
subset
of
our
business
logic
right
now,
with
basically
a
system
on
gke.
So
you
can
kind
of
show
you
how
it
works
a
little
bit
here.
We
have
a
storage
bucket.
It
is
just
a
bunch
of
test
media.
We
have
folders
with
a
bunch
of
WAV
files.
This
is
raw
PCM
data.
A
We've
got
an
empty
storage
bucket
called
the
BP
coop
con
demo
and
P
fours,
and
then
here
using
this
is
Henning
Jacob
he's
currently
a
solando
I
believe
great
contributor
in
the
community.
He
built
this
front-end
that
basically
shows
you
all
of
your
nodes
in
the
kubernetes
cluster.
So
all
these
purple
boxes
are
nodes
in
a
gke
cluster
that
we
provision
specifically
for
this
purpose
and
all
the
little
green
boxes
that
are
currently
highlighted
are
like
things
in
coop
systems
that
are
being
managed
for
the
kubernetes
clusters,
such
as
calico
pods,
and
things
like
that.
A
Normally
we
do
this
from
just
like
a
laptop
or
a
management
machine,
but
what
I'm
gonna
do
right
now
is
I'm
going
to
use
a
script
and
a
CSV
file
to
just
submit
a
couple
messages
to
the
queue
and
once
those
messages
are
on
the
queue,
the
dispatcher
our
little
Python
program.
That's
currently
running
right
there
on
the
pods
list
is
going
to
see
those
messages
on
the
queue
and
it's
going
to
template
a
bunch
of
compute
for
the
kubernetes
api.
A
A
The
UI
isn't
really
built
for
this
kind
of,
like
queueing
use
case,
because
right
now,
there's
a
bunch
of
pending
pods
in
the
cluster,
and
so
it's
actually
kind
of
funny,
like
they
just
start
to
get
in
line
over
here
and
they're,
not
currently
scheduled
in.
You
know
it's
good,
that's
kind
of
cute
right,
but
so
right
now
we've
got
a
couple
bunch
of
kubernetes
pods
on
the
cluster.
Well,
let's
go
ahead
and
describe
what
our
workload
looks
like.
A
A
But
also
here
we
can
see.
Okay
in
this
container,
we've
got
a
couple
of
volume
mounts,
it's
got
some
secrets
and
then
here's
like
a
shell
script.
You
know
because,
like
we
write
since
we're
infrastructure
engineers
see
here,
you
can
see
it's
like
running
ffmpeg
and
like
stat,
you
know
it's
getting
like
digests
and
putting
things
into
a
Google
bucket
with
gsutil.
A
If
you
look
at
the
mounts,
it
just
has
a
local
amount.
Well,
that's
because
this
container
up
here,
which
is
called
fetch
asset,
has
a
mount
or
actually
this
just
has
a
secret
that
allows
it
to
copy
things
from
the
Google
bucket
and
actually
populate
the
local
storage
bid,
all
right.
So
that's
kind
of
how
the
compute
workload
is
structured,
and
if
you
see
that
these
pods
in
purple,
they're,
actually
workloads
that
have
completed
computing,
their
assets,
and
so,
if
I
go
to
my
previously
empty
Google
bucket
I
should
now
have
tracks.
A
I've
got
some
folders.
This
is
why
we
did
the
sound
check
earlier.
Let's
make
sure
that
we
can
actually
play
a
song
I
hear
that
this
is
like
from
Finland
it's
a
new
song
has
just
dropped
so
hot
right.
Now
things
like
a
you
know,
this
artist
is
not
sure
he's
like
he's
really
up-and-coming
anything.
It's
like
I'm
gonna,
say:
oh
wow
yeah.
Those
are
good
sing.
I
saw
it.
Let's
turn
it
out.
A
B
So,
there's
a
lot
of
things
that
we
need
to
look
at
when
we're
designing
this
system
that
you
might
not
immediately
think
about
in
other
contexts.
Our
previous
system
was
long-running
Damons
that
we're
listening
off
of
the
cue
they're
each
responsible
for
a
specific
task
and
like
compressing,
audio
or
calculating
BPM,
so
we
were
fetching
the
asset
multiple
times
based
on
the
job
that
was
being
performed.
This
is
a
lot
different
than
approach
right,
so
we
need
to
keep
track
of
our
pod
failure
rate.
How
many
of
those
are
succeeding?
B
How
many
completions
we
have
per
minute
concurrency
garbage
collection
was
a
big
one.
I
know
there's
some
efforts
being
made
there
now,
but
what
we
ended
up
doing
was
writing
a
cron
job
to
basically
pull
4,
completed,
pods
and
then
delete
them,
and
then
we
need
to
riku
and
record
failed
jobs.
We
need
to
know
what
went
wrong
with
the
system
and
how
to
diagnose
it.
A
So
this
is
an
example
of
the
graph
that
we
have
in
Griffin
ax.
We
decided
to
install
the
coop
Prometheus
stack.
This
was
actually
kind
of
to
monitor
some
of
our
other
workloads
in
kubernetes.
Coop
Prometheus
is
a
solution
from
the
quarry
OS
team
originally
designed
to
help
service
their
tecktonik
kubernetes
platform
as
a
service
product
include
Prometheus
comes
with
several
things
right
out
of
the
box.
You
can
install
it
as
a
helmet
art
from
their
git
repository
and
it's
got
coop
state
metrics,
which
is
an
exporter.
A
A
Pod
phase
counts,
alright,
so
pause
in
a
specific
phase
such
as
pending
or
running
or
completed,
or
failed
or
unknown.
Luckily,
there's
none
that
are
unknown
on
this
graph,
and
what
you
can
see
happening
here
is
that
purple
is
the
line
that
represents
the
total
number
of
pods
in
our
cluster.
You
can
see
that's
kind
of
like
spiking
up
right
every
couple
of
seconds
and
that's
because
the
dispatcher,
the
little
Python
program
that
we
wrote
so
watch
the
queue
it's
trying
to
keep
a
thousand
pods
in
the
cluster
at
any
one
time.
A
It's
kind
of
a
number
that
we
just
picked
for
back
pressure.
You'll
know
that
if
you've
ever
written
a
queueing
system
that
you
need
to
write
back
precious
back
pressure,
algorithms,
right
or
some
kind
of
back
pressure
threshold,
and
so
we're
using
the
number
of
pods
that
are
currently
in
the
API
as
a
limiter
on
how
many
things
we
should
pop
up
off
the
queue
at
any
one
moment,
we
have
the
cron
jobs
in
the
cluster
to
clean
up
things
such
as
completions
and
failures,
and
then
you
see
this
yellow
line
of
pending
pods.
A
So
when
we
immediately
submit
things
to
the
cluster
they're
immediately
going
to
be
pending,
but
as
they
start
to
get
scheduled
and
as
they
go
into
their
container
creation
stage,
making
sure
that
the
docker
images
are
pulled
down
at
the
proper
image
tags,
then
we're
going
to
have
running
pods.
You
actually
see
right
here,
I'll
point
on
my
mouse
right.
A
This
is
the
important
bit
about
having
persistent
metrics
across
your
cluster
at
any
one
time
we
have
a
metrics
from
back
when
we
had
a
failure
of
our
API,
which
is
an
external
system
to
this
batch
encoding
workload,
and
at
this
time
we
were
dealing
with
lots
of
different
failures,
and
so
we
wanted
the
failures
to
stop
the
system,
which
meant
that
our
resubmit
cron
job
was
disabled
and
we
had
our
failures.
Spiked
up
hit
the
total
pods
threshold
and
the
dispatcher
stopped
submitting
things
to
the
cluster.
A
At
this
point,
I
responded
to
the
incident
and
basically
just
ripped
off
all
those
jobs
and
put
them
back
in
the
queue
a
fix
the
after
I
fixed,
the
API.
So
that's
a
this
workload
and
kind
of
monitoring.
We
wrote
this
query
on
kubernetes
metrics,
not
anything
specific
to
our
system
or
anything
that
we
had
instrumented
we're
just
running
shell,
scripts
and
pots,
and
it
was
really
cool
to
be
able
to
just
install
Cooper
Mathias
and
then
suddenly
get
this
kind
of
workload.
A
Visibility
in
the
way
that
we
were
like
running
things,
so
the
advantage
of
using
the
kubernetes
api.
This
one
is
a
really
funny
graph
as
well.
It's
just
system
utilization,
so
something
we
really
struggled
with,
even
though
our
system
was
fast,
it
didn't
seem
to
always
be
saturating
our
nodes
for
quite
a
while.
We
were
running
at
a
hundred
course
and
I
decided
to
basically
stop
the
dispatcher,
which
is
to
basically
stop
submitting
things
into
the
cluster,
letting
the
queue
just
stay.
A
Persistent
and
I
modified
the
dispatcher
to
basically
ignore
resource
requirements
so
to
submit
compute
workloads
to
kubernetes
that
didn't
actually
ask
for
like
two
CPUs
right
or
limit
them
at
five
CPUs
and
a
gig
of
memory
and
and
then
this
happened,
and
we
don't
know
why
this
was.
This
was
a
good
thing.
We
had
a
hypothesis
that
maybe
the
that
there
was
some
temporal
slicing
from
the
the
way
that
the
C
groups
were
getting
put
together.
A
We
thought
maybe,
let's
just
remove
the
resource
requirements,
it's
actually
talking
with
the
solando
folks
and
they
found
that
sometimes
this
helps
with
saturation
and
we
jump
up
immediately
to
like
350
CPUs,
which
they're
like
oh
yeah.
This
is
awesome
like
it's
gonna
go
three
times
faster
and
that's
not
what
happened.
They
didn't
improve
our
throughput
at
all,
so
we
don't
really
know
what
happened,
but
it
was
using
a
lot
more
CPU
and
we
reverted
that
change.
A
You've
gotten
like
apparently
know
these
things
about
requests
and
limits,
but
I
have
so
many
questions.
If
you
read
the
community's
documentation
about
resource
requirements.
This
is
a
huge
area
where
we
can
improve,
because
it's
not
incredibly
clear
when
you're
reading
the
documentation
there
seems
to
be
this
conflation
of
pods
and
containers
like
that.
Multiple
you
know
a
pod
can
have
multiple
containers
inside
of
it,
but
then
it
will
be
talking,
be
talking
about
a
container,
see
group
and
then
swap
to
using
pods,
and
there
may
be
some
semantic
errors
in
this
documentation.
A
It's
really
difficult
to
understand
how
things
are
structured
and
like
say,
if
I
provide
one
container
that
has
resource
requirements
and
then
provide
another
container
that
doesn't
like
what
happens
to
the
pod,
see
group
and
how
does
the?
How
do
those
see
groups
get
shared
across
multiple
CPUs?
If
the
pod
see
group
is
then
bigger
than
a
single
container
or
sorry
big
bigger
than
a
single
CPU?
The
canonical
folks
have
this
hacker
noon.
Article
called
job
concurrency
in
kubernetes,
with
lxd
and
CPU,
pinning
to
the
rescue
actually
really
interesting,
I
recommend
reading
it.
A
If
this
whole
resource
requirements
see
group
mapping,
conversation
is
peaking
your
interest
because
there's
some
nuances
here
with,
like
even
the
CRI
runtime.
You
pick
like.
What's
the
docker
default
of
like
asking
for
1100
Miller
Coors,
the
answer
that
question
is
not
simple
and
they
also
cite
this
guy,
a
St
grader,
who
seems
to
be
from
the
lxd
project,
probably
one
of
their
peers,
who
has
more
detail
about
how
lxd
can
handle
these
handles
these
things,
and
they
actually
suggest
that
for
a
workload
similar
to
ours,
they
were
benchmarking.
A
Video
encoding
was
that
the
best
way
to
get
through
put
in
the
system
was
to
run
several
lxd
kubernetes
workers
on
a
single
bare
metal,
node,
so
to
run
a
kubernetes
worker
in
an
lxd
container
right
and
then
to
have
that
lxd
container
run
the
Kubla
inside
multiple
times
per
bare
metal
nodes,
so
that
the
cpus
would
be
pinned
precisely
to
the
hardware
underlying
the
actual
compute
and
yeah.
This
was
something
that
was
not
very
simple.
Luckily,
we
were
able
to
scream
past
our
business
requirements
anyway.
A
And
then
we
looked
at
the
pods
in
the
monitoring
namespace,
and
we
saw
that
the
coop
state
metrics
pods
on
every
or
the
single
coop
state
metric
pod
in
the
cluster
was
in
a
crash
loop.
Well,
when
you
deploy
coop
state
metrics,
it
has
a
sidecar
inside
of
it.
That
comes
from
the
kubernetes
autoscaler
repository.
It's
called
the
add-on,
resizer
or
pod
nanny.
Some
of
you
may
have
interacted
with
this
component
before
the
pod
nannies
job
is
to
take
a
per
node
resource
requirement.
A
Look
at
the
number
of
nodes
in
the
cluster
at
any
one
time,
and
basically,
if
the
number
of
nodes
in
the
cluster
changes,
it
will
multiply
that
resource
requirement
and
basically
rewrite
its
pod
definition.
So
it'll
grant
itself
more
resources
in
the
kubernetes
api
as
the
cluster
grows
and
shrinks,
which
is
a
really
interesting
way
to
run
a
workload.
A
Unfortunately,
the
per
node
resource
requirements
are
kind
of
conservatively
tuned
to
normal
usage
of
kubernetes
and
when
you
start
putting
as
many
objects
as
we
were
putting
in
the
API,
we
were
just
like
totally
blowing
this
per
node
number
out
of
proportion.
We
had
to
scale
the
coop
state,
metrics
per
node
resource
requirement,
using
the
helm
values
to
probably
about
20
times
the
size.
That
was
in
order
to
handle
the
number
of
objects
we
were
putting
into
the
cluster
and
that
fixed
our
ability
to
view
the
needed
metrics
to
observe
the
system.
A
So
the
solution
was
to
you
know
just
scale,
the
KSM
helm
valley,
and
this
one
is
more
interesting,
because
the
coop
controller
manager
is
a
core
kubernetes
component
and
so
cook
controller
manager.
We're
seeing
basically
right
now
that
it
times
out
on
the
metrics
endpoint
in
prometheus.
This
is
an
interesting
line
like
if
there
are
any
contributors
in
the
room.
A
This
is
a
safe
operation
to
perform.
You
can
just
like
go
delete
the
docker
container,
backing
the
static
pod
on
the
control
plane.
So
I
guess
you
can
put
that
in
a
cron
job,
but
we
should
fix
this.
It
should
be
trivial,
I'm,
not
sure
if
it's
fixed
in
more
recent
versions
of
kubernetes
or
not
yeah,.
B
So
you
might
have
noticed
in
Lee's
demo,
there
wasn't
really
any
talk
or
kind
of
demonstration
of
jobs.
There's
a
reason
for
that.
We
ran
into
some
issues
early
on
kind
of
pastor
point
where
we
were
like:
okay,
let's
shoot
like
10,000
messages
and
create
that
many
jobs
we
weren't
seeing
issues
the
first
time
we
started
kind
of
a
sustained
load
test
of
our
system.
To
kind
of
you
know,
kind
of
like
do
a
pre
trial
of
this
long.
You
know
three
months
of
encoding.
We
had
to
do.
B
B
So
we
we
think
it
might
have
to
do
with
the
job
controller
and
how
it
waits
for
its
children
pods
to
get
cleaned
up.
There
might
be
some
additional
logic
that
has
to
be
done
there.
We
weren't
sure-
and
we
still
don't
quite
know
yet
we're
still
sort
of
figuring
it
out.
There's
a
lot
of
variables
we
kind
of
had
it
like
I
said
we
had
a
business
problem
to
solve
and
not
a
lot
of
time.
B
So
what
are
we
losing
out
on?
If
we
don't
use
jobs
right?
These
are
sort
of
the
basic
features
of
jobs.
We
have
parallelism,
we
have
retries,
we
can
do
things
over
again.
It
should
they
fail
and
we
have
completions
right.
We
don't
really
see
a
need
to
complete
the
same
job
multiple
times
we
get
retries
because
we
have
this
queuing
system
and
we
have
an
operator
that
will
go
in
and
grab
those
failed
jobs
and
put
them
into
a
dead
letter
q.
So
we
didn't
need
to
worry
about
that
and
parallelism.
B
It
was
just
not
something
we
needed
for
this
encoding
project.
I
also
point
out
that
there
is
garbage
collection,
I,
think
automatic
garbage
collection
as
of
1:12
alpha
but,
like
I,
said
we're
running
at
111
at
this
point
still
so
yeah.
This
is
how
we're
cleaning
things
up.
At
the
moment.
We
just
have
a
cron
job
that
checks
for
successful
pods
and
we'll
just
go
ahead
and
delete
those,
and
then
we
will
dead
letter
our
failed
stuff,
so
we
can
add
more
work
to
our
system,
so.
B
Yeah,
we're
like
I,
said
we're
not
keeping
state
and
kubernetes
we're
just
gonna
use
rabbit
to
store
all
that
work
that
we
need
to
do.
This
is
kept
us
out
of
trouble
so
far
and
a
lot
of
this
stuff
I've
touched
on.
We
have
a
max
number
of
pods
and
the
API,
and
we
have
some
pretty
good
performance
at
this
point
that
we
can
live
with.
A
We
ran
into
some
other
interesting
things
when
we
started
to
use
the
system.
This
was
actually
independent
of
any
kubernetes
centric
api.
Once
we
had
like
thousands
of
things
in
the
API
at
any
one
moment
in
time,
we
actually
had
a
coop
CTL
formatter
issue
that
was
plaguing
us.
The
default
formatter
would
panic
past
like
nine
hundred
objects,
but
somebody
fixed
it
because
now,
if
you
use
a
newer
version
of,
could
see
tail,
it
doesn't
happen.
A
That
was
frustrating
though,
because
then
you
had
to
like
pass
your
own
format
or
flags
in
order
to
get
it
to
work,
and
it
probably
cost
us
like
four
hours
of
our
lives.
We
also
noticed
some
things
about
the
API
quality
of
batch
v1
jobs
versus
core
pods.
Obviously,
pods
is
like
the
basic
unit
of
kubernetes,
like
it's
one
of
the
most
well
tested
well
exercise,
most
written
about,
like
best
documented
regression,
tested
performance,
scale,
tested
API
that
we
have
in
the
code
based
and
it's
a
big
codebase.
A
It's
not
like
a
like
a
very
public
or
documented
thing
that
some
of
the
fields
in
the
jobs
API
actually
have
typos
in
them
and
there's
not
as
many
of
the
field
selectors
in
the
jobs
API
as
there
are
on
pods,
and
that
makes
things
really
useful.
If
you
wanted
like
say,
put
together
a
run
book
on
like
just
using
coop
CTL
to
respond
to
a
system.
Incident
I
actually
have
an
example
of
something
that
we
did
with
the
pods
API.
Here's
our
internal
run
book
for
the
system.
A
We
wrote
a
JQ
program
on
the
coop
CTL
JSON
output,
and
so
you
can
like
basically
parse
in
a
container
and
container
statuses
and
then
get
the
exit
codes
and
put
it
out
in
a
nice
table
for
every
single
pot
in
your
cluster.
This
is
a
truncated
version
of
the
output
that
says,
oh,
you
know,
like
you,
had
some
errors
in
your
run-of-the-mill
ian's
things:
here's
like
some
audio
a/c
failures.
Here's
some
BP
API
auth
failures.
A
A
There
were
some
storage
incidents
and
API
issues
and
some
deployment
things
where
we
had
to
basically
stop
the
workload
in
the
cluster
to
free
things
up
and
because
that
was
done
in
a
kubernetes
native
way
like
we
could
use
the
cluster
to
kind
of
perform
other
tasks,
and
this
was
a
pretty
successful
project
for
our
company.
So
I
know
that
when
we
wrote
this
talk,
submission
and
when
we
submitted
it,
we
haven't
actually
built
the
system
yet
or
operated
system
in
production.
A
Right,
like
we
were
just
in
the
middle
of
load
testing
and
the
title
of
this
talk
was
encoding,
250,000,
songs,
a
day
with
batch,
v1
jobs
and
what
we
were
able
to
actually
deliver
after
our
estimations
and
running
through
all
the
problems
that
we
kind
of
told
you
about,
as
well
as
some
others
was
encoding,
140,000
songs
a
day
with
core
v1
pas,
and
this
was
this-
was
a
really
successful
project
for
us.
Thanks
for
coming
to
listen
to
our
story,
yeah.
A
A
That's
a
really
good
way
to
do
that.
So
the
question
is:
why
use
RabbitMQ
in
your
data
center
and
why
not
use
Google
cloud
functions
to
perform
the
compute
workloads,
and
this
has
to
go
back
to
a
little
bit
of
our
business
constraints.
So
the
first
thing
is
cost-effectiveness
and
the
reuse
of
our
data
center
hardware.
A
The
rapid
answer
to
the
RabbitMQ
question
is
that
we
already
used
rabbin
and
q
for
some
other
things
and
we
know
how
to
operate
it,
but
you
know
you
could
plug
the
dispatcher
onto
pub/sub,
probably
with
some
very
simple
Python
client
changes
and
then,
as
far
as
using
the
compute,
we
would
I.
You
know
really
love
to
do
service.
I.
A
The
cool
thing
about
this
system
is
that
it
is
built
on
kubernetes
like
you
could
run
it
on
and
say
a
server
list,
kubernetes
platform
such
as
asher
with
their
virtual
couplet,
but
yeah.
We
basically
wanted
to
use
a
computer
PI
that
could
be
run
inside
of
our
data
center
service
would
be
a
great
way
to
do
this.
We
couldn't
just
move
a
petabyte
of
our
storage
out
of
our
data
center
like
quickly,
though
so.
No.
A
Mm-Hmm,
so
the
question
is,
since
you
run
a
lot
of
short-term
jobs,
did
you
learn
anything
about
operating
DNS?
In
kubernetes
we
run
core
DNS
in
our
cluster,
which
is
GA
as
of
111.
Our
cluster
is
installed
on
bare
metal
as
a
single
node
control
plane
with
Covidien
and
DNS
has
been
relatively
boring.
A
We
do
have
our
core
DNS,
also
advertising
to
our
internal
VPN,
and
we
have
like
routable
service
IPS,
so
that
was
done
with
a
couple
of
core
DNS
plug-in
modifications,
but
we
didn't
run
into
any
performance
issues
and
the
number
of
DNS
clusters
in
our
DNS
queries.
Internal
to
the
cluster
is
relatively
minimal
because
the
API
that
it's
talking
to
is
run
in
a
GK
cluster
external
to
the
workload.
Does
that
answer
your
question?
Cool,
hey,.
B
Yeah,
it
kind
of
varies
wildly.
We
have
some
tracks
that
are
like
two
minutes
long.
We
also
still
host
like
60
minute,
extended
mixes,
so
it
can
go
anywhere
from
like
two
to
three
seconds
to
like
30,
30
plus
minutes,
sometimes
like
an
hour
and
for
those
really
long
extended
mixes.
So
I
mean
we
haven't
done
analysis
and
like
removed
outliers,
to
figure
that
out,
but
I
would
say
the
average
track
like
what
we
expect
is
like
a
five
six
minute
track
that
takes
about
six
or
seven
seconds
with
with
a
single
core.
A
Yeah
and
the
question
was
what
was
the
average
runtime
of
the
word
load
also,
we
did
take
this
into
account
because
we
were
worried.
Our
dispatcher,
like
doesn't
batches
of
submissions
to
kubernetes,
and
we
were
worried
about
like
a
stampeding
herd
kind
of
problem,
but
because
the
our
songs
on
average
are
so
varied
in
length,
it
kind
of
tends
to
randomize
out
and
the
workload
becomes
relatively
stable
after
about
an
hour.
So
yeah
good
question
does
that
answer
everything
you're
looking
for
yeah,
hey?
What's
your
question.
A
Yes,
that's
exactly
what
we're
doing
exactly
yeah.
So
the
question
is
with
your
RPC
spec,
where
you
have
a
list
of
assets
to
produce.
Can
you
dynamically
change
the
workload
in
the
dispatcher
so
that
it
doesn't
include
all
of
the
unnecessary
containers
and
that's
exactly
how
we
built
it?
So
the
dispatcher
will
only
mint
a
pod
spec
based
off
of
what
the
RPC
actually
specifies.
A
B
Another
thing
to
point
out
too
is
we
also
map
all
of
our
different
types
of
encoding
or
derivitive
assets
to
single
purpose-built
containers.
So
we
have
a
docker
registry
and
even
for
things
like
ffmpeg,
we
don't
kitchen-sink
stuff.
So
if
you
want
to
encode
like
AAC
files,
we
have
a
special
compiled
binary
for
ffmpeg.
That
only
does
AAC.
A
The
asset
so
that
that's
part
of
the
round
trip
the
network
consideration
is
like
not
pulling
files
over
the
network
more
than
once,
and
a
single
pod
is
working
on
an
asset
and
produces
basically
everything
that
you
need
for
that
asset
at
at
one
time,
and
we
didn't
find
the
need
to
do
any
caching
beyond
that,
because
you're
already
preventing
like
seven
or
eight
poles
of
the
audio
file
for
different
things.
Oh.
A
Present,
where
is
the
good
one
doing
this
yeah?
So
that
was
the
question
about
that
was
what
settings
are
using
for
docker
image
pulse
in
your
pod
specs
and,
if
not
present,
will
basically
reuse
the
image.
It's
important
to
tag
your
images
properly
us
that
you
don't
bust
that
cache
yeah
good
question,
oh
cool
anything
else.
Oh
it's
a
decision
when
asked
something
yeah
cool.
Thank
you
well
well,
we're
around.