►
From YouTube: Lessons From Scheduling 20 Million Windows Containers a Month - Julian Portillo, Relativity
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from April 17-21, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Lessons From Scheduling 20 Million Windows Containers a Month - Julian Portillo, Relativity
Speakers: Julian Portillo
Relativity schedules almost a million Windows containers per day to a globally distributed set of Kubernetes clusters. Two years ago we started to break apart our enterprise .NET monolith into microservices hosted on Kubernetes. At that time our developers had a multi-month release cadence. Now we have automated vulnerability patching, can do zero downtime migrations of workloads between clusters, have automated failover for critical services in the event of regional failures, and have have happy developers who can test and push to production immediately. How did we get here? By covering a rocky road full of issues. Come learn from our mistakes so you don't have to repeat them. We will talk about application and orchestration design patterns that have been successful for our teams, custom operators for Windows node problem identification that we have have built and found useful, and monitoring patterns that have helped us stay ahead of issues.
A
For
the
first
nine
months
of
this
year,
we've
gone
through
192
million
Windows
containers
that
are
actually
doing
like
production
workloads,
50
million
that
are
doing
Canary
workloads
and
like
testing,
see
if
nodes
are
working
and
we
found
105
failed
Windows
nodes.
So
obviously,
if
you
don't
have
an
automated
way
to
catch
failures
like
that
you're
going
to
have
a
very
bad
time
so
help
you
figure
out
how
you
can
catch
failures
such
as
those
the
number
one
lesson
to
take
away
as
you
migrate
to
kubernetes
on
a
Linux
world
or
a
Windows
world.
A
Is
these
things?
Are
cattle,
not
pets?
When
you
start
your
migration,
pods
can
be
the
cattle
and
then,
as
you
get
higher
and
higher
in
your
migration.
Eventually,
nodes
can
be
a
cattle
and
shoot
those
things
in
the
head
and
find
new
things,
and
then
eventually,
you
can
get
to
the
world
where
clusters
can
be
moved
around
and
swapped
at
will
and
that's
where
you're
in
a
very
happy
spot
from
a
devops
world.
A
So
here's
we're
going
to
go
through
first,
who
are
we?
What
is
the
team
that
I
work
on
and
what
is
relativity,
then
a
quick
understanding
of
what
our
first
kubernetes
setup
was,
why
we
started
to
use
Windows
containers
and,
like
the
general
motivation
for
it,
since
it's
not
a
super
common
way
to
move
two
years
ago
and
then
Windows
container
pain
points
we
run
into
how
you
can
avoid
those
and
some
recent
very
positive
changes
like
literally
this
morning
for
Windows
containers
that
you'll
probably
be
pretty
excited
about.
A
A
A
So
what
is
relativity,
if
you
all
remember
tannenbaum's
statement
about
never
underestimating
the
bandwidth
of
a
station
wagon
full
of
cassette
tapes
going
down
the
highway
relativity
is
attempting
to
solve
the
same
thing
for
747s
full
of
documents
when
two
companies
Sue
each
other.
You
go
through
a
discovery
process,
there's
tons
and
tons
and
tons
of
paperwork
in
the
old
days
like
the
80s.
A
So
around
198
of
the
top
200
law
firms
now
use
our
platform
for
moving
around
all
their
documents
for
e-discovery
and
we're
pretty
ubiquitous
inside
the
legal
space.
If
you're,
not
a
legal
nerd,
then
you
probably
have
never
heard
of
us,
but
we
do
a
lot
of
documents
and
a
lot
of
NLP
on
those
documents.
A
How
do
we
use
kubernetes?
Now
we
have
kubernetes
clusters
in
20
regions
spread
around
the
world.
We
have
to
have
them
in
certain
regions
for
data
protection.
Laws
turns
out
that
you
can't
really
move
EU
data
into
the
US.
Without
getting
a
lot
of
regulators
very
upset,
U.S
stuff,
you
can
move
around,
but
that's
another
story.
We
have
an
orchestration
system
that
can
handle
millions
of
containers
per
day.
We
have
automated
vulnerability,
patching
and
dynamic
image
changes,
so
we've
actually
just
on
Tuesday,
updated
os's
that
we
were
running
on
in
production
during
business
hours.
A
Without
any
customer
Interruption,
we
can
do
some
pretty
cool
stuff.
We
run
hybrid
Linux
and
windows
clusters,
so
we
do
workloads
that
are
doing
Linux
and
windows
at
the
same
time
how
he
started
when
we
started
our
kubernetes
journey.
We
basically
had
a
lot
of
VMS
that
were
doing
just
machine
learning
and
NLP
things
where
they
would
run
for
a
very
long
time.
They'd
need
to
use
a
lot
of
CPU
for
a
little
while
and
then
they
would
just
kind
of
sit
there
and
waste
money
for
us.
A
So
our
very
first
thing
was
moving
some
machine
learning,
analytics
Linux
workloads
into
kubernetes,
and
we
could
scale
up
and
down
at
will.
We
got
all
the
really
nice
things
about
having
like
a
neat
scalable
multi-tenant,
a
highly
available
architecture
with
full
CSD
to
production.
When
we
started
on
this
road,
it
would
take
a
relatively
developer
six
months
to
a
year
for
their
code
to
go
from
off
their
machine
and
into
production,
which
that
is
a
terrible
experience.
A
Nobody
likes
that
that
is
like
I
would
cry
if
I
dealt
with
that
on
a
day-to-day
basis,
I
cry
thinking
about
it,
sometimes
too,
just
for
fun.
Now
we
have
full
CSD
to
production,
we've
actually
for
one
of
our
demos
for
vulnerability
scanning.
We
are
vulnerability
patching.
We
ordered
a
pizza,
deep
dish
and
tried
to
race
our
deploy
of
our
patched
images,
all
the
way
through
our
production
clusters
before
the
pizza
was
delivered.
A
Also,
after
the
talk
we
can
talk
about
which
kind
of
pizza
is
the
best
Detroit
or
Chicago
or
New,
York
I
think
it's
Detroit
I'll,
say
it
I'm
here
so
Chicago
all
right,
so
a
large
portion
of
the
workloads
at
relativity
were
on
Windows.
We
were
a.net
shop,
we
started
out
doing
you
know
Microsoft
Word
documents
PDFs
and
like
just
moving
things
around
that
way,
so
the
general
gist
is
we
had
a
lot
of
Windows
VMS
that
saw
how
exciting
our
Linux
workloads
were
working
and
then
said.
A
Oh,
that's,
great
I
want
to
deploy
all
the
way
to
production
with
CI
CD
I
want
to
be
able
to
get
my
code
out
there
in
my
users
hands
very
quickly.
That
should
be
really
easy
right
sure.
It
is
really
easy
if
you
think
about
how
you're
going
to
do
the
migration
and
if
you
take
the
time
to
pay
back
tech
debt
and
to
think
about
how
you're
going
to
adjust
your
architecture.
As
you
move
from
VMS
to
the
kubernetes
world.
A
A
So
as
Tolstoy
or
Liz
Fong
Jones
from
Google
SRE
said
happy.
Kubernetes,
migrations
are
all
alike.
You
take
your
time,
you
think
about
what
you're
going
to
do.
You
become
an
expert
on
what
you're
going
to
work
on
in
Dev.
You
scale
up
in
Dev.
Make
sure
you
understand
how
things
are
going
to
work.
You
pay
back
all
your
Enterprise
Tech
debt
and
you
go
make
the
argument
to
all
of
your
business
and
development
leaders
saying
hey.
We
need
to
do
this,
or
else
we're
going
to
run
into
problems.
A
Unhappy
migrations
are
all
peculiar,
because
there's
going
to
be
certain
things
that
say,
hey
I
can't
change
this.
This
is
not
a
thing
that
can
change
for
the
cloud
native
world
and
they
don't
realize
that
if
you
don't
pay
back
your
Tech
debt,
your
customers,
will
it
isn't
just
a
thing.
You
can't
just
say
I'm
going
to
make
this
migration
I'm
going
to
change
nothing
and
everything's
going
to
work
really
well
because
you're
going
to
run
into
problems.
A
Obviously
what
we
did
was
we
tried
to
go
and
just
power
through
it.
We
took
our.
We
had
the
set
of
systems
that
was
running
a
ton
of
processes.
Basically,
whenever
a
user
would
click
a
button
inside
relativity's
UI,
it
would
go
it's
minimum
process.
Do
some
work
or
you
could
schedule
a
cron
and
it
would
go.
Do
some
work
on
different
things
inside
a
set
of
VMS
scaling
up.
Those
VMS
was
a
manual
process
for
our
Ops
teams
that
took
up
to
10
hours
to
do
10
hours,
look
Ops
teams
actually
working
on
them.
A
There
was
a
fair
amount
of
detected
inside
our
pipelines
to
do
those
scale-ups.
So
now
everyone
is
very
excited
about
being
able
to
scale
up
like
at
the
push
roll
button
and
move
things
forward.
That
way,
so
we
went
with
some
very
happy
happy
ideas.
We
were
just
going
to
take
basically
each
of
those
processes
that
were
running
on
our
VMS
package
them
up
into
a
container
like
you
would
do
in
Linux
and
then
run
it
on
Windows
nodes.
A
There's
for
people,
who've
worked
in
Windows
containers,
already
you'll
notice
that
there's
a
lot
of
very
interesting
problems
there.
The
average
Windows
2019
container
as
of
like
two
years
ago,
started
at
three
gigs.
When
you
actually
add
in
all
the
stuff
you
need
goes
up
to
six
gigs
or
15.
If
you
aren't
very
careful
on
Linux,
you
can
do
this,
you
can
run
you
know,
sub
100,
megabytes,
sub,
15,
megabyte
processes
that
do
everything
you
need.
A
Distros
containers
are
really
great
and
you
can
just
run
things
very
quickly:
have
an
average
life
of
30
seconds
and
be
off
for
the
races.
When
you
try
to
do
that
with
Windows,
you
will
have
relatively
long
pull
times
if
you're,
not
careful
with
how
you're
pulling
onto
the
node
when
you're
starting
these
things,
you
can
also
get
into
some
very
nasty
situations
where
nodes
will
just
fall
over
when
you're
pulling
like
40
Pods
at
once
that
all
have
you
know
10
gigabytes
of
data.
A
So,
as
you
can
guess,
our
initial
results
ended
up
in
sad
cloud.
We
had
some
common
networking
failures.
We
had
some
common
container
start
issues.
There
was
a
huge
lack
of
visibility
and
there
was
a
huge
lack
of
common
open
source
tools
like
Linux
node
problem
detector
is
what
everybody
uses
in
Linux
right.
You
can
identify
when
your
node
has
fallen
over.
There's
lots
of
cool
statuses.
A
A
We
couldn't
install
hose
containers
or
privileged
containers
at
the
time,
so
you
could
not
get
access
to
the
host
without
setting
up
a
Bastion
server
and
then
rdping
into
the
host,
which
is
a
real
pain,
and
nobody
wants
to
do
that
and
no
security
team
will
let
you
do
that
in
production,
where
you
generally
run
into
issues.
If
you
can
catch
scaling
issues
in
your
non-production
environments,
you
can
stop
your
promotion
to
production
if
you
can
only
catch
them
in
production
and
you
can't
RDP
into
your
node
in
production.
A
You
are
in
a
very
sad
spot,
so
you
have
to
build
better
and
better
scaling
tests
in
your
non-production
environment
and
get
all
sorts
of
fun
things
there.
So
no
privilege
containers
no
metrics
as
well.
That
has
recently
changed
with
host
containers
and
no
and
windows
node
exporter
and
there's
no
easy
logs
and
we
couldn't
do
very
easy
scanning
and
defense
for
vulnerabilities.
We
had
to
do
some
interesting
changes
in
terms
of
how
we
were
scanning
nodes
or
scanning
containers
with
Linux.
A
If
you
use
like
an
off-the-shelf
container
scanner,
you
can
generally
do
your
scanning
on
your
production
nodes.
If
you
have
a
large
enough
pool
of
nodes,
you
won't
run
into
huge
issues
with
performance.
We
try
to
do
that
with
Windows
and
random
massive
issues,
so
we
started
to
do
that
kind
of
scanning
off
off
in
our
non-production
environments
instead
and
then
tracking
shots
of
what
we're
actually
deploying
to
keep
track
of
whether
there
were
vulnerabilities
and
the
general
debugging
nightmares
that
we
had
so
our
first
solution
to
get
around
these.
A
You
can't
solve
problems
if
you
can't
measure
them
so
we
started
to
run
code
on
the
host.
We
set
up
different
code
to
export
it's
using
it
to
punt
out
the
blob.
If
we
saw
there
were
issues
to
identify
what
things
we
were
dealing
with,
we
started
to
set
up
things
where
we
could
actually
like
punt
out
metrics,
that
we
cared
about
from
the
containers
that
we
cared
about,
and
we
logged
made
sure
that
all
of
our
teams
had
really
solid
logging
from
the
containers.
A
One
really
key
thing
that
let
us
do
that
is
we
have
a
base
image
for
all
of
our
agent
teams
or
all
the
like
developer
teams
that
are
building
these.
These
processes,
that
are
these
windows
container
processes
and
we
can
easily
update
for
all
of
our
user
teams,
everything
that
they're
running
from
a
centralized
location.
So
if
we
do
find
this
a
vulnerability,
our
orchestrator
plus
that
Imaging
service
can
go,
make
it.
A
What
we
learned
was
that,
on
a
fairly
significant
number
of
Windows
pod
startups
we're
getting
different,
Windows
node
failures,
sometimes
those
will
clear
up.
Sometimes
they
would
not
the
we
were
seeing
a
large
number
of
HCS
shim
errors,
and
what
that
is
is
this
is
a
beginner
talks,
we're
not
going
to
like
go
into
huge
details.
A
If
anyone
wants
to
go
into
details,
though
I'm
going
to
grab
launch
after
this
so
feel
free,
we
can
go
talk
and
commiserate,
but
anyway,
HCS
shim
errors
were
the
bane
of
our
existence
for
a
little
while
they
would
cause
random
pod
failures.
We
catch
them
by
those
events,
and
then
we
found
that
the
node
generally
did
not
recover.
After
those
HS
shimmers.
A
We
actually
worked
with
some
Microsoft
folks,
thanks
guys,
they're
over
there
to
we
shared
some
testing
that
we
had
and
showed
that,
and
they
started
to
patch
a
fair
number
of
those
errors
so
first
shot.
When
is
the
projector?
We
looked
at
all
the
events,
and
then
we
tried
to
Cordon
that
worked
actually
really
well.
We
were
able
to
catch
things.
We
were
able
to
get
a.
It
was
very
fast
too.
We
could
easily
catch
an
event
and
then
corn
a
node,
and
we
saw
that
event
go
away
on
new
every
slot
event
go
away.
A
A
So
as
an
example,
one
of
the
failures
we're
going
to
talk
about
today
is
a
hose,
knocking
system,
failures,
h
s
failures
and
we're
going
to
talk
about
how
that
has
actually
had
with
a
live
demo,
we're
going
to
show
how
this
actually
had
a
pretty
huge
change
in
the
last
five
hours
in
four
regions
in
AKs,
which
is
kind
of
cool.
So
we
also
ran
into
new
failures
only
at
scale.
So
this
was
not
a
great
way
to
go
about
trying
to
catch
things.
We
had
to
continually
update
our
event
logger.
A
We
had
continually
event
update
what
we
were
looking
for,
and
it
was
just
not
a
very
fun
thing
and,
as
you're
scheduling
tens
of
thousands
of
containers
in
20
minutes,
you
actually
tend
to
put
a
fair
amount
of
stress
on
the
control
plane.
So
you
don't
want
to
have
a
lot
of
different
things.
Looking
at
all
the
events
coming
off
of
your
cluster,
yes,
you
can
send
them
to
another
spot
and
then
watch
over
a
centralized
location,
but
we
weren't
really
set
up
for
that
at
the
time.
A
So
our
second
version,
which
is
actually
what's
been
in
use
as
we've
scaled
up
on
our
Windows
container
usage
and
it's
a
really
simple
solution
that
has
worked
very
well,
which
is
my
favorite
kind
of
solution.
We've
had
to
do
just
very
minor
code,
commit
updates
for
it
as
we've
found
vulnerabilities
and
dependencies.
A
All
we
do
is
we
schedule.
We
have
a
canary
pod
scale,
Journal
scheduler,
on
a
cron
that
schedules
to
every
node
that
comes
up
and
if
the
canary
fails,
we
corn
the
node.
Then
if
it
passes,
we
wait
a
few.
We
wait
some
period
of
time
schedule
a
pod
again
see
if
that
pod
can
connect
to
a
variety
of
things.
A
We
know
the
Pod
needs
to
connect
to
and
do
stuff
it
needs
to
connect
to
and
or
checks
a
little
bit
of
performance,
and
if
we
notice
some
really
bad
performance,
we
also
get
rid
of
the
node
and
if
the
canaries
fail
and
more
times
drain
the
node,
if
they
were
passed,
then
on
quarter.
The
node
super
easy
super
simple,
and
it's
worked
really
well.
A
This
is
what
our
actual
growth
pattern
of
Windows
containers
that
are
running
customer
workloads
has
looked
like.
So
in
June
of
2021
people
were
kind
of
excited.
We
tried
to
test
it
out.
That's
around
40
000.
For
our
first
month
we
started
notice
issues
we
cut
back
and
switched
back
to
our
non-kubernetes
VMS
to
run
our
agent
workloads
as
a
platform
team.
A
We
started
to
look
at
like
what
issues
we
were
having
and
tried
to
start
fixing
those
working
with
Microsoft
working
on
our
own
to
try
to
solve
these
problems,
and
then
we
restarted
the
migration
in
August
and
basically
for
the
last
two
years.
We
have
an
8
000
year-over-year
growth,
which
sounds
cool,
but
is
not
a
good
thing.
We
don't
actually
get
paid
money
per
Windows
container
that
we
run
I've
tried
to
make
an
argument
for
that
for
our
customers,
but
nobody
has
taken
me
up
on
it.
A
So,
as
you'll
notice,
we've
actually
dropped
in
the
past
month,
we've
been
switching
a
lot
of
our
deployments
from
this.
You
know
really
fast
schedule.
A
ton
of
containers
to
instead
have
a
multi-tenant
shared
container
that
runs
workloads.
We
can
get
the
same
amount
of
customer
work
done
and
we
do
a
lot.
We
have
a
lot
less
like
platform
work.
It
makes
life
a
lot
better
for
our
user
teams
and
it
makes
life
a
lot
better
for
our
platform
teams.
A
It's
a
little
bit
less
fun
because
it's
kind
of
fun
to
think
about
like
if
we
can
get
to
a
billion
containers,
but
if
I
ever
come
up
here
and
give
a
talk
on
scheduling,
a
billion
Linux
or
Windows
containers
in
a
month
that
isn't
just
like
a
performance
check.
I
will
not
be
happy
and
nobody
will
be
so
this
is
kind
of
a
you
know.
We
were
talking
about
105
Windows
nodes,
fit
105,
000,
Windows,
node
failures.
It
was
sort
of
scary,
the
general
gist
here,
though,
is
life,
has
gotten
a
lot
better.
A
A
Think
a
lot
of
other
people
inside
the
windows
container
Community
started
to
notice
these
HS
failures
and
like
tried
to
debug,
where
they're
coming
from.
We
also
have
been
hit
by
this
in
a
major
way.
Our
mitigations
that
we
had
in
place
were
starting
to
fall.
A
little
bit
at
scale
turns
out
when
you're
scheduling,
like
30
million
containers
inside
a
single
cluster.
A
We
noticed
on
9
out
of
10
of
our
node
failures
at
the
time
we
started
grabbing
these
these
failures
the
hns
was
crashing
on
and
for
anybody
who's
looked
into
this
on
2019.
If
you
try
and
reboot
h
s
and
like
Q
proxy,
it
just
won't
work,
it
will
take
a
very,
very
long
time.
It'll
take
110
minutes
to
be
like
a
decent
sized
cluster
on
2022
it'll
come
back
relatively
quickly,
but
there's
also
other
issues.
A
If
you
don't
be
careful
with
relying
upon
just
rebooting,
hns
and
qproxy,
if
you
don't
also
check
for
failures
of
certain
pods
when
they,
when
the
networking
moves
back,
you
can
have
pods
stuck
in
pods,
stuck
in
running
with
no
actual
working
load.
Balancer
so
side
note
build
a
check
for
that
with
your
pods.
A
Anyway,
we
found
that
out.
We
talked
to
our
friends
at
Microsoft
and
we
said:
hey
we're
noticing
this
a
lot.
We
noticed
that
there's
seg
fault
when
we
schedule
a
ton
of
PODS
and
we
built
a
little
test
script
that
goes
and
schedules
even
more
pathologically
than
our
production
use
case
and
can
recreate
this
failure
pretty
much
at
will
so
the
NG
engines
that
Microsoft
started
to
figure
out
where
that
segment
was
in
hns
and
then
they
gave
us
an
unsigned
binary
to
test
in
our
regression
environments.
A
A
The
other
very
important
thing,
if
you
guys
have
not
made
the
move
yet
is
switch
to
containerd
with
the
exact
same
production
cluster,
where
we're
seeing
436
on
average
node
failures
per
day,
which
is
not
a
fun
experience,
make
sure
you
automate
catching
these
things.
We
only
had
82
once
we
switched
to
container
d
yay
and
then
once
we
did
containerdy
we
can
comparing
2019
and
2022
we've
caught
37
in
our
production,
environment,
you'll
notice.
This
is
different
than
our
test
environment.
Our
test
environment
showed,
like
you,
know,
90
drop.
This
was
only
50.
A
Production
is
different
than
test.
You
can
try
to
do
as
much
stuff
that
is
similar
to
test
or
similar
to
production
in
your
non-prone
environments,
but
you
will
still
run
into
issues.
Is
what
I
found
okay
and
then
here's
the
live
demo
portion
this
morning,
like
7
30
Detroit
time
I
got
a
note
from
some
Microsoft
devs
in
Shanghai
that
they
had
released
the
AKs
build
for
the
HMS
patch
to
for
the
Regents
that
are
are
busy,
so
we
asked
them
to
release
two
when
I
started.
A
A
A
It's
going
to
roll
out
to
the
other.
I
can
put
this
in
alt
day
new
slide.
I
can
put
the
exact
like
GitHub,
commit
to
follow,
but
it'll
roll
out
to
the
other
regions
over
the
next
two
or
three
weeks
too.
So
make
sure
you
update
make
sure
you
grab
that
you'll
be
a
lot
happier
Okay.
So
tail
shoe
designs.
Things
been
really
sad,
really
scary.
This
is
only
for
some
subset
of
our
developers.
The
developers
who
built
long-lived
Windows
containers
that
were
just
doing
workloads
their
work
just
got
the
job
done.
A
They
had
to
handle
failures
from
nodes
and
move
over
to
non-busted
nodes.
We
eventually
actually
set
up
some
node
pools
for
relatively
long
run,
contain
or
Long
Live
containers
that
let
them
live
in
a
happier
environment
for
a
while.
So
there's
some
real
pros
out
there,
like
you,
get
the
job
done,
your
customers
are
happy.
They
give
you
more
money,
there's
a
huge
con,
though
you
have
a
lot
more
free
time
and
then
your
PMS
keep
asking
you
to
put
out
features.
A
Instead
of
you
know,
fighting
fires,
and
that
is
you
know
it's
a
downside,
30
second
Windows
containers.
However,
from
a
platform
perspective,
there's
a
lot
of
really
fun
problems
to
solve,
and
you
have
to
solve
those
very
quickly
as
you
get
more
and
more
containers
thrown
at
you.
So
we've
had
you
know
some
pretty
interesting
times
solving
these
problems
at
scale
as
they
come
up
and
big
numbers
are
really
fun.
I
like
to
see
if
we
can
get
another
comma
put
onto
things.
A
Sometimes
con
is
obviously
lots
of
problems
to
solve
and
long
startup
times
in.
The
critical
path
are
one
of
my
least
happy
things
in
the
world.
I
started
my
career
in
high
frequency
trading
and
we
had
to
get
all
of
our
code
out
the
door
and
under
sub
8
microseconds.
So
the
fact
that
it
takes
us
like
four
minutes
to
pull
a
Windows
container
really
may
be
very
sad
takeaways,
please
upgrade
to
container
D
and
grab
this
HMS
patch
as
soon
as
possible.
A
If
you're
using
2022
it's
going
to
be
released
very
soon,
2019,
it's
not
going
to
be
out
till
November.
You
guys
can
go
talk
to
the
Microsoft
folks
and
like
ask
them
to
move
things
faster.
I
think
they'll
have
like
nice
name,
badges,
so
hunt
them
down,
they're
very
friendly
host
containers,
if
you're
not
using
them.
A
Yet
they
make
cluster
configuration
almost
like
Linux
and
it's
very
nice
I'm
pretty
excited
to
see
what
other
companies
start
building
and
if
I
can
convince
some
people
at
relativity
to
let
us
start
open
sourcing,
some
of
the
tools
that
we're
building
instead
of
just
stealing
things
from
the
open
source.
Community
I'm
just
kidding
we
we
are,
we
are
actually
actively
working
on
making
sure
that
our
legal
team
lets
us
open
source,
some
things
and
also
as
you're,
making
a
kubernetes
migration,
whether
it's
Windows
containers
or
Linux
containers.
A
B
A
A
B
B
So
I've
actually
ran
into
a
similar
problem
of
the
sort
of
short-lived
containers.
Even
on
Linux
there's,
this
sort
of
anti-pattern
I
see
a
lot
in
the
sort
of
Legacy
production
of
this
whole,
like
Tron
servers
the
server
it
runs.
It
has
like
a
whole
bunch
of
cron
jobs,
and
this
conjure
and
it
like,
runs
all
the
different
Crown
jobs
and
it
runs
them
in
parallel
and
it
just
you
just
throw
enough
see
CPUs
added
and
maybe
it
works,
but
when
you
kind
of
migrate
that
into
Cube
what
I've
noticed
is
like.
B
Okay,
like,
of
course,
the
system
is
a
lot
more
kind
of
distributed
and
sane,
but
there's
a
lot
more
overhead
for
kubernetes
Cron
job
compared
to
like
having
like
70
crons
on
like
a
Linux
server.
Basically,
so
what
do
you
think,
like
there's
any
kind
of
Solutions
like
either
potential
or
like
that
currently
exists?
That
could,
like
maybe
bridge
that
Gap
when
it
comes
to
that
kind
of
massive
increase
in
overhead
that
you
get
when
you're
migrating
Crown
jobs
to
kubernetes,
especially
short-lived
ones,.
A
So
I
think
the
problem
there
is
that
there
are
some
solutions
that
solve
like
80
of
those
problems
right,
but
a
hundred
percent
of
the
problem
is
with
that
last
20.
There's
nothing
that
I
know
of
that
will
make
it.
So
you
can
just
take
exactly
what
we're
running
NVM
transport
it
to
the
cloud
without
making
any
changes
and
have
things
work
perfectly
you're
right,
though,
that
is
a
thing
that
you
run
into
in
a
major
way
and
I've
seen
that
happen
at
two
different
companies.
A
That
I've
worked
at
figuring
out
how
to
make
the
changes,
as
you
make
the
migration
and
instead
like
running.
If
you
do
some
performance
testing
on
your
workloads
and
see
where
you're
actually
getting
caught,
if
it
is
the
container
startup
time
if
it
is
like
you're
getting,
I
o
blocked
or
like
what
kind
of
what
kind
of
issues
you're
running
into
and
try
both
ways.
The
cool
thing
about
kubernetes
and
like
the
cloud
native
world
is
you
can
move
really
fast
and
test
your
assumptions
so
do
it
get
some
data.
A
Oh
I
agree
with
that
and
I
think
the
old,
so
the
for
anybody
who's
listening
virtually.
He
was
saying
when
you're
saying
it's
running
every
minute,
you're
going
to
run
into
these.
These
pains
and
I
think
you're
right.
The
only
way
to
change
that
is
to
change
the
system
design,
so
you
don't
run
into
that.
We
I
didn't
go
into
detail
on
what
our
agent
framework
looks
like,
but
basically
we
have
a
ton
of
virtual
machines
that
are
running
relatively
software,
that
our
customers
use
and
every
minute.
Yes,
it's
a
minute.