►
From YouTube: Cluster-in-a-Box: Deploying Kubernetes on lxd [B] - Rye Terrell, Canonical & Marco Ceppi
Description
Cluster-in-a-Box: Deploying Kubernetes on lxd [B] - Rye Terrell, Canonical & Marco Ceppi, The Silph Road
Deploying kubernetes on top of lxd allows you to build and operate one or more clusters within a single machine, virtual or bare metal. Architect your cluster to be used for development, testing, and more. Once you're satisfied, create a machine image of your host VM for fast, reproducible deployments.
About Marco Ceppi
Marco Ceppi: Operations for The Silph Road, the largest grassroots Pokemon Go community, whose infrastructure is run on Kubernetes
Rye Terrell
Engineer at Canonical working on the Canonical Distribution of Kubernetes, a cloud-agnostic deployment and management tool for kubernetes.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Hey
everyone,
how
y'all
doing
I
know
last
couple
sessions
the
coupe
con
wide
it
down.
I'll
get
everyone
like
a
couple
seconds.
Some
people
are
still
flooding
from
the
hallways,
so
this
stuff
is
cluster
in
a
box.
That's
the
title
on
the
schedule,
so
we'll
be
talking
about
kind
of
be
going
over
different
ways
to
utilize
different
types
of
containers
to
deploy
different
kinds
of
Cabrini's
architectures.
So
with
that
said,
I
guess
we'll
get
started.
I
have
to
catch
a
flight
very
shortly
after
this
as
well.
So
we'll
get
going.
A
And
my
name
is
Marco
chappy,
amongst
other
things,
I
do
operations
at
the
Silph
Road.
How
many
people
here
I
imagine
most
people
here
know
what
I'm
going
to
is
they're
part
of
the
boom
I.
Don't
think
we
have
to
really
give
them
much
of
an
introduction
who
here
has
ever
heard
of
the
Silk
Road
yeah.
That's
just
like
two
people,
that's
ok,
I!
Imagine!
Most
of
you
might
actually
know
the
the
community
that
this
this
that
this
group
Anders
anyone
here
remember
Pokemon,
go!
Remember
that
thing!
A
A
Outside
of
this,
we
don't
have
like
an
income
plan,
we're
not
here
to
generate
revenue,
we're
just
here
to
make
sure
the
site
stays
afloat
and
we
don't
have
to
pay
money
for
it.
We
get
a
ton
of
hits
to
this
thing,
you'd
be
surprised
how
popular
and
how
continuous
our
traffic
grows.
Even
after
the
initial
dull
and
spike,
these
numbers
are
a
bit
old
now,
but
we're
doing
well
over
plus
million
hits
and
tons
of
uniques
every
month
and
because
we're
volunteer
run
and
operated.
A
We
have
a
very
interesting
kind
of
development
cycle
where
we
actually
leverage
people
from
the
community
at
any
one
time.
It
could
be
just
me
or
someone
else
up
to
most
recently
kind
of
ramped
up
for
this
upcoming
there's
an
in-game
release.
It's
happening
literally
right
now:
I'm
the
operations
guy,
giving
a
talk,
but
somewhere
behind
the
scene,
there's
a
bunch
of
code
being
deployed
and
I'm
just
gonna
pretend
like
that's:
okay,
but
we've
scaled
up
to
things
like
55,
even
60
developers
and
they're
all
volunteers.
From
around
the
world.
A
We
have
a
giant
volunteer
organization
for
leadership
and
community
management,
there's
a
1000
plus
research
group
that
all
gets
together
to
collect
and
use
scientific
information
to
determine
how
mechanics
work
in
game.
All
of
these
things
are
great,
but
it
leads
to
a
huge
nightmare
in
that.
How
do
we
make
sure
that
a
we
keep
running
a
really
smooth
operating
website
so
that
we
can
keep
being
that
premier,
grassroots
pokomoko
community,
and
how
to
make
it
so
that
we
can
keep
enticing
volunteers
to
support
and
supply
these
things?
And
so
it
was
obvious.
A
Containers
would
be
a
way
for
us
to
really
expedite
that
development
workflow
and
still
keep
sanity
from
us
who
are
all
volunteering
our
time
so
before
I
start
talking
about
Linux
containers,
machine
containers,
Dockers
and
everything.
I
want
to
talk
really
briefly
about
the
kinds
of
containers
that
exist
in
the
ecosystem
today,
for
a
lot
of
you,
this
may
be
a
review
for
some
of
you.
This
may
be
kind
of
new
and
we'll
kind
of
walk
through
why
these
different
types
of
containers
are
important
and
where
we've
found
ways
to
best
leverage
those
things.
A
So
this
is
this
is
a
server
it's
a
machine
could
be
your
laptop
could
be
a
piece
of
bare
metal.
It
could
be
a
cloud
instance
a
VM
running
somewhere,
but
most
for
the
most
part.
It
is
a
machine
in
the
most
traditional
sense
you
installed
an
operating
system
on
it,
that's
a
boot
to
its
it's
slack
or
its
Arch
Linux
or
whatever
it
happens.
A
To
be
you've
got
a
bunch
of
processes
running
you
know
things
like
an
it's
cron
SSH,
daemon,
loggers
things,
we've
known
to
come
and
love
as
a
core
operating
system
processes.
It's
got
networking
this
kind
of
cute
orange
dot
down
here.
It's
got
things
like
disks
where
you
can
go
and
store
and
retrieve
data
from
and
from
here
you
can
kind
of
slice
this
up
in
a
bunch
of
different
ways.
The
first
that
were
probably
most
familiar
with
is
virtual
machines.
Vms
I
mean
they've
been
staple
for
people
from
laptops
through
to
servers
most
famously.
A
All.
Public
clouds
are
basically
just
virtual
machines
running
on
big
giant
servers,
and
virtual
machines
are
just
that
their
virtualization
of
Io.
So
it's
emulation
of
hardware,
emulation
of
firmware
emulation
of
all
these
bits
and
carving
out
physical
resources
from
a
machine
in
order
to
have
them
appear
as
a
smaller
chunk
of
isolated
running
operating
system
and
it's
an
operating
system.
It's
an
it
processes,
its
cron.
It's
loggers!
It's
networking,
it's
disks.
This
thing
we've
come
to
love.
A
Then
we
talk
about
things
like
docker
containers,
I.
Imagine
most
people
here
are
pretty
familiar
with
things
like
docker
run,
sea
rocket,
the
gambit
of
things
that
provide
you
with
a
docker
like
experience,
a
way
to
manage
and
run
processes
and
confined
fashions
the
most
noticeable
difference
between
things
like
process,
containers
or
virtual
machines
is
a
quick
review
is
that
virtual
machines
are
on
all
these
supporting
processes
and
init
process,
SSH,
loggers,
Cron's,
Damons,
etc.
A
You're
not
running
all
of
these
additional
ancillary
support
services,
just
the
process
inside
of
some
form
of
disk
I
like
to
make
it
inverted
because
you
can
write
to
it,
but
you
can't
really
persist
it
I
mean
you
can
you
can
commit
snapshot
but
they're
really
event
to
be
ephemeral,
and
you
can
attach
networking
and,
most
importantly,
is
you
could
run
these
really
dense
the
number
of
docker
containers?
You
can
run
per
host
Trump's
in
the
thousands
compared
to
virtual
machines.
You
can
spin
up
on
that
same
host.
A
Do
that
same
workload,
because
you're
not
physically
carving
out
resources
and
virtualization
hardware
you're,
just
simply
confining
and
constraining
a
process,
the
last
container
type
is
a
machine
container.
This
is
the
natural
hybrid
model
of
a
virtual
machine
and
a
process
container
you've
got
a
machine,
that's
running
processes
and
NIT,
etc.
It's
got
disk.
It's
got
networking,
it's
got
your
application.
It's
everything.
A
You've
come
to
know
and
love
from
the
last
20
years
of
Linux
system
administration,
but
you're
able
to
run
it
just
as
densely
or
if
not
as
densely,
very
close
or
more
densely
than
virtual
machines,
because
it's
leveraging
those
same
primitives
that
docker
containers
are
it's
not
doing
virtualization
of
Io,
it's
simply
using
the
same.
Primitives
docker
does
to
leverage
things
like
isolation
of
processes
and
stuff,
but
it
gives
you
a
machine
context,
a
disk
you
can
write
and
persist
to
networking
spaces
and
it
processes
you
can
SSH.
A
You
can
manage
it
just
like
you
would
a
normal
machine
and
that's
kind
of
this
kind
of
spread
of
what
we
see
from
containers
today.
So
there's
things
like
virtual
machines.
There's
things
like
machine
containers.
Are
we
talking
about
lxd
in
this
case,
but
you
could
count
things
like
the
clear
containers
project?
A
Well,
not
technically,
a
container
is
a
is
a
very
similar
idea
to
how
to
run
really
lightweight
isolation
of
machines,
and
then
you
have
things
like
docker
run
sea
rocket,
ocid,
the
gambit
of
different
kinds
of
container
technologies
and
that's
kind
of
how
the
space
plays
out.
So
you
wouldn't
necessarily,
although
you
could
run
an
init
process
inside
of
a
process
container
like
docker,
but
you
could
definitely-
and
it's
expected
to
and
only
runs
inside
of
things
like
machine
containers,
so
that's
kind
of
where
we
started.
A
We
said:
okay,
docker
is
gonna,
be
a
great
thing
for
us,
but
all
of
our
developers,
all
of
our
volunteers.
They
all
have
different
environments.
We've
got
people
from
literally
around
the
world
on
Macs,
on
arch,
on
windows,
on
FreeBSD,
on
every
platform
you
can
imagine
and
to
be
able
to
present
them
with
a
standard
development
environment,
saying
here's
what
production
looks
like
it
looks
like
this
operating
system.
It's
got
these
characteristics.
This
software
installed
was
really
difficult,
so
we
turned
to
LXE
as
our
first
start.
So
LXE
was
the
to
us.
A
It
was
that
oh
gee,
Linux,
container
and
I
say
Linux
container,
because
it's
very
similar
to
things
like
Solaris
and
jails,
I'm,
sorry,
solaris
and
zones
and
freebsd
in
jails
which
all
give
you
that
same
idea
of
an
isolation
of
a
machine.
But
this
was
for
Linux
and
that's
what
we
were
using
and
we
found
out
that
things
like
Heroku,
the
paths
I'm
sure
you
may
all
be
familiar
with
Heroku
their
entire
infrastructure,
their
entire
path
offering
is
built
off
of
LXE.
A
So
it
gave
us
a
nice
comfort
layer
of
knowing
that
this
isn't
gonna
be
some
spiraling
security
issue
or
some
problem
where
it's
not
gonna
be
compatible,
also
found
out.
Pibbles
Cloud
Foundry
the
engine
behind
that
uses
LXE
is
a
way
to
confine
and
constrain
for
their
delivery
for
their
paths
as
well.
So
that's
when
we
started,
we
started
with
LX
scene
I
like
riot
I
kind
of
just
walked
through
real
quick
to
show
you
all
what
LXE
kind
of
looks
like
and
how
to
interact
with
it.
So
where
do
we.
B
A
So,
what's
it
doing
right
now,
it's
fetching
I,
guess
the
abouttwo
image,
that's
right,
yeah,
all
right
cool,
and
so
this
this
container.
This
is
effectively
we'd,
be
talking
about
where
it'll
boot
off
and
use
the
same
host
kernel.
It's
basically
kind
of
how
you
would
normally
do
like
a
docker
pull.
Then
a
docker
run
kind
of
a
scenario
set
that's
pulling
in
OS
image
down
instead.
Is
that
right,
that's
right!
Yeah.
A
Yeah,
that's
what
Kristen,
what
what
about
the
colonel?
So
this
is
an
Uzi
Niall
host
machine.
That's
that's
super
awesome
for
us
in
this
case
yeah
exactly
that's
really
compatible.
But
what,
if
we
were
to
launch
say
like
I,
don't
know
Santa's,
6,
machine
or
Fedora
machine
I,
don't
know.
Is
anyone
know
what
the
current
CentOS
7
kernel
is?
Is
it
less
than
4
4?
Maybe
3
10
there
you
go
well
to
be
honest.
We
we
tested
this.
We
literally
spent
all
day
testing
these
demos
make
sure
that
work
and
we'd
already
cashed.
A
He
blew
two
images,
so
I
wasn't
expecting
it
to
download
it
again.
So
we
would
have
been
well
into
your
answer
already,
because
that
was
definitely
part
of
the
demo,
because
you're
right
there
sharing
the
same
host
kernel.
So
what
happens
with
kernel?
Mismatches
turns
out
it's
actually
not
that
bad
of
a
thing
most
kernels
going
backwards
are
fine.
So
as
long
as
you
have
a
new
enough
kernel
running
a
user
land
that
expects
the
older
kernel
tends
to
not
break
it's.
A
When
you
run
a
super
old
host
kernel
and
expect
newer
user
land
features
inside
the
containers
to
access
that,
we
like
using
a
boo
to
riot,
maybe
I
mean
we
may
be
a
bit
a
bit
biased,
but
abouttwo
tends
to
ship
very
recent
kernels.
Even
in
xenial
we
can
install
a
4:13
kernel.
You
really
wanted
to
and
I'm
looking
for
it.
The
next
long-term
support
release.
So
that's
why
we
use
it
as
a
host
OS.
A
A
Where
to
download
that
image,
that's
a
great
question,
so
this
is
lxd
installed
on
top
of
a
nobuta
machine,
so
each
distribution
that
installs
LXE
kind
of
also
can
install
a
preset
of
like
in
remote
image
repositories.
So
there's
an
image
repository
called
a
boon
to
which
points
to
a
remote
source
that
has
a
list
of
all
the
abouttwo
images.
If
this
wasn't
downloading,
I
would
already
answered
your
question
as
well.
How
do
you
see
all
the
list
of
images?
Where
are
those
things?
How
do
you
hook
into
them?
That's
another
fantastic
question:
I'm!
A
A
A
A
A
Red
Hat
release
that
will
give
us
every
one
and
then
finally,
to
answer
the
last
question:
can
you
run
a
you
name
yeah
absolutely
so
this
is
the
entire
CentOS
user
lands
everything
you've
known
to
come
and
love
and
enjoy
from
CentOS,
but
it's
running
on
a
44.4
Linux
kernel
from
the
igloo
to
host,
which
is
interesting
actually,
but
we
haven't
had
any
problems
at
all
running
different
types
of
images.
Right.
Can
you
show
us
all
the
images
that
kind
of
come
available
by
default?
Yes,
we'd
launched
CentOS
with
Washington
blue.
A
Legible,
oh
yeah,
that's
super
legible.
Now,
I
sort
of,
though,
for
those
of
you
in
the
back
are
those
in
the
front
squinting.
Like
me,
we've
got,
let's
see
Alpine.
What
else
is
there?
Arch
Linux
sent
toss
six
and
seven
all
the
Debian's.
You
can
imagine
cyd
Jesse
Weezy
stretch,
fedora
openSUSE,
gent,
Oracle,
Linux.
A
List
goes
on
down
to
a
boom
tube
beyond
so
Lexie
has
like
a
default
image.
Repository
is
a
part
of
the
Linux
containers
project,
so
they
ship
a
bunch
of
these
images.
That
being
said,
though,
it's
you
can
take
and
create
your
own
lint
in
images
put
them
into
your
own
image
repository
just
like
you
would
maybe
a
docker
container
and
distribute
kind
of
starting
os's
or
even
golden
images.
On
top
of
that,
so
that's
super
cool.
So
this
is
how
we
started.
We
used
LX
e
and
then
ultimately,
we
started
using
lxd.
A
So
lxd
is
nothing
different.
It's
just
a
hypervisor
for
LXE,
so
before
Alex
II
was
really
burdensome.
You
had
to
like
go
and
run
all
these
long
commands
I
like
Steve
made
super
hypervisor
like
creating
containers,
starting
and
stopping
them
migrating
container,
snapshotting
them
everything
you
want
to
do
with
a
VM,
except
at
a
machine
container
layer,
it's
just
the
restful
api
with
either
a
local
clients
or
over
the
network,
and
we
just
integrated
into
our
pipeline.
A
So
what
we're
able
to
do
is
on
board
a
new
developer,
create
them
a
new
container
that
they
can
SSH
into,
and
then
they
have
the
tool
chains
that
we
use
that
the
environment
they
had
everything
that
we
basically
have
in
production
for
them
to
play,
break
repeat
and
rinse
and
that
ultimately
led
to
a
better
velocity
for
our
developer.
So
they
have
everything
they
expected
everything
they
needed
in
order
to
start
contributing
and
being
effective
as
a
volunteer.
Their
time
is
precious
to
us.
So
the
least
amount
of
friction.
A
We
can
have
the
more
we
can
get
from
contributions
so
on
the
cluster
in
a
box,
so
switching
tracks
a
little
bit.
Ryan
and
I
had
the
pleasure
of
working
with
a
really
small
team
of
people
in
a
Linux
Foundation.
It's
helped
design
the
certified
kubernetes
administrator
exams.
Anyone
taking
the
CKD,
a
GU,
crazy
trader,
exam
yeah.
Wasn't
it
brutal
I'm?
Sorry,
it's
a
really
tough
exam,
but
it's
a
great
exam.
I
recommend
everyone
here
go
out.
Take
the
exam.
A
It's
super
tough,
but
it
kind
of
covers
the
gambit
of
things
that
you'd
ever
need
to
know.
In
order
to
administer
a
kubernetes
cluster
and
that
certification
holds
quite
a
lot
of
weight,
if
you're
looking
for
jobs
or
shopping
around
well
CNCs
analytics
foundation
that
produces
and
runs
the
exam.
So
we
had
a
bit
of
a
problem
when
we're
designing
the
exam
and
that
this
is
kind
of
what
you
expect
today.
When
you
run
a
cluster,
you've
got
a
bunch
of
machines,
probably
in
a
cloud
somewhere,
Etsy
demasters
workers.
A
However,
you
do
your
architecture,
whether
you're
self
hosting,
whether
you're
keeping
the
opponent
separate
you've
got
VMs
running.
So
when
we
approached
when
we
were
working
with
the
Linux
Foundation
team
and
the
CNC
F
to
kind
of
design,
this
exam,
we
said
okay.
Well,
we
can't
use
just
a
single
cluster,
because
if
the
user
messes
up
on
a
previous
question,
we
don't
want
that
to
impact
the
rest
of
the
exam.
One
wrong
question
shouldn't
basically
just
discount
you
from
being
able
to
take
the
rest
of
the
exam.
A
So
we
said
we're
gonna
need
at
least
six
clusters.
In
fact,
we
ended
up
with
something
like
eight
or
nine.
At
the
end
of
the
day
when
we
designed
all
the
questions
to
make
sure
that
we
had
properly
isolated
all
of
these
pieces
and
the
Linux
Foundation
and
the
CNC
F
came
back
and
said,
that's
great.
However,
this
exam
is
$300,
so
at
that
price
point
in
order
to
run
a
five
or
six
hour
exam
with
the
requisite
set
up
times
and
burn
and
review
times.
A
A
Now
it's
sto
I
remember
this
talk
from
Berlin
where
she
talked
about
using
LXE
containers
as
a
way
to
do
colocation
and
multiple
clusters
in
a
single
machine,
and
so
this
was
very
fortuitous
because
we
worked
on
this
project
over
the
summer,
so
we
started
leveraging
a
lot
of
those
primitives
in
order
to
drive
that
as
let's
deploy
multiple
clusters
in
a
machine,
so
the
Linux
Foundation,
CN
CF,
said
you
have
one
VM.
This
is
the
side
we're
gonna
run.
A
It
in
a
cloud
instance,
and
that's
it-
make
it
work
inside
of
there,
and
so
we
did.
We
set
up
Aleksey
cluster.
We
deployed
all
the
components
that
you'd
come
to
know
and
love.
They
are
all
isolated
from
each
other.
They
all
have
IP
addresses
you
can
connect
to
them,
they're
all
networked
up.
It
was
effectively
like
we
were
doing
in
a
previous
diagram
only
it
was
on
a
single
machine
and
we
were
able
to
take
that.
A
We
were
able
to
scale
it
up
and
we
said,
let's
create
just
a
bunch
of
clusters,
let's
destroy
the
I/o
on
this
machine
and
see
how
big
we
can
stretch
Ian's.
At
the
end
of
day,
we
were
able
to
deploy
quite
a
lot
of
consecutive
clusters,
all
isolated
from
each
other,
all
independent
of
each
other,
so
that
as
exam
takers
move
through
the
questions,
any
wrong
question
against
a
bad
cluster
wouldn't
affect
the
rest
of
the
exam
and
as
of
right.
A
Now,
if
you
go
and
take
the
exam
you'll
be
taking
it
in
an
environment
similar
to
this,
so
I'm
gonna
have
ride.
Take
us
through
what
that
kind
of
looks
like
in
practice.
This
isn't
the
actual
exam
environment,
it's
similar
in
construction,
but
it
would
be
disingenuous
if
we
showed
you
what
the
invent
exam
environment
looked
like.
So
this
is
just
a
using
similar
technologies
and
processes
setting
up
something
like
that.
So
what
do
we
have
here
right?
So.
B
A
B
B
A
A
B
A
Looks
like
the
15-minute
load
is
around
eight,
so
at
eight
cores,
that's
pretty
much
about
a
hundred
percent
utilization.
So
that's
five
kubernetes
clusters
across
50
ish
machines
running
on
a
single
instance
at
$300
a
month,
that's
right,
cool
I!
Imagine
most
of
the
bottleneck
is
not
CPU,
but
actually
disk
IO
or
is
it
something
else
yeah.
A
Sweet,
so
that's
that's
an
example
of
how
we're
able
to
do
things
like
using
LXE.
You
can
use
whatever
flavor
of
tool
you
like
to
deploy
kubernetes
with
whether
it
be
cube,
ADM
or
the
gambit
coop
spray,
cdk
cetera,
to
deploy
a
bunch
of
machines
into
a
single
cluster.
So
from
a
silver
perspective,
this
was
wildly
successful
for
us.
A
A
As
you
can
imagine,
we
have
quite
a
lot
of
code
running
for
those
of
you.
Who've
used
the
site,
which
is
none
of
you.
You
should
check
it
out
if
you
don't
that's
fine
too,
but
we
have
a
lot
of
supporting
services.
Things
like
in
a
geolocation
based
game,
it's
most
important
to
help
find
people
that
are
in
similar
areas
to
each
other.
So
we
have
lots
of
code
that
runs
to
help
figure
out.
Who
is
close
to
you
to
help
you
achieve
certain
tasks
in
game.
A
We
also
have
a
giant
world
map
which
we
help
display
where
points
of
interest
are
and
other
things
of
importance
within
the
game
itself,
and
so
because
of
that
we
do
a
lot
of
traffic
and
if
you've
ever
tried
to
embed
Google,
Maps
or
any
other
map
vendor
onto
your
website,
you
might
find
it's
pretty.
Ok.
When
you
start
pushing
30
million
plus
hits
those
things
it
actually
becomes
really
expensive.
It
turns
out
maps
and
base
maps
are
super
costly,
and
so
we
are
into
a
problem
where
our
map
vendor
was
saying.
A
We
can
no
longer
support
you
at
the
frites
here,
you're
going
to
need
to
pay
$10,000
a
month
for
your
utilization,
considering
we
pull
in
less
than
2,000
a
month
in
ad
revenue.
That
was
not
going
to
be
sustainable
for
very
long
and
I.
Don't
feel
like
paying
out
of
pocket
very
much
for
these
things,
so
we
figured
okay,
we'll
run
our
own
map
servers.
We
use
open
street
maps,
there's
open
source
software.
A
Do
all
of
these
things,
we'll
take
our
take
our
destiny
destiny
into
our
hands
and
we'll
build
and
run
these
things
ourselves
turns
out.
It's
super
expensive
to
generate
map
tiles.
No
wonder
it's
so
freaking
costly,
so
we
realized
that
running
in
the
cloud
is
not
very
effective,
we'll,
invest
in
some
bare
metal
servers.
We
found
a
really
decent
Colo.
Some
good
secondhand
servers,
sockets
about
400,
gigs
of
ram
and
tons
of
disks
to
store
this
data,
and
then
we
hit
our
next
roadblock,
which
is
the
100
pod
limits.
Does
anyone
know
about
this?
A
A
Now,
for
those
of
you
that
running
decent
capacity
workloads,
most
average
web
workloads,
don't
really
hit
those
kind
of
things,
especially
in
cloud
instances
where
you
have
like
4
to
8
cores
on
average
per
machine
or
less
and
like
less
than
16
gigs
of
ram
a
hundred
pods
in
that
space
is
actually
quite
dense.
But
when
you
start
looking
at
things
like
bare
metal,
this
was
the
average
size
of
our
server.
It's
a
two
socket
kind
of
standard,
almost
white
label
box.
A
We
had
24
CPUs
and
500
gigs
of
ram
and
about
20
terabytes
of
space,
and
if
you
start
dividing
that
up
and
saying
okay,
I'm
gonna
run
on
these
servers
on
average
a
hundred
pods,
that's
not
going
to
work
very
well,
it's
awesome
if
you're
doing
machine
learning
or
rendering
farms
or
anything
that
requires
tons
of
compute
space
but
very
little
workloads.
But
if
you're,
not
it's
kind
of
awful,
and
so
what
we
found
is
that.
Well,
if
this
is
the
model
today,
one
piece
of
bare
metal,
one
pod
limit
to
100
pods.
A
A
Don't
need
to
worry
about
installing
anything
to
manage
my
VM
layer
I'm,
just
using
the
traditional
hypervisor
API
to
say,
give
me
four
machine
containers,
there's
no
vert
IO
overhead,
so
I'm
not
limiting
the
resources
on
my
bare
metal
machine,
but
that
I'm
able
to
slice
it
up
and
say
instead
of
one
couplet
I,
can
actually
fit
four
couplets
at
400
pod
limits.
We
went
further
and
started
playing
with
different
architectures,
so
we
said
all
right.
A
Well,
let's
do
it
this
way
where,
instead
of
having
this
previous
model,
where
we
JC
said
each
one
of
these
was
pinned
to
a
CPU
socket
so
via
two
machine
containers
per
sockets
and
then
half
the
RAM
divided
across
them.
We
went
further.
Let's
do
CPU
sharing
models,
let's
see
how
we
can
slice
and
dice
and
limit
the
quality
assistance
quality
of
service.
On
these
things
we
got
up
to
a
900
Pub
limit.
We
never
hit
that
I.
A
A
A
A
A
We
haven't
hit
any
we
do
this
today
in
production,
I
know
of
a
couple
other
customers
that
are
doing
something
like
this
in
production
as
well.
We
just
had
to
make
sure
we
allowed
and
whitelist
certain
kernel
modules
into
the
machine
container,
so
things
like
VLAN,
IP,
V,
I,
P,
b6
and
stuff,
but
outside
of
those
things
we
did
haven't
had
any
problems,
and
we
view
this
as
a
safeguard
for
us.
We
never
touched
the
bare
metal
anymore.
A
It's
just
very
thin
lightweight
a
boom
OS
at
this
point
and
we're
pretty
confident
that
if
we
ever
got
an
exploit
because
again
it's
volunteers
giving
us
code,
we
we
do
our
best
for
security,
but
we're
also
not
PCI
HIPAA
combined
by
any
means.
But
if
a
container
did
break
out
of
confinement
in
docker
or
whatever
CRI
we
were
using,
we
feel
okay,
knowing
that
it
likely
won't
escape
this
isolation
and
affect
the
bare
metal
or
eject
that
firmware
or
something
yeah.
A
A
No,
these
are
all
connected
to
one
cluster
just
span
across
multiple
pieces
of
hardware.
So,
from
our
perspective,
we've
got
it's
actually
just
three
pieces
of
metal
and
a
coal.
Oh,
it's
not
like
a
huge
bare
metal
footprints,
but
we
have
a
lot
of
capacity
that
we
use
so
inside
of
kubernetes.
We
actually
see
15
kubernetes
nodes
in
our
single
cluster
and
that
just
happened
to
be
spread
across
these.
A
We
use
the
combination
of
stuff
and
open
source.
It's
not
super
relevant.
This
will
work
anywhere.
We've
done
it
with
Kubb
ADM.
We
weren't
quite
ready
for
cube
ADM,
there's
some
H
a
and
upgrading
problems.
We
want
to
make
sure
we
always
upgrade
at
our
kubernetes,
so
we
used
a
combination
of
things
like
canonical
distribution
of
kubernetes
and
some
Verone
scripts
to
manage
how
we
do
our
bit.
So
we
just
kind
of
borrowed
from
vendors
that
stall,
stuff
yeah.
A
Yeah,
it's
not
at
the
end
of
my
slide,
but
I'll.
Add
that
I
had
a
bunch
of
links
in
there.
I'll
make
sure
that
link
is
in
there
as
well.
Just
some
of
the
infrastructure
code
we
have
yeah
Alexi
has
a
way
to
modify
profiles,
so
inside
of
elux
see,
I
can
say
here's
the
general
profile
that
gets
applied
to
all
machines,
or
this
machine
in
particular.
Has
these
CPU
limits
these
memory
limits
even
down
to
network
and
disk
IO
throttling
as
well?
A
Yeah
it
does
it
does
a
lot
of
this.
We
had
some
pretty
unique
architecture
stuff,
so
we
utilize
a
lot
of
tools
underneath
conjure
up
to
do
that,
but
we
basically
built
and
scripted
our
own
way
conjure
up,
works
great,
that's
how
we
did
a
lot
of
our
proof
of
concepts,
but
when
we
went
to
production
we
we
use
the
same
tooling
underneath
conjure
up
to
do
that,
but
we
just
kind
of
scripted
our
own
way.
Basically
yeah.
A
So
juju
is
one
of
those
tools
that
we
bolted
on
some
scripting
and
processes
to
kind
of
map
how
we
dynamically
resize
these
misty
machines
or
says
they
handled
them,
juju,
yeah.
So
the
end
of
the
day,
sorry,
we
did
actually
start
with
virtual
machines
initially,
and
we
found
that
the
density
was
quite
poor.
A
In
fact,
we
ended
up
seeing
about
10x,
more
density,
with
the
same
workload
on
these
pieces
of
bare
metal,
10x,
probably
more
actually,
but
as
a
results,
that's
kind
of
how
we
ruled
out
virtual
machines
and
still
sit
firmly
in
the
land
of
containers,
just
a
varying
range
of
them.
I
think
that's
about
it.
For
our
talk.
Yeah.
Thank
you.
Thank
you
all
for
coming.
I
really
appreciate
it.
A
I,
don't
know
I
think
we're
out
of
time.
Yeah
we're.
Definitely
at
a
time
here
are
those
links.
I
mentioned
before
Linux
containers,
org
for
everything,
LX
c
LX
d,
CN
CF,
that
io
certification
experts.
All
you
should
get
certified.
It's
a
great
thing
to
have
do
study
before
you
get
certified.
It's
like
a
four
six
hour
exam
there's
lots
of
questions
about
kubernetes
from
both
administration
and
usage,
and
then
that's
that
cluster
large
page.