►
From YouTube: From Screen to Pods: Bootstrapping a Cloud Agnostic System using Kubernetes [I] - Patrick McQuighan
Description
From Screen to Pods: Bootstrapping a Cloud Agnostic System using Kubernetes [I] - Patrick McQuighan, Algorithmia
Today, Algorithmia runs multiple Kubernetes clusters each with CPU and GPU nodes, 100s of pods, and 10,000s of containers created daily. We can create a copy of our entire stack in a variety of cloud environments in about an hour. Twelve months ago, Algorithmia was limited to AWS and reliant on an enterprise product for deployment management. In that time, we learned how to ensure a highly-available setup in multiple environments, handled networking issues between old applications and pod-based applications, discovered many quirks with cloud components (such as AWS ELB), learned what wrong assumptions we held about the cloud, and migrated our live production services to run within Kubernetes. We also learned the limits of Kubernetes and when to control components on our own. Ultimately, we reduced the number of servers needed to run our full stack, simplified the process of adding services, reduced dependency on particular cloud services, and have a hardened way to deploy our platform.
In this talk I’ll cover why we moved to Kubernetes to build our enterprise product, the benefits it entailed, difficulties we encountered with Kubernetes, containers, cloud providers, and what we’re most excited about in the future of Kubernetes.
About Patrick McQuighan
I joined Algorithmia in December 2015 and have been improving system performance and creating the Algorithmia Enterprise product. In that time we have grown from a single pool of workers to heterogenous groups of machines, and developed support for running the platform in multiple cloud providers. Previously I worked in the display advertising analytics platform at Amazon creating big data pipelines and managing a platform that runs thousands of jobs per day.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
So
before
we
get
started,
I'm
just
curious,
how
many
about
you
out?
There
have
migrated
your
clusters
or
systems
to
kubernetes
or
docker.
Anybody
know.
So.
Basically,
in
this
talk,
my
I'm
going
to
talk
through
what
we
used
to
do
why
we
decided
to
move
to
docker.
While
we
move
to
kubernetes
the
problems
we
faced
and
sort
of
the
system
that
we
have
now,
which
works
across
multiple
different
cloud
providers
and
we
can
install
it
on
on-premises
as
well,
but
first
a
little
bit
about
me.
I'm
Patrick,
mcquigg
and
I
worked
at
algorithm
yeah.
A
A
What
we
do
is
we're
basically
a
surrealist
AI
cloud.
The
idea
is
that
we
have
a
common
API
for
algorithms
functions,
machine
learning,
models
to
run
as
micro
services
users
can
create
algorithms
and
seven
or
eight
different
languages.
Whenever
they
push
to
a
git
repo,
it
automatically
creates
an
version
API
endpoint,
which
then
anybody
can
call
its
set
up
and
as
a
marketplace,
so
users
can
publish
publicly
if
they
desire
to
you
could
set
a
royalty
fee
or
you
can
make
it
private.
A
You
could
share
it
within
an
organization,
there's
a
whole
lot
of
different
options
that
happen
there.
The
key
is
that
we
will
scale
our
cloud,
both
the
number
of
replicas
of
a
particular
algorithm
that
exists
and
the
number
of
servers
that
we
have
running
at
any
given
point
in
time
based
on
demand,
as
it
happens.
So,
for
example,
we
have
this
one
algorithm
for
doing
filtering
like
you
might
see
in
snapchat,
or
anything
like
that.
It
has
a
whole
bunch
of
different
possible
filters.
So
you
take
an
input
picture.
A
It
gives
you
give
it
a
path
to
save
it
and
one
of
the
filter
names,
and
you
get
this
nice
little
stylized
picture
of
my
dog,
thousands
of
algorithms
out
there
there's
50,000
different
users.
Anyone
can
create
anything,
some
things
or
utilities
or
sentiment,
analysis
and
LP.
It
kind
of
goes
all
over
the
place,
but
we
do
support
all
the
major
deep
learning
frameworks:
tensorflow
MX,
Caffe,
torciano
and
Karis
I'm
sure
I'm,
forgetting
some.
But
the
point
is
we're
trying
to
do
anything
all
right.
A
So
with
that
in
mind,
I'd
like
to
first
start
by
saying
what
we
used
to
do
and
then
why
we
moved
to
docker
and
kubernetes
getting
to
the
problems
and
then
how
we
generalized
for
multiple
cloud
providers
and
then
I'll
finish
up
with
the
things
I'm
sort
of
excited
about
with
kubernetes
I'm.
Sure
you've
heard
a
lot
so
far
this
week
on
where
things
are
going
and
I'll
talk
a
little
bit
about
what
I'm
personally
excited
about.
A
So
this
may
look
familiar
to
you.
It's
a
pretty
traditional
web
application
diagram.
You
know
we
have
DNS
that
has
entries
for
the
website
and
for
the
API
servers
those
will
point
to
load
balancers.
We
were
using
AWS,
so
that's
ELB
behind
the
load
balancers.
You
have
a
bunch
of
virtual
machines
that
are
running
servers,
for
example
a
web
server,
maybe
on
port
9000
and
an
API
server
on
port
9100.
A
When
an
API
request
comes
in,
it
goes
to
the
load
balancer
round
robins
to
one
of
the
virtual
machines
that
you
know
does
some
authentication
and
we
have
our
own
schedule
or
to
figure
out
where
we
actually
wanted
to
run
that
work.
It's
pretty
standard,
but
just
in
case
some
of
you
haven't
experienced
this
before
here's
how
the
deployment
might
work.
So,
first
of
all,
we
have
to
use
write
a
whole
bunch
of
our
own
small
bash
or
Python
scripts.
A
To
do
each
step
of
the
deployment
which
was
managed
by
a
third
party
called
estelí
and
the
way
that
I
would
typically
work.
Is
you
have
our
three
servers?
We
pick
one
of
them
to
deregister
from
the
load
balancer.
The
load
balancer
will
stop
sending
new
connections
to
that
server.
We
wait
for
work
to
drain
off
of
that
machine.
Whenever
it's
done
servicing
all
active
requests,
it
gets
deep
registered
fully.
A
We
can
then
update
the
application
binary
and
then
reattach
to
the
load,
bouncer
and
start
responding
to
health
checks
after
30
seconds
or
whatever
our
configured
interval
is
of
healthy
things.
Then
the
load
balancers
start
sending
work
to
it
again.
This
is
pretty
standard.
This
is
what
people
have
been
doing
and
some
variation
of
this
for
the
you
know
10
15
years
in
any
sort
of
web
serving
architecture,
but
it
has
a
whole
bunch
of
limitations
to
it.
A
If
you
want
to
want
to
run
multiple
copies,
you
can
do
things
like
run
a
CH,
a
proxy
to
balance
between
ports
or
something
along
those
lines,
but
it
starts
to
get
a
little
bit
more
messy
and
it's
usually
just
easier
to
create
more
machines
running
one
copy
of
the
service.
Additionally,
it
can
be
really
difficult
to
add
services.
You
have
to
install
libraries
on
the
host.
You
may
need
to
configure
your
particular
directories.
We
have
to
write
the
deployment
pipeline.
A
A
On
top
of
that,
like
I
said,
we
have
a
public
marketplace,
but
the
customers
came
to
us
who
a
public
marketplace
doesn't
always
work
for
them.
They
may
have
data
privacy
requirements
such
as
a
HIPAA
requirement,
or
maybe
they're
an
EU
and
there's
data
privacy
laws
where
you
can't
send
data
across
or
outside
of
the
EU
there's,
also
a
bandwidth
or
latency
requirements,
users
who
are
sort
of
in
the
big
data
space
they
have
all
of
their
data
in
a
certain
data
center.
A
Why
should
they
send
all
of
that
from
say
Europe
to
our
data
center
and
US
East,
one
just
to
do
some
processing
really
they'd
like
it
to
be
within
their
own
data
center?
So
but
those
motivations
we
felt
like
we
can't
just
handle
or
can't
only
have
a
public
marketplace
but
wanted
to
have
an
enterprise
offering
where
we
would
install
our
system
and
whatever
cloud
whatever
region,
on
premise
that
the
users
wanted.
A
But
before
we
can
do
that,
there's
a
whole
bunch
of
barriers.
We
have
to
figure
out
how
you're
actually
going
to
deliver
your
applications.
You
have
to
figure
out
how
to
work
on
different
operating
systems
based
on
the
customers
requirements.
Like
I
said
we
have
third-party
software
for
called
Esteli
for
doing
our
deployment
management.
We
would
also
use
some
things
like
Algol
II
as
a
power
search
on
our
website.
There's
a
high
proof
of
concept
and
the
list
kind
of
goes
on.
But
what
I'd
like
to
talk
about?
A
So
the
benefits
of
container
izing
is
that
in
general,
you're
no
longer
tied
to
the
host
libraries
or
operating
system
with
a
small
asterisk
I'll
come
back
to
it,
gives
you
a
easy
way
to
distribute
and
upgrade
the
application
images.
You
can
simply
take
the
docker
image.
You
can
push
it
to
a
registry.
You
can
dump
it
to
a
tar
file.
You
could
send
it
via
email,
whatever
you'd
like
to
do,
there's
multiple
ways
to
distribute
it.
It
also
gives
a
unified
way
to
view
logs
and
kill
and
manage
your
applications.
A
So,
for
example,
if
you're
running
H
a
proxy
and
some
shell
scripts
and
a
web
server
using
Apache
and
a
different
web
server
and
Python
or
whatever
else
they
might
all
right
logs
to
random
spots
on
the
operating
system
with
docker
single
docker
logs,
you
can
view
application
logs
for
anything.
It
also
makes
it
somewhat
easier
to
run
multiple
copies
on
the
same
machine,
but
not
fully
so
I,
dr.
A
There's
a
couple
caveats,
as
I
said,
you
aren't
really
completely
free
from
knowing
what
operating
system
you're
running
on
fully
for
us
when
we're
using
GPU
devices
and
other
drivers.
Those
actually
need
to
be
installed
on
the
host
operating
system
itself.
It's
not
something
that
you
can
only
have
existing
in
the
container,
because
you
have
kernel
modules
and
other
things
that
you're
dependent
on
then
the
kernel
is
not
a
part
of
the
container.
It
exists
outside
of
the
container
additionally,
and
this
is
a
little
more
convoluted.
A
Well,
it
turns
out
that
in
Ubuntu
or
sense,
OS
or
a
couple
other
different
operating
systems
when
you
install
docker,
they
get
a
different
group
ID,
and
that
means
that
your
container,
your
application
is
running
with
a
group,
ID
of
say
999.
That
means
that
it
can
only
actually
access
the
dock
socket
on
other
hosts
that
have
docker
group
999.
A
If
you
are
using
fully
using
kubernetes,
you
don't
really
have
to
worry
about
that,
because
usually
you're
not
interacting
with
docker
directly
you're
gonna
interact
with
the
kubernetes
api
server,
or
something
like
that
and
when
you're
using
the
kubernetes
api
server
either's
like
role-based
access
controls
and
tokens
that
will
handle
all
the
access
that
you
need.
But
if
there's
some
sort
of
middle
ground
where
you
are
using
docker,
but
not
fully
using
kubernetes
or
some
other
Orchestrator,
this
might
be
a
problem.
A
If
you
need
to
target
different
operating
systems,
we
worked
around
that
by
sort
of
customizing
the
the
base
image
that
we're
going
to
use,
even
if
we're
using
Ubuntu
or
sent
to
us
or
anything.
We
install
the
docker
and
have
a
specific
group
that
we
use
for
all
of
our
a.m.
is
that
we
share
with
customers.
A
There's
a
couple
other
issues
and
it's
something
where
dr.
people
like
to
think
that
it's
very
easy.
But
in
some
ways
it
can
be
complicated
and
have
sort
of
unknown
performance
issues.
When
you
go
to
figure
out
what
storage
driver
you
want
to
use
to
with
docker,
you
kind
of
get
this
whole
little
matrix
here,
if
you're,
using
a
Bunty,
Aur,
Deb,
Ewing
or
whatever,
there's
just
this
whole
sort
of
list
and
you're
like
okay,
well,
which
one
do
I
choose
turns
out.
A
The
recommended
driver
has
changed
multiple
times
over
the
last
three
years
that
we've
used
docker
from
device
mappers
to
AU,
FS
and
now
it's
at
overlay
and
that's
something
that
they,
the
community
sort
of
realized.
Oh
this
one
has
performance
issues
or
that
one
has
performance
issues
and
it
kind
of
keeps
evolving
for
us.
We
ended
up
using
a
ufs.
We
might
be
changing
that,
but
there
are
some
tricky
bits
with
a
particular
storage
driver
that
they
may
not
explain
to
you
very
clearly
in
some
of
the
documentation,
for
example
with
a
ufs.
A
A
So
your
reboot,
the
host,
the
kernels,
knew
we
were
missing
this
extra
kernel
package
and
then
the
docker
daemon
is
that
they're
crashed
looping
can
easily
be
fixed
by
you
know,
being
a
little
more
intelligent
and
now
you're
up
creating
your
packages
and
making
sure
you
upgrade
everything
you
really
need
to.
But
it's
something
that
sort
of
caught
us
off-guard
when
we
you
know
a
month
or
so
after
we're
using
docker
and
our
system,
so
I'd
only
had
these
weird
edge
cases
we
were
hitting.
A
Secondly,
with
a
AWS
specifically
EBS
volume
performance
is
really
bad
until
the
images
or
until
the
volume
is
fully
warmed
up
depending
on
how
big
your
volumes
are.
This
can
take
several
hours
to
read
all
the
data.
You
don't
actually
have
to
write
anything
you
just
have
to
read
and
scan
the
entire
volume.
A
In
our
case,
it
was
taking
45
minutes
to
an
hour
and
a
half
to
actually
fully
read
the
volume
and
when
you're
trying
to
be
an
auto
scaling
system
bringing
up
and
down
lots
of
hosts
an
hour
and
a
half,
you
can't
wait.
That's
not
reactive
in
the
way
we
want
to
be
because
of
that
issue.
We
found
that
with
very
write,
heavy
workloads
like
when
we're
spawning
lots
of
containers.
We
a
lot
of
churn
in
our
system.
A
The
docker
Damon
would
ended
up
taking
45
seconds
to
a
minute
to
create
a
container
or
do
anything
and
that's
something
that
typically
takes
a
hundred
milliseconds
or
less
something
like
that,
and
all
that
time
was
just
file
system
sink
times
or
way
too
high.
So
one
way
to
work
around
that
is,
if
you
mount
empty
EBS
volumes,
they
don't
need
actually
be
fetched
or
pre-warmed,
or
anything
like
that.
A
So
you
can
mount
empty
eve,
EBS
volumes
and
do
a
bunch
of
work
on
those
and
they'll
function
way
better
than
if
you
have
something
that
needs
to
be
pre-warmed.
This
only
really
matters
for
situations.
We
are
doing
lots
of
auto
scaling.
If
you
just
bring
up
a
host-
and
you
expect
it
to
be
around
for
a
long
time-
you
might
not
actually
need
to
worry
about
this
very
much.
But
oh
I
have
questions
at
the
end.
Sorry,
thank
you.
A
But
overall
you
know
we
able
to
move
everything
to
docker.
It
works
great,
but
it
doesn't
do
everything
for
us.
So
the
next
question
is
kind
of.
Why
do
we
move
on
to
kubernetes
so
docker?
You
know
we
kind
of
abstracted
around
our
operating
system
when
we
have
a
nice
way
to
bundle
our
applications,
but
there's
still
a
whole
bunch
of
other
problems.
A
So
how
does
kubernetes
get
us
that,
since
this
is
the
end
of
the
week
of
kubernetes
talk,
I
don't
think
I'm
gonna
run
through
all
of
this,
but
the
most
important
things
I
found
our
services,
which
give
you
ability
to
do
load,
balancing
service
discovery,
DNS
entries
deployments,
slash
replica
sets
they
allow
you
to
have
n
copies
of
thing
running.
You
can
do
rolling
updates,
it's
all
native.
You
can
configure
how
the
updates
going
to
go
say
you
know,
do
one
at
a
time.
A
Two
at
a
time
wait
a
minute
in
between
instances,
it's
highly
configurable
lots
of
things
you
can
do
there
and
thirdly,
the
cni
plugin,
which
is
the
networking
plugin,
which
allows
you
to
get
the
actual,
giving
IP
addresses
to
every
container
and
infrastructure,
and
it
also
establishes
routing
between
all
the
containers
and
it
allows
you
to
create
network
policy.
So
you
can
say,
I
want
my
API
servers
to
talk
to
Redis,
but
I
don't
want
the
web
servers
to
talk
to
Redis
or
whatever
situations
you
may
have.
A
A
We
could
probably
do
everything
we
wanted
to
with
sort
of
native
kubernetes
ideas
and
like
operators
and
stuff
like
that,
but
we
kind
of
chose
not
to
and
in
part,
because
we're
a
small
team,
it's
a
huge
project,
there's
a
lot
of
things
going
on
and
we
didn't
want
to
throw
a
ton
of
variability
into
everything
else
that
we
were
already
doing.
But
on
top
of
that,
we
had
spent
a
lot
of
time
optimizing
our
system
for
the
way
that
we
are
scheduling,
algorithm
containers,
people.
A
We
would
look
at
things
like
whether
or
not
images
were
there,
which
kubernetes
would
also
do,
but
we
have
higher
level
data
about
what
will
this
algorithm
need
in
terms
of
the
data
files
that
and
what
are
the
likely
files
it
might
be
using
based
on
characteristics
we've
seen
in
the
past.
Things
like
machine
utilization
is
something
in
kubernetes
is
also
very
likable,
natively
track
for
you,
but
pending
requests
and
queue
sizes
of
something
a
little
bit
more
abstracted
might
not
know
about.
A
Unless
we
added
that
on
top,
we
also
have
some
special
security
concerns
that
go
as
part
of
our
scheduling.
We
will
proxy
requests
for
users,
maybe
put
them
on
an
isolated
network.
If
we
don't
trust
it
or
the
user
wants
it
to
be
on
an
isolated
network,
and
all
of
this
we
were
still
getting
within
single-digit
millisecond
performance.
It's
one
of
those
things
where
it's
like.
Well,
you
spent
all
this
time
optimizing
and
investing
in
this
codebase.
A
We
didn't
feel
there
was
a
need
to
move
all
of
that
inside
of
to
be
a
little
bit
more
like
kubernetes
native
instead
of
running
in
our
application,
something
we
may
change
in
the
future,
especially
as
we
want
to
extend
those
capabilities
further.
But
for
now
it's
something
you
know
you
should
consider.
Do
we
need
to
move
every
single
aspect
of
our
system
to
be
kubernetes
native,
or
can
we
leave
some
of
it
within
the
application?
A
Also
GPU
and
device
management
kubernetes
will
allow
you
to
expose
the
devices
to
a
particular
container
or
a
particular
pod.
You
can
say:
I
need
one
two,
four
whatever
it
may
be,
but
there's
a
lot
of
metrics
that
you
can't
get
out
of
a
GPU,
so
in
particular
GPUs
will
track
track
memory,
usage,
etc
by
the
process
ID.
So
when
you're
looking
at
a
GPU,
if
you're
just
exporting
okay
process,
Ida,
you've
one-two-three
is
using
this
much
memory
that
doesn't
tell
you
anything
well,
you
need
to
know
what
was
that
process?
A
Who
was
it
what's
the
sort
of
meta
information
about
it,
which
would
be
like
the
user
of
the
algorithm?
That
kind
of
information
or
application
knows
so
we
had
to
find
our
own
way
to
expose
that
information
that
wasn't
sort
of
natural
with
the
advisor
and
other
metrics
that
are
already
generated
for
you.
A
They
might
need
to
validate
x.509
certificates
from
the
clients
who
are
coming
in
and
all
of
these
configuration
options
didn't
exist
in
any
current
ingress
format,
so,
rather
than
go
through
and
figure
out
how
to
get
all
these
annotations
and
other
things,
we
found
a
way
to
template
our
own
sort
of
hot
proxy
config
and
just
kind
of
rolled
with
that.
It's
very
close.
It
is
almost
like
the
ingress
controller,
spec
and
so
I'd
like
to
get
it
back
in
there.
A
But,
overall
you
know
we
completed
a
migration
there's
in
production.
In
January
this
year,
everything
was
running
on
kubernetes
fully.
We
were
able
to
drop
the
minimum
number
of
servers.
We
needed
from
that
twelve
to
six
and
six
is
because
there's
three
kubernetes
masters
for
highly
available
set
up
and
we'd
have
three
kubernetes
nodes
for
highly
available
Redis
API
servers
all
those
other
things
that
we
run
in
our
system.
We
also
only
need
a
single
load
balancer
now
and
that's
part
of
like
that.
A
Ingress
mindset,
where
you
have
a
single
nginx,
hop
proxy,
whatever
sitting
in
front
of
all
of
your
services
and
then
it
routes
based
off
of
what
the
incoming
you
know
address
was
now
we're
able
to
replace
our
third-party
search
with
using
elastic
search
is
very
easy
to
get
elastic,
search,
configured
and
deployed
within
kubernetes
and
kubernetes.
Again
we
have
native
deployment
and
rolling
upgrades,
so
we
could
get
rid
of
dis
daily
entirely.
A
Overall
is
very
easy
to
add
services
to
the
stack
as
well
such
as
Prometheus
turn
Cortana.
If
you
hadn't
hear
about
them
the
rest
of
this
week,
there
are
amazing
tools
for
graphing
monitoring,
etc
and
only
took
about
a
day
for
us
to
integrate
coober
integrate
Prometheus
with
our
stack
because
there's
so
many
deployment
templates
already
out
there.
A
So
what
went
wrong
in
this
process?
There's
a
lot
that
can
go
wrong
if
you
have
an
application
that
currently
exists,
not
in
the
context
of
kubernetes,
and
you
want
to
migrate.
It
piece
wise,
it's
always
possible
to
duplicate
your
entire
stack
inside
of
kubernetes
and
sort
of
just
point,
your
dns
to
the
kubernetes
one,
but
that
may
or
may
not
work
depending
on
your
situation,
and
you
might
not
even
be
able
to
migrate
everything
to
kubernetes.
If
say,
there's
some
other
team
you
depend
on
and
they
aren't
willing
to
do
so.
A
Something
like
that,
and
so
the
projects
are
moving
fast.
Like
really
fast.
You
might
have
seen
that
from
this
whole
week,
all
the
features
people
have
talked
about
in
the
last
year,
the
things
that
are
upcoming,
but
they
aren't
always
compatible
with
each
other.
So,
for
example,
docker
1.13
came
out
earlier
in
this
year
and
at
that
time
kubernetes
1.5
was
out.
However,
it
was
not
compatible
and
also
docker.
A
1.13
wasn't
targeted
for
kubernetes
1.6
and
it
wasn't
until
1.7
that
it
was
targeted
and
things
were
fixed
and
it
resulted
in
docker
was
making
some
changes
to
the
way
that
they
did
iptables
rules,
so
they
added
a
default,
deny
or
default
drop
policy
or
something
to
the
top
of
your
chain
and
that
broke
everything.
So
people
try
and
upgrade
and
didn't
work.
A
Secondly,
there's
a
ton
of
different
versions
that
are
in
development
and
use
at
the
same
time,
just
in
November
I
think
there
was
1.6
12,
117,
183
and
1.9
Oh
alpha
all
released
in
kubernetes
in
addition
to
other
versions.
So
if
you're
gonna
be
taking
time
to
migrate
your
service,
what
kind
of
version
are
you
going
to
target?
A
That's
just
something
worth
considering
and
probably
looking
at
what
the
chart
is
and
estimate
how
much
time
you
might
need
and
then
try
and
target
something
appropriately,
there's
also
a
lot
of
new
concepts
which
can
break
sort
of
the
way
your
deployments
may
work.
So
readiness
checks
with
the
deployment
or
are
very
similar
to
a
health
check
from
a
load
balancer,
but
there's
lots
of
things
in
the
application
and
pod
life
cycle
that
are
different.
You
might
have
to
change
the
way
that
your
application
gets
stopped.
You
have
to
change
your
rolling
update
strategy.
A
You
may
be
used
to
doing
certain
things
that
are
initialization,
which
now
you
have
to
create
another
container
to
be
in
a
NIC
container,
and
none
of
this
is
particularly
difficult,
but
it
is
something
where
every
service
you
migrate.
You
we'll
have
to
look
at
all
of
these
things
and
figure
out.
What
do
I
need,
what
don't
I
need,
possibly
creating
five
or
six
containers
if
you
have
a
lot
of
initialization
that
needs
to
get
done,
but
the
real
interesting
thing
is
the
technical
challenges.
A
So,
as
we
said,
the
CNI
plugin
will
generate
IP
addresses
for
every
single
pot
in
your
cluster,
but
those
are
not
routable
from
things
not
within
the
kubernetes
cluster,
and
this
can
become
really
awkward
if
your
services
are
trying
to
talk
to
each
other
via
IP
address.
For
example,
we
have
API
servers
and
we
have
Redis
and
Redis
needs
to
talk
to
itself
to
you
know,
replicate
data
and
if
the
master
goes
down,
one
of
the
replicas
needs
to
take
over.
So
if
you
migrate
Redis
before
migrating
the
API
servers.
A
Well,
your
API
servers
won't
be
able
to
talk
to
Redis
unless
you
put
some
sort
of
proxy
or
something
in
front
of
it
that
they
can
address,
and
then
that
would
be
able
to
address
the
things
inside
of
Redis
something
you
just
need
to
understand
your
own
application.
What's
talking
to
what
before,
you
can
really
migrate
anything
safely.
A
Secondly,
classic
load.
Balancers
are
not
really
meant
to
address
containers
again,
the
IP
address
isn't
real.
So
what
may
happen
is.
Is
that
you're,
exposing
your
containers
or
your
pods
on,
say,
port
9000
and
the
traffic
is
getting
proxy
to
that
container
from
the
host,
but
it's
just
as
likely
that
the
traffic
is
getting
forwarded
to
a
container
on
a
different
host,
because
that's
the
way
that
QB
proxies
works.
A
It
will
forward
to
something
that
is
active
across
the
fleet
and
then
what
happens
if
host
B
goes
down
or
deployment
comes
along
and
updates
that
pot
or
whatever?
Well,
if
that
connection,
that
it
was
going
to
host
B
gets
reused
for
some
reason,
it's
possible
that
LBO
will
mistakenly
thank
host
a
is
down
because,
from
elby's
point
of
view,
it's
talking
to
host
a,
but
in
reality
the
track
is
getting
forwarded.
This
can
happen
with
any
sort
of
HTTP
client
and
any
other
application
you're
using
as
well.
A
The
main
reason
this
comes
about
is
because
of
connection
pooling,
so
the
connection
from
ELB
that
was
going
to
host
B
if
L
B
were
to
create
a
new
connection,
it
would
cou
be
proxy,
would
only
forward
it
to
an
active
pod.
You
don't
have
to
worry
about
that.
The
problem
is,
is
that
these
connections
may
or
may
not
get
reused
based
on
your
application
can
and
if
you're,
not
careful
about
setting
headers
such
as
their
connection
clothes.
You
may
end
up
stuck
with
this
problem.
A
A
Secondly,
when
you
have
the
cni
plugin,
you
have
a
whole
bunch
of
hosts
and
they
get
networked
together.
Somehow,
for
example,
in
this
diagram,
so
we
use
the
weave
net
plugin,
which
works
great
and
has
lots
of
configuration
options,
but
there
was
one
we
weren't
aware
of
until
it
sort
of
bit
us
since
we
down
scaler
cluster,
it's
possible,
you
can
end
up
with
disconnected
nodes,
so
say
like
we
have
this
diagram
here
and
host
BNE.
We
turn
off
because
we
don't
have
traffic
or
something
like
that.
A
Well
now
you
can
see
your
cluster
is
divided
into
two
different
groups
and
they
can't
talk
to
each
other
across
that
divide.
That's
something
where
you
can
actually
configure
the
number
of
connections
a
given
host
is
going
to
have,
and
you
know
you
can
easily
prevent
if
I
expected,
downscale
and
nodes
and
then
make
at
least
n
plus
1
connections,
simple
heuristic,
but
it
works.
It
gets
the
job
done,
but
you
do
need
to
be
aware
of
your
sort
of
topology
there.
A
For
us,
we
highly
monitor
the
number
of
connections
that
each
host
has
how
many
are
failed,
having
or
whatever,
instead
alerts
on
that
all
very
easy,
using
Prometheus
and
graph
on
a--,
and
you
could
consider
your
network
topology
while
scaling
these
nodes
will
not
refresh
their
node
peers
list.
So
if
a
node
goes
down,
it
may
not
actually
try
and
to
find
a
new
node.
You
could.
You
know,
make
all
the
nodes
aware
of
that.
We
thought
that
was
a
little
too
complicated
and
haven't
had
a
need
based
on
our
heuristics
and
monitoring.
A
Yet
so,
how
do
we
go
from
here
to
multiple
clouds?
There's
still
a
lot
of
moving
pieces
of
infrastructure
that
you
need
to
create
VP
sees
networks,
you
know
virtual
machines
etc,
and
we
also
have
to
get
it
to
work
on
a
whole
bunch
of
different
cloud
providers.
So
the
first
step
is
that,
oh
sorry,
but
there's
a
bunch
of
considerations
with
your
application.
You
might
want
to
think
of.
First
vendors
should
like
to
try
and
lock
you
into
certain
to
using
them.
For
example,
AWS
has
services
like
DynamoDB
Kinesis,
ECS,
all
great
services.
A
Additionally,
you
can
use
kubernetes
to
replace
a
lot
of
things
you
might
otherwise
use
infrastructure.
For,
for
example,
you
can
use
a
service
to
replace
having
physical
load
balancers
as
we've
discussed
before,
but
we
still
have
to
figure
out
how
to
provision,
and
this
is
where
it
believed
this
was
mentioned
in
the
last
talk
as
well,
where
hashey
Korps
terraform
is
great.
It
allows
you
to
create
dot,
TF
files.
You
can
commit
infrastructure
as
code.
You
can
version
it.
A
You
can
plug
in
a
whole
bunch
of
different
variables
for
say
region
and
AWS
account
or
whatever
else,
and
then
we
can
bring
up
a
cluster
with
those
parameters,
but
once
you
have
a
whole
bunch
of
TF
files
for
editable
us,
you
can't
actually
just
change
those
to
the
azure
equivalents.
It's
not
just
a
find
and
replace
virtual
machine
with
the
virtual
machine
for
a
matcher
and
there's
a
number
of
reasons
for
that.
A
Don't
try
and
go
quickly
through
these,
so
AWS
has
an
internal
DNS
to
your
V
PC,
but
other
cloud
providers
do
not,
so
you
might
have
to
reconsider
your
architectures
a
little
bit
there
again.
I
would
rely
on
kubernetes
to
do
in
services
to
use
dns
and
not
actually
have
to
use
of
DNS
within
the
cluster,
but
you
can
use
a
kubernetes
level
DNS
and
that
solves
the
problem
for
you
right
there
in
Rackspace.
A
If
you
have
a
public
and
a
private
IP
address
those
come
as
two
different
network
cards
and
there's
this
issue
with
weave
and
the
Kubla,
where
it
end
up
sending
traffic
to
the
wrong
network
card.
If
it's,
basically,
if
you're
trying
to
send
traffic
within
the
cluster,
but
a
route
didn't
exist,
then
it
would
actually
send
it
to
the
through
the
public
network
interface
instead
of
the
private
network
interface,
and
there
are
ways
to
work
around
that.
It's
on
that
issue
there,
not
a
workaround
it
lasts.
A
Api
versions
are
never
simple.
For
Rackspace
is
not
exactly
the
same
as
OpenStack
within
a
different
OpenStack
clusters.
You
can
have
multiple
versions
of
authentication
even
and
the
types
of
and
api's
that
are
available
may
or
may
not
change
between
different
customers
who
have
an
openstack
cluster.
So
it's
one
of
those
things
like
you
can
just
got
all
the
services
available
within
OpenStack
for
a
particular
environment.
A
Load
balancers,
you
know
if
you're
doing
a
lot
of
copy
and
paste
between
terraform
files,
you
might
see
Oh
AWS,
HTTP
load,
balancer
cool
copy,
that's
at
OpenStack,
but
it
turns
out
that
with
AWS,
if
you
have
HTTP
that
means
terminate
SSL,
but
an
OpenStack
HTTP
means
do
not
terminate
SSL.
So
there's
just
these
like
really
small
things
that
can
get
lost
in
this
mix
of
so
much
going
on
that
you
end
up
not
noticing
very
fine
details
like
that.
A
Additionally,
there
are
differences
where
you
can
only
bind
to
a
single
internal
load
balancer
for
a
virtual
machine
and
again
it's
because
it's
meant
to
be
a
classical
one.
Application
per
virtual
machine
AWS
as
you're
in
Google,
compute
I,
believe
they
all
have
concepts
of
like
an
api
gateway
or
something
along.
Those
lines,
which
is
a
Appler
application
load,
balancer
I,
think,
is
what
it's
called
an
AWS,
which
can
route
a
little
more
natively
to
containers
instead
of
virtual
machines.
A
It's
something
that
it
again
doesn't
exist
in
every
environment
certainly
want
to
exist
in
an
on-premise
environment.
But
if
you
only
want
to
target
a
particular
cloud
provider,
you
could
use
those
types
of
load
balancers
instead,
but
with
Azure
traffic.
If
it
goes
from
a
you
know,
say
like
you
have
to
containers
a
web
and
an
API
server
on
the
same
machine,
and
you
want
to
talk
to
the
web,
wants
to
talk
to
the
API
server
through
a
load
balancer
from
Adger
and
actually
can't
do
that.
A
Algorithms
and
all
of
that
and
less
than
an
hour,
which
I
think
is
pretty
good
when
it
takes
25
minutes
to
provision
an
RDS
instance
in
AWS,
so
not
much.
We
can
do
about
that
one.
So,
looking
forward,
there's
a
lot
of
performance
improvements
coming
up
just
the
last
year,
the
GPU,
exposing
GPUs
to
a
container
was
just
added
to
kubernetes
I
believe
in
1/4,
I,
think
and
since
then
it's
become
easier
to
expose
multiple
devices.
A
Maybe
they
could
be
faster,
but
I
have
no
idea
how
to
get
kubernetes
to
run
and
expose
an
FPGA,
I,
don't
know,
but
it'd
be
really
interesting
to
see
what
can
happen
in
the
future
there
next
there's
a
sort
of
when
we're
building
a
whole
bunch
of
images
and
shipping
them
around
our
cluster
a
lot
we
found
that
the
biggest
the
whole
Nick
ends
up
being
just
how
big
some
of
the
images
are
and
being
able
to
compress
them
better
would
be
an
ideal
situation
that
I'm
not
sure.
This
is
likely
to
change.
A
Multi-Cloud
is
sort
of
becoming
a
little
more
of
a
native
thing
within
kubernetes.
So
what
we
have
is
a
system
we
can
deploy
inside
of
a
particular
data
center,
but
you
can
imagine
it
would
be
ideal
to
be
able
to
run
across
multiple
data
centers,
which
has
problems
of
security
concerns
and
data
traveling
over
the
internet.
You
know
you
can
also
get
a
lot
of
latency
and
it
gets
confusing
and
hard
to
track.
A
All
of
that,
and
that's
where
things
like
with
the
service
mesh,
would
enable
you
to
track
better
how
your
how
your
services
are
talking
to
each
other
across
all
of
these
regions,
there's
the
concept
of
a
federated
uber
Nettie's,
where,
if
you're
trying
to
run
you
know
it's
sort
of
the
difference
between
running
kubernetes,
that's
has
masters
in
one
data
center
and
all
the
nodes
are
part
of
one
cluster
versus.
Do
you
have
you
know
kubernetes
running
and
five
different
clusters
in
some
way
to
federate
access
to
each
one
of
those
in
a
common
format?
A
So
I
think
that's
about
all
I
have
right
now,
so
you
can
contact
me
for
any
other
questions
at
Patrick
at
algorithm
eucom,
if
you're
interested
in
infrastructure,
how
artificial
intelligence,
machine
learning
surrealist
computing
kubernetes,
how
all
that
sort
of
matches
together
we're
hiring
for
pretty
much
everything,
and
you
know
we
have
a
coupon
code.
We
give
people
free
credits
every
month,
but
there's
also
a
free
coupon
code.
A
A
A
And
then
you
have
self-signed
certificates
and
it's
a
little
bit
annoying
to
deal
with
one
of
those
things
where,
when
we
are
controlling
the
stuff,
that's
running
you've
mounted
the
docker
socket
to
only
a
single
thing
and
again
it
only
has
limited
permissions
and
you
can't
do
anything.
We
don't
run
any
privilege,
containers
or
anything
like
that.
A
A
It's
an
AWS
thing:
it's
in
their
documentation.
It's
basically,
if
you
have
an
EBS
volume
that
has
data
on
it.
The
way
that
the
volumes
are
stored
is
that
they're
backed
up
in
s3.
So
when
you're
create
a
new
volume
and
you
attach
it
to
an
instance
from
a
snapshot
that
volume
gets
attached
immediately,
but
it
doesn't
actually
fetch
all
the
data.
A
So
if
you
have
a
whole
bunch
of
blocks
it
basically
every
time
you
access
a
block
for
the
first
time
it
has
to
get
downloaded
from
s3,
which
can
be
slow,
and
if
you
have
say
you
know,
250
gigabyte
Drive,
then
it's
got
to
download
all
of
that
data
from
s3.
Even
if
there's
nothing
there.
If
it's
an
empty
block,
it
doesn't
know
it's
an
empty
block
until
it
fetches
it.
Yeah.