►
From YouTube: Automating and Testing Production Ready Kubernetes Clusters in the Public Cloud - Ron Lipke
Description
Automating and Testing Production Ready Kubernetes Clusters in the Public Cloud - Ron Lipke, Gannet/USA Today Network
As a large enterprise organization with legacy infrastructure, we were interested in adopting Kubernetes in our internal Platform as a Service in the public cloud. However, we faced several challenges not addressed by the turn key offerings on the market, such as:
- Maintain control over network architecture within the public cloud to integrate with our internal resource
- Allow teams to easily spin up kubernetes clusters on their own for faster development cycles while retaining cost boundaries and charge-back insight
- Quickly iterate as new kubernetes versions are released and make new features available to end-users (most recently: Role Based Access Controls and StatefulSets)
We will share our experience of using configuration management to automate the testing, building and deployment of production ready cloud agnostic kubernetes clusters to the AWS and Google clouds. We will also discuss examples of moving some of our largest application workloads to these clusters.
About Ron Lipke
Senior Developer, Platform as a Service, Gannet/USA Today Network
Nuclear plant operator turned cloud person
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Hello,
everybody
thanks
for
coming.
We
can
get
started,
I
guess
so.
My
name
is
Ron
Lukey
I'm,
a
senior
engineer
cloud
type
person
on
the
platform
as
a
service
team
at
Gannett,
I'm
gonna
talk
to
you
a
little
bit
about
our
container
journey:
full
disclosure.
This
is
my
first
time
ever
speaking
at
a
conference,
so
in
addition
to
dying
a
little
inside
right
now,
I'm
also
a
person
who
stutters
or,
if
you're,
from
the
UK
its
stammer.
So
if
it
looks
like
I'm
rebooting
up
here
or
stalling,
that's
probably
why
so.
A
A
A
So
what
does
a
platform
as-a-service
do
forget?
We
provide
the
central,
the
central
location
for
self-service,
our
provisioning
and
and
tooling,
for
infrastructure
for
about
40
internal,
a
dev
teams
across
cadet
and
that's
pretty
much
where
they
can
test
and
deploy
and
and
all
the
way
up
to
you
up.
There
apps
all
the
way
up
to
production
workloads,
and
this
is
all
in
the
public
cloud.
We
do
have
some
data
centers,
but
everything
on
the
platform
is
in
in
the
public
cloud.
A
So
for
us
not
only
are
we
are
we
managing
the
environment
that
runs
all
of
all
of
us
a
today
apps,
but
our
customers,
dev,
is
our
prod.
From
from
our
point
of
view,
we
have
a.
We
have
a
five
teams
on
paths
I'm
on
the
integration
team
and
we
are
responsible
for
architecting
and
maintaining
the
core
features
of
the
platform.
A
Cash
in
front,
cashing
back,
protect
your
origin
at
all
costs
and
you
should
be
able
to
handle
all
those
new
spikes
that
are
seem
to
be
happening.
Much
more
frequently,
then
there's
events
like
the
Super
Bowl,
the
Olympics,
the
presidential
election,
which
have
a
sustained
high
volume
of
traffic
pattern
and
the
data
footprint
is
of
it's
very
different
because
your
users
want
the
most
accurate,
most
up-to-date
information
as
possible.
A
So
you
can't
protect
your
origin
behind
those
really
high
cache
details
and
the
content
and
the
product
teams
are
probably
going
to
be
requesting
a
lot
of
but
deployments
to
change
how
the
data
is
being
presented,
fix
any
bugs
or
anything
like
that,
so
the
faster
we
can
do
those
up
deployments
and
safer.
We
can
the
more
satisfied,
our
internal
and
our
external.
What
customers
are
so.
A
This
was
shaping
up
to
be
the
largest
night
of
traffic
in
USA
today
history
and
we
figured
why
not
make
that
or
first
time
we
run
a
container
app
up
to
this
point.
We
had
not
a
container
on
the
platform,
so
this
was,
it
was
new
for
us.
It
was
something
that
we
had
been
considering,
but
really
hadn't
and
out
yet,
and
then
our
core
product
team
came
to
us
and
said:
hey.
A
We
were
interested
in
running
the
Elections
app
in
a
container,
we
said
sure,
and
we
scoped
it
as
kind
of
a
stretch
goal
if
we
could
run
maybe
5
or
10%
of
election
night
election
night
traffic
in
a
container.
We
would
call
that
a
success,
so
we
started
out
by
creating
a
list
of
our
requirements.
This
is
just
a
short
short
list
of
those.
We
are
an
old
enterprise.
We
have
a
data
centers
all
across
country,
so
we
have
a
lot
of
legacy
infrastructure
that
we
needed
to
just
be
aware
of
that.
A
There
were
dependencies
here
that
we
couldn't
we
couldn't
engineer
out
of
and
make
everything
just
a
perfect
cloud
native
app.
We
really
have
to
take
advantage
of
our
existing
bootstrapping.
Our
existing
up
features
on
the
platform
to
bootstrap
these
at
clusters.
We
had
limited
time
to
even
do
this,
so
we
couldn't
learn
something
new
like
stand
up
up
for
prometheus
a
cluster
or
something
we
had
to
use
what
we
were
using
on
the
platform
and
we're
a
chef
shop.
So
we
had
to
use
chef
2
to
bootstrap
these
things.
A
Auto-Scaling
was
non-negotiable
for
nodes
in
containers
because
of
the
breaking
news
thing,
and
we
really
didn't
have
time
to
like
write
our
own.
So
it
would
be
better
if
this
was
inherent
in
whatever
we
chose
to
use
and
pretty
much
everything
on
the
platform
is
a
self
service.
So
these
a
coup
daddies
like
these
container
clusters
have
to
be
in
the
same
vein.
A
A
We
all
of
these
are
great
products.
There
were
things
that
in
and
of
itself
they
were
great,
but
we
kept
a
finding
that,
as
we
got
a
little
bit
down
in
the
development
cycle
with
one,
we
would
hit
a
deal
breaker
or
just
something
we
couldn't
engineer
around
in
the
in
a
time
that
we
had,
but
this
is
a
cube
con.
So
it's
no
surprise
that
the
winner
was
kubernetes.
That
would
be
really
awkward
ECS.
A
So
a
good
part
of
the
reason
why
we
chose
we
chose
this
is
something
I
like
to
call
the
Kelsie
Hightower
effect.
We
all
know
who
he
is.
If
you
don't
know
who
he
is
I
highly
recommend
checking
out
his
Twitter
and
github
accounts,
our
Google
team-up
put
us
in
front
of
Kelsey
after
we
just
kept
bombarding
them
with
questions
that
they
were
like.
A
Someone
else
needs
to
answer
these,
and
after
that
meeting
we
came
out
like
really
confident
that
cube
was
was,
was
the
choice
that
that
was
going
to
work
the
best
for
us
so
just
to
talk
about
some
of
other
requirements.
I
was
talking
about
and
how
they
informed
our
choice
of
kubernetes.
We
probably
spent
a
lot
of
time,
probably
the
most
time
on
on
networking
and
I,
want
to
give
a
shout
out
to
my
teammate
Dane
who's
doing
this
now
for
our
team.
A
He
was
a
network
engineer
in
a
former
life,
so
his
expertise
here
was
super
super
invaluable
and
Q
has
some
pretty
some
pretty
like
gnarly
a
network.
It's
where
everything
has
to
be
able
to
see
everything
else
without
you
know
what
the
magic
of
nat
and
since
we're
running
these
in
a
public
cloud,
we
really
weren't
comfortable
with
delegating
a
cluster
the
ability
to
edit
or
network
routes
in
like
AWS,
a
PCP,
V
PC
or
in
GCE
plus
there's
inherent
limits.
A
You
know
you
can
only
about
50
routes,
so
we
went
with
an
overlay
network
which
is
pretty
common.
We
looked
at
weave,
we
looked
at
flannel,
we
looked
at
calico
weave
we
eliminated
early
flannel.
To
be
honest,
we
just
didn't
have
time
we
never
like.
It
was
planned
that
we
were
gonna
look
at
it,
but
we
just
ran
out
of
time,
and
that
was
okay
because
it
turned
out
that
calico
was
really
good
fit
for
us.
A
It's
policy
management
was
was
a
super
helpful
with
our
app
deployment
strategy,
which
I'll
get
to
in
a
slide
pretty
soon
IP
type
II
encapsulation
meant
we
could
just
drop
it
into
an
already
existing
V
PC.
We
didn't
have
to
stand
up
anything
extra
or
reconfigure.
Everything
and
a
SS
was
probably
happy
since
we're
not
trying
to
set
up
a
BGP
routes
between
our
clusters
and
their
routers
and
yeah.
A
This
is
what
scaler
looks
like
to
an
end-user
on
the
platform.
It
has
a
really
great
UI.
It
also
has
a
really
extensive
API
and
it
handles
all
of
the
creating
and
managing
of
our
cloud
user.
Our
cloud
resources
and
supports
really
extensive
governance
of
policy
and
roll
base
access.
So
we
can
empower
our
users,
but
we
can
also
fine-tune
what
they
have:
access
to
and
scalar
runs
on
a
farm
paradigm.
A
So,
like
one
cube,
cluster
would
be
a
farm
and
then,
within
that
farm
on
the
left,
you
have
your
farm
roles
and
they
map
to
cube
master.
The
workers
at
C
D,
your
API,
a
load
balancer
and
each
of
those
are
formals,
has
its
own
separate
settings,
for
instance,
size,
auto
scaling
of
settings
orchestration
scripts,
and
this
really
worked
well
with
for
how
we
were
envisioning
our
or
clusters
and
then
kind
of
more
on
the
side
of
non
technical
challenges.
A
We
really
have
to
architect
our
Q
clusters,
knowing
that
they
be
owned
and
a
provision
by
just
a
single
team.
So
that
means
we
kind
of
have
to
automate
and
abstract
all
of
the
little
things
that
an
app
team,
just
probably
does
want
to
deal
with
like
making
sure
SCD
is,
is
backed
up
setting
our
name
spacing
of
standards
upfront,
because
we
were
going
to
have
a
lot
of
teams
with
a
lot
of
clusters
and
that
was
going
to
get
really
annoying
soon
and
and
automating.
A
So
we
really
have
to
make
sure
that
our
documentation
was
good.
We
really
wanted
to
incentivize
them
to
care
and-
and
part
of
that
is
also
eating
or
eating
our
own
dog
food.
So
to
speak.
So
that's
we
were
using
cube
on
our
team
and
running
our
apps
on
it
and
kind
of
getting
a
perspective
of
what
their
user
experience
looks
like.
A
So
that
takes
us
to
what
our
current
cube
architecture
is
like,
and
this
is
pretty
much
what
happens
when
any
Atene,
when,
when
anytime
a
team
comes
to
us
and
says,
hey
Paz
I
want
to
do
some
containers
and
we
say
hold
up
start
here,
and
everyone
has
to
a
complete
our
Paz
labs
as
we
call
them,
they
basically
run
through
all
the
features
of
the
platform,
basically,
what
it
takes
to
start
using
them
how
they
can
make
your
team
as
successful.
The
cube
lab
is
our
longest
one.
A
A
A
This
is
a
really
bad.
Drawing
of
what
our
cube
kind
of
looks
like
right
now.
Some
things
to
note
here
are:
we
are
masochists
and
we
basically
roll
our
own
cube
RPMs
using
the
Google
binaries,
we're
not
running
them
as
containers
themselves
and
that's
for
everything
in
the
Q
control
plane
and
then
queue
Thank,
You
proxy
on
the
workers
we
run
sed
and
its
own
cluster.
A
There,
it's
usually
always
a
3-node
a
quorum,
but
for
some
of
our
bigger
clusters
we
go
up
to
five,
and
then
we
put
AJ
proxy,
which
is
heavily
orchestrated
in
front
of
all
the
worker
nodes,
and
then
that
gets
updated
with
where
to
find
the
API
server
and
anything
that
needs
to
talk
to
the
cube
API.
It
goes
through
that
HR
proxy.
A
So
we
also
have
this
really
awesome,
API
team,
on
on
paths.
They
build
all
these
great
tools
and
go
and
they
had
built
a
deployment
API
that
was
initially
used
for
deploying
apps
on
cloud
instances
and
managing
that
entire,
the
entire
lifecycle,
so
they
built
some
a
new
some
new
functionality
into
it
to
handle
a
container
deployments
on
a
cube
cluster.
A
Basically,
it's
managing
like
three
three
key
things:
it's
taking
a
cube,
a
deployment
object,
so
anything
you
can
do
in
in
a
cube
deployment.
You
can
do
in
our
API.
It
just
takes
that
DC
realises
it
and
sends
it
to
the
cube
API.
It
creates
a
service
to
go
with
that
deployment
and
it
picks
a
node
port
from
this
list
of
node
ports
assigned
to
that
cluster.
A
Then
it
goes
and
updates
the
HF
proxy
config,
with
the
IP
addresses
of
all
of
the
worker
nodes
in
the
farm
and
the
node
ports,
where
you
can
find
the
currently
a
deployed
version
of
your
app
and
in
the
new
version,
then
the
user
can
use
the
API
and
start
shifting
a
traffic.
However,
they
want
and,
however
they
are
comfortable
with,
they
could
also
do
a
canary
deployment.
A
They
can
just
shift
a
little
bit
of
traffic
and
then
do
some
automated
eye
testing
with
a
pass/fail
mechanism
in
it
once
they're,
okay
with
the
deployment-
and
they
say
it's
it's
good
to
go.
They
send
a
complete
a
complete
message
to
the
complete
endpoint
and
the
API
will
go
in
and
delete
the
old
appointment
and
services
and
update
HR
proxy
that
config
again
or
if
it
didn't
go
well,
they
could
ask
for
a
rollback
and
it'll
just
put
everything
back
to
the
way
it
was
before.
A
A
So
a
Cubism
is,
is
really
good
at
like
abstracting
all
these
things
from
people
who
just
want
to
run
container
workloads
but
I'd
say
that
a
benefit
comes
with
a
lot
of
added
complexity
from
operations.
A
standpoint
there's
a
valid
argument
also
that
like
whether
or
not
taking
on
and
that
added
a
complexity
is
worth
it
at
this
point
we're
where
we're
at
to
run
some
containers.
A
One
of
those
things
is,
we
run
a
server
spec
in
our
CI
CI,
a
pipeline
against
a
fully
running
cube,
stack,
we're
checking
for
making
sure
that
the
system
is
configured
right
and
we're
also
checking
actual
functionality.
We're
running
a
commands,
we're
checking
in
the
output
of
it
we're
making
sure
everything
in
the
cube
system.
Namespace
isn't
just
saying
it's
running,
but
it's
actually
also
working.
A
We
maintain
a
set
of
run
books
if
you're
not
doing
this
I
highly
suggest
it,
and
not
just
for,
like
the
you
know,
the
new
person's
first
night
on
call
it's
it's
really
good.
Just
as
a
reference
for
the
team,
we
have
everything
in
there
from
common
troubleshooting
techniques,
also
links
to
documentation.
A
They
are
continuously
updated
any
time
we
encounter
something,
that's
just
weird:
it
goes
in
there
and
they're
all
in
a
github
repo,
and
then
we
link
those
to
all
of
our
our
New
Relic
alerts
and
our
Victor
ops,
like
our
paging
services
alerts,
and
then
these
last
two
things
are
kind
of
just
like
standard
ops,
things
I
think
in
the
keynote.
Yesterday
morning
we
heard
observability
engineering,
I
kind
of
like
that
term,
and
this
is
kind
of
like
in
a
cute
cluster,
which
is
different
from
our
normal
stuff.
A
It's
very
dynamic
and
it's
volatile
things
are
being
spun
up
and
torn
down
all
the
time,
so
we
shove
all
of
our
cue
blogs
and
our
container
logs
off
to
our
logging
vendor.
This
is
also
where
it
helps
to
have
some
good
name
spacing
up
patterns
up
front
and
then
we've
designed
and
created
some
pretty
nifty
dashboards
that
our
success
team
has
even
automated
as
part
of
the
provisioning
step
when
you're
when
you're
building
a
new
acute
cluster.
This
is
just
an
example
of
what
one
looks
like
just
standard
worker
stuff.
A
You
know
it's
CPU
disk
memory
and
then
just
checking
what,
like
all
the
important
services
on
a
worker,
are
still
there
like
docker
and
queue
proxy
and
cube
lid
stuff.
And
then
we
look
at
things
like
how
many
of
a
container
and
a
version
that
container
are
on
all
the
workers.
We
look
at.
We
look
at
the
container
at
distribution
that
gives
us
a
little
bit
of
insight
into
how
the
bin
packing
is
working
and
then
just
some
random
bio
stuff
over
there.
So
we
did
run
into
somewhat
challenges.
A
One
of
them
was,
we
had
problems
with.
Nodes
are
terminating
and
terminating
cleanly,
not
only
draining
pods
but
removing
themselves
out
of
a
calico.
So
we
didn't
have
routes
to
knows
that
it
weren't
weren't
there
anymore.
We
handle
this
by
creating
an
orchestration
scripts
that
scaler
will
trigger
before
a
worker.
Node
is
removed
and
terminated
in
it's
a
cloud
provider
and
all
that's
really
doing
is
it's
hitting
it.
It's
hitting
at
the
cube
API
for
its
own,
a
node
name.
A
There's
like
inherent
ways
I
to
do
this
in
kubernetes
or
you
can
just
use
your
cloud
provider.
You
know
if
you're
in
AWS
you're,
using
auto
scaling
groups
in
GCE
or
using
managed
instance
groups,
but
there
are
our
chances
are
that
you
probably
want
to
do
things
after
the
node
is
there
and
it
like
post,
config
management,
post
boot
steps.
A
A
Initially,
on
our
or
smaller
initial
clusters,
we
were
taking
the
average
of
those
metrics.
So
when
we
got
to
the
larger
clusters
we
started
at
seeing
that
there
were
a
few
at
worker
nodes
that
were
just
getting
crushed,
even
though
everything
else
was
fine.
So
we
rewrote
those
scripts
to
scale
off
of
an
individual
workers
resource
at
consumption
and
we
made
those
both
available
to
our
users.
So,
whichever
works
better
in
in
their
cluster,
they
could
think
it's
use.
We
had
some
issues
with
contract
limits.
A
This
was
kind
of
a
race
condition
and
our
testing,
because
we
were
disabling
a
firewall
D
in
chef
and
also
in
our
pre-baked
images,
but
we
knew
that
a
calico
was
running
its
own
firewall
dia
pretty
much,
but
it
was
also
reloading.
The
net
filter
contract
kernel
module,
so
it
was
putting
back
into
play.
Those
sis
ETL
contract
limits
and
we
started
seeing
just
random
nodes
in
our
larger
clusters,
just
dropping
out
TCP
connections.
A
But
there's
currently
some
issues
here
or
we're
getting
bit
by
an
open
issue
where,
if
you
set
the
cloud
provider
equals
GCE
cubelet
flag,
which
is
required
to
get
any
of
the
GC
specific
functionality.
Like
persistent
volumes
and
you're
using
a
CNI
plugin,
it
will
still
try
and
create
routes
in
GCE
and
then
you'll
have
nodes,
come
up.
That
will
say
that
ready,
but
there's
no
routes
to
them.
A
If
we
were
just
like
a
team
working
on
just
this,
we
probably
would
be
able
to
do
it,
but
you
know
we
have
a
whole
other
platform
to
manage
and
we're
also
we're
also
integrating
a
features
into
our
cube
clusters.
Like
our
our
vault
integration
or
our
console
integration,
so
we're
trying
to
get
better
at
this
and
we're
looking
at
some
alternatives
that
can
help
us
out
with
the
pace
of
development
here.
A
A
We
managed
over
a
hundred
and
seventy
at
deployments
that
night
all
success
which
we
couldn't
have
done
if
we
were
running
this
on
the
old
old
vm
way.
So
it
was
a
long
night
for
a
lot
of
reasons,
but
it
was.
It
was
a
rewarding
a
finish
to
a
very
challenging
project
and
a
very
transformative,
a
project
for
us
and
I.
Just
won't
talk
about
this
up
real
quick,
because
this
is
a
a
little
bit
more
recent.
A
So
our
success
team
recently
took
all
of
the
desktop
sites
for
all
of
our
properties
and
move
them
into
the
current
kubernetes
cluster,
and
that
resulted
in
reducing
our
daily
operating
costs
by
a
several
hundred
bucks
and
they're.
Looking
at
further
optimizations
there,
our
deployment
times
went
from
fully
optimized
getting
all
140
sites
out,
took
over
two
hours
and
now
we're
down
to
25
minutes,
and
we
could
probably
get
even
faster
there
and.
A
During
the
recent
tragedies
of
Hurricane
Harvey
in
Houston
area
and
Hurricane
Irma
in
Florida,
we
dropped
our
pay
wall
for
all
of
the
local
sites
in
the
in
those
areas.
So
all
the
residents
could
get
you
know
the
most
up-to-date
and
accurate
information
that
they
needed
to
stay
safe
and
informed.
We
ended
up
making
1500
deployments
during
that
a
time
period
which
again,
we
could
not
have
done
the
the
old
way.
So
that
was
a
pretty
big
win
for
us
and
real
quick
open.
A
What's
next
for
us,
we're
eventually
going
to
finish
our
move
into
Etsy
d3
cube
9,
like
I,
talked
about
we're
relooking.
Looking
we're
we're
taking
another
look
at
cube
ATM.
When
we
first
started
this,
it
wasn't
really
ready
and
now
it's
it
looks
like
it
will
help
streamline
our
how
we
handle
our
a
control
plane
with
those
growths
rpms
vault
integration
is
actually
live
now,
we're
using
the
the
kubernetes
vault
off
back
ends
in
conjunction
with
a
pre-configured
role
for
that
team,
a
policy
for
that
team
and
a
role
for
that
cluster.
A
If,
if
any
deployment
needs
to
access
a
secrets
in
vault,
you
just
add
a
vault
in
a
container
to
that
a
deployment
and
then
it'll
mount
a
token
with
a
very
short
a
TTL
in
a
shared,
a
secrets
volume
for
any
of
the
containers
in
that
pod
to
access
we
started
looking
at
service
mesh.
We
think
it's
really
cool.
We
really
want
to
do
awesome
things
with
it,
but
no
one's
really
asking
for
it.
A
So
it's
it's
taking
a
backseat
to
some
higher
priority
stuff
and
last
but
not
least,
the
elephant
in
the
room
are
terrible,
not
terrible,
but
are
terribly
over
orchestrated
H,
a
proxy
per
application,
ingress
load,
balancer
solution
that
we're
looking
at
some
cloud
native
stuff
like
using
al
bees
in
Amazon
or
GCE,
a
load
balancers
or
maybe
something
like
envoy,
but
that's
where
we're
at.
So.
Thank
you
for
listening
to
me.
If
you
have
any
questions,
feel
free
to
ask
or
find
me
ordained
or
we're
also
hiring.