►
Description
Mission Critical Kubernetes - Andy Repton & Michael Russell, Schuberg Philis
A talk about the challenges we faced and the solutions we came up with to utilise kubernetes to deliver an enterprise mission critical system for ourselves and our customers that delivers 100% functional uptime; including high availability, monitoring, multi cloud, multi tenancy, lifecycle management, testing and more.
About Andy Repton
Working on developing Mission Critical solutions for customers at Schuberg Philis in Amsterdam, from Banks to eCommerce to Cancer research centres, focusing on innovation and 'cool stuff'.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Good
morning,
everybody
how's
everyone
doing
you
enjoying
conference,
yeah,
good,
good,
all
right
cool
everyone's,
a
bit
yeah
quiet
this
morning.
My
name
is
Andy
Repton
I'm,
a
mission-critical
engineer
at
shoe
burg.
Phyllis.
We
are
a
mannerless
integration,
specialist
based
out
of
Amsterdam.
The
main
thing
that
we
kind
of
pride
ourselves
on
is:
we
guarantee
100%,
functional
uptime
for
our
customers.
A
B
A
Second,
what
see
if
I
can
get
this
working?
Okay,
just
a
brief
overview
of
what
we're
going
to
talk
about
kind
of
why
we
chose
coop
inators
in
the
first
place,
what
about
it?
I
mean
I.
Think
that's
gonna
be
a
very
similar,
familiar
story
to
all
of
you,
guys,
the
tooling
that
we
chose
something
that
we
call
sub-clusters,
I'm,
not
sure.
If
anyone
here
saw
Brandon
Phillips,
there's
excellent
talk
on
self-hosted
kubernetes,
we
did
a
relatively
similar
idea.
A
Ours
is
slightly
different
but
running
cube,
inators
on
top
of
kubernetes
and
kind
of
what
that
brings
us
testing
and
monitoring
and
kind
of
the
continuation
of
our
journey,
because
you
never
really
stop
with
this
stuff
and
kubernetes
is
constantly
evolving.
So
to
start
off
with,
why
did
we
choose
kubernetes?
We
need
to
go
faster.
We
we,
as
a
company,
recognized
the
software,
is
eating
the
world
and
fast
as
the
new
you
know,
faster
is
the
new
fast.
So
we
wanted
to
go
containerization
and
we
wanted
to
move
into
public
cloud
eventually
in
the
future.
A
B
So
this
is
just
a
quick
overview
of
the
tooling
we'll
go
into
more
detail
later,
but
just
for
the
very
sort
of
high-level
overview.
We're
using
kubernetes,
of
course,
which
were
then
running
on
core
OS,
which
were
deploying
with
terraform,
announce
table
and
we're
using
Voltas
sort
of
a
middleman
for
secret
storage.
For
why
we're
actually
storing
tokens
and
things
like
that
before
actually
deploying
them
to
a
stack
Jenkins
for
CI
testing,
sensu
and
Prometheus
for
monitoring
and
flannel
as
the
overlay
network,
which
has
just
been
replaced
as
of
last
week.
A
One
of
the
other
things
that
we
should
mention
is
we've
kind
of
built
this
all
on
top
of
existing
technology
that
we
also
have
within
our
company.
So
we
have
container
registry
private
as
running
as
a
container.
We
have
an
on-premise
s3,
compatible
object,
store,
which
has
given
us
a
lot
of
benefits,
because
we
can
actually
use
s3
compatible
tooling
with
it.
As
long
as
the
SDK
is
not
hard-coded.
The
region's
into
that
particular
SDK
Java
is
a
key
one
where
they,
actually,
you
can't
use
it
with
anything
outside
of
AWS,
so
kubernetes.
A
We
wanted
to
basically
be
deploying
this
on
both
AWS
and
on
our
private
cloud.
Our
private
cloud
is
cloud
stack,
we've
actually
forked
cloud
stack
now
as
of
about
six
months
ago,
and
our
Fork
is
called
cosmic
or
mission-critical
cloud,
and
we
basically
had
to
write
a
lot
of
the
functionality
into
kubernetes
that
would
allow
it
to
work
with
stack
a
lot
of
the
magic
that
you
see
in
a
lot
of
the
tutorials
and
the
guides
that
you
find
online.
A
A
lot
of
them
are
relying
on
AWS
or
GCE,
and
the
functionality
that
those
cloud
providers
give
you
and
for
us
trying
to
then
reverse-engineer
those
and
get
them
into
our
private
cloud
was
a
really
interesting
journey,
but
actually
gave
us
a
lot
of
insight
into
the
underlying
of
how
kubernetes
is
actually
working.
So
to
start
off
with,
we
just
did
terraform,
and
so
terraform
is
a
pretty
standard
tool
from
Hoshi
Corp
we
used
know
convergence.
It
was
a
mutable
infrastructure.
A
We
used
terraform
combined
with
terraform
templates,
where
you
can
actually
pass
in
variables
and
using
those
configure
cloud.
Config
people
here
familiar
with
the
concept
of
cloud
config
yeah
vaguely
we're
using
core
OS,
so
cloud
config
for
those
who
are
not
aware,
you
can
create
a
Yama
file
basically,
and
this
Yama
file
will
describe
to
Koro
as
how
it
should
configure
itself.
You
pass
this
through
on
the
router
VM.
The
router
VM
then
passes
it
through
to
your
core
OS
host,
it
configures
itself
and
you
have
a
configured
system.
A
It
was
simple
and
simple
as
always
good,
but
unfortunately
there
was
no
convergence,
so
this
meant,
if
we
wanted
to
do
a
rolling
upgrade
of
the
cluster
terraform,
would
quite
happily
destroy
your
entire
infrastructure
and
rebuild
it
for
you,
which
meant
downtime,
which
for
us
was
just
not
really
an
option.
In
addition,
because
core
OS
is
an
immutable
server
or
a
mutable
operating
system,
it
meant
that
we
had
to
configure
everything
through
systemctl
scripts
and
the
systemctl
scripts.
You
then
have
to
add
bash
bash.
Can
then
change
slightly
depending
on
how
you're
deploying
it?
A
B
So
intensive
also
bit
of
complexity
saj
to
get
in
we,
we
do
miss
the
good
old
days
of
cloud
init
where
we
could
just
create
and
destroy
clusters.
My
life
was
simple:
we
didn't
need
to
worry
about
data
migrations
and
and
having
compatible
upgrades
and
things,
but
we're
starting
to
have
a
few
customers
starting
to
use
it.
So
it
was
becoming
a
bit
unreasonable
to
say:
hey.
B
New
features
are
out
by
the
way
we've
deleted
your
cluster,
so
this
is
where
ansible
came
in,
and
this
gave
us
for
the
first
time
the
ability
to
do
rolling
upgrades
so
with
terraform
out
of
the
box,
there's
no
actual
way
to
say
upgrade
master
1,
then
master
2,
then
master,
3
and
there's
definitely
no
way
to
say
upgrade
master
1.
Do
a
health
check,
then
upgrade
master
2.
So
that
was
really
the
the
main
plus
side
of
using
ansible
and
another
big
big
gain
that
we
got
from.
B
That
was
being
out
of
stole
the
secrets,
client
side,
so
from
actual
laptops,
we're
pulling
the
seekers
out
of
volt
with
a
vault
token
and
then
actually
deploying
those
to
the
cluster.
So
this
removed
the
need
to
have
to
put
any
sort
of
secrets
inside
a
cloud
in
it
which
was
a
bit
of
an
insecure
way
of
doing
things.
B
So
the
other
really
really
big
game
we
got
from
this
is
it
gave
us
much
better
templating
engine,
so
this
meant
that
we
could
do
certain
tasks
for
Amazon,
hosted
instances
or
certain
tasks
for
cloud
stack
or
if
we
wanted
to
potentially
support
anything
else
and
we'll
tell
turn
a
little
bit
later,
but
with
the
sub
coasters,
there's
also
slight
changes
needed
for
the
nodes
running
in
sub
clusters.
So
with
ansible,
that's
a
very
simple:
oh!
B
We
just
need
to
do
this
if
it's
a
sub
coaster
and
do
this
if
it
isn't
one
so
this
gave
is
just
a
single
single
codebase
that
we
could
use
across
all
of
our
configurations
and
cloud
providers.
So
one
of
the
downsides
that
we
hadn't
yet
solved
was
how
to
handle
having
lots
of
different
clusters
and
lots
of
master
components,
and
this
was
also
a
bit
of
a
practical
side.
B
But
the
way
we
work
where
we
need
to
actually
pay
for
infrastructure
from
our
cloud
team
and
the
customers
actually
using
this
environment
also
have
to
pay
for
it.
So
if
we
were
able
to
find
a
way
to
actually
consolidate
all
of
the
master
components
and
ec
DS
in
a
really
nice
tool,
such
as
kubernetes,
then
that
would
be
a
big
game
for
us
and
also
for
our
customers.
So
a.
A
Little
bit
of
background
on
story
for
this
one,
somebody
came
to
me
a
engineer
and
one
of
the
other
customer
teams
and
said:
look
I
really
need
a
private
kubernetes
cluster
I,
don't
want
to
configure
it
I,
don't
want
to
have
to
do
anything
with
it.
I
just
want
an
API
and
I
want
nodes
on
it
and
I
just
want
to
start
using
it.
Can
you
give
this
to
me
and
we
kind
of
thought
about
this,
and
we
came
up
with
this
idea
of
subclusters,
so
this
is
basically
kubernetes
running
on
top
of
kubernetes.
A
We
were
trying
to
figure
out
with
life
cycle.
You
know,
how
do
you
store
the
secrets
for
kubernetes?
You
know,
I,
don't
know
if
anyone
set
up
multi
master
kubernetes
before
but
trying
to
sort
out
the
the
certificates,
for
example,
and
the
keys
is
really
quite
difficult.
How
do
we
sort
out
logging?
How
do
we
sort
our
upgrades?
How
do
we
sort
out
a
che?
All
of
these
things
are
problems
that
kubernetes
itself
excels
at
solving,
and
then
we
kind
of
thought
about
it
and
we
realize
well.
A
Why
don't
we
just
put
kubernetes
on
top
of
another
kubernetes,
so
we
created
something
which
we
call
the
management
cluster,
which
is
just
a
random
name?
We
assigned
that's
deployed
using
terraform
and
ansible,
and
then,
on
top
of
that,
we
create
namespaces
per
additional
cluster,
which
we
call
a
sub
cluster.
A
A
Now
we
are
just
using
kubernetes,
so
we
have
a
deployment
which
contains
three
replicas
set
of
three
parts.
Each
of
those
pods
has
three
images
inside
it:
a
cube,
API
server,
a
cube
controller
manager
and
a
cube
scheduler,
those
three
leader
election
and
they
float
somewhere
within
the
management
cluster
control
by
kubernetes.
They
talk
to
their
own
etcd,
and
this
allows
us
all
of
the
benefits
of
a
deployment.
You
know
we
have
high
availability.
A
In
addition,
logging,
if
we
have
any
problems
with
any
of
the
components
we
just
connect
to
the
management
cluster
and
just
do
a
cube,
CTL
logs
command,
we
can
even
do
an
exec
into
those
particular
containers
and
see.
What's
going
on
there,
we
can,
on
the
fly,
adjust
the
velocity
of
each
of
those
binaries
to
increase
or
decrease
logging.
Based
on
what
we'd
like
it
to
see
secrets,
we
can
now
push
the
secrets
into
the
management
cluster
and
mount
them
inside
of
the
pots
as
volumes.
This
allows
us
to
push
anything.
A
We
need
to
very
quickly
into
the
management
cluster,
and
then
we
can
just
wreak
ich
the
pods
and
they
will
basically
get
their
new
secrets
we'll.
Actually,
we've
actually
are
starting
to
use
hash.
Your
core
vote
for
that,
but
that
comes
at
the
end
and
in
a
ditch
sorry,
in
addition,
ease
of
upgrade.
We
have
teams
who
come
to
us
and
say
I
really
want
to
use
alpha
cube.
Inators,
you
know,
I
want
to
use
one
dot
for
today
and
I
have
another
team
who's
on
one
dot
3.5.
A
Well,
now
it's
as
simple
as
cube
CTL
edit
deployment
change,
the
image
tag
to
be
v1
dot,
four,
do
a
right
quit
in
them
and
cube
inators
automatically
does
a
rolling
upgrade
of
the
deployment
it'll
turn
off.
Two
of
my
masters,
it'll
turn
on
two
more
of
them
when
they
were
port,
healthy
it'll
turn
off
the
last
one,
and
then
we
create
my
H,
a
kubernetes
masters.
It
takes
about
five
seconds.
In
addition,
with
deployments,
you
have
robach,
you
can
roll
back
a
kubernetes
deployment.
A
So
if,
for
whatever
reason,
the
upgrade
goes
badly
is
just
as
simple
as
a
cube.
Cto
rollback
and
the
main
thing
for
us
was
multi-tenancy.
We
can
put
our
nodes
completely
separate
from
our
master
components.
The
nodes
and
the
master
components
do
not
need
to
shower
a
network
overlay.
It
makes
life
easier
sure,
but
they're
quite
happy
to
run
separately.
A
We
can
go
as
far
as
actually
running
our
master
components
inside
of
Cuba
native
cluster
in
our
data
center
and
run
the
nodes
for
those
master
components
in
AWS
and
they
actually
communicate
over
the
public
Internet.
This
allows
us
to
very
quickly.
You
know,
control
the
master
components
on
our
IP
space,
which
allows
us
to
restrict
it
by
firewalls,
but
the
nodes
themselves
can
win
in
AWS
and
take
advantage
of
elastic
block
storage,
IO
bees
and
better
replication.
A
I
would
offer
a
demo
of
this,
but
the
terraform
takes
forever
and
I.
Try
to
do
this
earlier.
I
took
twenty
minutes
and
I
won't
bore
you
with
it.
If
anyone
is
really
interested
in
seeing
this
I'm
quite
happy
to
show
you,
this
is
what
it
looks
like
from
the
management
cluster.
This
is
one
of
our
sub
clusters,
with
the
etcd
pet
set
at
the
top,
and
then
the
three
master
components
running
as
an
H.
A
down.
The
bottom
I
think.
B
One
thing
I'll
add
to
the
sub
clusters:
is
that
having
the
nodes
separated
from
the
master
components
also
solves
another,
really
big
problem
that
we
have,
and
that
is
that
different
customers
we
have
will
have
their
own
environments
already
with
very
look
down
firewall
rules.
So
we
want
to
be
the
ones
actually
hosting
kubernetes
and
in
control
and
updating
that,
and
they
want
to
have
the
nodes
in
their
own
environment
where
they
can
actually
connect
locally
to
their
databases
and
other
service
needs
access.
A
And
I
would
like
to
stress
that
this
is
just
a
deployment.
There's
no
custom
go
scripts.
There's
no
dashboard,
there's
no
front
end,
there's
nothing!
It's
literally
just
pure
kubernetes,
so
I'm
hoping
that
I
will
get
this
up
on
github
before
the
end
of
next
week
and
try
and
publish
this
out.
But
it's
very
simple
and
anyone
can
do
it
inside
their
own
clusters.
B
So
we
ended
up
sending
up
a
nice
CI
job
to
try
and
test
all
the
functionality
we
knew
about,
including
the
deployment
and
secrets,
management
and
sub
clusters.
So
I'm
gonna
try
and
talk
you
through
it.
It's
quite
a
bit
that
it
goes
through
each
time.
So
at
the
start,
we
we
destroyed
the
testing
environment
completely.
So
this
is
deleting
the
actual
project
and
cloud
sucks,
so
nothing's
left
so
at
this
stage,
terraform
is
then
creating
the
infrastructure,
which
is
nine
chorus
nodes.
So
three
for
the
Masters.
B
Through
d
and
through
for
the
nodes,
this
makes
up
the
management
cluster.
At
that
stage,
ansible
takes
over
and
actually
deploys
kubernetes
and
during
the
ansible
run,
it's
also
doing
some
health
checks.
So
this
is
actually
in
there
for
the
rolling
upgrade.
So
after
deploying
a
master
component
doing
a
health
check.
Is
it
healthy
before
continuing
on
so
the
same
thing
is
actually
happening
during
the
Jenkins
run.
B
So
at
this
stage
we've
got
a
COO
Bonelli's
cluster
running
and
we
have
a
really
nice
tool
written
by
one
of
our
NGO
developers,
which
is
an
end-to-end
tester.
So
this
is
actually
a
bit
of
go,
which
is
testing
things
like
is
port
forwarding
working
is
cube.
Ttl
proxy
working
can
I
do
pod
pod
networking
all
of
these
kinds
of
things
that
can
be
broken,
but
still
look
like
it's
working
because
up
until
this
stage
we
would
just
do
a
cube.
Ctl
get
RC,
something
like
that
and
see.
B
Oh,
look,
it's
okay
and
just
assume
that
all
the
functionality
is
working,
so
we
haven't
yet
touched
on
it,
but
we
then
actually
run
all
that
I'm
monitoring
checks
manually.
So
that's
part
of
the
pipeline
as
well,
and
at
this
stage
we
can
actually
start
the
point,
the
real
stuff,
which
is
a
sub
clusters.
So
this
involves
creating
another
new
project,
which
is
where
the
nodes
are
going
to
be
hosted.
So
none
of
the
nodes
ever
hosted
in
the
same
network
as
a
management
cluster.
A
B
So
this
stage
we
sort
of
start
again
so
we're
deploying
a
new
sub
costa
kubernetes,
stop
deploying
the
monitoring
again
run.
The
end-to-end
tends
to
stir
again
so
this
is
now
testing
a
kubernetes
inside
of
kubernetes,
including
the
monitoring
once
again
and
I.
Think
that's
that's
pretty
much.
It
takes
around
3035
minutes,
like
Andy,
said:
yep
do
to
terraform
having
to
create
VP,
sees
and
everything
each
time,
but
it's
a
pretty
good
turnaround
for
a
full
stack
test
of
kubernetes
cluster
with
a
actual
sub
cluster
as
well
and
the
monitoring
we.
A
We
triggered
this
on
merge
request,
so
we
use
git,
lab
wealth
and
get
high.
We
previously
used
Enterprise
github
on
premise,
but
we
found
that
they
weren't
creating
the
features
that
we
wanted
and
it
was
an
extortion
of
price
tag,
so
we've
moved
to
get
lab,
and
so
now
we
have
a
single
repository
which
contains
our
terraform
plans.
It
contains
the
ansible
playbooks
it
contains
the
kubernetes
add-ons,
for
example,
and
then
we're
just
running
bash
scripts
on
top
of
this
to
basically
trigger
each
of
these
in
turn
as
a
Jenkins
job
from
gitlab.
B
Yep,
so
one
during
monitoring
was
a
fun
one,
because
there
was
quite
a
few
challenges
that
I
had
never
had
to
do
before
with
monitoring.
So
first
one
was
core
OS,
which
is
you
go
okay?
How
do
I
install
my
monitoring
clients,
and
then
you
realize
you
don't
have
a
package
manager
or
anything
like
that,
so
that
was
the
first
challenge.
B
The
the
second
challenge
was
at
the
time
we
were
still
using
the
cloud
and
in
terraform
setup,
so
any
kind
of
monitoring
wheel
setting
up
had
to
be
reconfigurable
without
destroying
and
recreating
the
cluster,
because
we
weren't
going
to
do
that
just
for
adding
a
new
monitoring
checks
and
then
the
other
new
fund
challenge,
of
course,
is
monitoring
stuff
inside
kubernetes.
The
question
of
what
do
we
actually
want
to
monitor
what
is
responsibility
of
the
infrastructure
monitoring?
B
What's
the
responsibility
of
the
developers
to
monitor
so,
as
you
know,
from
the
sub
busters,
we
like
to
do
kubernetes
inside
of
kubernetes,
so
we're
also
monitoring
kubernetes
from
inside
kubernetes
as
well.
So
it
looks
a
little
bit
something
like
this.
So
we
have
node
exporter
running
on
all
of
the
core.
B
Os
nodes,
so
that's
deployed
with
the
initial
deployment
and
for
those
of
you
not
familiar
with
node
exporter
this,
this
exposes
their
port,
which
has
just
got
a
very
funky
looking
API,
that's
used
by
the
Prometheus
standard
and
that's
in
scraped
by
Prometheus,
which
is
running
from
inside
the
kubernetes
cluster.
So
if
you're
talking
about
management
clusters
and
sub-clusters
inside
the
management,
cluster,
there's
a
Prometheus
and
then
a
sensor
client
for
monitoring
and
then
in
each
of
these
sub
clusters,
there's
also
another
Prometheus
and
sensor
client
which
monitor
the
nodes
of
the
sub
foster.
B
So
at
this
stage,
we've
now
actually
got
basic
sort
of
os
monitoring.
Are
we
running
out
of
inodes
or
the
services
running?
Is
a
load
too
high
that
kind
of
stuff,
and
then
on
top
of
that,
we're
then
using
the
great
kubernetes
sensor,
plugins,
which
are
actually
doing
API
checks,
checking
that
all
the
nodes
are
ready,
checking
that
there
aren't
any
other
errors
going
on
and
that
kind
of
stuff,
and
then
finally,
we
get
onto
actually
monitoring
the
stuff
running
inside
kubernetes.
B
So
what
we
ended
up
coming
up
with
this
is
we're
sort
of
doing
this.
On
a
namespace
level,
our
developers
were
using
different
namespaces
of
different
environments,
and
their
request
was
oh.
This
is
my
production.
Namespace
I
want
you
to
check
all
the
health
checks
inside
this
namespace.
So
what
we
do
is
we're
actually
monitoring
on
namespaces
based
on
labels,
so
we
just
have
a
label
I
think
it's
called
page.
B
This
equals
true
and
a
developer
can
enable
out
on
a
single
pod
on
a
service
on
a
namespace
and
then
send
to
or
actually
pick
that
up
and
start
monitoring
it.
So
at
this
stage
the
the
first
question,
which
normally
happens
is
have
you
heard
of
Prometheus
events,
and
so
that's
kind
of
what
I'm
doing
with
node
exporter
and
there's
a
few
reasons
why
I'm
actually
doing
that
manually
instead?
B
And
one
of
that
is,
if
you
look
at
the
the
nice
dashboard
here,
we
can
actually
see
the
current
state
and
everything
is
green,
and
this
is
I.
Think
more
of
a
personal
choice
where,
with
the
events
in
Prometheus,
is
that
it
will
sort
of
tell
you
when
something
is
wrong
rather
than
telling
you
that
everything
is
OK
and
when
you're
spawning
a
bunch
of
sub-clusters
and
you've
got
a
whole
bunch
of
new
services
being
deployed
by
developers,
you're,
not
sure
whether
everything's,
ok
or
whether
everything's
actually
not
working
and
you're
monitoring
hasn't
said
anything.
A
A
A
lot
of
things
that
we
need
to
kind
of
clean
up.
You
know
it's
always
one
of
those
situations
where
you're
embarrassed
by
your
own
code.
So
you
want
to
make
sure
it's
nice
and
tidy
first,
but
we
will
put
it
out
there.
We
use
send
you
because
we
have
it
already
available
within
our
infrastructure,
but
the
nice
thing
about
Senzu
is:
there
is
a
push
based
monitoring
system,
so
you're
only
sending
JSON
payloads
and
then
sends
you
will
alert
if
that
heartbeats
stop
oops.
A
This
is
quite
nice
because
you
can
actually
put
any
sort
of
other
monitoring
system
you'd
like
in
place
as
long
as
it
can
receive
those
JSON
payloads.
You
can
theoretically
just
reuse
this
kind
of
push
based
monitoring
system
and
it
actually
scales
very
nicely.
We
don't
need
to
configure
anything
on
the
Senzu
side
when
we
bring
up
additional
clusters.
Each
additional
cluster
just
send
this
information
to
send
to
and
then
based
on
that.
We
can
then
actually
page
during
the
night
for
our
engineers,
yeah.
B
Just
to
give
you
a
bit
more
info
on
how
that
check
is
actually
working.
Is
it's
querying
the
Prometheus
API
and
when
you
do
that
for
something
like
uptime,
it
then
returns
a
list
of
all
of
the
hosts,
so
the
check
is
then
actually
going
through
that
list
and
since
it
allows
you
to
send
checks
on
behalf
of
something
else,
so
there's
a
single
sensor,
client
actually
running
all
the
checks.
B
But
when
you
look
inside
the
dashboard,
you
see
more
a
traditional
setup,
which
is
each
of
the
hosts
and
each
of
the
checks,
but
it
also
gives
you
the
ability
to
do
cluster
level
checks.
So
if
you've
got
a,
for
example,
cube
API
service
is
that
available
as
a
cluster
thing,
rather
than
having
that
go
off
three
times?
If
there's
an
issue
for
each
of
the
three
masters,
you
can
just
have
a
centralized
check.
A
Next
steps,
so
we've
we've
intentionally
left
out
some
of
the
stuff
we've
done
otherwise
we'd
be
here
all
day
and
you
guys
would
fall
asleep,
but
some
of
the
things
we're
working
on
now
are
Federation,
so
we're
trying
to
spread
our
clusters
across
different
cloud
providers.
So
we
have
one
POC
running
right
now,
where
we
have
one
cluster
in
AWS,
one
classroom
in
our
private
cloud
and
then
spreading
Federation
across
those
particular
control
planes.
A
B
Networking
yep
this
summer
writing
these
slides
a
few
weeks
ago.
We
are
looking
at
replacing
flannel,
mostly
because
we're
having
a
few
issues
with
it.
There's
a
bug
at
the
moment
where
it
starts
to
forget
routes
and
finally,
and
etcd
is
pretty
fast.
So
when
it
does
forget
these
routes
that
can
find
it
again
and
around
a
second,
but
when
you're
actually
running
the
kubernetes
in
kubernetes,
it
doesn't
really
like
the
latency
too
much,
so
it
was
causing
a
lot
of
weird
issues
and
luckily
a
wee
are
a
great
blog
posts.
B
I
think
that
week,
which
was
talking
about
doing
host
native
networking,
which
is
just
injecting
the
rats
with
ansible
and
luckily
for
us
all
of
our
hosts,
were
connected
layer
to
level.
So
we
gave
that
a
go
and
it
worked
pipeline
was
green.
So,
okay,
let's
move
across
the
native
networking
and
we've
been
writing
that
for
the
last
few
weeks,
with
a
lot
of
success.
A
The
next
thing
is
automated
vault
integration,
I'm,
not
sure
if
any
of
you
saw
the
digital
ocean
talk
about
using
vault
as
a
certificate
authority.
So
we've
been
we've
been
using
vault
for
a
while
to
actually
push
secrets
into
vault
and
then
provide
a
token
to
the
notes.
They
would
then
use
a
bash
script
to
then
request
that
the
secrets
in
terms
of
deploying
stuff
we're
now
moving
to
using
console
templates
or
using
the
vault
sidekick
container.
A
So
the
idea
being
that
we
can
push
secrets
into
volt
and
then
automatically
our
API
servers
will
automatically
watch
that
location
redownload,
the
new
secrets
reload
themselves.
This
would
basically
allow
us
to
then
do
very
short-lived
certificates
and
rotate
our
certificates
every
week.
In
addition,
we're
storing
cube
configs
inside
vault,
which
then
we
can
create
ACLs,
which
then
allows
us
to
then
say:
development
team,
a
authenticates
to
vote
via
a
LDAP,
sorry
Active
Directory.
There
we
go
because
that's
not
currently
supported
by
kubernetes,
so
they
authenticate
to
vote
based
on
their
Active
Directory
groups.
A
A
We're
going
to
continue
to
develop
classic
integrations.
So
as
part
of
this,
we
we
are
having
to
reverse
engineer
a
lot
of
the
GCE
and
the
eight
of
us
integrations
that
are
written
basically
just
writing
that
into
CloudStack.
We're
hoping
to
then
use
this
knowledge
that
we
gained
to
try
and
help
commit
to
the
documentation.
A
A
A
A
A
What
we
do
is
because
you
have
cube
DNS
running
inside
your
management
cluster.
If
you
create
a
service
with
an
endpoint
with
the
public
IP,
then
the
actual
pods
inside
the
management
cluster
will
actually
resolve
the
node
names
as
that
public
IP
and
will
then
connect
to
them.
So
this
allows
things
like
queue,
TTL
log
and
cube
CTL
exec
to
work.
There
are
a
couple
of
assumptions
in
the
code
base
which
assumes
that
the
masters
are
able
to
communicate
over
overlay,
but
I
think.
A
A
So
the
question
was:
have
we
looked
at
uber
Nate?
It's
now
called
Federation.
I
did
briefly
mention
it
where
we're
trying
to
create
multi
cloud
clusters,
sorry
we're
trying
to
create
clusters
that
are
in
various
cloud
and
then
use
Federation
to
spread
across
them.
Yes,
the
current
problem
that
we're
having
is
that
AWS
or
when
you're
doing
multi
clouds,
especially
on-premise
versus
AWS
versus
GCE,
some
of
the
things
like
persistent
volumes,
don't
exist
on
our
on-premise
cloud
stack
and
so
trying
to
deploy
something
on
the
federated
control
plane.
A
It
will
then,
quite
happily
spread
out
across
your
clusters,
which
works
great
for
small
things
and
stateless
services,
but
trying
to
then
deploy
one
thing
on
the
federated
control
plane
that
actually
works
across
all
three
different
types
of
clouds
has
proven
to
be
tricky.
We're
still
working
on
that.