►
From YouTube: Managing and Running Multiple Kubernetes Clusters in Hybrid Setups [I] - Sebastian Scheele
Description
Managing and Running Multiple Kubernetes Clusters in Hybrid Setups [I] - Sebastian Scheele, Loodse & Simon Pearce, SysEleven
As hosting provider, SysEleven, runs and manages multiple Kubernetes clusters for various customers on different platforms. In this talk, we will give you a breakdown on how we run one single Google-like container engine for various clouds and also for bare metal. Moreover, we show how we provide high-availability clusters by running Kubernetes on Kubernetes.
About Simon Pearce
Simon Pearce is a System Architect at SysEleven in Berlin Germany since 2013. He has over 15 years of experience in the web hosting industry. With a focus on building distributed systems on public and private clouds. He is responsible for the kubernetes service team at SysEleven. Working on improving the experience of running multiple kubernetes clusters on a openstack cloud with a quobyte storage cluster.
About Sebastian Scheele
Sebastian Scheele is the CEO and co-founder of Loodse. With Loodse, he wants to empower IT teams to focus on their core: write groundbreaking applications and design the digital future. Sebastian is passionate about the potential of container and cloud native technologies and has been a major contributor to the development of the Open Source projects K8SNIff and Kube-machine. He is a KubeCon EU and Cebit speaker and has published several articles on Kubernetes in leading tech media including The New Stack.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
B
I'm
Simon
Pearce
from
sis
11
from
Berlin
and
atz's
Levin
we're
a
managed
hosting
company
from
Berlin
we've
been
offering
our
customers
managed
hosting
for
the
last
10
years,
and
up
to
now
it's
been
a
fairly
traditional
platform.
We've
been
offering
virtual
Linux
containers,
we
have
about
4000
of
them,
which
the
majority
are
managed
with
puppet
as
configuration
management
and
orchestrated
by
our
own
hauling
over
various
Hardware
nodes,
distributed
over
a
few
data
centers.
B
We
also
have
our
own
OpenStack
cloud,
which
we
run
for
infrastructure
as
a
service,
and
we
basically
wanted
to
extend
that
portfolio
with
kubernetes.
So
we
were
looking
in
a
way.
What
would
be
the
best
way
for
a
medium-sized
German
hosting
provider
to
run
a
managed
communities
offering,
of
course
we're
not
Google
we're
not
AWS,
so
we
needed
to
find
something
else,
and
our
partner
Lutz
are
here,
helped
us
integrate
great
installer
into
our
network.
Yeah.
A
I'm
from
the
Bastion
I'm
Sebastian
I'm
from
Lutzer,
we
are
startup
from
Hamburg
and
we
built
a
platform
how
you
can
run
and
manage.
Multiple
Cuba
needs
cluster,
like
a
Google
container
engine
in
your
own
environment,
no
mistake
on
bare
metal
or
on
different
cloud
providers,
and
together
was
this
11.
We
built
up
the
platform
for
them,
yeah.
B
So
we
would
like
to
take
some
time
to
talk
with
you
about
that
today
and
basically
show
us
how
we
did
that
and
give
you
a
short
live
demo
of
everything
works
out
as
well.
So
you
can
actually
see
how
that
installer
runs
and
how
you
can
basically
set
up
a
communities
cluster
where
the
few
mouse
clicks.
B
So
basically
we
start
off
what
was
the
challenge
for
us
and
also
for
the
looter,
basically
getting
a
management
interface
up
and
running,
which
would
allow
us
to
maintain
and
view
multiple
clusters
we
didn't
want
to
have
to
log
on
to
each
individual
so
to
check
the
update
status
of
the
different
customer
clusters.
We
wanted
to
have
like
a
unified
web
interface,
where
a
system
engineer
can
look
at
and
could
see
the
state
of
the
clusters
we'll
be
able
to
start
an
installation
for
a
customer.
B
If
you
didn't
want
to
do
himself
or
to
see
basically,
the
version
of
kubernetes
we're
just
installed,
maybe
even
add,
or
remove
a
node.
Also,
a
thing
that
was
very
important
to
us
was
multi-tenancy,
which
is
something
which
most
of
the
offerings
that
we've
seen
up
to
today
seem
to
lack.
They
all
seem
to
focus
on
single
projects
or
on
single
companies.
What
we,
our
idea,
was
basically
to
have
this
multi-tenancy
approach,
where
you
could
have
more
than
one
customer
running.
B
And
we've
also
wanted
the
master
components
as
a
complete
managed
service.
So
we
want
to
take
care
of
the
API
server,
the
queue
controller,
the
scheduler
and
also
the
etcd.
We
want
to
take
all
that
hassle
away
from
the
customers,
so
they
don't
have
to
deal
with
those
sort
of
things
and
give
them
the
ability
to
be
able
to
update
that
this
very
quickly
if
they
want
to,
and
also
an
administration
interface
where
you
could
maybe,
as
a
later
date,
change
certificates,
add
users
and
do
various
things
with
the
cluster.
B
And
also
offer
a
choice
of
add-ons
which
could
be
installed.
One
to
be
named
here
would
be
a
CNI,
so
you
could
choose
between,
for
instance,
calico
kernel
flannel,
so
customer
would
be
able
to
have
things
like
network
policies
if
you
needed
it
or
other
people
might
be
more
interested
in
bgp
networking
so
give
the
customer
the
choice
and
allow
him
during
that
installation
to
choose
between
the
different
network
interfaces
which
exist.
B
We
also
have
a
single
service
endpoint,
in
which
we
maintain,
which
has
also
been
a
challenge.
We
have
like
one
IP
address,
which
is
the
API
service
and
that
endpoint
is
for
all
of
the
customers.
So
we
have
multiple
clusters
running
behind
one
service
endpoint,
which
also
allows
us
to
do
a
lot
of
nice
things
with
that
as
well,
and
we
can
also
upgrade
multiple
clusters
as
well.
So
we
are
able
to
view
all
of
the
clusters,
see
them
check
the
upgrade
status
and
maybe
upgrade
them
all
at
once.
Otherwise,
you'd
be
sitting
there.
B
You'd
have
some
form
of
terraform
or
maybe
Anza
ball
playbooks,
and
you
don't
need
to
upgrade
every
individual
cluster
which
will
take
forever
to
be
honest,
and
it
would
also
bring
lots
of
errors
with
it
as
well
trying
to
do
that.
We
tried
it
in
the
past
and
it
didn't
work
very
well.
It
scales
up
to
about
maybe
10
20
clusters,
but
after
that
you
need
so
many
members
of
staff
to
actually
do
unmanaged
all
the
clusters.
It
just
doesn't
work
out.
B
B
B
The
access
to
the
Kas
master
is
also
slightly
different,
because
behind
that
one
IP
address,
we
don't
just
have
one
Kade
as
master
running.
There
are
multiple
masters
running,
so
we
need
to
find
a
way
to
actually
get
that
API
call
to
the
right
master
and
I'll.
Show
you
how
we
did
that
later
on
as
well.
There
various
ways
that
this
can
be
done.
B
And
we
require,
within
our
Staller
a
minimum
of
three
VMs,
because
we're
running
this
on
top
of
OpenStack,
you
can
basically
choose
the
flavor.
Do
you
want
similar
that
you
would
do
when
you
spin
up
a
traditional
VM
with
a
cloud
provider?
You
can
basically
choose
the
flavor.
You
keep
mixed
different
flavors,
but
we
require
that
you
have
a
minimum
of
three
VMs
so
that
you
can
distribute
your
pods
between
the
VMS.
B
Of
course
you
can,
you
can
add
more
anytime,
you
want,
we
can
add
Prometheus,
you
can
check
the
utilization
of
your
cluster
and
you
can
add
new
ones
at
any
time,
and
that
can
also
be
automated
as
well
and,
of
course,
to
accomplish
this.
Additional
tooling
is
required.
One
thing,
maybe
a
specific
proxy
to
get
into
the
correct
cluster.
A
Yeah,
and
so
when
we
started
to
in
Drive
years
ago,
was
customers
working
on
kubernetes,
we
also
Google
Google
partner,
and
when
we
were
on
the
blue
cloud,
it
was
every
time
quite
easy
press
a
button
and
you
have
a
cluster
and
for
other
customers.
It
was
every
time
at
challenge.
Okay,
we
built
it
with
their
tools
and
then
it
starts
every
time.
The
same
same
way.
So
first
we
build
up
a
first-class
stars
and
they
ask
hey:
can
you
help
us
with
updates?
Then
they
come.
We
also
want
to
have
some
more
clusters.
A
Can
you
help
us
building
up
more
classes
to
manage
them
and
yeah?
We
want
to
have
something
like
self-service
and
it
ends
mostly.
It
was
like
yeah.
We
want
to
have
something
like
a
Google
container
engine,
but
on
our
platform,
and
then
we
did
this
a
few
times
for
customers
and
we're
asking
ourself
can't
we
do
this
better.
A
Can't
we
build
something
which
feels
and
works
like
probably
a
Google
container
engine,
and
the
customer
can
run
this
and
managers
in
their
own
environment
on
their
own
platform,
like
old
mistake
per
metal
or
even
on
their
own
cloud
provider,
and
so
what
we
really
want
to
achieve
is
like
providing
a
self-service
for
the
developers
the
developers
can
create
and
manage
their
clusters,
so
a
tea
can
concentrate
on
operating
the
infrastructure,
but
the
developers
can
create.
Clusters
can
decide
on
their
own
when
it's
time
to
upgrade
my
cluster.
How
do
I
want
to
size?
A
My
cluster
and
different
teams
can
have
different
clusters.
Instead
of
putting
everything
in
one
big
cluster,
everyone
can
decide.
Ok
should
I,
have
a
bigger
cluster
or
do
I
want
to
have
smaller
clusters,
also,
probably
building
up
trust
us
in
your
CI
pipeline.
Like
we
have
work
drop
running,
you
want
to
test
something
spin
up
quickly,
a
cluster.
Do
your
jobs
sawed
away
again.
A
Another
challenge
was,
of
course,
updates
of
a
cluster,
so
we
really
want
that
the
developers
can
focusing
updating
the
cluster
and
it's
not
a
big
deal
for
the
operations.
Updating.
Many
of
the
clusters,
as
Simon
said,
if
you
are
have
to
five
classes,
it's
working,
ok
with
tools,
but
if
you
go
into
like
10,
50,
100
and
even
more
clusters,
it's
getting
quite
difficult
to
update
and
especially
also
to
provide
to
your
developers
different
version
of
clusters
so
that
you
can
manage
and
run
those.
A
Of
course,
we
also
want
to
install
like
add-ons
see
CNI,
add-ons
ham,
charts
and
the
dashboard
so
that
the
customer
was
a
developers,
can
immediately
start
and
don't
must
think
about.
How
do
I
integrate
this?
How
do
I
need
additional,
or
what
do
I
must
install
additional
to
the
plain
setup
and
of
course
we
want
to
have
flexibilities
in
our
cluster.
So
we
add,
we
want
to
add
and
remove
worker
notes,
and
the
developer
should
decide
when
it's
time
to
add
the
worker
notes.
A
We
currently
also
working
on
like
cluster
auto-scaling,
that
in
the
future,
we
don't
want
to
rely
on
the
cloud
provider
specific,
auto
scaling.
We
want
to
have
an
integration
of
the
cluster
autoscaler
so
that
we
can
do
auto
scaling
on
any
platform
and
we
don't
must
look
into
it.
How
is
Google
AWS
or
how
is
OpenStack
doing
this?
We
can
then
doing
it
anywhere
yeah,
as
I
said,
auto-scaling
of
worker
notes
we're
currently
working
on
this
to
get
this
done.
A
Of
course,
there
is
a
question:
how
do
I
work
with
external
load
balancers,
so
we
we
integrate.
We
want
to
integrate
the
same
with
networking
with
the
existing
tooling.
Our
focus
is
really
spinning
up
all
this
set
ups
and
yeah.
Let's
a
customer
choose
okay,
what's
the
best
networking
option,
what
load
balancers
do
I
have
in
my
own
cloud
and
how
do
I
can
work
closely
together
with
cuban
eaters?
A
Of
course,
one
important
thing
is
also
automatically
backup
and
recovery
from
the
cuban
ITA's
master,
so
that
we
ensure
that
your
EDD
master
is
every
time
he'll
see
and
in
in
the
case
that
something
happens.
You're
one
of
the
node
crash
or
the
complete
program
is
lost
that
we
can
recover
your
EDD
and
that
you
don't
have
manually
interact
with
it,
so
that
it's
completely
automated
and
the
system
is
completely
running
out
of
the
box
and
you
don't
have
much
to
do
and
what
we
mainly
do
is
we
come
up
with
the
idea?
A
What's
the
best
way
or
how
we
can
run
a
lot
of
cuban.
It
is
cluster
and
we
ask
ourself
can't
we
run
cuban.
It
is
on
cuban
eaters.
So
what
we
spin
up
is
we
have
one
management,
kubernetes
cluster
and
on
this
cuban
it
is
cluster.
We
spin
up
for
each
cluster
in
a
namespace
or
the
computer
component
humanities
needs,
and
then
we
can
connect
from
the
outside
world,
like
from
OpenStack
the
worker
nodes
from
the
specific
cluster,
so
that
it's
connected
and
connected
to
the
specific
cluster.
A
We
only
need
one
dedicated
IP
address
for
all
of
these
clusters
connected
and
yeah
heaven
SSH
tunnel.
That's
a
API
server
can
talk
to
the
Twizy
nodes
and
do
some
stuff
there,
but
that's
our
main
setup.
What
we
technically
build.
It's
like
kubernetes
operator,
which
knows
how
to
deploy
and
to
upgrade
queue,
benitez
cluster,
but
in
that
way
that
we
only
need
this
operator
more
or
less
for
the
for
the
start-up
and
for
the
upgrade
phase
after,
but
you
could
completely
remove
our
operator
out
of
it.
A
The
clusters
are
working
and
the
good
thing
is
because
it's
running
on
Cuban
eaters,
when
something
is
failing
like
when
the
API
servers
crashing
it's
automatically
restarted
and
updates
of
the
master
control
plane
is
for
us
quite
easy.
We
do
a
rolling
update
from
Cuban
Eaters
and
we
are
done
so
yeah
and
this
we
can
easily
then
move
also
to
different
cloud
providers
and
integrating
new
cloud
providers
now
takes
us
between
like
two
to
ten
days,
because
we
only
must
look
into
it.
A
How
we
can
connect
the
worker
nodes
to
it
and
on
the
worker
nodes
we
only
need
like
a
container
runtime.
We
need
the
couplet.
We
must
configure
the
couplet
with
a
token
or
of
the
API
server
and
the
URL
of
the
API
server,
and
then
we
are
done
and
all
of
the
rest.
We
then
roll
out
again
with
kubernetes
itself.
A
So
the
networking
is
rolled
out
with
daemon
sets
and
all
the
rest
also,
and
we
saw
on
the
talk
before
they
had
the
discussion
about
machine
API
and
class,
the
API
we
come
up
with
a
similar
concept.
We
called
it,
we
call
it
node
set.
We
are
also
commonly
in
discussion
with
the
guys
how
we
can
combine
this
both
concepts
together
because
they
are
from
from
the
concept
side
quite
close
together
from
the
technical
implementation
sides.
A
There
are
some
difference,
but
we
want
to
make
it
yeah
why
we
won't
have
an
alignment
and
yeah
in
best
case,
get
it
as
close
to
cube
Anita's
as
possible,
and
the
idea
is
like
really
when
we
started.
We
were
thinking.
Ok,
why
can
I
not
manage
with
kubernetes
nodes?
So
we
come
up
with
the
idea:
hey.
We
need
something
like
nodes.
A
Of
course,
we
are
dealing
on
the
one
side
with
quite
a
lot
of
cloud
providers,
but
on
the
other
side,
also,
enterprise
customers,
who
want
to
run
hybrid
setups
and
their
authorization
and
access
management,
is
every
time
a
challenge.
So
what
we
support
is
different
identity,
identity
provider
like
what
we
mainly
do
versus
all
out
or
add
up.
You
can
use
Google
as
logon
possibility
use.
A
So
what
we
want
to
have
is
like
a
seamless
management
and
cluster
login,
so
really
single
sign-on
for
the
users
that
don't
need
additional
users
on
our
side
and
the
next
step.
We
want
to
also
we
support
Eric
and
network
policies
inside
our
clusters,
but
we
want
to
like
push
it
from
the
outside
provider
inside
to
the
cluster.
This
is
something
where
we're
currently
working
on.
A
Yeah,
and
what
would
you
want
to
F
is
like
we
want
to
provide
multiple
providers,
so
we
want
to
have
the
same
setup
of
the
or
what
we
have
now
is
the
same
setup
of
the
kubernetes
cluster
master
on
every
provider,
because
we
every
time
we
run
queue
bonitas,
we
can
do
this
more
or
less
anywhere
and
what
we
only
need
is
like.
We
must
deploy
a
VM,
we
need
docker
or
a
container
runtime
on
it
and
couplet
and
configures
a
couplet,
and
then
we
are
done
so
the
complex
part
running
and
maintaining
the
cluster.
A
We
can
take
it
every
time
and
put
it
to
the
new
cloud
provider
or
to
a
different
provider.
As
long
as
we
have
a
queue
benitez
running,
we
are
fast
and
easy
to
set
this
up,
and
the
good
Cygnus
is
also
when
you
have
different
setups.
So
in
this
case,
like
OpenStack
and
permettre,
the
same
team
can
operate
all
of
them
because,
like
the
master
control
plan,
works,
I
would
every
time
the
same.
So
you
don't
have
this
challenge.
Oh
now,
I'm
on
this
platform,
I
have
complete
different,
tooling
or
have
a
different
deployment
method.
A
Then,
on
another
platform,
all
of
the
setup
is
more
or
less
the
same.
Like
99%
is
same.
There
are,
of
course,
flavors
which
is
different,
like
load,
balancer
storage
and
how
we
deploy,
especially
on
metal
the
nodes,
but,
like
all
the
communication,
all
the
stuff
is
done
the
same,
and
it's
easy
for
the
team
to
operate
all
this
clusters
and
to
manage
orders.
B
Okay,
so
now
we
would
like
to
talk
a
bit
about
the
hybrid
setup
and
the
way
we
set
all
this
up
and
got
it
running.
Some
of
you
might
think.
Oh,
this
is
kind
of
unusually
these
guys
are
taking
like
bare-metal
servers,
they're
installing
hypervisors
on
our
OpenStack
and
all
this
stuff,
and
on
top
of
that
they're
putting
kubernetes
what
and
on
top
of
that
they're
putting
the
kubernetes
into
communities.
How
does
this
work?
And
is
this
a
good
idea?
B
Yes,
it
is
it's
a
good
idea
because
we
can
use
existing
api's,
which
we
already
have
an
open
stack.
It
allows
us
to
leverage
things
like
storage
as
well
via
the
cinder
api
and
use
images
and
also
scheduling
underneath
which
we
need
with
an
open
stack.
So
it
also
one
thing:
we're
working
on
the
moment
is
our
second
cloud
region.
B
It's
going
to
go
online,
I
think
in
January
or
February
and
next
year,
so
that
will
then
allow
us
to
integrate
to
cloud
regions
within
the
Installer,
and
the
customers
will
be
able
to
use
something
similar
to
federate,
I.
Suppose
you'll
be
able
to
distribute
your
worker
nodes
between
the
two
regions.
B
We
also
have
certain
customers
have
very
specific
demands.
They
don't
want
to
run
on
a
shared
platform.
They
say.
Ok,
OpenStack
and
hypervisors,
with
shared
by
different
customers,
is
not
an
option
for
them,
so
we
can
also
integrate
bare-metal
servers
that
we
have
in
our
data
center.
A
customer
may
rent
three
or
four
bare-metal
servers
or
even
a
whole
rack
of
servers
for
his
own
personal
use,
with
his
own
switching
and
routing
equipment
to
make
sure
it's
completely
isolated
from
other
any
other
customers.
B
We
also
run
two
storage
zones.
We
work
with
a
storage
provider
from
Berlin,
Cork,
whoa
boy,
cool
byte,
and
they
offer
us
a
storage
which
we
utilize
via
the
cinder
API,
which
allows
us
have
two
different
storage
zones
in
both
data
centers,
all
of
the
node
type
SSDs
for
the
storage.
So
it's
very
fast
as
well.
B
Integration
of
additional
data,
centers
or
cloud
providers
as
possible
loads
have
done
a
lot
of
work
on
that
for
other
customers,
they've
already
integrated
digital
ocean
ws
Google,
and
it
would
be
easy
to
integrate
any
further
partners
that
we
may
had,
or
even
maybe,
on
an
on-prem
solution.
If
the
customers
got
a
form
of
API
which
we
can
authenticate
and
some
form
of
provisioning
servers
or
VMs,
it
can
be
used.
So
it's
fairly
easy
to
extend
this
setup
as
well,
which
is
great
for
us
as
well,
because
it
offers
a
lot
of
flexibility
as
well.
B
What
I'd
like
to
show
you
next
is
a
live
demonstration.
So
hopefully
fingers
crossed
is
it's
gonna
work
out
we're
gonna
start
up
a
live
demonstration
on
our
servers
in
Berlin,
so
long
distance
is
hope.
It
works
out.
So
basically,
this
is
the
interface
at
its
current
state.
It's
a
very
simple
interface.
We
did
this
deliberately
so
there's
nothing
not
much
of
not
many
menu
options
here
or
anything.
You
can
basically
what
you
start
off
with.
Is
you
upload
your
SSH
key,
which
I've
already
done
here
to
distribute
on
all
the
worker
nodes?
B
We've
basically
got
the
manage
cluster
and
the
create
cluster
and
we're
gonna,
try
and
create
a
cluster
here.
Oh
cube
Khan
yep,
and
then
we
continue
to
the
provider.
So
you
can
see
here
we
have
our
OpenStack
and
our
bare-metal
service
here.
So
we're
not
going
to
try
provision
a
bare-metal
service.
Akers
will
probably
take
too
long.
But
what
we're
gonna
try
and
do
is
start
up
a
cluster
on
OpenStack,
so
we
select
OpenStack
as
provider.
We
continue
to
then
select
the
data
center.
B
B
So
that's
basically
all
that
we
need
to
do
we'll.
Also
add
my
SSH
key
sorry
pull
it
down
here.
Add
my
SSH
key
and
that's
basically
all
we
do
so
what
we've
got
here,
we're
using
the
all
the
default
settings.
Basically,
I've
got
my
tenant
already
added
I'm,
going
to
spin
up
three
machines
with
Ubuntu
1604
and
afterwards,
hopefully
install
the
cubelet
on
them
and
connect
them
to
the
master.
B
So
we
can
review
the
settings
here,
what
we've
created
and
basically
after
that
will
create
the
cluster,
and
if
everything
works
out,
we
should
see
some
master
components
coming
off.
So,
as
you
can
see
now
on
the
dashboard,
we've
got
these
twirling
symbols
at
the
top,
showing
us
the
health
of
our
master
components.
So
we're
running
a
cube.
Annette
is
one
point,
eight
point
four
here
we
can
also
decide
which
versions
we
want
to
provide
for
our
customers.
B
B
Yes,
we've
got
a
cube
controller
and
a
scheduler,
so
we've
nearly
got
all
of
the
master
components
up
once
they
are
up
and
running.
We
should
see
the
creation
of
our
worker
nodes
to
actually
distribute
our
workload
on.
So
we'll
just
wait
for
a
moment
for
this
to
happen,
and
then
I
can
quickly
show
you
in
the
OpenStack
dashboard
the
games
that
are
created.
B
So
how
many
of
you,
if
I
quickly
ask
question,
were
running
our
demonstration
or
actually
running
kubernetes
in
a
production
environment,
the
moment
Oh
quite
a
few
people,
and
how
many
of
you
have
actually
got
a
working,
C,
icd
pipeline
up
and
running
Oh?
What's
a
quite
a
few
people's?
That
was
a
something
that
was
quite
interesting
because
you
see
lots
of
talks
about
CID
pipelines
at
the
moment
it
seems
to
be
an
issue
at
the
moment
in
the
community.
Okay,
so
we
see
our
nodes
already,
they
are
being
created.
B
B
A
Yeah
I'm:
what
do
you
see
also
on
the
top
is
currently
we
run
the
latest
version,
but
it's
it's
grayed
out,
there's
the
upgrade
button.
So
when
there's
a
new
version
available,
the
developer
can
decide.
Ok
up,
press
the
Update
button
and
then
first
we'll
upgrade
the
master
and
later
upgrade
the
nodes.
And
of
course
the
developer
can
add
or
remove
nodes
depending
on
their
requirements.
So
they
are
completely
flexible.
Can
you
find
ok
what
workloads
I
do
I
run
when
I
want
to
scale
up
when
I
want
to
scale
down?
A
And
yes,
then,
a
plain
vanilla,
Cuban
eaters
with
dashboard
and
all
the
stuff
running
can
use
storage
classes
out-of-the-box
from
old
mistake,
and
it
really
feels
like
for
the
developer
like
a
Google
container
engine.
They
easily
can
also
move
from
there
into
the
setup
and
can
work
that
deploys
application.
There
yeah.
B
So
we
see
our
first
worker
nodes
just
come
up
now,
so
we
see
we've
got
our
green
indicator
here,
so
the
other
one
should
come
up
within
a
few
minutes.
It
normally
takes
about
roughly
about
five
minutes
for
everything
to
come
up,
so
we
can
basically
then
download
the
cube
corn
fig
here.
If
you
want
to
and.
B
A
B
Here
we
can
see
two
of
the
machines
already
ready,
as
you
can
also
see
here
on
the
side
they've
already
signed
up
to
the
master
and
the
others
are
just
about
to
be
registered,
so
we
should
see
them
all
coming
up
now,
yep,
so
we've
got
the
last
two
aren't
quite
ready,
but
that
will
happen
in
a
few
seconds
yeah.
So
that's
basically
it
from
the
demonstration
side.
A
Yeah
one
more
thing:
I
talked
about
already.
Our
cube
node
is
open
source
if
you're
interesting
into
it.
We
are
in
discussion
with
a
machine
API
where
we
want
to
integrate
additional
we're
currently
working
on,
because
what
we
see
from
the
customer,
it's
one
to
get
like
clusters
up
on
the
other
side.
We
must
get
work
load
and
on
it
and
we
were
looking
around
and
also
had
a
discussion
with
Simon.
Okay.
A
What's
the
best
way
to
do
this
and
play
the
road
was
wet,
Jenkins
and
other
tools
and
I
was
saying
we
every
time
want
to
run
native
on
cue
Benitez,
so
we
were
singing
okay
is.
Why
is
there
not
a
native
tool
for
communities
to
run
Cuba
Cuba,
neeta's
workloads,
oversea,
ICD
workloads,
and
we
come
up
with
an
idea,
building
cube
C
I,
which
is
extension
to
drone
CI
and
which
makes
the
once
AI
possible
to
run
on
cue
Benitez.
A
So,
instead
of
connecting
to
the
docker
socket,
it
will
in
the
future
spin-ups
pots
and
also
I,
have
some
plugins
already
developed
for
cube
City
and
for
him.
So
we
want
to
make
a
package
so
that
Simon
can
go
to
their
customers
and
say:
hey
here's
an
easy
pipeline
put
this
in
your
git
repository
and
then
we
get
it
on
our
platform
and
everything
works
natively.
We
can
easily
manage
this
yeah.
Okay,.
B
So,
let's
come
back
to
our
last
few
slides
and
yeah.
Thank
you.
So
basically,
I
want
to
quickly
wrap
this
up
and
talk
to
you
about
some
of
the
lessons
we
learned
and
about
the
future
roadmap.
Where
were
actually
going
and
some
of
our
targets
that
we
have
for
2018.
So
basically,
as
Sebastian
said,
we
need
to
wrap
this
thing
up
with
some
tooling.
We,
the
Cade
as
a
service,
is
not
enough.
People
need
a
workshops.
People
need
the
correct
tooling,
to
enable
them
to
be
able
to
deploy
their
applications.
B
Most
people
don't
or
most
developers.
Don't
really
have
this
great
interest
in
communities.
Most
people
have
the
great
interest
in
their
apps,
which
is
good.
They
want
to
be
able
to
deploy
them
in
a
standardized
way
and
they
don't
want
to
be
having
to
go
through
the
internet
or
read
for
ages
until
they
can
find
out
how
to
do
that.
So
we
want
to
allow
enable
them
to
do
that
with
an
easy
way.
What
we
did
first
is
we
use
the
same
port
for
every
single
API
request.
B
What
we
did
is
we
were
using
sni,
so
we
were
sending
HTTP
headers
to
distinguish
the
different
clusters
that
worked.
It
worked
well
with
like
standard
tooling,
but
we
had
quite
a
few
problems
to
accomplish
there.
For
instance,
things
like
Prometheus.
They
want
to
do
a
service
discovery
within
the
cluster
and
they
don't
use
the
hostname.
They
send
like
a
direct
request
to
the
IP
address,
so
it
wasn't
accessing
the
correct
cluster
and
there
was
also
a
few
other
tools
that
had
problems
with
that.
B
So
we
decided
to
change
this
and
to
give
every
customer
a
unique
port
for
his
API
server,
but
always
used
the
same
IP
address
so
that
we
got
that
unique
service.
Endpoint
to
that
I
mentioned
at
the
beginning
of
the
talk
yeah
one
of
the
things
we
definitely
run
into
when
we
were
first
had
our
first
15
20
clusters
running
I
think
was
a
severe
etcd
problem.
B
We
actually
decided
that
it
would
be
a
good
idea
which
it
wasn't
to
limit
the
resources
the
etcd
can
use
it
crashed
on
us
because
we
had
so
much
data
because
we
were
running
kubernetes
in
these
namespaces.
It
created
a
lot
more
etcd
data
than
we
had
previously
and
it
basically
crushed
the
database
and
we
needed
to
recover
from
it,
which
took
me
quite
a
while.
B
Take
regular
etcd
snapshots
on
persistent
volumes,
always
make
sure
that
you
a
decent
backup
of
all
those
etcd
data.
It
is
definitely
a
lifesaver
if
anything
goes
wrong.
Do
that
for
every
individual
cluster,
but
also
for
the
master
cluster
as
well.
That
was
definitely
also
a
very
important
thing
that
we
found
that
we
need
it
to
do
and
also
restrict
access
to
master
components
may
sound
weird,
but
you
saw
this
also
from
AWS
and
also
Google
won't
allow
you
anywhere
near
the
etcd.
B
It's
normally
a
good
idea
to
keep
your
customers
away
from
these
sort
of
things.
I
mean
give
them
a
button
where
they
can
update
it,
but
don't
allow
them
to
just
update
to
any
version
because
it
would
break
and
we
had
customers
which
actually
decided
they
would
try
an
alpha
version
of
communities
which,
of
course,
didn't
work
correctly
on
our
platform.
So
that's
why
we
make
sure
that
we
are
the
people
that
actually
give
give
out
the
or
hand
out
the
versions
which
can
be
installed
and
last
but
not
least,
is
our
roadmap.
B
One
of
the
things
we're
working
at
the
moment
is
improving
the
cholesterol
identification,
we're
working
on
a
role
based
access
control
system,
similar
to
what
AWS
was
talking
to
in
the
opening
keynote
like
I,
am
sounds
something
interesting,
something
that
we're
gonna
need
to
later
on,
to
make
sure
that
we
can
restrict
developers
to
certain
namespaces
and
also
to
certain
tasks
as
well
within
the
cluster
and
maybe
give
people
read-only
access
and
other
things.
Quite
a
few
of
our
customers
need
these
worker
node.
Auto
scaling
is
also
a
thing
I
mean
due
to
communities.
B
Most
of
you
know
you
have
the
horizontal
pot,
auto
scaling,
which
makes
it
easy
on
a
CPU
load
or
maybe
on
HTTP
requests
to
scale
the
pots,
but
what
you
actually
do
when
you've
reached
the
capacity
of
your
three
worker
nodes.
We
need
to
know
this
and
we
need
to
be
able
to
scale
the
worker
nodes
up
to
a
limit,
so
the
customer
could
maybe
say:
okay
I
want
to
scale
between
3
and
10
worker
nodes.
B
Support
for
different
Linux
distributions
is
also
a
thing
at
the
moment.
Our
cloud
is
running
on
Ubuntu
and
we
are
running
Ubuntu
for
the
work
notes,
but
we
had
quite
a
few
people.
That
said,
why
don't
you
support
Center
ass?
Why
don't
you
support
core
OS?
What
about
all
these
auto
updating
features?
Of
course
we
want
this.
We
don't
want
to
use
Ubuntu.
So
that's
also
something
we're
looking
into
at
the
moment.
B
B
We
have
three
or
four
data
centers
which
we
can
run
on,
and
we
would
like
to
use
an
external
load
balancer
for
this,
so
one
of
the
things
that
can
be
used
as
CloudFlare,
we
basically
modified
and
also
put
it
in
a
recap
or
request
for
external
DNS.
It's
on
the
contract,
repo
of
communities
and
that
allows
us
then,
to
monitor
ingress
resources
and
automatically
update
CloudFlare
with
IP
addresses
of
new
worker
nodes
and
new
ingress
resources.
B
So
a
CloudFlare
actively
does
health
checks
on
the
worker
nodes
and
we
can
also
add
new
ones
and
get
them
in
the
load
balancing
pool
automatically
without
touching
CloudFlare.
So
we
just
use
the
CloudFlare
api
for
that,
and
also
drone
which
Sebastian
just
talked
about
is
we
would
like
a
standard
way
for
CI
CD,
something
which
a
new
customer
may
start
off
with
us
and
he's
looking.
What
what
can
I?
What
can
I
do?
How
can
I
deploy
there's
so
many
different
deployment
tools
around
and
we
think
drone
could
be
a
good
idea.