►
From YouTube: Building a Kubernetes on Bare-Metal Cluster - Alexandros Kosiaris & Guiseppe Lavagetto
Description
Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).
Building a Kubernetes on Bare-Metal Cluster to Serve Wikipedia - Alexandros Kosiaris & Giuseppe Lavagetto, Wikimedia Foundation (Advanced Skill Level)
Starting about a year ago, the Technical Operations team of the Wikimedia Foundation (aka WMF), the organization that runs wikipedia, decided to embark on a journey pushing for streamlined service/microservice delivery. The project is still ongoing. During the build-out of the kubernetes clusters that are going to power the services responsible for Wikipedia, there were a number of interesting decisions that had to be takenr regarding the networking implementation, the routing of external traffic, the chosen bootstrap method for the clusters and the building methodology for containers and last but not least the transition plan from the old style services to the kubernetes style services. This talk is about sharing these implementation decisions with a wider public
About Alexandros
A Linux sysadmin, turned FreeBSD sysadmin, turned Linux sysadmin, turned systems engineer (somewhere along that path there's a Devops hat as well), Alexandros has been in the space since 1999, starting as a hobbyist, then a professional. Currently working with the Wikimedia Foundation, he has pushed forward for more virtualization and better orchestrated microservices and environments for their execution. The kubernetes project is a current passion.
About Giuseppe
I am a Site Reliability Engineer for your favourite free encycolopedia. I love to model and improve large, distributed systems, and to abstract complexity away from the individual developers, and my work from the individual hardware parts it relies on. Kubernetes seems like the natural environment where to make that happen.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
Join us for KubeCon + CloudNativeCon in San Diego November 18 - 21. Learn more at https://bit.ly/2XTN3ho. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Okay,
yeah,
okay,
I
can
speak
hello.
You're
Alexandra
play
we're
going
to
talk
about
our
experiences
in
building
kubernetes
cluster
on
bare
metal
hardware
for
Wikipedia.
So
first
some
information
about
who
we
are
I,
think
most
of
you
are
familiar
with
our
main
project,
which
is
Wikipedia
leukemia
foundation
is
a
nonprofit
organization
that
works
on
maintaining
the
infrastructure
for
Wikipedia,
but
not
just
Wikipedia
our
sister
projects
like
wiki
data,
or
we
travel
with
users
and
so
on,
not
with
leaks
as
you
can
see.
A
We
have
half
the
amount
of
traffic
monthly,
so
we
are
one
of
at
top
10
websites.
As
far
as
traffic
is
concerned,
still
we
have
our
infrastructure
as
two
primary
data
centers,
where
the
application
layer
stays
and
persistence,
layer
stays,
and
then
we
have
free
caching
data
centers,
one
in
San
Francisco,
one
in
Amsterdam
and
a
new
one
in
Singapore,
but
are
basically
just
keeping
the
caches
of
a
content
so
that
like
we
can
serve
a
content
nearer
to
a
user.
A
So
it's
kind
of
our
own
home-baked
CDN,
even
with
the
amount
of
traffic
that
we
have.
Our
cluster
is
relatively
small.
We
just
have
1,200
hardware
machines,
we
don't
typically
use
VMs
in
production,
just
lately
we're
using
a
couple
of
vanity
clusters.
What
was
that
about
hundred
VMs
and
we're
not
just
lean
in
terms
of
hardware,
we're
also
lean
in
terms
of
the
size
of
the
organization,
as
you
can
see,
around
160
ingenious
overall
and
the
people
that
care
about
the
application
layer,
which
is
what
we're
moving
to
kubernetes,
is
just
four
people.
A
So
why
we're
moving
to
kubernetes?
Just
a
few
words
about
this
and
up
until
2014,
we
just
had
the
big
monolithic
application,
which
is
MediaWiki
and
one
service
which
is
called
par
soil.
That
was
the
helping
with
our
visual
editor
fast
forward.
Four
years
to
today,
we
have
much
more
services.
We
are
moving
somehow
towards
a
micro,
service-oriented
architecture
slowly,
but
even
with
slow
motion.
A
The
toyline
on
our
team,
which
is
very
small,
is
increasing,
increasing
and
we
need
a
way
both
for
us
to
work
faster
and
for
the
developers
to
be
able
to
deploy
our
services
better
and
with
less
weight
on
our
time.
But
these
are
not
the
only
reasons
to
move
to
kubernetes.
One
big
reason
for
us
is
elasticity,
so
V
ability
to
scale
up
and
scale
down
each
function
that
we
have
in
a
short
amount
of
time
after
traffic
bursts
and
also
well.
A
And
finally,
we
are
giving
the
same
developers
with
people
that
deploy
code
on
the
cluster
more
control
over
their
applications,
because
the
configurations
about
realisations
is
not
anymore
in
puppet
which
is
controlled
by
us.
So
they
can
have
much
more
flexibility
about
what
they
do,
but
we
also
see
also
saw
some
challenges,
so
the
first
one
kubernetes
is
complex
as
we've
just
seen
in
the
previous
talk
right.
Just
the
QoS
is
very
complex
and
everything
else
is
complex
and
there
are
a
lot
of
moving
parts.
A
We
didn't
want
to
sacrifice
the
best
ability
that
we
have
just
to
get
flexibility.
Also,
it's
a
new
paradigm.
Anyone
has
heard
cloud
native
in
last
few
days,
yet
it's
a
new
paradigm
and
it's
a
good
thing,
but
it's
also
a
problem
for
all
the
tooling
that
we
have
around
our
cluster,
which
is
based
on
butt
concept
of
having
physical
servers.
A
We've
services
installed
on
them
and,
finally,
containers
ourselves
are
also
a
challenge
for
us,
because
we
used
to
be
very
efficient
in
doing
security
upgrades
and
everybody
that
has
dealt
with
containers
knows
that's
that's
kind
of
almost
a
salt
problem
on
containers.
So
why
we're
doing
things
on
bare
metal?
Why
are
we
at
the
cloud
native
conference
talking
about
standing
things
of
them
they're
made
aware?
A
Well,
there
are
some
reasons
why
we
can't
use
a
public
cloud
first
and
foremost,
is
that
we
value
the
privacy
of
our
users
very
much
and
the
contract
that
we
have
with
our
users
in
terms
of
their
privacy,
and
we
feel
that
if
we
are
the
third
party
actor
in
the
middle,
we
have
to
have
to
trust
that
for
party
fully
in
order
to
guarantee
gave
the
use
of
the
same
guarantees-
and
that's
not
gonna
happen
with
public
cloud.
Also,
we
already
have
our
infrastructure,
you
already
maintaining
it.
So
it's
not
like.
A
We
have
an
upscale
cost
for
just
maintaining
a
bare
cluster,
a
bare
metal
cluster
and
finally
costs.
As
I
said
in
the
first
slide
right,
we
have
quite
some
traffic
what
quite
some
amount
of
traffic.
Think
of
the
Bill
of
just
taking
all
of
us
by
it's
out
of
cloud,
what
what
wet
would
be,
and
we
don't
really
have
money
being
a
non-profit
and
relying
on
donations
of
individuals.
A
A
We
felt
it's,
it's
not
really
something
we
want,
because
you
want
to
reduce
the
number
of
moving
parts
right
and
OpenStack
is
actually
more
moving
parts
when
kubernetes,
so
we
didn't
receive
a
reason
to
do
that,
because
the
elasticity
we
need
is
already
given
us
by
Kuber
needs
itself.
Since
we
want
to
move
all
the
application
layer
to
kubernetes,
we
have
a
set
of
machines
that
are
dedicated
to
that
and
we
just
want
to
switch
resources
between
one
or
and
the
other,
and
so
we
don't
need
the
additional.
A
B
Everyone
nice
to
be
here
so
talking
about
how
our
clusters
are
set
up.
We
have
a
very
Debian
developer
populated
as
a
reteam.
There
are
a
lot
of
those
people
there
and
that's
why
we
decided
to
go,
of
course,
with
the
BIM
packages
way.
So
we
package
our
own
debian
versions
of
communities
and,
very
very
recently,
in
this
conference
we
found
out
that
Google
is
also
providing
tabs
for
people.
So
thank
you
for
that.
B
B
Why
are
you
not
on
3
point
X
and
the
answer
is
yes,
we
need
again
to
grade
we're
packaging,
all
these
software
on
our
own,
and
we
need
to
maintain
it.
Our
clusters,
well
we're
a
very
big
pet
shop.
If
you
go
to.
If
you
want
to
go
to
our
Gerrit
or
you
go
to
github,
where
we
have
a
mirror
of
our
Gerrit,
you
can
see
our
pipette
repo.
It's
I
think
the
biggest
open
source
puppet
repo.
There
is
out
there,
even
though
it
doesn't
have
a
license,
but
don't
please
judge
us
on
that.
B
So
we
configure
everything
via
a
puppet,
so
we
configured
all
of
the
community's
components
by
puppet
as
well,
and
that
had
one
very
nice
thing
it
has
our
list.
It
allowed
us
to
have
Els
for
all
the
components
everywhere
from
the
go
just
by
reusing
the
puppet,
see
a
and
also
showed
a
couple
of
problems
with
a
puppet
ca
in
the
process.
We
would
like
at
some
points
all
these
or
migrate
away
from
it.
We
will
see
what
we
do
not
maintain
by
a
puppet,
even
though
we
pondered
the
idea
was
the
community's
resources.
B
There
is
even
a
puppet
module
in
puppet
Forge
that
allows
you
to
do
that,
but
on
thinking
about
it,
we
realize
that
maintaining
such
a
puppet
module
and
making
generic
and
nos
and
all
that
and
publishing
it
in
puppet
Forge
would
be
a
pretty
big
toriel
and
then
we
even
realized
later
on.
It
would
not
make
sense
for
our
case
because
we
end
up
maintaining
most
of
the
resources,
if
not
all,
of
them
via
helm.
At
the
end,
a
little
bit
more
about
how
those
processors
are
set
up.
B
True,
and
the
reason
for
that
is
that
it
allows
us
to
produce
two
components
to
be
on
the
same
node
as
the
key
API
server
and
talk
to
it
on
the
unfit,
ik,
aided
port
and
just
rely
on
firewalling
in
order
to
protect
our
API.
That
allowed
us
to
avoid
for
now
authentication
and
authorization
that
we
would
need
to
do
for
those
two
components
of
communities
to
talk
to
the
API
server
a
little
bit
more
about
those
two
production
clusters.
B
They
are
currently
three
well,
one
is
not
really
production,
it's
the
staging
cluster,
which
we
are
now
binding
to
our
built-in
deployment
pipeline
and
we
have
one
primary
DC.
We
were
talking
about
the
primary
basis.
Previously
we
have
separated
these
two
clusters
for
every
one
of
those
kubernetes
clusters
and,
of
course,
we
use
a
basic
idea
of
three
in
order
to
maintain
quorum.
They
are,
of
course,
DC
local
in
order
to
maintain
level
of
latency
and
funnily
enough.
Those
are
on
VMs
from
Ghana
DBMS
gannett
is
a
mature,
virtualization
orchestration
framework
by
Google.
B
Thank
you,
google,
for
that
as
well.
You
can
expect
what
we
run
q
proxy
in
the
default
way,
which
is
a
tables
mode,
but
we
are
evaluating
at
PBS.
We
already
have
very
heavy
lvsd
our
shop,
and
so
this
is
actually
interesting
in
with
for
us,
and
we
do
have
expertise
already
in
this
and
we
host
our
own
registry
and
we
enforce
it.
B
You
cannot
guarantee
what
you're
running
and,
most
importantly,
if
and
if
you
can
trust
the
person
doing
it,
has
he
updated
hash,
he
updated
the
specific
image
that
you
are
running,
so
what
we
do
is
we
run
all
lowering
we
build
and
run
all
our
images
from
scratch,
and
you
can
even
go
around
check
out
our
images.
Your
docket
Pokemon
is,
there
feel
free
to
see
what
kind
of
images
we
have
there,
of
course
very
specific
to
our
use
cases,
but
your
feel
free.
B
Of
course,
they
are
well
they're
being
based,
as
I
said,
we're
Debian
shop,
the
backend
for
this
specific
docker
registry,
which
uses
the
reference
docker
registry
from
docker
is
OpenStack
Swift
and
it
works
reliably.
Well,
remarkably,
well,
we
went
with
our
back
rule
based
access
control
from
day
one
and
the
reason
for
that
is
very
simply
we're
not
cloud.
We
cannot
have
our
comedians
classes
in
the
cloud,
which
means
that
we
cannot
assign
a
kabillion,
this
class
super
team
or
/
service
or
whatever.
B
B
For
those
of
you
who
do
not
know
what
Conway's
law,
it
says,
basically,
that
the
organizational
structure
or
the
structure
for
your
organization
is
going
to
show
up
in
your
products
and
what
has
happened
in
the
past
in
organization
is
that
there
have
been
a
lot
of
changes
and
we
were
kind
enough
locks
in
the
last
few
years.
So
we
decided
instead
of
going
to
team
centric
approach
to
go
for
a
product
centric
approach
where
product
is,
of
course,
the
service
authentication
wise.
We
use
parently
token
for
our
file.
B
We
have
evaluated
all
the
various
authentication
modules.
There
is
actually
a
fabricator
task
and
our
fabricator
it's
fabricated
at
Wikimedia,
dad
org
so
say
if
it
were.
If
you
want
it's
written
by
yours
truly
and
it
valuates
more
each
one
of
those
vindication
methods,
and
why
on
we
ended
up
with
toka
North,
which
is
a
little
bit
difficult
to
maintain.
But
we
would
like
at
some
point
to
go
to
the
web
hook,
authentication
mode,
but
we
first
need
to
run
a
service
that
actually
allows
you
to
off
against
the
emission
controls.
B
Is
we
have
all
the
standards
that
are
the
standard
mission
controllers
that
we
run
for
keep
the
RQ
news
version?
We
have
not
deviated
from
that.
What
we
have
done
in,
however,
is
that
we
have
firm,
which
is
a
wrapper
around
IP
six
tables
and
IP
tables
that
makes
managing
rules.
Why
are
these
tools
way
way
way
more
easier?
So
we
have
this
across
the
entire
fleet.
There
is
not
a
box
without
well
there's,
not
a
box.
B
That
cannot
be
absolutely
true
but
anyway,
so
where
most
of
the
boxes
have
firm
there
and
they
have
default
policy
of
drop,
and
that
specific
thing
also
is
applied
on
the
kubernetes
nodes
and,
of
course,
the
API
servers.
We
were
kind
of
worried
about
this,
because
we
also
applied
firm
to
our
open,
stack
boxes
and
well.
There
are
race
conditions,
you
reload,
firm
and
flashes.
B
These
are
for
rock
rows
F,
as
you
can
see
we're
in
one
of
those
adding
a
fifth
one,
and
every
rock
row
is
comprised
of
eight
tracks
and
every
Rock
low
has
one
tour
top
of
the
rock
switch
and
they
are
stacked
into
being
a
single
single
managed
switch,
which
means
that,
basically,
if
you
reboot
that
specific
switch,
the
entire
rack
row,
which
is
32
racks,
is
going
to
go
down
for
the
duration
of
the
reboot,
which,
by
the
way,
jr.
predicts,
does
take
a
little
bit
of
time.
B
In
that
big
switches,
it's
a
little
bit
of
a
collapsed
cloth
topology.
If
you
want
to
be
not
very
pedantic
about
it
in
the
form
that
we
have
those
two
effects
switches
without
which
are
basically
10g
port
switches
that
are
the
spines
and
there
are
the
next
witches
that
are
the
the
Leafs.
The
thing
to
note
in
this
specific
diagram
is
the
fact
that
if
you
have
a
box
in
one
rack
row,
we
know
de
to
reach
any
other
box
in
any
other
row.
You
have
to
go
through
the
routers.
B
B
So
we
have
currently
the
clusters
are
rather
small,
therefore
machines,
and
so
we
have
one
per
row,
but
we
plan
to
increase
these
to
12
very
soon,
and
we
had
this
requirement
since
the
beginning
that
the
kubernetes
clusters
needed
to
be
fully
compatible
with
the
legacy
infrastructure,
we're
legacies,
what
we
currently
have,
which
meant
that
we
could
not
do
not
know
network
and
its
relation
every
anywhere.
We
started
looking
at
the
various
solutions
and
two
years
ago,
at
FOSDEM
in
2016
there
was
a
calico
talk,
we
looked
at
it
and
we
said
whoa.
B
B
So
what
we
have
done,
we
have
it
configured
again
by
a
puppet
well,
partly
by
a
puppet
part
in
manual,
because
there
are
two
things
that
we
do
not
currently
do
by
a
puppet
in
calico
and
those
are
the
IP
pools
and
the
BGP
peers,
the
routers.
Basically,
and
it's
working
quite
fine
and
we're
probably
happy
with
it.
B
Not
nothing
is
that's
the
same
IP
space,
but
a
different
different,
slash
24
for
the
service
IPS,
and
this
is
the
part
where
you'll
say,
service
type.
These
are
not
really
route
or
bullion
cabinet.
Is
there
they're
not
really
a
piece
to
start
with?
They
are
tags.
Basically,
you
say
the
pod
just
needs
to
have
an
IP
for
to
talk
to,
so
those
are
effectively
just
reservations.
B
We
just
reserved
IP
spaces
because
we
do
not
want
to
have
192
168
IP
is
in
our
network
because
whenever
we
met
those
for
some
reason
it
was
due
to
a
either
a
miss
configuration
something
very
confusing
going
on,
so
it
was
just
maintaining
the
status
quo
and
one
thing
that
calico
has
allowed
us
to
do
and
was
very,
very
nice
and
thank
you
calico
for
that
is.
We
now
have,
even
though
communities
does
not
support
ipv6,
or
at
least
in
our
version.
B
Our
paths
are
ipv6
enabled,
and
in
order
to
do
that,
it
was
very
easy
to
do
that
in
the
ipam
for
colic
or
just
the
sign,
maybe
before
true
a
sign
of
pv6
true,
and
then
we
had
to
go
to
every
node
and
set
those
to
see
CTL
configurations
that
you
see
there
now
for
those
that
I.
Perhaps
you
do
ipv4
forwarding
and
haven't
done
ipv6
forwarding
on
Linux.
That
thing
does
not
do
what
it
does
in
ipv4
go
read
the
darks.
They
are
completely
different.
They
do
different
things.
B
So,
for
those
who
don't
know
one
point:
seven
does
not
support
egress.
One
point:
eight
does
so
we
had
to
work
around
that
we're
hoping
to
drop
that
workaround
in
the
future.
In
one
point,
eight
calico
does
support
us
and
we
asks
how
you
implemented
a
workaround,
and
the
second
thing
that
we
met
was
the
fact
that
one
point
seven
does
not
allow
you
to
change
a
network
policy.
B
You
have
to
bring
down
and
reinstall
it
in
order
to
to
change
the
policy,
and
we
had
to
do
that
via
hell,
the
authority
that
we
have
actually
implemented.
Is
we
patch
the
calculate
s
controller?
The
version
was
zero
point,
six
zero
and
it's
a
Python
one
and
it's
no
longer
there
because
it
was
rewritten
in
go
so
our
solution.
Well,
it's
not
needed
anymore.
Thankfully,
with
one
point
eight,
we
don't
even
need
that
it's
what's
really
really
small.
We
haven't
default
ingress
policy
and
our
patch
is
like
very
minimal
fourteen
lines
of
code.
B
What
about
ingress,
we
evaluated
back
in
2017
and
said
nope,
not
yet
not
ready,
not
for
us.
So
how
do
we
route
traffic
to
our
classes?
You
asked
that's
a
pretty
valid
question,
so
what
we
do
is
we
use
that
Python
in-house
built
daemon
that
we
have
been
having
for
like
what
it's
now
eight
years.
Maybe
it's
called
PI
Bo
and
there
is
a
URL
for
you
and
github.
If
you
want
to
go
and
take
a
look,
what
it
does
is
it
monitors
lvsd
our
entries
in
the
kernel
lvsd
are
obvious
direct
routing.
B
What
it
basically
does
is
when
the
packet
reaches
the
load
balancer.
The
only
thing
that
it
does
it
through
the
IP
packet
is
change.
The
Mac
it
doesn't
alter
in
any
way
the
IPP
headers
or
anything
like
that,
so
it's
very,
very
lightweight
and
the
traffic
goes
directly
to
the
backend.
That
means,
however,
that
the
backend
needs
to
have
that
IP
set
up
in
its
interfaces.
Otherwise
it's
not
going
to
accept
the
traffic.
B
The
good
thing
is
that
it
does
not
have
to
reply
back
to
the
load,
balancer
and
then
load
balancer
does
not
have
to
reply
to
the
client.
Outgoing
traffic
leaves
all
the
backends
and
goes
straight
back
to
the
client,
instead
of
going
through
your
load
balancers,
which
is
good
in
our
case,
because
we
have
minimal
into
traffic
bits,
basically
get
slash
and
article,
but
we
have
a
lot
of
outgoing
traffic
from
those
boxes.
B
We
had
a
lot
of
expertise
over
this
software
and
we
decided
to
reuse
it,
and
we
know
from
a
blog
post
from
github.
They
have
more
or
less
a
similar
approach
a
little
bit
bit
different,
but
we're
happy
where
at
least
we're
not
alone
in
this
and
we're
using
node
port
in
order
to
with
external
IDs
that
change
by
the
way
in
Bernie's
1.65.
Member
correctly,
it
used
to
be
different
in
the
API
that
caught
us
off
surprise
for
a
little
bit
then
figure
out
that
this
is
actually
a
better
way
about
metrics
collection.
B
We
actually
connect
metrics.
Thank
you
from
easiest
for
that.
It
works
quite
well,
and
the
only
thing
that
we
did
was
have
pop-eyed
populate
the
configuration.
So
what
we
do
have
is
that
Prometheus
discovers
the
API
and
pause
the
couplets,
the
couplets
eadvisor
it
all
so
Paul's
the
the
couplet
API
itself
and
we've
met
a
bug
in
1.73,
where
the
API
it's
a
little
bit
was
a
little
bit
different.
B
The
interesting
thing
with
that
was
the
Prometheus
community
already
documented,
is,
in
the
example
configuration
that
was
great
great.
However,
applications
are
a
little
bit
more
interesting,
so
we
also
use
Fermi
fix
for
that
too.
What
we
did
is
all
other
implications.
You
surgically
use
stats
D,
so
we
have
stats
the
exporter.
That's
a
sidecar
container
to
all
our
parts.
B
Applications
talk
to
localhost
and
push
all
the
stats
team,
metrics
and
then
Prometheus
discovers
all
the
pods
and
Paul's
all
the
pods
and
gathers
the
information,
and
we
we
visualize
all
of
that
through
grow
fauna
and
here's
a
link
if
you
want
to
get
our
graph
on
our
installation,
big
kudos
to
the
graph
on
a
team.
This
is
an
excellent
software.
Thank
you
very
much.
Rather
on
pretty
sure
you
all
know
it.
It's
just
a
shout
out.
B
Alerting
is
not
that
great
currently-
and
this
is
mostly
for
historical
reasons.
We
use
Prometheus
again,
but
partly
because
we're
still
on
I
shall
I
sing
a
sharp
eye
single.
You
know,
for
those
of
you
do
not
know
it's
I'm,
all
software
from
the
90s,
it's
very
node
centric,
not
service
centric.
We
won
on
my
grade
off
of
it.
B
We
have
started
by
integrating
check,
Prometheus
metric
script
in
bash
and,
of
course,
we
had
problems
with
it,
because
it's
not
support
floats
and
we
and
various
other
small
corner
cases,
and
then
we
ended
up
rewriting
it
in
Python.
And
if
you
add
in
puppet,
which
is
how
you
manage
all
of
our
alerts,
you
can
get
that
very,
very
weird-looking
alert
like
what
does
this
thing
even
do
we're
fixing
this
one
of
my
colleagues.
B
Actually,
while
we
were
here
submitted
patches
to
fix
this
and
all
of
the
alerts
themselves
are
now
in
the
Prometheus
engine,
and
we
just
all
the
very
specific
name
alert
for
that.
The
last
thing
that
we
did
is
that
we
decided
to
go
forward
with
a
different
way
of
monitoring
applications
that
what
a
singer
did,
because
basically
applications
are
a
little
bit
difficult
to
monitor
when
it
comes
to
api's,
because
each
of
them,
one
of
the
eight
points,
can
fail
in
a
myriad
ways
and
all
of
the
other
points
may
not
fail.
B
So
it's
difficult
to
go
around
and
create
a
software
that
monitors
all
of
these
api
is
and
have
a
die
single
compatible
and
all
that.
So
what
we
did
is
we
extended
our
swagger
specs
for
all
these
applications,
with
a
very
specific
expect
that
allowed
us
to
create
an
applicator
that
would
go
around
walk
all
the
API
and
the
endpoints
and
submit
a
a
test
query
to
that
specific
endpoint
and
that's
where
I
pass
the
ball
again.
Gizelle.
A
Okay,
so
we
call
the
project
stream
nine
service
delivery,
because
the
main
point
for
us
was
that
kubernetes
allows
you
to
change
the
way
you
manage
this
lifecycle
of
software
and,
specifically
micro
service
from
development
to
production.
There
is
this
big
scheme
of
what
we
did,
that
which
is
pretty
detailed.
I
won't
get
into
that
or
let
you
leave
at
6
o'clock
I'm
just
going
to
concentrate
on
the
details
of
deploy
stage.
If
you
want
to
say
the
details
of
everything
where
it's
also
all
documented
on
wiki.
A
So
if
you
just
search
with
enough
tenacity,
you
will
find
all
the
documentation
about
this.
Let's
just
concentrate
on
deployments.
So,
as
we
said,
we
have
one
namespace
per
service
basically,
and
we
will
have
several
people
which
should
be
allowed
to
the
product
of
that
service.
But
we
want
people
to
be
extracted
from
the
problem
of
knowing
the
details
of
our
kubernetes
works.
So
we
use
Elm,
which
is
working
very
well
for
us
I.
Will
it
did
we
set
up
element
production
so
that
we
could
secure
registration
as
much
as
possible?
A
What
we
did
is
that
we
have
one
Taylor
instance
running
in
each
namespace
with
specific
Arabic
rules
which
are
basically
what
they
what
is
suggested
in
guide
from
tiller.
Apart
from
the
fact
that
we
also
need
to
set
up
a
networking
rules
for
egress,
because
we
basically
white
least
the
ability
of
a
pod
to
call
our
pods
to
make
connections
to
our
pods
and
when
we
have
a
deploy
user,
which
is
just
allowed
by
our
buck
to
speak
with
tiller.
A
Basically,
so
we
only
think
that
somebody
that
has
we
credentials
for
way
the
pro
users
can
do
is
use
them.
Basically,
on
the
cluster,
we
can't
use
cube
Caudill,
so
we
restricted
the
user,
coop
kado
to
administrators
and
normal
the
players
just
have
to
use
them.
We
don't
have
to
care
about
when
treatises
wait
can
basically
shoot
themselves
in
the
foot
with
avocado.
A
A
This
rapper
does
some
more
things.
We
said
we
have
to
make
primary
data
centers
most
things
that
we
deploy
all
things
that
we
deploy
on
kubernetes.
At
the
moment
are
active
active
between
two
data
centers,
so
we
want
release
to
go
to
both
data
centers
and
this
rapper
does
that
and
also
we
are
working
at
the
moment
we
hope
to
be
finished
before
the
conference,
but
we're
not
on
doing
cannery
releases.
The
way
we
do
it
is
pretty
simple:
we
just
have
we
cheat.
A
Basically,
we
have
two
LM
releases
for
each
practical
degrees,
a
canary
release
with
a
small
set
of
replicas
and
when
production
release,
but
as
larger
set
of
replicas
and
the
service
definition,
but
the
selectors
to
select
all
the
pots
from
both
releases.
Whenever
we
do
a
new
release,
we
first
really
do
Alan
first
upgrades
the
canary
release.
We
check
the
metrics
and
vlogs
to
see
if
everything's
green,
we
go
on
and
approach
of
a
production
part.
A
Finally,
as
I
said,
it's
a
shameful
bash
script
at
the
moment.
What
we
plan
to
do
is
to
try
to
rewrite
everything,
including
this
part
as
an
plugins,
so
that
we
can
basically
give
service
with
a
rest
of
the
community.
Finally,
we
have
our
very
own
and
very
new
elm
repo,
which
at
the
moment,
that's
just
one
chart
which
is
a
public
repo,
so
that
you
anybody
can
run
the
services
which
we
run
in
production
using
our
and
charts
in
the
future
and
where's
the
link.
A
And
finally,
since
we're
almost
done
and
I
think
everybody's
very
tired
at
this
point
of
a
conference,
I
want
to
thank
you
for
sticking
with
us
until
five
o'clock
and
a
shameful
plug.
As
I
said,
our
team
is
too
small
and
we
are
hiring
at
the
moment
and
I'm
living
this
slide,
because
you
have
some
links
to
the
Fayetteville
resources
that
you
can
consult.
A
Remember
we're
a
compounder
percent
free
software
shop,
so
everything
we've
talked
about
and
everything
we
do
is
publicly
available
as
free
software,
and
you
can
take
a
look
and
even
contribute
if
you
want
to
so.
That
concludes
our
talk
with
a
couple
of
minutes
available
and
if
somebody
has
questions
we
are
here
to
answer
them.
A
C
A
So
the
question
is
about
why
we
evaluated
rocket
I.
Think
rocket
has
some
interesting
characteristics,
the
most
interesting
of
which,
from
our
perspective,
is
the
ability
to
use
the
TPM
to
verify
the
images
and
to
check
consistency.
We're
very
concerned
with
security.
Remember
our
site
is
a
site
that
accepts
arbitrary
input,
both
in
text
and
binary
form,
because
Wikimedia
Commons
accept
binary
forms
as
well.
A
So
we
have
to
be
very,
very
careful
with
security,
and
that
was
one
thing
that
seemed
appealing:
I,
don't
think
that
the
support
for
rocket
was
there
when
we
decided
to
deploy
communities
in
production
into
kubernetes,
so
we're
stick
with
sticking
with
docker
for
now,
and
we
have
quite
some
tooling
around
docker.
We
would
have
to
adapt
that
rocket
whenever
we
want
to
do
that.