►
Description
Scaling Microservices Beyond a Single Cluster w/ Kubernetes - Dan Wilson, Concur
All things fail, including clustered technologies that are designed for failure. Learn how Concur uses external load balancing and the existing k8s tools (pre-ubernetes) to provide cluster failure tolerance for our services in a fully automated way.
About Dan Wilson
I have a passion for collaborating across the business and using data to drive decision making, microservice architecture, api first, performance optimization, cloud service architecture, scale out technologies, cost reduction, design for failure and open source technologies.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
My
name
is
Dan
Wilson,
thanks
for
for
being
here.
This
isn't
about
me.
It's
about
you,
so
I'd
like
to
know
a
little
bit
about
the
audience.
If
that's
okay,
how
many
folks
here
are
using
kubernetes,
already
nice,
nice
quite
a
bit
and
how
many
are
new
to
goober,
Nettie's,
okay,
so
a
pretty
even
split
there.
A
It
looks
like
and
then
how
many
are
developers
very
good
good
group
of
developers,
so
any
ops
here,
a
little
bit
some
QA
or
program
management,
scrum,
master,
okay,
so
I've
already
failed
because
there's
no
way
I
can
please
everybody.
There's
like
kind
of
50/50
split
between
all
those
things,
so
I'll
do
my
best
to
cover
content,
that's
appropriate
for
each
of
you,
but,
like
I
said
it's
gonna
be
tough
but
I'm
glad
to
see
a
high
amount
of
developers.
So
that
means
I'll
focus
a
little
bit
more
on
some
of
the
code
pieces.
A
So
I
concur.
This
is
a
quick
agenda.
I,
don't
think
it's
much
there,
but
if
you
don't
know
about
concur,
we
make
travel
and
spend
management
easier
for
businesses,
and
we
have
quite
a
few
tools
that
do
this.
There's
our
concur
mobile
app,
there's
TripIt.
Recently
we
have
Hipmunk
as
well
as
one
of
the
pieces
of
concur
technology.
The
whole
focus,
though,
is
on
our
end
users,
the
folks
actually
traveling
and
getting
reimbursed
for
the
travel
while
they're
on
the
road.
So
it's
really
important
for
us
that
our
tools
and
services
are
available.
A
All
the
time
and
easy
to
use,
we've
been
growing
at
a
pretty
steady
clip
a
few
years
ago.
We
were
purchased
by
us
AP
and
that
helped
accelerate
our
growth.
Actually,
we've
we've
been
growing
even
faster,
since
s
AP
acquired
us
and
we've
done
a
fairly
good
job
of
keeping
that
concur
culture
in
place
prior
to
the
AP
acquisition.
So
it's
it's
important
to
us
that
the
company
didn't
change
too
much
as
a
result
of
that
acquisition.
A
These
are
the
guys
that
helped
make
kubernetes
possible
at
concur
so
really
happy
to
work
with
the
folks
on
the
slide.
There's
a
few
remote
people
that
couldn't
be
in
the
picture
I
feel
bad
cuz,
because
one
of
them
is
right
here
Chris,
but
thank
you
for
the
team
and,
as
you
all
know,
today
is
election
day.
So
to
take
a
little
bit
of
pressure
off,
I
thought
you
know
I'd,
just
let
you
know
who
I'm
voting
for,
because
you
know
that
way.
A
So
how
do
we
build
kubernetes
that
concur
because
we're
doing
this
in
all
of
our
data
centers
and
also
in
the
cloud?
So
we
use
a
number
of
tools,
some
of
which
you
wouldn't
expect
on
the
slide
like
yeoman
yeoman
is
actually
really
important
for
us,
because
from
a
developer
perspective,
we've
got
a
whole
bunch
of
different
dev
teams,
hundreds
of
different
dev
teams,
in
fact-
and
each
of
those
teams
needs
a
quick
way
to
get
started
on
micro-services.
We're
currently
transitioning
between
our
large
monolith
to
micro-services
and
yeoman.
A
Helps
for
that
because
it
it
with
a
few
commands,
can
scaffold
at
concur:
nodejs
app,
a
java
app
a
Python,
app
c-sharp
whatever,
and
it
ties
it
into
our
whole
pipeline
that
the
dev
teams
can
use
to
get
started
granted.
They
can
still
customize
it,
because
this
is
another
thing.
That's
really
important
to
us.
Second
curve
is
that
dev
teams
actually
make
good
engineering
decisions.
A
So
it's
in
my
best
interest
as
a
website
as
a
principle
architect
to
make
it
so
it's
easy
for
them
to
use
the
stuff
that
works
well
and
not
as
easy
to
use
the
stuff
that
doesn't
work
in
our
platform.
Well,
at
the
same
time,
I
need
to
do
so
in
a
way
that
gives
them
that
freedom
to
choose.
So
we
don't
want
an
ivory
tower
telling
people
what
to
do.
We've
also
built
a
few
additional
tools
internally
that
I'll
talk
about
today
that
helped,
with
this
whole
platform,
the
whole
equal
ecosystem.
A
A
Terraform
is
used
as
our
deployment
system
of
the
kubernetes
clusters
and
chris,
along
with
another
member
of
our
team,
helped
create
all
of
those
terraform
templates
jenkins
and
circle
cir.
Both
up
here,
you
expect
those
to
be
competing
technologies,
but
we
consider
that
tier
to
be
composable,
so
any
CI
system
can
plug
in
to
our
platform.
A
We
need
to
have
our
code
internal,
so
that's
why
we,
we
went
with
github
Enterprise
and
we
have
f5s
in
our
data
center,
so
we
needed
to
integrate
with
f5.
Our
monolith
currently
uses
that
as
a
primary
way
of
communicating.
So
it's
important
for
us
to
make
sure
that
everything
works
inside
our
data
centers,
just
like
it
does
for
the
monolith
with
these
new
micro
services,
a
few
ones
that
you
definitely
don't
recognize
the
penguin.
A
We
like
this
is
not
a
real
view,
but
but
in
each
location
we
have
many
kubernetes
clusters
and
we
did
this
on
purpose.
We
did
this
because
everything
fails.
Yes,
even
clustered
technologies
fail
as
much
as
I
wish.
The
every
cluster
of
technology
was
actually
truly
100%
resilient
fault,
tolerant,
always
on
there's
always
gotchas.
That
happened.
You
can
look
through
the
high
availability
guide.
There's
a
lot
of
great
information
about
their
about
how
to
make
kubernetes
high-availability
are
highly
available.
A
Things
like
making
sure
you
have
the
right
number
of
SED
nodes
and
all
the
other
things
in
the
guide.
It's
it's
helpful,
but
let's
face
it.
This
is
an
unknown
unknown.
I,
don't
know
if
you've
ever
heard
the
term
or
not,
but
things
like
my
sequel
and
the
ways
in
which
it
fails
are
pretty
well
known
at
this
point
in
time.
Right,
there's
there's
some
known
unknowns
with
my
sequel
like
when
am
I,
going
to
run
into
lock
contention
on
a
particular
table.
I
know
that
that
it
could
happen.
A
You
could
look
at
this
either
way
like
some
people
will
probably
look
at
this
and
say
no
way,
you
should
definitely
not
have
multiple
clusters
in
one
location.
You
should
do
your
best
to
make
sure
that
the
kubernetes
clusters
in
your
environment
are
highly
available
and
always
online
and
I.
Don't
disagree.
I!
Think
that
we'll
get
there
too,
but
there's
some
really
interesting
things
that
came
out
of
us
architecting
around
having
multiple
clusters
that
I
want
to
share
here
and
get
some
feedback.
It's
get
some
understanding
from
a
larger
group.
A
Does
this
make
sense
or
not,
because
for
us
it
makes
sense
so
far?
So
what
does
this
look
like
inside
each
location?
There
are
multiple
kubernetes
clusters.
I
show
it
here
with
cluster
a
and
class
derby,
we
had
to
tie
them
into
load,
balancing
and
the
load.
Balancers
are
at
the
top
of
the
diagram.
We
had
to
still
make
every
cluster
highly
available,
so
we
have
multiple
Etsy
D
nodes
in
each
cluster.
We
have
multiple
master
nodes
in
each
cluster
and
obviously
the
pool
of
worker
nodes
running
the
actual
services.
A
In
order
to
properly
monitor,
we
had
to
move
monitoring
outside
of
the
cluster,
so
we
run
ELQ
independently
of
kubernetes.
We
run
from
Atheists
independent
of
kubernetes.
When
services
are
running
on
the
clusters,
they
get
then
get
tied
into
the
load
balancers.
So
traffic
hits
the
load
balancers
and
then
gets
directed
to
all
of
the
worker
nodes
across
both
sets
of
clusters.
A
Ok,
that's
great,
but
how
do
you
expose
services
because
there's
not
really
an
easy
way
to
do
that
today,
all
right
we
we
needed
to
make
it.
So
if
a
cluster
fails,
the
service
is
still
up
and
running,
but
nobody
wants
to
manually
go
in
and
configure
the
f5
every
time
you
add
in
a
cluster
node
or
every
time
you
add
any
service
and
expose
the
node
ports
to
each
of
the
nodes.
So
we
came
up
with
this
list
of
requirements
that
we
had
and
we
ended
up
building.
A
Our
own
plugin,
it's
a
it's,
a
go:
Lane
docker
container
that
runs
inside
kubernetes.
It
watches
the
kubernetes
api
uses.
The
the
new
client
go
api's
that
this
was
originally
written
using
the
the
go
client
as
a
part
of
the
the
whole
kubernetes
project,
which
became
very
painful
because
of
all
the
dependencies.
So
if
there's
anyone
here
that
was
involved
in
actually
breaking
that
out
and
to
the
stuff
client
go.
Thank
you.
That
was
a
huge
benefit.
A
A
I
think
it's
a
great
thing
that
the
the
kubernetes
community
has
developed
the
ability
to
watch
the
events
in
the
cluster
and
then
handle
them
using
the
kubernetes
api,
so
how's
this
work,
there's
the
main
package
just
package
go
make
this
a
little
bigger
for
you
and
there's
a
few
load,
not
load,
but
a
few
test
scripts.
So
I
have
this
cloned
locally
here
and
I'll
just
run.
It.
A
Actually
hold
on,
let
me
do
the
one
without
docker
I
set
up
two
different
scripts,
one
to
test
it
inside
docker
and
one
to
test
it
without
docker,
because
testing
inside
docker
is
a
little
bit
slower.
S
actually
build
the
container
and
pushed
out
to
docker,
but
they
both
work
the
same
way
as
you
can
see
from
the
little
diagram
here
it
uses
mini
cube.
So
whoever
built
mini
cube.
Thank
you
that
that
is
also
awesome.
It
stands
up
mini
cube
in
the
script.
A
So
there's
just
this
trickery
to
make
it
work
locally,
but
if
you
build
and
push
this
docker
container
to
a
repo
and
then
run
it
inside
a
kubernetes
cluster,
it
will
work
the
exact
same
way.
It's
running
here,
it'll
just
be
running
inside
kubernetes,
there's
a
nice
way
of
testing
that
that
works
and
that
you're
getting
events
and
that
you're
processing
the
events
locally
without
having
to
push
it
end
and
all
the
way
out
to
kubernetes
each
time.
You're
iterating.
A
A
A
Okay,
so
that,
oh
that
solves
one
problem
for
us,
but
but
what
about
the
the
problem
of
deploying
new
versions
of
kubernetes?
So
right
now
in
this
example,
there's
kubernetes
version
1.2,
that's
deployed
to
all
the
locations.
What
about
one
1.3
comes
out?
Well,
it's
already
out,
but
we're
not
quite
there.
Yet
we
like
to
really
harden
things
before
we
deploy
them
so
so
this
is
what
we
do.
A
We
create
a
whole
new
set
of
1.3
clusters
and
we
actually
blue-green
kubernetes,
so
we
make
it
available
for
all
of
our
deployments,
but
this
gets
messy
too,
and
the
reason
why
is
because
if
you
have
these
Cube
control
scripts
that
you're
using
to
deploy
well,
the
scripts
need
to
know
where
kubernetes
is.
It
needs
to
know
what
the
context
is,
what
the
name
of
it
is
to
deploy
against.
A
It
needs
to
handle
credentials
for
each
of
the
kubernetes
systems
and
there's
a
lot
of
different
commands
for
deploying
into
kubernetes
and
they
change
quite
a
bit.
In
the
past
year,
we've
gone
from
replication
controllers
to
deployments,
for
example,
and
then
there's
a
whole
slew
of
other
api's
that
are
available.
A
A
So
what
we
do
we
came
out
with
skipper:
skipper
is
a
lightweight
layer
on
top
of
the
kubernetes
api
that
takes
the
deployment
primitives
and
translates
them
into
what
the
right
cube
control
commands
are
to
run
against
the
different
kubernetes
clusters
in
each
of
the
data
centers.
So
as
a
developer,
I
can
just
say:
I
want
to
deploy
this
image
to
the
US
location
and
the
toilet.
Api
then
we'll
look
up
okay.
A
There's
some
extras
in
here
too,
which
I
think
are
really
cool
like,
for
example,
the
set
up
API,
which
I'll
show
here,
allows
you
to
configure
your
kubernetes
client
locally.
There's
a
configuration
IEP
I
that
was
written
for
the
ops
teams,
so
they
can
add
and
remove
kubernetes
clusters
as
they
become
available
in
each
of
the
zones
and
well
you
guys
can
read.
I
put
the
docs
of
what
this
API
is
out
on.
Github
and
I
have
been
given
approval
to
open-source
it
if
people
are
interested.
A
A
A
Danger
right,
how
am
I
gonna
know
where
my
kubernetes
clusters
are
so
I'll
just
show
you
an
example
of
what
the
API
call
is
here.
I
just
provide
my
kubernetes
namespace
and
the
token
to
connect
up
to
the
kubernetes
clusters
and
then
I
provide
that
to
the
skipper,
setup,
API
and
I
pipe
the
results
tube
in
bash.
A
What
this
thing
is
going
to
do
is
it's
going
to
take
that
information
and
translate
it
into
the
right
cube,
control,
config
commands
to
actually
create
my
configuration,
so
I'll
just
show
you,
the
config
commands
that
it
creates
here.
It's
a
bunch
of
different
commands,
setting
certificates
and
whatever
else
so
I'll
just
run
that
now
for
real
I'm
using
a
different
copy.
That
has
real
credentials,
not
the
one
that
I
showed
you
on
the
screen.
A
A
A
A
So
so
that's
that
was
my
argument
for
it
and
that
that
went
really.
Well,
obviously,
we
have
a
process
that
we
need
to
go
through
for
open
sourcing
things,
and
sometimes
that
can
be
a
little
bit
heavy,
but
but
it
just
depends
on
what
we're
doing
something
like
this.
That's
just
an
API
or
a
shell
sample
program
with
no
business
logic
is
much
easier
to
get
open
source
than
something
that
had
internal
anything
baked
into
it.
A
Great
question:
how
does
this
overlap
with
native
kubernetes
Federation,
so
we
haven't
figured
that
out
yet
I
think
it's
a
little
bit
different.
What
we're
doing
then
the
native
Federation.
We
would
like
to
transition
towards
Native
Federation
as
well,
but
I,
don't
know
that
the
goals
of
Federation
line
up
with
our
goals
here
either.
So
one
of
the
things
that
was
really
important
to
us
was
abstracting
the
kubernetes
api
from
developers
so
that
their
deployment,
privet
primitives,
would
be
stable,
no
matter
which
direction
the
kubernetes
community
goes
in.
A
Another
benefit
here
is
that
we
can
deploy
to
multiple
clusters
in
a
single
location
and
from
what
I've
seen
so
far
of
Federation
I'm.
Certainly
not
an
expert
is
that
it's
more
designed
around
having
individual
clusters
in
multiple
locations,
so
the
other
thing
that
I
love
about
this
is
it
is
very
opinionated
for
us
right.
A
It
solves
our
needs
based
on
what
tools
we
have
in
our
environment
and
we
can
decide
what
the
configuration
logic
is
that
ends
up
getting
pushed
out
for
our
services
in
each
of
the
various
locations.
Things
like
this
is
my
name
for
my
service
are
plugged
in
as
an
environment
variable
into
the
container
or
here's
how
to
get
to
the
service
discovery
endpoint
in
this
location,
also
plugged
into
an
environment
variable
here's.
A
What
your
proxy
server
is
to
reach
the
internet
for
this
location
and
all
that's
configurable
when
the
ops
team
plugs
in
a
new
location
or
a
new
cluster
they
decide.
Here
are
the
values
that
we
want
to
advertise
and
it's
it's
not
strict
and
the
design
of
the
application
for
what
you
can
plug
in
there.
But
it's
strict
for
us
and
what
we've
decided
to
provide
is
that
bear
another
question,
sir.
A
Have
I
considered
trying
to
roll
this
back
up
stream?
It
I
certainly
will
I'm
not
against
it,
I
liked
the
freedom
of
being
able
to
work
on
our
problem
and
get
it
deployed
quickly
for
concur,
so
that
we
could
move
forward
very
fast,
and
we
had
this
Multi
cluster
configuration
working
early
this
year
and
it's
been
tremendously
beneficial
for
us
to
get
teams
on-boarded
quickly.
A
A
A
So,
in
the
background
there
you
can
see
it
going
through
and
doing
some
stuff.
There's
a
401
there
that
I'm
sure
someone
will
notice,
but
that's
just
another
API
call
we
have
built
into
this
to
plug
it
into
service
discovery
which
it
can't
reach
from
here,
and
it
deployed
it
to
two
clusters
in
Seattle,
and
it
gave
me
my
one
endpoint
for
both
of
those
clusters.
I
can
keep
control,
config,
use
context,
SCA,
3,
I
get
the
pods
and.
A
A
Okay,
so
that's
a
look
at
the
pipeline,
we're
using.
We
are
using
artifactory
as
well
the
the
way
the
artifactory
works
for
us
is.
We
have
a
development
environment
that
CI
systems
can
push
images
to
and
then
when
they
do
a
promotion.
Well,
they
have
to
be
logged
in
to
do
a
push.
First
of
all,
they
have
to
have
a
named
account
for
their
team
when
we
do
a
promotion
with
artifactory.
A
It
first
goes
to
a
staging
environment
where
they
can
do
their
testing
and
then,
if
they
do
another
promotion,
then
it
goes
to
release
when
it
goes
to
release.
Then
artifactory
replicates
the
images
across
to
all
of
our
data
centers,
so
that
that
image
can
then
be
pulled
in
from
kubernetes
locally.
Just
again,
something
that's
really
important
for
us.
A
We
don't
ever
want
to
be
in
a
situation
where
a
we're
putting
strain
on
network
or
work
or
coupled
tightly
to
network,
whether
it's
swen
or
Internet,
to
pull
down
our
outer
facts
and
run
them
and
artifactory
helps
us
with
that.
So
skipper
then
another
one
of
those
config
items.
It
replaces
the
docker
image
path
with
the
right
artifactory
location,
so
that,
when
I
deploy
to
all
clusters,
I
don't
have
to
know
for
this
location.
I
need
to
set
up
my
configuration,
the
amyl
to
pull
to
a
different
or
pull
from
a
different
artifactory
location.
A
A
Already
showed
you
the
skipper
demo,
so
what
do
we
learn?
This
is
more
like
what
we
learned
from
a
developer
perspective.
First,
testing
locally
is
really
critical
and
testing
locally
inside
a
container
is
nearly
as
critical
I've
had
number
of
situations
where
developers
have
said
hey,
you
know,
I
ran
my
XYZ
app,
doesn't
matter
if
his
goal
laying
node
asp.net
whatever
app
locally
and
then
when
I
pushed
it
to
kubernetes
it.
It's
not
running
it's
in
this
crashed
loop
back
off
state
and
it's
throwing
an
error.
A
What's
what's
wrong
with
kubernetes
and
almost
a
hundred
percent
of
the
time
when
this
happens,
it's
because
the
developer
when
they
tested
they
tested
it
natively
in
their
Mac
or
on
their
Windows
box,
and
they
didn't
test
it
inside
the
container.
Well,
it
turns
out
those
are
different
operating
systems
and
your
code
might
work
fine
locally,
but
by
the
time
you
produce
a
container
running
on
LP,
Linux
or
Ubuntu
or
whatever
else
it's
gonna
behave
differently.
One
of
the
the
main
things
we've
run
into
here
is
file
systems
being
different
on
your
Mac
OSX.
A
Your
file
system
is
actually
case-insensitive
by
default
and
then
inside
your
Linux
container,
it's
guaranteed
to
be
case
sensitive,
so
we've
run
into
issues
where
packages
were
named
and
correctly
and
work
just
fine
locally.
But
when
you
push
it
out,
it
doesn't
work
anymore.
So
it's
something
to
be
aware
of.
Definitely
an
education
thing
for
for
us
was
just
getting
all
of
the
development
teams
to
run
that
test
locally.
Having
a
small
container
is
really
important
too,
especially
if
you're
pushing
that
container
out
to
your
global
infrastructure.
A
You
don't
want
to
have
to
tie
up
a
large
amount
of
bandwidth,
just
replicating
containers
around
and
almost
equally
important
to
having
a
small
container
size.
This
layering
it
properly
because-
and
you
can
notice
this-
that
when
you
build
the
docker
image,
sometimes
it
has
to
pull
down
the
Internet,
especially
if
you're
working
with
Java,
and
if
you
find
that
it's
doing
that
every
time
do
a
docker
build
your
docker
layering
is
not
set
up
right
because
it
should
be
cashing
those
packages.
A
Well,
if
it
can't
cash
locally,
do
you
think
it
can
cash
well,
when
you're
pushing
the
image
out
to
all
of
your
data?
Centers
no
I
can't
can't
cash
there
either
and
another
lesson
learned
for
developers
is
starting
small.
So
it's
super
easy
to
deploy
a
hundred
copies
of
my
container
into
kubernetes.
A
But
if
I
do
that,
when
I'm
first
starting
an
application
and
I
start
to
run
into
errors,
it
can
be
really
tough
to
troubleshoot
the
problem,
because
then,
if
I
send
in
a
request
to
kubernetes
and
I,
get
the
error
back
and
I
want
to
check
how
that
failed
on
a
specific
container
well,
which
one
do
I
look
at,
you
can
find
it
an
elk
and
figure
it
out,
but
man
it's
a
lot
easier.
If
you
just
start
with
one
make
sure
everything's
working
and
then
scale
it
out.
A
So
what
about
some
benefits
to
the
way
we
designed
this?
We
talked
about
a
few
of
those
already
it's
it's
a
easy
to
upgrade
our
clusters,
who
we
blue/green
them.
The
the
team
rolls
out
the
the
new
version
skipper
automatically
deploys
on
to
new
and
old,
and
then
when
teams
are
ready,
they
feel,
like
their
services,
are
running
correctly
across
both
the
new
and
old
clusters.
We
tear
down
the
old
ones.
If
we
ran
into
an
issue,
we
would
just
shut
down
the
new
ones.
A
Let's
see
cluster
and
node
issues
do
not
need
immediate
action.
This
is
important
for
the
ops
guys
right
yeah.
You
got
to
have
a
team
running
kubernetes.
Do
you
really
want
them
to
be
slaves
to
the
pager?
If
a
node
goes
down
in
kubernetes?
Well
with
this
architecture,
they
don't
have
to
be,
it
can
be,
you
know,
wake
up
the
next
morning
and
fix
the
the
node
down
instead
of
you
know,
have
to
address
it
right
away.
A
We
have
loose
coupling
between
the
different
pieces
and
the
architecture,
for
example,
skipper
deploys
the
service
to
kubernetes,
but
kubernetes
itself
is
the
thing
plugging
into
the
f5.
So
since
those
are
two
very
independent
pieces,
the
the
skipper
complexity
is
lower
and
swapping
out
to
from
f5
to
console,
which
is
the
direction.
We're
probably
heading
long
term
would
be
much
easier.
I
don't
have
to
make
an
update
to
skipper
well
same
thing
from
the
CI
system
to
the
kubernetes
there's
very
loose
coupling
there
with
an
API
layer.
A
So
if
people
change
CI
systems,
this
easy,
if
we
decided
to
go
with
a
different,
dare
I
say
a
different
scheduler
at
some
point
in
time,
then
that
would
be
easy
to
change
as
well.
I,
don't
think
we're
gonna
do
that,
and
it's
opinionated
for
our
needs
some
challenges,
so
it
adds
complexity
for
troubleshooting.
That's
why
we
had
to
pull
ELQ
outside
another
piece
that
the
team
wrote
was
a
little
cute,
kubernetes
logstash
modification
to
to
send
the
logs
in
the
in
our
specific
elf
format,
from
kubernetes
to
our
external
out
cluster.
A
We
we
have
a
bit
of
a
challenge
right
now
that
we're
working
on,
which
is
how
do
we
put
state
and
the
clusters
right
now?
Our
state
is
entirely
outside
of
kubernetes
we're
not
ones
to
just
like
jump
in
and
run
everything
stateful
on
a
new
platform
right
away,
so
we
wanted
to
test
the
waters
a
little
bit
first,
one
of
the
thoughts
we
have
for
solving
this
is
instead
of
having
the
flannel
network
or
canal.
A
If
that's
what
we
end
up
going
to
running
just
in
a
single
kubernetes
cluster,
we
could
expose
that
across
all
of
the
kubernetes
clusters,
in
a
location
that
way
state
could
live
across
a
set
of
services
running
at
that
lower
level
and
communicating
with
each
other
across
the
clusters.
But
this
is
a
challenge.
Oh
this
is.
This
is
one
of
the
things
that
we
have
to
work
out
with
this
architecture.
A
And
then
there's
this
other
one
here
where
you
know
what,
if
my
deployment
works
great
and
one
kubernetes
cluster,
but
it
doesn't
work
and
the
others
kubernetes
cluster.
So
my
code
is
a
v1
over
here
and
at
v2
over
there
and
there's
not
really
a
good
answer
for
this
I.
Don't
want
to
add
the
complexity
and
a
skipper
to
manage
that
and
I.
Don't
think.
A
We
need
to
honestly
I
think
that
this
is
a
little
bit
of
an
education
thing
for
us
and
a
learning
for
us
as
a
developer
team,
that
your
deployments
should
always
be
immutable
and
they
should
always
be
Bluegreen
able,
in
a
way
that
both
version,
1
and
version
2
are
online.
At
the
same
time
now
you
should
obviously
get
notified
when
that
happens,
so
you
can
rectify
it
and
decide
whether
you
rollback
version
2
to
version
1
or
fix
it
in
a
way
that
makes
version.
A
1
go
to
version
2,
but
I'd
say
if
this
ever
becomes
an
issue.
The
problem
is
probably
that
your
code
isn't
designed
to
work
with
both
versions,
because
usually
when
I
see
this,
it's
because
there's
a
breaking
change
and
they
don't
really
have
a
really
good
rollback
plan.
Anyways,
it's
something
like
a
tight
coupling
to
a
database
change
which
should
really
be
three
deployments
instead
of
one
one
where
you
create
the
new
code
that
can
recognize
both
versions
of
the
database.
A
So,
a
few
of
the
other
pieces
that
you
saw
up
there
that
we're
working
on
which
I
can't
share
the
details
of
right.
Now
one
is
Haku,
which
is
our
internal
secrets
and
service
discovery
system.
This
is
a
thin
layer
on
top
of
console.
We
decided
to
do
it
this
way,
for
various
reasons,
happy
to
talk
with
someone
over
that
over
beer,
and
we
also
are
in
the
process
of
developing
a
security
service
Citadel.
What
this
does
is
similar
to
our
f5
plugin
this.
A
Where
do
you
put
that
right
now,
yeah,
you
can
put
it
in
a
secret
in
kubernetes,
but
it's
not
encrypted
and
any
operator
can
look
at
it.
So
this
for
us
is
a
more
secure
way
of
doing
that,
where
the
the
secrets
pulled
out,
push
to
the
Container
and
only
stored
in
memory.
Somebody
at
the
command
line
can't
go
get
that
secret.
A
Sorry
I
went
a
little
over
I'm
sure
everybody
wants
to
get
some
lunch
I'm
happy
to
stay
and
answer
questions
though
too,
but
this
is.
This
is
what
we
built
like
I
said
it
might
not
be
for
everybody
works
really
well
for
us.
We
like
it
because
of
the
the
reasons
I
talked
about
and
don't
like
it
because
of
the
reasons
I
talked
about
too.