►
Description
Switching From External Load Balancing to consul & ingress [I] - Dan Wilson, Concur
At Concur we integrated our kubernetes clusters to our own internal F5 ecosystem which worked well for internal data center deployments but turned out to be cumbersome when trying to maintain a consistent setup for our cloud environments. We'll discuss what the issues are that we faced and the new architecture that we're using which combines consul for dns service discovery with kubernetes ingress in a multi kubernetes cluster design.
About Dan Wilson
I have a passion for collaborating across the business and using data to drive decision making. My primary areas of focus include container orchestration, developer pipeline, cloud service architecture, scale out technologies, design for failure and open source technologies.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Okay,
well,
hey
thanks
for
being
here.
I
know
it's
near
the
end
of
the
day
at
the
two-day
conference
here
it
was
probably
kind
of
tired,
and
some
of
you
are
probably
jet-lagged
like
I,
am
and
looking
forward
to
hanging
out
and
relaxing
a
little
bit
this
evening.
So
appreciate
you
showing
up
and
being
here,
I
hope
I
can
make
it
worth
your
while.
A
So
a
little
bit
about
concur.
We
are
in
the
business
of
controlling
spend
for
companies
and
at
the
same
time,
we
make
it
easier
as
an
employee
to
get
reimbursed
quickly
for
that
spend
when
you're
on
the
road
and
our
users
expect
our
services
to
be
available.
So
availability
is
super
important
to
us
and
speaking
of
users,
we've
been
growing
like
mad
and
it
hasn't
slowed
down.
Fact
it's
accelerated.
Since
s
ap
acquired
our
business
a
few
years
ago.
A
Little
about
me,
oh
yeah,
sorry
I,
been
at
concur
forever
and
worn
a
lot
of
different
hats
throughout
the
years,
thankfully,
otherwise
I'd
be
really
bored
and
I've
been
working
on
kubernetes
for
about
a
year
and
a
half
now
and
I
currently
participate
in
the
Federation
sig
and
have
been
a
contributor
to
that
regard.
So
I
love
working
with
the
product
a
little
bit
about
kubernetes
second
curve.
This
is
a
very
high
level
view.
A
A
A
We
we
still
make
every
cluster
highly
available
with
multiple
sed
nodes
and
master
nodes.
We
plug
the
pool
of
master
nodes
into
external
load
balancers
inside
our
data
center,
so
that
API
calls
can
be
distributed
to
those
masters
and
if
a
service
is
deployed
on
our
clusters,
that
service
is
also
tied
into
external
load.
Balancing
I'll
talk
a
little
bit
more
about
how
we
do
that
in
a
minute.
We've
pulled
monitoring
and
elq
outside
of
the
cluster.
A
So
zooming
out
just
a
little
bit
to
where
we're
looking
at
multiple
clusters.
Now
in
a
single
location,
we
we
tie
those
clusters
into
load
balancing
with
a
single
endpoint,
so
a
service
that
is
being
accessed
would
have
a
single
DNS
name,
a
single
VIP
and
a
single
pool,
but
inside
of
that
pool,
are
kubernetes
nodes
from
each
of
those
clusters.
A
So
if
a
single
cluster
is
having
an
issue,
then
requests
can
still
get
routed
to
the
other
kubernetes
clusters
and
we
we
made
a
conscious
decision
early
on
to
go
with
this
approach,
because
the
technology
was
still
very
new
any
time
you're
dealing
with
new
technology.
There's
some
unknowns
you
could
hit.
We
didn't
want
to
risk
those
unknowns,
impacting
our
end-users
and
we
wanted
to
go
into
a
production
state
with
kubernetes
as
fast
as
possible.
So
that's
why
we
we
want
with
this
architecture,
okay.
So,
but
how
do
you
do
that
right?
A
How
do
you
get
those
kubernetes
nodes
plugged
into
a
load
balancer
if
you're
in
a
private
data
center
we're
in
private
cloud-
or
you
know
our
own
data
centers
and
a
lot
of
locations
we're
also
in
public
cloud?
So
we
wanted
it
to
work
the
same
as
much
as
possible,
so
we
wrote
our
own
little
kubernetes
plugin,
it's
just
a
simple
golang
container
that
runs
on
kubernetes.
It
watches
the
API.
A
Some
of
you
may
have
attended
some
talks
today,
talking
about
how
to
leverage
the
kubernetes
api
and
that's
exactly
what
we
did
super
90s
api
is
awesome.
You
can
get
a
watch
endpoint
you
can
get
updates
and
to
fire
off
whatever
events
or
changes
that
you
need
to
make
externally
or
inside
the
cluster.
A
But
how
does
this
help
you
guys
right?
Because
you
don't
have
our
load
balancing
technology
inside
your
data
center
chances?
Are
you
don't
have
our
abstracted
API?
Either
we
wrote,
we
have
our
own
load-balancing
as
a
service
inside
our
data
centers
that
were
we're
leveraging
here,
instead
of
reaching
out
to
the
vendors
API
directly.
A
Well,
you
could
do
this
yourself
too,
if,
if
you
have
private
data
center
needs,
whether
it's
load,
balancing
or
logging
changes
to
the
cluster
or
whatever
else
we,
you
know,
created
a
little
sample
project
on
how
to
do
this.
It
builds
a
docker
container.
You
can
also
use
a
sample
project
to
test
locally
against
mini
cube.
It's
all
the
test
scripts
are
set
up
for
you.
A
Ok,
but
what
are
some
of
the
challenges
we
ran
into
with
this
design,
because
clearly
it
wasn't
our
design
of
choice
if
we
wanted
to
move
away
from
external
load
balancing
well,
one
challenge
is
that
the
plugin
has
to
be
pretty
smart,
because
it's
manipulating
a
single
pool
that
has
nodes
in
it
from
many
clusters.
So
you
know.
A
The
other
thing
is
when
we
try
to
bridge
this
technology
into
the
cloud
we
decided
to
go
with
the
software
load,
balancer
version
of
our
vendors
hardware
load
balancer
and
again
this
is
a
conscious
decision.
We
made
mainly
because
our
monolithic
apps,
that
we're
all
supporting
to
the
cloud
relied
on
specific
features
for
that
load
balancer.
A
A
A
Balancer,
and
then
the
vendor
recommends
for
high
availability
reasons
that
you
also
add
nodes
from
clusters
in
different
regions
or
different
availability
zones.
So
we
ended
up
having
to
run
three
copies
of
our
integration
on
each
cluster
in
order
to
plug
the
nodes
into
the
three
load
balancers
and
it
just
gets
really
messy
right
at
the.
It
creates
a
lot
of
spaghetti
in
our
cloud
environment
that
really
shouldn't
be
there.
A
Okay,
so
we
wanted
a
better
approach.
We
started
to
write
down
our
requirements.
We
wanted
the
environments
to
still
look
as
much
the
same
as
possible
in
every
location.
We
wanted
to
use
the
same
technology
at
every
location
as
well.
If
we
could
SSL
needs
to
be
as
close
to
our
actual
pods
as
possible
and
make
sense
for
the
workload
and
it
still
needs
to
work
with
our
own
micro
services,
client,
certificate,
validation
and
job
validation.
A
A
Okay,
so
physical
load
balancers
don't
work
because
there's
no
real
equivalent
in
the
cloud.
This
is
like
a
nice
classic
car,
that's
fast,
but
doesn't
have
all
the
bells
and
whistles
that
we
need
in
a
cloud
or
like
a
really
nice
steamer
and
the
cloud
load.
Balancing
technology
doesn't
work
for
us
because
we
can't
run
cloud
load
balancing
and
our
data
centers.
It's
not
currently.
A
So
we
started
to
look
at
console
and
ingress
and
pairing
them
together.
So
a
little
bit
about
console
and
when
I'm
talking
about
console
here,
I'm
talking
about
console
services
and
not
console
key
value
store
very
different.
The
services
allow
you
to
take
a
set
of
systems
and
tie
them
into
a
single
service
that
then,
when
you,
when
you
do,
that,
you
get
a
DNS
name
and
console
for
that
service.
That
resolves,
if
you
use
the
console
as
your
DNS
server
itself
or
you
forward
to
it
from
your
your
existing
DNS
servers.
A
Every
time
you
ask
for
what
the
the
a
record
is
of
that
DNS
you'll
get
one
of
the
servers
that
are
behind
the
name
and
there's
a
bunch
of
logic
that
you
can
have
console
do
there,
so
that
it
decides,
which
note
is
responding
the
fastest
and
gives
you
that
IP
or
gives
you
the
IP
address
of
the
the
node,
that's
closest
to
you.
So
it's
very
valuable
it.
It
doesn't
do
anything
at
the
the
higher
networking
layers
right.
A
It's
just
giving
you
an
IP
address
and
that's
why
it
pairs
really
well
with
ingress,
because
ingress
can
do
that
layer,
seven
stuff.
So
if
you
wanted
to
route
to
service
based
on
the
host
header
or
based
on
the
URI
path,
then
ingress
handles
that
well,
with
each
console
service,
you'd
have
a
unique
DNS
name
and
with
ingress
you
could
look
at
that
name
and
route
it
to
the
to
the
correct
server
in
our
testing
within
graphs,
we
saw
two
or
three
millisecond
overhead.
We
use
the
nginx
load.
A
A
So
the
the
question
came
to
be.
Is
this
the
perfect
match,
so
we
put
together
a
experiment
using
the
technologies
and
we
leveraged
a
similar
plugin
to
what
you
see
in
the
community
coop
the
console
plug-in
that
would
take
changes
to
services
and
nodes
and
kubernetes
from
the
watch.
Endpoints
excuse
me
and
translate
that
late,
that
into
the
API
calls
that
you
need
to
add
and
remove
the
the
correct
nodes
and
node
ports
and
console.
A
Well,
we
ran
into
some
issues
also,
you
have
to
create
the
ingress
control
on
kubernetes
and
the
ingress
resources
themselves
in
order
to
make
this
work
and
ingress
works,
the
same
way
actually
right.
The
ingress
controller
is
watching
for
the
ingress
objects
in
kubernetes
and
manipulating
the
load
balancing
rules
dynamically
okay,
so
we
ran
some
problems.
A
Accessing
services
across
datacenters
was
an
issue
for
us
because,
let's
say
my
services,
my
site
console
dot
IO
well,
if
I'm
in
a
Mia
and
I'm
looking
for
my
site
at
console,
dot,
IO
it'll
give
me
the
local
location
for
that
site.
If
I
need
to
reach
out
to
the
u.s.,
though
there
wasn't
a
way
for
us
to
do
that,
because
we
hadn't
set
up
clustering
of
console
between
all
of
our
data.
Centers
console
supports
this.
Where
you
could
say
my
site,
us
console
dot
IO,
but
we're
just
not
set
up
for
that.
A
Yet
it
wasn't.
A
part
of
our
rollout,
also
console
DNS
by
default,
is
set
up
with
a
TTL
of
zero.
So
what
that
means
is
every
single
DNS
query
is
going
to
reach
out
to
the
console
DNS
server.
Also,
it's
paxos
under
the
covers,
and
it's
going
to
make
sure
that
it
is
responding
with
a
valid
result
by
checking
with
the
other
nodes
in
the
cluster.
This
takes
time,
so
what
we
saw
is
on
our
untuned
system
was
between
150
and
300
milliseconds
of
latency
to
do
each
console
query,
which
that
would
definitely
not
fly.
A
A
A
None
of
them
supported,
console
ACLs,
so
we
had
again
needed
to
make
some
code
changes
there
in
order
to
support
that
I'm
happy
to
contribute
those
back.
If
somebody
can
tell
me
what
the
route
Coupe
de
console
project
is
to
contribute
them
to
be
happy
to
do
it
a
side
benefit
of
using
console,
is
it
stores
the
SRV
record
data?
So
not
only
do
you
get
the
dns
name,
but
you
also
get
the
port,
the
kubernetes
node,
for
that
the
service
is
running
on
for
each
node.
A
Ok,
so
having
hit
some
stumbling
blocks
there,
we
decided
to
go
and
look
at
other
alternatives
like
is
there
some
other
way
to
solve
this
problem?
And
we
went
back
to
our
load,
balancing
keyer
and
we
said,
okay?
Why
don't
we
couple
what
we're
doing
today
with
load
balancing
inside
our
data
center
with
ingress?
So
what
this
involves
is
having
a
lighter
weight
version
of
our
plugin.
A
So,
instead
of
tying
services
into
the
load,
balancer
for
every
service,
running
or
third
entries
into
the
load
balancer
for
every
service
running
on
kubernetes,
it
would
just
tie
in
the
ingress
controller
service
itself
into
load
balancing
so
that
there's
a
small
number
of
VIPs
on
the
load.
Balancer
that
then
route
directly
to
ingress
and
then
ingress
from
there
would
forward
the
request
to
the
correct
service
based
on
DNS
names,
and
we
have
the
capability
inside
of
our
data
center.
A
We
have
a
DNS
API,
so
it
would
just
be
a
matter
of
creating
a
cname
for
each
service.
That
would
then
point
to
either
the
main
regional
load,
balancer
pool
or
VIP
or
the
individual
pool
that
the
service
is
running
on
if
it's
not
running
across
tall
clusters,
and
by
doing
that
we
would
get
essentially
the
same
thing:
we're
getting
out
of
console
Plus
in
graphs,
but
just
using
native
DNS
and
using
the
native
load
balancing
that
we
have
in
our
data
center.
A
A
So
why
not
just
leverage
what's
available
already
there,
instead
of
having
any
plug-in
running
on
the
kubernetes
clusters
and
it's
less
rust
to
maintain
overall,
so
this
would
end
up
making
it
so
from
a
end,
end
developer
team
perspective:
kubernetes
is
acting
the
same
way
inside
the
cloud
as
it
is
inside
our
data
center.
They
just
have
ingress
objects
and
services
and
deployments
on
kubernetes
and
the
cname
would
then
get
tied
into
either
the
the
main
load
balancer
or
each
individual
load
balancer,
depending
on
what
the
needs
are.
A
I've
heard
a
few
other
talks
today
and
yesterday
with
similar
needs
that
we
have
so
it
might
make
sense
for
us
to
do
that.
Also
I've
been
making
skipper
thinner
as
much
as
I
can.
So
it's
a
it's
less
code
for
us
to
maintain
and
it
gives
value
back
to
the
community
by
contributing
the
features
that
we've
rolled
into
this
product
internally
into
kubernetes
Federation,
and
that's
really.
The
direction
we
want
to
go
a
long
term
is
to
just
leverage
Federation.
A
A
A
Okay,
so
the
question
is:
who
is
the
consumer
for
the
console
DNS
in
this
situation?
It
is
not
external
consumers,
its
internal
consumers,
of
the
console
DNS.
All
of
our
external
calls
go
through
a
separate
tier
of
API
gateways
in
our
our
data
centers
and
then
from
there.
It
would
end
up
hitting
the
console
DMS
yeah.