►
From YouTube: Weaving a Mesh for Multiple Clusters at bol.com - Remco Overdijk, bol.com & James Brook, Google
Description
Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Weaving a Mesh for Multiple Clusters at bol.com - Remco Overdijk, bol.com & James Brook, Google
Bol.com, the largest online retailer in the Netherlands, has many teams running services on Kubernetes in cloud and on-prem. Over time, they have grown out of a single cluster. They will explain how adopting Envoy and service mesh gives them resilient and uniform connectivity across clusters and datacenters, while optimizing costs. The goal was to go all in on service mesh with a cautious and incremental approach. Injecting Envoy into hundreds of mission-critical pods needs care. You'll learn about the reasons for choosing service mesh, the journey, architecture, challenges and the lessons learned along the way.
https://sched.co/ZesW
A
B
A
A
As
such.
We
feel
it's
important
for
our
featured
teams
to
be
able
to
work
independently
without
having
to
wait
on
platform
teams
to
deliver
or
maintain
their
infrastructure
for
them.
For
example,
that's
why
we
believe
that
self-service
infrastructure
is
the
future
for
us
at
bolicom.
We're
used
to
doing
things
ourselves.
A
A
Historically,
we've
been
statically
skilled
for
the
busiest
day
of
the
year,
plus
a
little
safety
margin.
We
do
have
lots
of
automation
and
manage
our
machines
using
puppet
software.
Development
teams
can
deploy
their
applications
in
our
data
centers
using
automation,
but
all
of
the
infrastructure.
Interaction
on
their
behalf
is
left
to
site
reliability
teams.
A
We
really
believe
in
devops,
and
we
want
to
allow
teams
to
really
focus
on
building
applications
and
not
all
the
other
stuff
in
a
data
center
self-service
is
partially
possible,
but
it's
a
bit
obscure
and
inflexible.
Our
philosophy.
Is
you
build
it?
You
run
it.
You
love
it
to
further
embrace
that
mantra.
We
try
to
increase
the
level
of
self-service.
We
offer
granting
autonomy
and
sharing
ownership
and
a
true
devops
way
of
working
turned
out
to
be
hard
in
our
current
setup.
A
A
A
Moving
to
the
cloud
together
with
regulations
like
gdpr
also
affected
the
way
we
needed
to
think
about
security.
The
platform
gives
freedom,
but
it
also
enforces
policy
guard
rails.
At
the
same
time,
given
the
scalability
of
kubernetes,
the
original
idea
we
had
was
to
have
a
single
cluster
per
environment
in
order
to
reduce
the
overhead.
A
These
multi-tenant
clusters
house
all
of
the
different
applications
and
services
isolated
into
namespaces
and
are
able
to
connect
to
each
other
using
network
policies.
This
worked
well
for
a
while,
but
some
mistakes
were
made
along
the
way,
one
in
particular,
since
we
want
our
gcp
services
to
be
reachable
from
our
data
centers
and
vice
versa,
they're
using
private
interconnects
to
form
a
large
private
network
sharing
the
same
class
a
network.
A
This
also
means
that
we
had
to
use
our
ip
subnet
space
sparingly,
given
that
we
had
around
400
microservices
in
the
data
center
at
that
time.
Initially,
we
figured
that
having
1024
services
in
a
single
kubernetes
cluster
would
be
plenty,
and,
as
such,
we
set
up
the
gke
clusters
with
a
slash
22
subnet
for
services
allowing
for
20
000
1024
services.
A
We
also
underestimated
the
popularity
of
the
self-service
model,
and
soon
we
saw
a
rampant
growth
of
new
services
being
used
for
all
kinds
of
proof
of
concepts
and
new
microservices.
This
ended
up
being
a
bit
of
a
crisis.
In
the
end,
we
came
very
close
to
hitting
the
surface
limit
effectively
halting
the
creation
of
new
services
in
that
environment.
A
I
guess
that's
our
first
lesson
to
share
today
put
plenty
of
thought
into
your
ip
subnet
allocation
strategy.
Believe
it
or
not,
this
was
actually
the
primary
reason
for
looking
at
the
service
mesh
in
the
first
place,
so
we
knew
we
needed
multiple
clusters.
We
had
quite
separate
doubt
on
the
cloud
and
data
center
worlds.
We
had
many
different
ways
to
load
balance
and
discover
services,
and
there
wasn't
even
really
an
easy
way
to
find
a
service
either.
A
Now,
service
mesh
definitely
seems
to
be
somewhere
on
the
hype
circle,
or
at
least
it
was
a
year
ago,
but
we
really
did
not
set
out
to
justify
having
a
service
match,
because
we
thought
it
was
cool.
In
fact,
quite
the
opposite.
We
were
a
little
scared
of
the
idea.
We
did
everything
in
our
power
to
go.
Multi-Cluster
without
service
mesh
service
mesh
is
definitely
complex.
A
It's
not
a
mature
technology
either.
We
had
experience
with
that
from
gambling
on
kubernetes
when
it
was
still
young,
but
we
started
to
think
seriously
about
the
service
mesh
and
some
bonus
features
started
to
become
quite
apparent
like
strong
identity,
mutual
tls,
authentication
and
authorization
for
services
would
make
it
much
easier
to
please
our
security
teams.
A
A
Our
close
cooperation
with
google
also
meant
that
we
had
access
to
istio
developers,
which
proved
to
be
very
useful.
I
will
touch
on
we'll
touch
on
that
later
as
well.
Service
tax
service
mesh
technology
has
come
a
long
way
in
these
past
18
months.
Any
available
options
are
now
more
mature,
more
diverse
and
tailored
to
specific
scenarios,
as
well.
Linker
d,
for
example,
recently
launched
multi-cluster
support
so
make
sure
that
you
do
a
proper
product
selection
based
on
your
requirements
before
diving
in
head.
First,
okay,.
B
A
B
B
So
it's
definitely
useful
to
have
a
pretty
good
understanding
of
envoy
terminology
if
you're
serious
about
doing
mesh,
each
endpoint
in
a
cluster
is
simply
the
destination
ip
address
for
a
port
and
a
port.
The
traffic
flows
between
pod
ip
addresses
and
doesn't
follow
the
usual
ip
tables,
routing,
which
is
programmed
by
cube
proxy
and
kubernetes
by.
B
Running
across
multiple
kubernetes
clusters,
we
hope
to
make
the
cluster
boundary
much
less
significant,
client-side
load
balancing
is
one
of
the
most
powerful
concepts
in
service
mesh
and
the
bulb.com
gke
clusters
are
set
up
to
use
vpc
native
networking,
which
basically
means
the
pods
get
an
ip
address
from
a
reserved
subnet
in
the
vpc
and
all
of
the
pod.
Ip
addresses
in
all
of
the
clusters
are
directly
routable
in
a
flat
network
with.
B
We
originally
expected
to
hear
an
audible
groan
when
we
heard
this
when
you
heard
this
jargon
mentioned
that
this
talk's
recorded.
So
it
doesn't
matter
now,
it
sounds
like
jargon
and
marketing
hype,
but
it's
important
security
is
really
important
to
bold.com,
especially
in
these
days
of
gdpr.
Regulation.
B
Unlike
an
ip
address,
which,
in
the
kubernetes
world,
is
something
very
ephemeral,
we
want
applications
to
be
able
to
communicate
with
each
other,
but
we
need
a
way
for
applications
to
trust
each
other
based
on
their
identity
and
not
an
ip.
And
for
that
to
happen,
we
need
a
few
other
things
credentials.
So
these
come
in
the
form
of
tls
certificates.
B
B
B
B
In
this
project
we
were
only
focused
on
enabling
the
sidecar
don't
start
using
any
of
the
features.
Yet
only
when
you've
got
the
side.
Cars
in
then
start
using
features
and
realizing
the
benefits
at
a
really
high
level.
We
plan
several
phases
to
weave
the
mesh
lots
of
preparation,
enabling
side
cars
in
permissive,
mutual
tls
mode
with
something
called
auto.
Mtls
turned
on
enabling
authorization
policy,
switching
mutual
tls
to
strict
mode,
disallowing,
plain
text
connections
and
then
eventually,
cleaning
up
all
the
loose
ends
and
start
adding
more
features.
B
Broadly,
each
step
depended
on
the
previous
step
for
the
whole
mesh.
We
knew
the
adoption
would
have
to
happen
incrementally
namespace
by
namespace
or
even
by
workload,
and
we
would
depend
on
lots
of
different
application
teams.
Initially,
developers
were
given
the
option
to
opt
in
to
having
the
sidecar
in
their
workloads.
B
Emails
were
sent,
documentation
was
shared,
people
were
nagged
at
some
point,
the
majority
of
applications
had
been
migrated
and
after
that
site
cars
were
injected
by
default.
Developers
still
had
the
option
to
opt
out
if
they
really
needed
to
so.
What's
the
big
deal
about
enabling
the
sidecar,
surely
you
just
need
to
label
a
namespace
or
a
deployment?
B
B
Talk
a
bit
in
more
detail
about
some
of
the
things
you
need
to
think
about
when
adding
side
cars
to
your
applications.
There
are
some
simple
things
that
you
just
have
to
get
done.
For
example,
if
you
want
http
related
functionality,
istio
needs
to
be
able
to
figure
out
that
you're
using
the
http
protocol.
B
You
need
to
prefix.
The
port
names
in
your
kubernetes
service
with
protocol
envoy
seems
to
do
some
http
response,
validation,
a
malformed
response
header
could
trigger
a
503
error
so
that
just
comes
down
to
testing
and
fixing
our
applications
are
monitored
with
prometheus
and
the
logs
go
to
elasticsearch.
A
B
B
A
A
B
A
A
A
A
A
A
B
B
B
The
configuration
scope
of
the
proxies
by
segmenting
the
mesh
more
about
that
in
a
moment,
if
you
use
the
horizontal
pod
autoscaler,
you
need
to
think
about
that
too.
It
works
by
summing
cpu
requests
for
all
the
containers
in
pods.
Adding
the
side
car
can
easily
skew
the
thresholds
at
which
auto
scaling
kicks
in.
So
you.
A
B
B
B
B
This
resource
lets
you
limit,
which
services
are
configured
as
destination
endpoints
in
the
client-side
load,
balancing
config,
so
you
can
use
it
to
segment
your
mesh
by
namespace
or
even
with
fine
grained
workload.
Selectors,
then,
the
proxies
are
only
programmed
with
config
for
the
destinations
that
they
need
to
know
about.
B
B
B
A
A
A
A
That's
a
problem
if
the
sidecar
isn't
ready
yet
native
support
for
sidecars
is
a
feature
that
the
kubernetes
team
is
currently
working
on,
but
due
to
the
many
prerequisites
it
might
take
some
time
for
the
particular
feature
to
appear
in
a
release.
Given
our
current
problems,
we
weren't
able
to
wait
that
out.
So
we
needed
a
temporary
solution.
B
B
Could
accept
both
mutual
tls
and
plaintext
connections,
allowing
both
regular
and
sidecar
enabled
services
to
communicate
with
them.
Initially,
we
thought
that
we
would
need
to
configure
which
routes
should
use
mutual
tls
and
then
a
feature
called
auto.
Mtls
appeared,
auto
mtls
uses
the
information
in
the
meshes.
A
B
Registry
and
will
automatically
use
mutual
tls
connections
wherever
they
are
supported
and
that
saved
us,
a
lot
of
extra
manual
configuration
hassle,
and
while
this
is
a
great
feature
to
allow
for
a
seamless
migration
at
some
point,
the
plan
is
to
enable
strict
mode
for
most
of
the
applications
to
ensure
security
of
in-flight
data.
On
all
of
the
connections
between
services,
there
are
all
sorts
of
applications
out
there.
We've
said.
A
B
A
A
A
B
Here's
a
final
summary
of
some
things
that
we
think
are
best
practices,
be
very
incremental,
prepare
and
educate
application
teams.
They
need
to
be
on
board,
understand
envoy
and
its
configuration
use.
The
istio
cni
plugin
for
injection
keep
a
careful
eye
on
data
plane.
Resource
usage,
avoid
config,
bloat
and
segment
with
the
sidecar
resource
and
allow.
B
We've
learned
a
lot
during
this
implementation
and
migration,
most
of
which
we've
already
mentioned
in
detail
during
this
talk.
But
one
last
thing
to
remember
is
that
service
measures
are
a
relatively
young
technology,
and
development
on
most
of
them
is
fast
paced
as
well.
So
you
should
be
prepared
to
keep
up
with
those
developments
in
order
to
make
the
most
out
of
your
implementation,
even
if
that
means
adjusting
your
plans
a
couple
of
times
along
the
way,
even.
A
Though
we
spent
quite
some
time
and
preparation
to
get
the
service
mesh
in
its
current
state,
we
realize
we're
only
at
the
beginning
of
the
journey
with
so
many
options
to
explore,
but
those
futures
adventurers
will
have
to
wait
for
another
cubecomp
presentation
at
a
later
date.
For
now.
I'd
like
to
conclude
this
talk
today
by
thanking
james
and
google
for
their
continued
support,
as
well
as
the
cncf
all
contributors
and
the
community
for
enabling
us
to
build
our
foundation
on
awesome
products
like
kubernetes,
envoy,
istio,
prometheus
and
the
open
policy
agent.