►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Service Mesh: There and Back Again - Hema Lee & Cody Vandermyn, Nordstrom
You might have heard about service mesh and its amazing benefits. Maybe you believe it’s the next big thing, but will it truly meet expectations? As any start to a relationship, things look fun and easy but once we talk performance at scale, compliance with internal security policies, and seamless onboarding, you might reconsider taking it home to meet your parents! With a highly distributed system that includes services running on Kubernetes clusters along with VM and Serverless workloads, vanilla service mesh would not work for us. In this talk, Hema & Cody will cover how Nordstrom’s relationship with service mesh evolved, what initial results revealed, what surprised us, and the open source contributions and adaptations we made to get to production readiness. We will share lessons learned and hopefully help with your service mesh relationship.
https://sched.co/UadC
A
Thank
you.
So
we
have
this
giant
reveal,
but
I'm
gonna
do
it
anyway.
So
in
the
land
of
distributed
micro
services,
how
can
we
secure
service
to
service
communication
in
a
highly
distributed
environment?
All
a
while
considering
system,
resiliency
and
reducing
our
developer
pain
points,
Cody
and
I
hope
to
highlight
this
in
Nordstrom's
journey
towards
service
mesh
there
and
back
again
morning.
Everyone
welcome
glad.
You
took
the
time
to
come
here
and
attend
our
talk.
A
My
name
is
hey
Molly,
as
Risha
introduced
us
and
I'm
a
senior
engineer
at
Nordstrom,
my
teammate
Cody
and
I
represent
the
networking
team
in
our
engineering
platform
organization,
a
team
devises
solutions
that
protect
the
flow
of
traffic
between
services
in
Nordstrom's
compute
environment,
a
little
bit
about
Nordstrom.
We
are
the
leading
fashion
retailer
that
was
founded
in
Seattle
in
1901
his
shoe
store.
Today
we
serve
our
customers
in
more
than
three
hundred
and
eighty
stores
in
40
states
in
the
US
Canada
and
Puerto
Rico.
A
In
fact,
we
recently
opened
our
New
York
City
flagship
store
in
midtown
Manhattan.
So
if
you
are
in
town
police
check
us
out
at
57th
and
Broadway,
so
our
tale
this
morning
evolves
from
why
we
considered
a
service
mesh.
The
investigations
we
considered
and
the
best
for
the
best
options
for
Nordstrom,
then
Cody,
will
walk
you
through
a
roll
up
and
some
of
the
gotchas
we
encountered
during
our
installation
which
led
to
some
contributions
to
the
community.
A
So
how
do
we
come
to
discuss
the
service
mesh?
Let's
look
at
what
a
service
mesh
is
one
of
the
most
common
definitions
that
I
found
states
that
a
service
mesh
is
a
dedicated
infrastructure
layer
for
facilitating
communications
between
micro
services
when
we
first
considered
a
service
mesh,
the
landscape
at
Nordstrom
looked
like
this.
Each
of
the
service
teams
owns
their
own
tech
stack
and
the
apps
were
built
with
a
variety
of
tools
and
languages.
A
So
what
else
can
we
accomplish
by
integrating
with
service
mash?
Our
list
of
internal
services
is
constantly
growing
to
be
able
to
fulfill
the
needs
of
our
customers,
our
technology
teams,
design
solutions
across
the
broad
range
of
store
and
online
services.
Additionally,
these
services
have
to
operate
seamlessly
by
our
mobile
app
as
well
as
our
website.
We
have
to
ensure
our
core
priorities
are
met.
So
what
are
our
core
priorities?
We
need
to
make
our
applications
and
services
highly
available,
so
they
can
serve
our
customers
and
employees
with
minimal
disruption.
We
cannot
just
been
accustomed.
A
Er
experience
by
taking
hours
to
bring
our
site
back
up
on
line.
Consistency
and
observability
is
also
key.
We
also
need
to
make
a
smooth
experience
for
our
developers
on
board
by
using
our
service
and
to
offer
them
access
to
metrics
and
logs,
so
they
will
not
have
to
configure
it
themselves.
We
need
to
offer
clear
benefits
so
that
our
developers
teams
want
and
asked
to
use
a
service
rather
than
be
forced
to
onboard
it's
a
necessity
that
we
can
handle
traffic
spikes
with
minimal
overhead
and
still
be
operational.
A
We
hold
two
big
events,
each
here,
our
anniversary
and
holiday
sale.
In
addition
to
these
events,
there
are
numerous
small
events
for
product
launches.
Latency
is
a
critical
metric,
especially
during
these
major
sales,
and
our
footprint
needs
to
be
small.
Last
but
not
least,
securing
communications
is
hard.
How
many
of
you
have
had
an
outage
due
to
an
expired
cert
you'll,
be
shy.
A
A
So,
what
environment
should
we
focus
our
priorities
on
when
we
started
down
this
path?
Last
year,
we
had
to
understand
our
what
these
environments
were
and
where
developers
like
to
deploy
their
services
to.
We
had
services
running
on
AWS,
ec2
instances
on
premises,
VMs
server
lists
functions
and
in
our
kubernetes
clusters
on
both
AWS
and
gke.
To
give
you
an
idea,
we
had
about
six
thousand
680
total
pods
in
both
of
our
production
kubernetes
clusters.
A
We
averaged
about
3,000
lambda
functions,
daily
7000
production
VMs
in
our
on-premises
servers,
and
we
had
about
6,000
production,
ec2
instances
running
with
numbers
like
these.
We
couldn't
focus
our
efforts
in
all
of
these
environments.
Our
mindset
was
to
ultimately
craft
a
solution
that
would
work
for
our
services
that
run
on
the
cloud
as
well
as
on-premises,
specifically
in
our
kubernetes
clusters.
A
We
anticipated
that
the
majority
of
our
services
in
the
future
will
be
deployed
indicates
to
give
you
an
idea,
our
during
our
anniversary,
sale
this
past
July,
we
scaled
up
to
running
more
than
12,000
pods
and
36
terabytes
of
RAM
in
first
30
minutes.
In
both
our
production
Cades
clusters,
we
had
one
service
alone
scale
to
running
3000
pods.
These
numbers
are
telling
us
that
implementing
for
the
cluster
was
first
was
the
right
path.
A
So
how
do
we
go
about
finding
a
solution?
We
considered
a
wide
variety
of
options,
including
spending
months,
designing
and
building
our
own
solution
from
scratch,
but
why
we
engineer,
what's
already
been
created,
so
we
scrapped
our
solution
and
looked
the
options
that
we
could
either
buy
or
looked
towards
the
open
source
community
to
use
an
open
source
project
would
allow
us
to
quickly
reliably
and
economically
implement
a
solution
to
help
serve
our
customers
better.
To
use
an
open
source
software
would
also
integrate
well
into
our
development
environment.
A
It
would
also
give
us
engineers
a
chance
to
contribute
and
collaborate
with
teams
outside
of
Nordstrom,
but
we
did
have
other
things
to
consider.
Besides,
how
performant
and
secure
a
project
is
being
a
large
enterprise?
We
needed
to
look
at
the
number
of
contributors,
the
community
support
for
the
project
and
the
engineering
teams
ability
to
work
with
us.
In
the
end
we
chose
the
open
source
route
as
one
of
our
company's
company's
values
is
to
contribute
back
to
the
community,
and
this
gives
us
engineers
a
chance
to
have
that
sense
of
accomplishment.
A
So
what
do
we
have
on
the
table?
Among
the
options
that
we
looked
at
where
sto
Aspen
mesh
and
Link
Rd?
It
was
great
to
be
able
to
work
with
the
engineering
teams
of
each
of
these
products
and
the
help
we
received
as
we
refined
our
configurations
and
testing
parameters.
But
ultimately
they
were
all
good
choices
for
us,
but
we
chose
linker
D,
so
link
Rd
is
described
as
an
ultra-light
service
mesh
for
kubernetes.
It
gives
you
observability
reliability
and
security
without
requiring
any
code
changes
the
performance
tests
we
conducted
here.
A
What
for
scalability
and
load
testing
our
benchmark?
We
used
a
benchmarking
tool
called
work
to
through
which
we
looked
at
two
use
cases,
an
above
average
daily
workload
and
an
anniversary
load.
The
latency
numbers
that
you
see
here
are
for
p90.
The
test
for
the
above
average
daily
load
were
run
at
500
requests
per
second
at
a
thousand
concurrent
connections,
resulting
in
a
latency
of
three
point:
seven,
zero
milliseconds.
A
A
So
who
gets
to
hold
the
keys
to
this
project,
since
our
team's
main
Charter
is
to
secure
communications
between
all
of
nordstrom
services,
whether
they
hosted
on-premises
in
AWS
or
GCP
instances
or
in
our
company's
KH
clusters.
We
should
be
involved
in
some
way.
I
think.
However,
the
logical
choice
to
lead
the
effort
would
be
arcades
team.
They
are
the
dedicated
experts,
laughs
tasked
with
maintaining
and
supporting
all
things
related
to
the
cluster,
but
with
so
many
features
available
that
can
extend
the
kubernetes
based
platform.
A
B
Thank
You,
Hema
and
hello
everybody.
My
name
is
Cody
and
I'm,
one
of
the
Norseman
engineers
responsible
for
implementing
our
service
mesh
solution
over
the
past
year.
We've
just
heard
why
we
want
to
use
a
service
mission
now
I'm
going
to
describe
how
my
team
managed
to
get
our
service
mesh
solution
production
ready.
B
So,
although
norstrom
has
been
using
kubernetes
for
years
now,
we
were
one
of
the
early
adopters
of
kubernetes.
My
team
was
most
experienced
with
designing
front
end
applications
for
mobile,
so
most
of
my
team
was
brand
new
to
the
kubernetes
ecosystem
and
we
found
ourselves
in
the
position
of
needing
to
learn
as
much
as
we
could
as
fast
as
we
could.
So
the
first
thing
we
did
was
go
to
the
kubernetes
website
and
read
it
about
what
is
kubernetes.
B
So
the
description
you
see
here
has
a
lot
of
keywords
that
I've
highlighted
that
stood
out
to
us
as
developers,
so
it's
portable
extensible
open-source
declarative
and
can
automation
so
given
nor
some
situation.
This
description
appealed
to
us.
We
had
been
breaking
up
large
monolith
services
for
years.
B
So
we've
learned
that
kubernetes
is
a
platform
for
managing
containers.
So
now
what
we
had
to
do
is
figure
out
what
a
container
was,
and
once
we
learned
about
containers
when
you
get
to
know
a
bit
about
the
kubernetes
networking
model
and
on
and
on
and
on,
and
it
felt
like,
we
could
get
lost
in
all
these
technologies.
B
So,
in
addition
to
these
various
technologies
that
we
needed
to
learn,
there
are
a
few
other
dragons.
We
need
it
to
slay.
We
had
to
learn
and
become
familiar
with
new
terminology.
Things
like
pods
deployments.
Replica
sets
cluster
rolls
web
hooks
even
words
that
we're
already
used
to
using
like
services
and
endpoints
were
being
used
in
a
new
way.
For
these,
the
kubernetes
documentation
we
found
is
really
well
done
and
it
helped
us
learn
immensely
as
we
were
going
through
this
process.
B
Finally,
we
needed
to
figure
out
how
we
could
deploy
test
and
development
clusters
so
that
we
didn't
bring
down
anything
in
production,
so
we
started
with
mini
cube,
as
most
people
do
and
while
mini
cube
is
certainly
great.
We
needed
something
that
was
more
closely
related
to
our
actual
environment
that
we're
going
to
be
deploying
to
so
we
needed
to
work
with
our
kubernetes
team
to
figure
out
how
we
can
run
these
clusters
in
our
AWS
account
at
nordstrom.
B
B
Also,
we
had
cost
concerns,
so
we
were
going
to
be
spinning
up
new
clusters
in
our
accounts,
and
there
were
some
concerns
about
how
much
that
would
cost.
We
ended
up
figuring
out
that
we
could
spin
up
and
down
clusters
on
a
daily
basis,
so
we
could
keep
those
costs
down.
Our
existing
kubernetes
team
has
a
ton
of
tooling
already
in
place
to
deploy
their
development
clusters
to
their
accounts,
but
we
needed
to
work
with
them
closely
to
adapt
their
tooling
for
our
needs.
B
So
now
that
we've
come
up
to
speed
with
kubernetes
it's
time
to
install
and
run
the
service
mesh,
but
first
we're
going
to
talk
a
little
bit
about
how
the
linker
d--
works.
You
can
see
here
an
architecture
diagram
that
shows
the
various
components
that
make
up
linker
D
on
the
top.
You've
got
the
control
plane,
it
has
components
for
things
like
aggregating,
telemetry
and
metrics.
It
provides
a
user
facing
vi4,
their
dashboard
and
the
command-line
tools
and
also
there's
a
service.
B
The
data
plane
at
the
bottom
consists
of
the
site
card
proxy
that
is
injected
into
each
kubernetes
pod
in
the
service
mesh.
The
sidecar
proxy
injection
typically
includes
an
init
container
that
is
responsible
for
writing
the
IP
tables
for
the
pod.
These
rules
are
configured
so
that
the
linker
de
proxy
can
intercept
all
the
outgoing
traffic
and
incoming
traffic,
and
this
is
how
they're
able
to
add
their
features
like
MT,
LS
traffic,
observability,
load,
balancing,
and
things
like
that.
B
B
So
first
we're
going
to
run
the
install
link
or
D
command
and
we're
going
to
see
what
happens
when
we
do
that.
So
we
see
some
services
being
created.
We've
got
the
identity
service
being
created
this
great
weave.
This
is
a
service.
That's
responsible
for
issuing
certificates,
we're
going
to
see
some
more
services
created.
We've
got
the
controller
and
the
proxy
injector
that's
going
to
help
us
inject
our
proxies
and
a
whole
bunch.
More
resources
were
created
about
45.
B
So
this
was
our
first
problem
here.
It
says
no
resources
found,
although
it
just
told
me
that
it
created
about
48
resources,
so
we
need
to
do.
We
need
to
figure
out
what's
going
on
so
we're
gonna,
look
at
our
deployments
and
sure
enough.
None
of
our
deployments
are
ready,
so
we
need
to
drill
down
and
figure
out.
Why
so
we're
gonna
take
a
look
at
our
replicas
sets
and
our
deployments
to
see
if
there's
any
kubernetes
events
that
can
help
us
figure
out.
B
B
B
B
So
over
time,
our
kubernetes
team
has
put
in
place
security
measures
than
include
pod
security
policies,
LDAP
access
control
and
the
principle
of
least
privilege.
These
measures,
along
with
many
others,
secure
a
cluster
for
the
hundreds
of
service
teams
using
it.
So
our
first
challenge
was
realizing
that
we're
not
cluster
administrators
and
really
this
shouldn't
have
been
a
surprise.
We
should
have
known
this,
but
it
was
a
surprise
for
us
at
the
time.
We
also
don't
want
to
be
cluster
administrators.
B
We
want
to
keep
the
closer
administrators
the
count
to
a
minimum
due
to
the
types
of
data
present
in
the
cluster
and
the
types
of
communication
between
our
services,
so
not
being
administrators
means
that
we
can't
just
create
cluster
roles
and
modify
policies
in
order
to
make
our
service
mesh
work.
So
if
we
need
a
cluster
wide
resource
or
if
we
need
to
modify
something,
we
need
to
talk
with
our
kubernetes
team.
B
We
need
to
reach
out
to
them
and
we
need
to
tell
them
what
they
need
or
what
we
need,
and
they
help
us
get
what
we
want.
They
ask
us
some
very
tough
questions
and
they
look
over
every
line
of
llamó
that
we
submit
and
they
ensure
that
we're
only
getting
what
we
asked
for
and
what
we
need.
The
security
of
our
cluster
is
a
top
priority,
so
the
second
challenge
was
figuring
out
what
to
do
with
this
pod
security
policy.
B
A
pod
security
policy
is
one
way
that
a
cluster
administrators
have
controlled
our
security
requirements
for
the
they
can
control
things
like
port
numbers
that
are
allowed
user,
IDs
and
specific
linux
capabilities
that
are
allowed.
So
the
errors
we
were
seeing
meant
that
we
are
in
violation
of
this
pod
security
policy,
specifically
that
must
run
as
non-root
property
and
the
request
for
net
admin
and
that
raw
capabilities.
B
B
Captain
net
admin
and
this
capability
allows
users
to
administer
the
firewall,
modify,
routing
tables
bind
to
any
address
and
generally
other
things
that
shouldn't
be
allowed
in
a
multi-tenant
cluster
for
most
users,
if
the
user
had
this
capability
and
they
wanted
to,
they
could
likely
inspect
all
traffic
on
from
coming
in
and
out
of
the
pod
or
even
on
the
node.
So
there's
a
very
good
reason
why
our
team
has
denied
this
capability
for
us
so
because
the
controls
are
linked
to
service
accounts
and
kubernetes
and
not
pods.
B
B
So
to
do
that,
we
worked
with
a
vibrant,
open
source
community
surrounding
linker
D.
We
chatted
with
the
link
of
the
engineers
and
our
own
kubernetes
cluster
team
to
figure
out
a
solution
that
would
work
for
us.
So
during
our
discussions
it
was
suggested
that
a
CNI
plugin
would
be
one
way
of
solving
our
issue.
This
would
allow
us
to
run
a
custom
binary
whenever
a
pod
is
created
and
that
binary
would
be
able
to
write
the
IP
tables
rules
for
that
pod,
because
the
CNI
plugin
process
runs
on
the
note
itself.
B
There
are
no
special
capabilities
that
you
need
to
give
a
service
account
and
the
pods
no
longer
need
to
ask
for
net
admin
or
net
raw.
So
the
other
appealing
feature
of
the
CNI
plugin
is
that
we
can
chain
them
together,
and
so
our
new
plugin
would
be
able
to
move
in
to
where
the
other
existing
plugins
that
we
had
in
place.
B
So
we
went
for
it
and
created
the
cni
plugin
I,
don't
know
how
many
of
you
have
ever
tried
to
Krita
C&I
plugin,
but
for
us
I
remember
this
particular
story
when
we
were
just
simply
trying
to
verify
that
what
we
had
written
that
was
going
to
work
and
because
you
know
we're
running
in
a
cluster
on
a
remote
machine.
The
typical
application
developer
tools
like
d
bugs
and
breakpoints
of
stuff
weren't
available
to
us
as
far
as
we
could
tell
so,
we
had
to
figure
out
another
way.
B
It
turns
out
that
CNI
plugins
communicate
via
the
standard
in
standard
out
using
J,
a
specific
JSON
payload,
and
what
we
were
doing
was
putting
our
print
statements
into
the
standard
out
and
that
wasn't
valid
JSON.
So
our
plugins
down
the
chain
we're
not
working
now.
It
was
interesting
because
we
finally
figured
this
out,
but
it
you
know
if
you,
if
we
would
have
read
the
documentation
for
sinaia
plug,
is
it's
all
right
there.
It
says,
use
standard
error
so
when
we
decided
to
use
standard
eret,
everything
was
working
just
fine
after
all.
B
B
So
there's
just
one
more
issue
for
us
regarding
the
scene
I
plug
in,
and
that
was
how
to
get
it
deployed
on
our
nodes.
So
our
kubernetes
cluster
knows
come
up
and
down
constantly
all
the
time,
because
our
kubernetes
team
is
doing
upgrades
and
maintenance.
Often
so
because
of
this
there's
a
very
small
chance
that
when
a
pod
comes
online,
it
will
give
scheduled
on
a
new
node
and
it
has
a
linker
D
proxy.
But
that's
before
the
CNI
plugin
has
written
the
IP
tables.
B
So
that
means
even
though
the
pot
had
only
could
be
proxy.
It
would
not
be
a
part
of
the
service
mesh,
so
we
had
to
figure
out
a
way
to
work
around
this,
and
so
because
we
couldn't
use
the
typical
daemon
set
installation
that
you
would
usually
see
with
the
CNI
plugin.
So
it
took
some
more
time
talking
with
the
kubernetes
team
and
we
were
able
to
basically
bake
the
the
plugin
directly
onto
our
base
image
for
our
notes.
B
And
so
finally,
we
had
a
solution
that
would
allow
us
to
install
linker
D
onto
our
clusters,
but
we're
still
not
quite
done.
There
are
two
more
security
requirements
that
we
had
to
meet.
We
had
to
be
able
to
audit
all
of
the
certificates
that
our
service
mesh
was
using,
and
we
also
had
to
make
sure
that
they
were
all
signed
by
the
Nordstrom
certificate
authority.
B
So
remember
that
link
Rd
has
that
identity
service
that
is
responsible
for
creating
and
issuing
certificates
to
the
proxies,
since
that
service
typically
handles
all
the
certificate
creation
and
generation
automatically
using
its
own
self
signed
certificates.
We
couldn't
meet
these
requirements
out
of
the
box
and
we
needed
to
figure
out
a
way
to
get
these
certificates
from
our
Nordstrom
certificate
authority
that
we
already
had
in
place.
B
So
we
chatted
with
the
link
OD
engineers
some
more
and
we
decided
to
go
ahead
and
code
up
a
solution
for
us
that
would
have
the
link
Rd
identity
service,
reach
out
to
the
Amazon
private
certificate
authority.
We
had
to
get
those
certificates
and
we've
been
using
the
solution
for
a
few
months
now,
and
it
works
great
for
us.
It
turns
out
that
this
particular
approach
is
it
actually
going
to
be
merged
into
the
upstream
code
and
that's
okay.
B
The
link
for
the
engineers
are
going
to
use
another
open
source
project
sort
manager
to
use
for
custom
certificates
and
I
think
overall
for
the
community.
You
know
this
will
be
a
better
solution,
because
we'll
be
able
to
support
more
than
just
our
particular
AWS
certificate
authority.
It
also
means
that
we
are
itching
to
contribute
back
to
any
cert
manager
pour
request
or
issue
that
needs
our
help
and
so
stay
tuned
for
that
work.
B
So
now
that
we've
implemented
their
changes
and
installed
the
linker
DCI
plug-in
on
our
cluster
notes,
let's
check
out
our
service
mesh
pods.
Well,
would
you
look
at
that?
All
of
the
link
of
the
pods
are
running
and
our
service
mesh
is
operational,
so
I'm
going
to
bring
him
Amy
back
on
to
wrap
it
all
up.
A
So
how
do
we
do
in
our
rollout?
We
had
challenges
that
I
think
any
enterprise.
Might
we
spend
a
lot
of
time
understanding
the
problem
space
and
the
tools
we
needed
to
integrate
our
services
into
the
mesh?
Even
though
we
chose
linker
D,
this
may
not
be
the
right
option
for
you.
We
went
through
a
lot
of
iterations
vetting
are
designed
with
other
teams
in
our
security
and
infrastructure
organizations.
A
We
also
encountered
problems
as
a
result
of
our
team,
not
being
cluster
administrators.
Installing
in
our
cluster,
was
not
easy,
but
it
gave
us
a
chance
to
collaborate.
The
filling
cardi
team,
which
resulted
in
contributions
towards
a
CNI
plugin
to
benefit
the
community.
Our
service
onboarding
procedure
surprised
our
development
teams.
Engineers
came
in
expecting
to
spend
a
full
30
minutes
and
to
make
a
ton
of
code
changes
they
when
they
were
done
in
the
first
ten
and
they
added
in
those
two
lines
of
code
that
you
see
up
there.
A
A
We
still
have
a
long
journey
ahead
to
make
this
a
true
service
mesh
by
integrating
all
of
our
environments
and
continuing
our
service
onboarding.
We
thank
you
for
coming
and
open
it
up
to
any
questions
you
may
have
about
our
initial
process,
the
challenges
we
encountered
and
how
we
made
it
enticing
to
our
developers.
Thank
you.
C
You
have
any
information
on
the
CPU
impact
of
using
linker
D.
One
of
the
reasons
we've
been
reluctant
to
move
down
the
ServiceMaster
path
is
some
anecdotes.
We've
heard
about
extremely
high
CPU
requirements.
If
you
have
high
transactions
per
second.
Has
that
been
your
experience
with
link
or
D?
What
kind
of
impact
has
it
had
for
you.
B
D
B
E
Again
great
presentation:
we
have
a
very
similar
situation
where
teams
are
using
a
wide
variety
of
technology
stacks.
So
one
of
the
challenges
we
faced
is
communicating
the
value
of
doing
this
kind
of
work
to
our
engineering
leadership.
So
what
was
what
was
most
effective
for
you
and
you
were
pitching
this
work
to
get
it
funded
and
then,
as
a
like
a
two-parter.
What
was
the
relative
size
of
the
team
that
actually
ended
up
implementing
this
solution?.
A
So
initially
I
think
our
main
selling
point
was
securing
service
to
service
communication.
That's
how
we
went
about
it
and
and
our
team
size
right
now
we're
about
six
to
seven
engineers
on
our
team
and
then
I
think
it's
an
additional
six
to
seven
engineers
on
the
kubernetes
team.
So
we
try
to
collaborate
with
them
to
learn
as
much
as
possible
so
that
we
can
actually
make
this
work.
A
So
that
was
one
of
our
primary
reasons.
We,
our
kubernetes
team,
had
given
us
a
set
of
requirements
for
it,
so
they
were
saying.
Please
me,
try
and
meet
these
requirements,
so
that
was
essentially
what
we
worked
off
of
and
at
the
time
when
we
tested
these
linker
D
worked
really
well
for
difficult
and
the
resources
that
we
had
so
a.
G
Great
presentation,
one
question:
two
questions:
how
do
you
allow
services
to
opt
out?
What
kind
of
considerations
goes
in
because
once
you
have
two
services
in
the
service
mesh,
do
you
always
open
a
non
mesh
path
as
well,
and
the
second
one
is
when
you
have
to
upgrade
a
service
mesh
I'm
assuming
your
team
is
the
mesh
operator
for
the
developer.
So
what
kind
of
considerations
have
you
tested
it?
What
kind
of
challenges?
If
any
you
have.
B
So
the
ability
to
opt
out
really
for
us
is
only
going
to
be
viable
during
the
migration
period.
You
know
we,
you
know,
we've
got
our
sales
that
happen
and
holiday.
Sale
is
coming
up
and
things.
So
it's
important
that
as
we're
migrating
to
all
in
that
teams
can
also
choose
out.
So
that
does
mean
that
you
know
there's
a
path
between
services
that
is
not
in
the
mesh.
So.
H
G
B
So
I
mean
for
us:
it's
been
really
straightforward
and
simple,
with
linker
D,
they
provide
these.
You
know
dashboards
and
things
that
we
can
show
teams.
We've
also,
you
know
used
our
own
metrics
and
things
that
we
have
dashboards
for
they're
really
so
far,
I
haven't
been
to
many
challenges
that
we've
had
to
overcome.
As
far
as
teams
using
a
service
mesh,
great.
B
B
Perhaps
yeah
so
I
know
that
the
fork
that
we
have,
we
don't
really
want
to
maintain
super
long
term.
So
we
want
to
eventually
do
something:
that's
going
to
benefit
everybody
and
get
back
into
the
main
line.
So
what
probably
happened
is
as
linker
D
cert
manager
support
grows
will
probably
create
a
cert
manager
issuer
for
AWS,
a
CMP
CA,
and
that
will
be
the
way
that
we
contribute
into
the
broader
community.