►
From YouTube: Rapid Experimentation Simplified with Linkerd - Alexander Simon Jones, DevOps Institute Ambassador
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Rapid Experimentation Simplified with Linkerd - Alexander Simon Jones, DevOps Institute Ambassador
The cloud engineering team of a multinational financial services corporation used Linkerd to introduce rapid experimentation.
It enabled them to create more resilient services and comparatively test changes. In this session, we'll explore how they did it and lessons learned.
With the proliferation of microservices architectures, developers need to rapidly set up prod-like dev environments locally.
Fortunately, local Kubernetes clusters to model high-scale, complex architecture within an enterprise are increasingly common.
This enables engineers to work with local abstractions of much larger systems.
Within this paradigm, service meshes also need representation locally as it has intrinsic differences to running a vanilla network layer within Kubernetes.
To speed up and ease experimentation, the service mesh must be simple and easy to use. A positive developer experience is key.
A
So
the
agenda
for
today
is
we're
going
to
talk
about.
Why
is
there
a
need
for
experimentation?
Why
do
firms
invest
in
tooling?
And
why
are
things
like
linker
d
becoming
extremely
exciting
for
very
low
bar
to
entry,
ways
of
measuring
those
experiments
and
testing
hypotheses
the
apparatus
of
that
experimentation?
The
technical
side
of
how
is
this
implemented?
How
difficult
is
it
to
use,
and
what
are
the
kind
of
things
I
can
do
you
know
is
that
a
b
testing
is
that
chaos
testing?
Is
that
gonna
be
canary
testing?
A
A
Let's
say
that
you're,
a
product
engineering
team
and
you
are
rolling
down
the
tracks
building
out
these
version
changes
now.
What
happens
along
here
is
that
we
start
to
see
our
sre
team
are
telling
us
that
hey
the
latency
of
your
application
is
increasing
over
time.
This
is
a
very
coarse-grained
approach
of
understanding
the
infrastructure
footprint
impact
of
our
application
right,
whether
that's
compute,
whether
that's
manifested
as
iops
or
some
other
signal.
A
We
are
making
a
change
in
the
environment
which
is
going
to
cause
additional
outcomes
over
time
in
terms
of
how
that
microservice
or
application
interacts
with
other
systems,
and
so
therefore,
it
makes
a
lot
of
common
sense
that
we
want
to
measure
scientifically.
What
is
the
delta
of
change,
not
just
in
code,
but
also
in
performance
right,
and
so
I
think
about
this
in
terms
of
reduced
signals
as
well
and
I'll
come
on
to
describe
what
I
mean
by
that
in
a
moment.
A
A
secondary
example
is
how
does
that
application
or
service
interact
in
a
complex
environment?
If
we're
changing
the
version
of
several
micro
services,
how
do
we
know
scientifically
what
the
change
will
be
to
a
queuing
mechanism,
and
how
do
we
understand
the
knock-on
implication-
and
I
think
this
shows
also
that
there's
a
real
need
to
be
able
to
not
only
inject
faults
but
to
understand
when
there's
service
degradation,
how
does
the
environment
perform?
A
There's
often
a
joke
that
you
know
the
disaster
recovery
plan
is
nothing
like
what
it
looks
like
when
you
actually
have
to
perform
it.
That's
because
that
it's
so
high
cost
in
many
organizations
to
actually
perform
a
dr
exercise
that
many
of
these
things
are
just
existential
and
aren't
really
testable.
A
Another
part
of
this
as
well
and
illustrated
by
the
previous
two
examples,
is
that
a
b
testing
should
be
easy
right.
This
idea
of
having
some
change
and
being
able
to
test
it
should
be
easy.
The
problem
with
this
sort
of
faz,
based
example
here,
is
that
between
these
two
functions
essentially
performing
a
failover
right,
I
have
to
test
the
new
optimized
database
table
against
a
new
function
and
then
fail
back
again,
and
this
could
even
be
a
code
change
right.
A
This
could
be
an
optimization
in
the
code,
not
necessarily
the
database,
but
the
idea
is,
you
have
to
manually
change
the
service,
whether
that's
through
automated
deployment,
config
or
some
other
human
activity.
It's
not
something
you
run
simultaneously
equally,
this
becomes
even
more
compounded
when
you
want
to
run,
say
20
to
30
faz
changes
with
many
small
nuances
between
them.
So
a
b
testing
needs
to
be
easy,
and
this
is
a
really
really
difficult
problem
to
solve
for
at
enterprise
in
a
safe
and
scalable
way,
but
wait
a
minute
right.
A
Many
people
will
tell
me
hey
well
in
our
environment,
we
can
deploy
multiple
versions.
We
have
no
problems
whatsoever,
we
can
have
you
know
1.1
1.2,
et
cetera,
on
branches,
prs,
yada
yada,
that's
fine,
but
let's
really
break
that
down
right.
So
you
have
your
micro
service,
alpha,
v1
and
v2.
The
delta
change
is
just
the
code
and
we
go
through
the
typical
kind
of
routine
of
you
know
committing
that
code
change.
We
deploy
that
through
a
pipeline.
A
We
get
a
new
replica
set
yeah,
that's
great,
but
how
do
we
then
see
what
changes
there
are
between
these?
We
have
to
do
some
sort
of
activity
where
we
observe
the
prior
state
and
then
look
at
the
new
state,
and
so
we
can
look
at
a
pattern
of
change
and
determine
that.
There's
been
a
regression:
okay,
then,
if
there's
a
regression,
we
have
to
go
back
to
the
drawing
board
and
we
have
to
figure
out
what
that
regression
is.
That
could
be
latency
that
could
be
saturation
some
other
penalty.
We
have
to
pay.
A
A
This
is
a
complex
and
often
impractical,
set
of
ideas
to
try
and
bring
across
to
an
organization
where
they
don't
know
where
to
start
right,
there
might
be
an
application
team
who
are
delivering
20
different
micro
services
and
each
one
of
those
micro
services
might
have
a
bunch
of
branches
with
a
bunch
of
changes
in
each
branch.
How
do
you
determine
which
branch
is
introducing
a
regression
in
terms
of
infrastructure
performance?
A
A
It
stands
to
reason
that
we
should
be
able
to
test
in
real
time
an
alternative
vision
to
this
architecture
and
in
this
case
it's
direct
calling
the
database
right.
So
this
is
fairly
well
known
and
well
trodden
this
kind
of
path,
but
we
find
it
difficult
to
do
because
it's
hard
to
be
able
to
tell
the
api
gateway
to
send
data
to
both
of
these
without
a
code
change
in
the
gateway
and
again
that
is
introducing
more
change.
A
Thinking
back
to
our
disaster
recovery
illustration
how
we
can
start
to
bring
failures
and
chaos
into
the
system
and
build
more
resilient
systems
and,
lastly,
observing
the
difference
across
generations
is
really
paramount
to
this
succeeding.
We
can
do
all
this
stuff,
but
if
we
can't
observe
it
in
a
way
that
isn't
super
coarse
grain,
then
it's
pointless.
How
do
we
bring
rate
errors,
duration,
utilization,
saturation
signals
all
to
the
table
and
say
between
the
version?
One
and
three
there's
a
massive
regression
right.
A
These
are
things
that
we
need
to
be
able
to
understand
and
understand
how
to
measure.
So
that
brings
me
to
the
apparatus
of
this
experimentation
and
really,
after
looking
at
a
lot
of
different
solutions.
What
we
settled
on
time
and
time
again
was
linker
d,
the
two
key
tenets
of
this
traffic
splitting
and
observability,
both
of
which
are
underpinned
by
super
easy
to
use
dx
that
time
and
time
again
have
saved
us
a
ton
of
effort
by
just
working
out
of
the
box
when
we
think
about
especially
how
these
things
work.
A
It
reminds
me
that
there's
a
lot
of
effort-
that's
been
put
in
at
the
smi
spec
level,
from
the
cnc
f6,
who
are
caring
about
the
future
of
these
kind
of
implementations
and
how
the
end
users
are
going
to
work
with
them
and
that's
very
much
appreciated,
because
when
we
look
at
the
customer
resource
definition
for
how
a
traffic
split
should
work,
it's
super
easy
to
understand
that
in
this
example,
there's
90
traffic
balance
to
the
v1
versus
10
on
the
v2.
Equally,
the
alpha
v4
of
traffic
splitting
is
taking
us
in
a
direction.
A
And
so,
when
you
have
super
crisp,
dashboarding
and
visualizations
of
what's
going
on,
then
it
makes
everyone's
lives
easier.
And
this
is
great
because
you
can
double
click
into
this,
so
that,
if
you
are
an
engineer
that
wants
to
understand
like
hey,
what's
going
on
with
the
response
codes,
what's
going
on
with
like
the
internal
host
headers,
you
can
do
that
within
the
mesh
within
the
kd.
A
Equally
for
sres
there's
that
super
deeply
ingrained
prometheus
and
grafana
installation
that
lets
you
be
a
bit
more
scientific
over
time.
It's
one
thing
just
to
deploy
a
service
and
say
yeah.
Okay,
it's
introduced
some
latency,
it's
another
thing
to
then
start
pushing
a
load
into
that
service
and
looking
at
how
it
performs
you
know
compared
to
its
prior
generations.
A
So,
let's,
let's
look
at
a
small
demo
now,
so
I've
got
this
linkedin
demo
repository
right
here.
I've
got
a
client
that
calls
a
version,
and
that
version
is
the
one
two
or
three
right,
and
so
what
this
client
does
is
it
creates
a
user.
The
user
will
then
just
sit
in
memory
now.
The
difference
between
one
two
and
three
is
pretty
small,
but
I'll
show
you
so
version.
A
Two
I've
changed
the
swagger
spec,
and
this
is
so
that
we
can
represent
some
code
change
that
an
engineer
might
make,
and
in
this
case
it's
adding
a
required
food
preference
field.
Now
my
client
is
super
dumb
and
all
it's
doing
is
hitting
with
the
default
user
field.
So,
what's
going
to
happen,
is
I'm
going
to
get
a
422
because
it's
going
to
say
hey,
I
can't
process
this.
I
don't
understand
where's
my
food
field
equally
in
the
open
api
v3.
What's
going
to
happen
is
that
I've
introduced
some
latency
right.
A
I've
just
put
some
times
dot,
sleeps
all
over
the
code
so
that
we
can
sort
of
emulate
what
would
happen
if
there
was
a
real
real
sort
of
regression
introduced
into
that
service.
And
what
brings
this
all
together
is
the
traffic
split,
so
the
default
behavior
of
the
open
api
client
is
to
hit
this
v1.
But
what
we're
saying
now
is
hey.
Actually
I
want
to
balance
equally
between
the
v2
and
the
v3.
So
let's
go
ahead
and
deploy
that.
A
Cool,
so
I
want
to
visualize
this
naturally.
So,
let's
go
link
to
the
there's
dashboard,
and
if
we
go
into
here,
we
can
see
the
default
behavior
as
expected.
Open
open
api
client
is
hitting
the
v1
back
end.
We
can
see,
though,
that
the
traffic
splits
has
come
online
and
we
have
some
prior
data
for
the
existing
service,
and
what
will
soon
happen
is,
as
the
live
data
starts
to
come
through
from
the
new
routes.
A
We'll
see
these
fields
start
to
get
populated,
so
you
can
see
ones
come
up
right
there
and
what's
really
really
exciting
and
useful
about
this,
is
that
it
just
works
right,
there's
no
additional
configuration.
It
was
just
what
you
saw.
I
just
applied
that
crd
and
away
we
go,
and
now
what
I'm
interested
to
see
is.
A
In
addition
to
this,
I
could
say:
okay:
well,
what's
the
what's
the
history
of
this
v3
api
and
again
clicking
through
to
grafana
it's
great,
I
can
look
and
see.
Oh,
you
know
what
I
can
see
over
the
past
couple
of
minutes.
This
is
you
know
something
that
I've
known
to
be
a
behavioral
change
in
this
service,
or
this
is
something
that's
absolutely
fine.
I
love
the
idea
that
you
can
combine
this
with
existing
dashboards
right.
A
So
if
I'm
working
in
a
compute
constrained
environment
and
I'm
deploying
a
hundred
different
micro
bets,
I
can
have
a
default
dashboard
that
I
add
in
that
focuses
purely
on
compute,
so
that
when
I
have
all
these
different
services-
and
if
I
refresh
now,
I
should
see
them-
I
can
tell
which
one
is
going
to
be
the
most
performant
change
right,
and
so
these
three
could
represent
three
different
hashing
algorithms.
You
know
it
could
be
anything
that
you
want
to
try
to
test
in
a
very
low
barter
entryway.
A
Well,
you
know
we're
essentially
creating
an
experiment
factory
we're
creating
the
ability
for
developers
on
their
local
machine
in
their
lower
environments,
wherever
they
might
be
to
put
a
load
of
different
small
changes
out
there
and
test
the
ones
that
work
the
best
and
when
you
start
cascading
this
to
not
just
a
single
micro
service
but
multiple
micro
services,
then
it
gets
really
exciting,
because
at
that
point
you
can
start
combining
it
with
chaos
testing.
So
it's
almost
like
an
evolutionary
darwinism
of
microservices
where
you're
starting
to
see
which
one
survives
the
best.
A
So
in
this
example,
let's
say
alpha,
v1
and
v2
actually
have
different
roots,
but
what
we
can
see
is
that
the
implication
of
those
roots
can
be
quite
significant
in
terms
of
which
is
more
resilient,
and
so
we
might
find
that
you
know
what
actually
alpha
v2.
If
we're
knocking
it
down.
It
impacts
the
the
beta
v1
service.
A
This
means
that
the
candidate
releases
that
actually
go
forward
to
production
environments
are
innately
more
stable
because
there's
been
a
consideration
around
the
signals
of
the
infrastructure
to
which
they're
measuring
now,
obviously
there
might
be
the
argument
of
hey.
Well,
you
know,
deploying
to
a
lower
environment,
isn't
completely
representative
of
a
higher
environment.
Well,
that's
where
the
adventure
gets
really
really
exciting
because
you
can
start
dipping
into
those
higher
environments,
especially
if
you
federate
your
mesh
across
clusters.
A
I
feel
like
there's
a
super
bright
future
for
this
sort
of
stuff
and
I
hope
that
you've
enjoyed
this
talk.
There's
so
many
questions
that
I
have
yet
to
answer,
and
so
many
things
we
can
get
to
talk
about,
and
if
anybody
wants
to
follow
me
offline,
we
can
talk
and
chat
about
all
this
stuff,
but
ultimately
watch
this
space,
because
canary
releasing
a
b
testing
and
chaos.
Testing
are
all
completely
possible
within
lincoln
d
and
are
all
being
done
today
in
hundreds
of
companies
with
a
lot
of
success.
Thank
you.