►
From YouTube: Dynamically Testing Individual Microservice Releases In Production - Matt Turner, Tetrate
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from April 17-21, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Dynamically Testing Individual Microservice Releases In Production - Matt Turner, Tetrate
Speakers: Matt Turner
A lot of us test new versions of services in our Production environment, since it's the best way to get representative, reliable results. If the new service is "on the edge" of the topology then hitting it is easy, as the test clients can directly call it. But if it's in the middle of a chain of services, then calling the current versions of all of them, except one beta version in the middle of the chain, is the dream. This kind of advanced traffic control is possible with a Service Mesh like Istio. But the configuration needed to enable this for all versions of all services is complex and error-prone. In this session Matt will show you how to use an Operator which auto-generates the necessary config. We'll see how just deploying a new version results in all the necessary config for sophisticated "override-based testing". Matt will walk through the technique, the underlying config, and the operator that generates it from Deployments.
A
So
welcome
I'm
Matt
Turner
I'm,
going
to
talk
about
dynamically
testing,
individual
microservice
releases
in
production,
which
is
a
bit
of
a
mouthful
but
I
wanted
to
get
across
what
we're
so
actually
going
to
be
talking
about,
which
is
to
test
on
an
ongoing
basis.
You
know
new
releases
of
an
individual
Service
as
part
of
a
more
complicated,
bigger
set
of
micro
services.
So,
let's,
let's
dive
into
what
that
means.
A
So
yeah
lots
to
talk
about
a
lot
of
these.
A
lot
of
these
topics,
I'm
only
going
to
be
touching
on
kind
of
briefly,
it's
a
bit
of
background.
The
meat
of
this
really
is
in
sort
of
the
istio
config.
What
we,
what
we
can
do
with
this
and
the
automation
that
I've
started
to
build
for
it,
but
I
will
just
touch
on
a
bit
of
background
as
well.
I
can't
see
my
own
slice
in
here,
so
I'll
have
to
lean
in
occasionally
remember
what
I
promised
I
was
going
to
say.
A
I
won't
have
to
leave
that
list
out.
So
a
little
bit
about
me,
I
am
a
software
engineer
at
tetrate.
Tetrate
was
founded
to
solve
the
problem
of
using
service
meshes
to
scale.
Our
management
plane
provides
a
layer
on
top
of
one
or
more
hdmes's,
it'll
install
them
for
you
upgrade
them
for
you
and
it
uses
the
tetrated
studio.
Distribution
for
that
which
is
a
build
of
Upstream
is
Joe.
A
Okay,
that's
probably
half
of
you
cool
who
is
sort
of
a
beginner
can
make
it
do
something.
That's
probably
the
other
half
and
anybody
consider
themselves
sort
of
an
expert
like
if
I
give
you
a
problem.
You
write
the
config
to
fix
it.
Okay,
a
few
people-
that's
cool,
I,
guess
I've,
sort
of
I've
sort
of
got
this
about
right,
then
so
I'm
actually
going
to
talk
about
this
from
the
perspective
of
of
the
problem,
we're
trying
to
solve
and
I
would
then
introduce.
A
A
It
might
look
like
this.
This
is
this
is
a
small
part
of
Netflix's
service
topology.
Apparently
they
wouldn't
give
a
higher
res
image,
so
maybe
they're
a
little
embarrassed
for
people
to
be
able
to
read.
What's
actually
going
on.
Of
course,
you
know
you
can
redraw
it
to
make
it
more
simple.
Make
it
easier
to
read.
This
is
also
Netflix.
A
Apparently
they've
maybe
got
a
little
more
connectivity
than
they
should
have,
but
you
know
when
we
do
a
when
we
do
a
talk
when
we
do
a
demo,
we're
probably
looking
at
something
like
this
right,
a
simplest
system.
This
might
be
you
know,
part
of
your
system
in
isolation.
This
might
be
your
whole
system
if
you're
a
new
startup
or
this.
This
might
be
for
the
reason,
with
the
Doma
paper
from
from
Uber.
That's
a
really
good
way
of
talking
about
breaking
things
up
into
little
isolated
sections.
I.
A
Think,
there's
some
animation
on
this
slide
that
I
forgotten
about
there.
We
go
I'll,
probably
come
back
to
this
point.
Actually
I
forgot
this
animated.
This
wasn't
meant
to
do
that.
So
there's
a
slightly
thicker
row
there.
So
when
we've
got
to
give
an
operation,
you
know
we've
said:
we've
got
a
user.
We've
got
some
some
web
apis
that
we're
calling
some
external
Services
we've
got
a
database
for
persistence
and
any
given
operation
that
a
user
performs
probably
isn't
going
to
hit
all
the
services
right.
A
You
know
users
request
the
distributed
transaction
might
even
end
up
going
through
a
linear
chain
of
services.
Like
this,
you
know
more
than
likely.
Some
of
these
services
are
going
to
call
multiple
others,
but
for
any
particular
path.
You
know
we
can
look
at
it
like
this
and
if
it's,
if
it's
linear,
then
it
sort
of
forms
a
chain,
and
we
can
reason
about
things
as
a
chain.
Now
this
isn't
what
I'm?
A
You
know
this
isn't
necessary
for
what
I'm
going
to
talk
about,
and
what
I'm
going
to
talk
about
applies
to
to
the
big
messes
that
we've
seen
before.
But
this
is
the
mental
model
I'll
be
using
right.
This
is
the
example
because
we
can
actually
reason
about
it.
So
imagine
you
know
One
path.
A
user
makes
a
request.
Service
call
service
course
service
and,
to
be
honest,
it
often
I
mean
it
often
is
true.
A
So,
depending
on
you
know
who
you
are
your:
what
service
plan
you've
bought,
whether
you're
roaming
or
native
to
their
Network
they're,
going
to
send
you
down
different
parts
because
they
might
firewalls
on
people
they
might
give
some
people
more
Media,
Services
than
others,
so
sort
of
linear
change
like
this
are
more
common
than
you
think
and
as
I
say,
it's
a
it's
a
mental
model.
We
can
use
for
more
arbitrarily,
more
complicated
systems.
A
This
is
why
the
animations
here,
if
you
and
if
you
think
about
your
services
a
little
bit,
then
they
probably
fall
into
a
few
different
categories
right.
So
we
probably
have
the
blue
things
that
sort
of
back
ends.
They
actually
do
the
business
Logic,
the
heavy
lifting,
but
in
order,
but
these
web
apis
right.
Maybe
one
returns
XML,
because
it's
from
the
90s,
maybe
one
is
accessed
over
an
ipsec
Tunnel
right
hands
up.
If
you
work
in
finance,
we
could
hide
those
complexities
and
those
nastiness
by
making
services
that
sort
of
shim
them
right.
A
A
Equally,
we
can
shim
databases
again
so
that
whatever
the
database
uses
for
auth,
whatever
the
database
is
why
a
protocol
is,
we
can
talk
a
sort
of
unified.
You
know,
for
example,
grpc
authenticated
protocol
to
it
and
then
maybe
we've
got
a
couple
of
front
ends
on
front
ends
on
that
right.
So
this
is
your
sort
of
back
end
for
front-end
pattern.
A
So
this
is
just
something
to
bear
in
mind
for
a
bit
later,
so
continuous
deployment,
continuous
release,
which
is
the
sort
of
problem
we're
actually
trying
to
tackle
I,
think
cicd
has
been
a
hot
topic
for
you
know.
Maybe
two
decades
now
I'm
going
to
look
at
the
continuous
deployment
part
of
that
and
what
that
actually
means
so
say:
we've
got
our
string
of
microservices
right
and
there's
a
new
version.
There's
a
putative
new
version,
a
candidate
of
one
of
them,
I'm,
going
to
call
these
things
V1
and
V2.
A
You
know,
obviously
they
might
be
1.035
and
1.036
or
whatever
you
might
have
multiple
candidate
versions
at
a
time.
So
multiple
people
working
on
multiple
branches-
that's
all
that's
all
about
it!
That
works,
but
V1
and
V2
is
simple.
So
if
I've
got
this
candidate
version,
it's
in
red
right
because
it
might
not
work
yet,
but
we
want
to
start
deploying
it
we
want
to.
We
want
to
test
it.
So
how
do
we
test
it
or
the
agile?
You
know
agile
testing
pyramid
says
you
should
do
something
like
this.
A
This
isn't
a
bad
model,
it's
not
perfect,
but
it's
definitely
not
bad.
You
know.
I
have
to
I
have
to
get
that
in
there.
I
have
to
add
I
think
before
even
the
unit
test,
you've
got
a
type
system
and
like
a
borrow
Checker,
if
you're
using
a
good
language,
but
I
mean
the
real
point
when
I'm
not
just
for
rust,
is
that
the
bottom
part's
like
happening
in
isolation
right,
you're,
they'll
happen
in
your
cic
system
anywhere.
You
can
run
a
Unix
process.
The
top
parts
are
sort
of
testing
in
context.
A
A
So
most
people
do
the
sort
of
integration
tests
and
above
the
end-to-end
tests
like
this,
you
have
you
know
the
build
environment,
that's
doing
the
unit
tests,
the
component
tests
and
then
you'll
have
a
test
environment.
Maybe
that
does
the
integration
test
and
a
staging
environment
where
we
do
the
system
tests
to
me.
I
couldn't
find
a
real
definition
of
this.
There's
a
lot
of
copies
of
this
picture
on
the
internet.
Nobody
seemed
to
want
to
actually
put
a
stake
in
the
ground
and
give
a
definition
for
each
of
these
levels.
A
A
So
how
do
we
run
these
integration
tests?
Well,
that's
easy
enough.
You
can
do
this
in
the
wild
west.
Sorry
I
mean
I
mean
test
right
any
way
you
can
deploy
software,
but
importantly,
not
CI.
This
isn't
a
sort
of
unit
test
this
in
order
to
do
an
integration
test,
a
black
box,
a
service
like
this.
It
needs
to
be
subject
to
you
know
real
runtime
resource
constraints,
real
runtime
security
constraints,
it
needs
representative
config
files
and
environment
variables,
but
we
can
deploy
it
to
an
environment
right
and
the
little
robots
are
like.
A
You
know,
test
scripts
that
maybe
call
in
call
out
tested
like
a
black
box.
But
how
do
we
do
that
into
end
testing
like?
How
do
we
do
this
sort
of
manual
testing
or
automated
end-to-end
testing?
We
need
to
emphasize
exercise
this
service
in
in
context
in
the
context
of
this
whole,
chained
or
more
complicated
graph.
A
How
do
we
do
that?
Well,
we
can
have
a
staging
environment
where
all
the
new
versions
are
deployed
right,
but
this
this
isn't
representative,
because
these
aren't
versions.
If
we're
testing,
you
know
this
one,
we
want
to
know
how
it
behaves
when
it
gets
deployed.
It's
not
going
to
see
the
new
version
of
this
and
the
candidate
version
of
this.
It's
going
to
be
sitting
in
between
the
two
production
versions.
Right,
you
know,
assuming
service
three
is
the
one
that
gets
through
test
and
gets
deployed.
A
A
So,
to
get
around
that
we
could
have
the
family
environments,
we
could
spin
up
a
new
environment
for
each
service.
You
know
even
for
each
each
PR
right
each
branch
of
each
service,
but
these
are
hard
to
build
the
the
automation
side
to
build
they're,
expensive
to
run
and
they're
still
not
prod,
they're
still
not
representative.
Anybody
has
ever
built.
One
of
these.
You
know
they're,
always
a
look-alike
of
prod,
but
they're,
never
really
quite
the
same
so
testing
in
production.
Why?
If
we
want
it
to
be
representative
actually
testing
product?
Why
not
testing
prod?
A
The
issue
with
that
is
right
is
that
our
software
is
is
so
charity
majors
has
this
eye
testing
prod
thing,
which
basically
says
you're
never
going
to
catch
all
there
is
in
testing
just
release
it
and
like
and
then
deal
with
it
when
the
users
find
things
you've.
Never
thought
of.
That's
I,
guess
a
little
stage
on
from
this,
but
when
our
software's
under
test
we're
not
ready
to
release
it,
we
don't
want
users
to
be
exposed
to
its
results
right.
We
don't
want
it
to
get
used
to
traffic,
because
the
flip
side
is.
A
We
don't
want
new
users
to
get
results
from
it,
because
they
might
be
nonsense
right.
It
might
still
be
broken,
so
we
get
to
win
if
we
can
separate
that
deployee
stage
from
the
release
stage
right.
So
if
we
can
deploy
it,
if
we
can
run
the
new
version
in
production
subject
to
the
all
the
all
of
the
quirks
of
the
production
environment,
but
we
don't
release
it
where
release
means,
you
know
it
doesn't
get
a
user
traffic,
so
there's
no
risk
to
the
business.
A
So
how
does
that
look
like?
What
does
that
look
like?
Well,
we
now
have
the
technology
right.
What
I
really
want
to
do
is
just
just
test
this
V2,
so
the
user
traffic
is,
you
know,
going
to
be
coming
in
and
getting
V1
V1
V1,
because
we
know
they're,
all
stable
I
want
to
be
able
to
put
it
under
test.
Get
some
test
traffic
to
go,
V1,
V1
up
to
V2
and
then
down
to
V1
again,
so
this
is
all
going
to
be
in
prod.
You
know
this
is
the
prod
database
that
I
can
read.
A
You
know
realistic
data
from
prod
constraints,
but
it's
still
under
test,
so
it's
only
getting
traffic
from
the
test
bot,
but
the
test
bot
or
you
know,
or
the
user.
The
developer
is
at
the
front
of
that
chain,
so
they
can't
just
sort
of
you
can
Cube
CTL
port
forward
to
right
to
here
right
and
you
can
do
integration
tests,
but
you
can't
you
can't
do
an
end-to-end
System
test
like
that,
so
the
test
agent
needs
to
be
opt-in,
able
to
opt
in
to
test
questions
as
V2
at
an
arbitrary
depth
down
that
chain.
A
A
A
Sorry,
we
can
well
but
would
be
deploy
them
into
prod,
but
we
don't
let
them
get
any
user
traffic,
because
these
sidecars
are
doing
Advanced
routing
for
us,
and
we
can
then
change
those
routing
rules
and
we
can
put
a
little
blip
in
the
chain
right
and
send
things
up
to
up
to
V2,
and
this
is
all
done
by
the
service
mesh.
This
is
all
in
kubernetes
configured
by
yamls,
we're
not
fiddling
around
with
IP
tables
or
VMware
Network
Shenanigans,
when
we're
not
doing
like
layer,
3
nonsense.
This
is
all
nice
and
kubernetes
Native.
A
A
So
this
was.
This
was
kind
of
inspired
by
a
talk.
I
saw
called
breaking
up
lifts
deployment
monolith
given
by
a
lift
engineer:
Jake
Hoffman
coupon
London
earlier
this
year,
essentially
doing
what
I
showed
with
the
proxies.
But
Lyft
has
that
so
lift
are
the
people
that
made
Envoy
right
and
Envoy
is
the
proxy
that's
used
by
istio
and
a
bunch
of
other
service
meshes
they,
but
they
don't
use
istio.
They
don't
use
one
of
their
available
service
meshes
because
they
were
the
first
members
with
Envoy.
A
They've
got
their
own
custom
thing,
their
own
custom
control
plane.
So
they
did
this.
They
managed
to
make
this
work,
but
it
can't
be
reproduced
by
anybody
else,
and
there
was
a
bunch
of
sort
of
custom
seats.
They
had
to
Fork
Envoy
and
inject
custom,
C
plus
plus
code
into
it.
So
it
I
think
it
was
a
great
idea
when
I
saw
that
talk
and
they
said
hey.
This
is
the
idea.
A
A
We
have
the
istio
virtual
service,
which
is
basically
like,
given
a
request
for
for
a
name
service
where
do
I
send
it.
So,
if
somebody's
trying
to
send
a
request,
a
service
Foo
you're
in
kubernetes
right
you're,
trying
to
send
a
request
to
Foo,
you
end
up
at
the
service
capital,
S
Foo,
but
with
this
Joe
like
the
virtual
service
kind
of
slots
in
before
that
it
says.
Okay,
you
wanted
food.
You
use!
A
You
know,
host
Foo
in
your
in
your
HTTP
request,
but
where's
it
actually
going
so
I
can
select
different
real
Services
if
I
want
to
I
can
select
parts
of
therm,
I
can
sort
of
identify,
subsets
of
them
and
say
well
yeah
that
one,
but
we're
only
going
to
part
of
it
and
I
can
I
can
make
those
routine
decisions
based
on
headers
or
all
kinds
of
other
attributes
of
of
the
request.
So
I
can
do.
A
Basically,
this
is
what
introduces
the
layer,
7
routing
right,
the
istio
that
sorry
that
kubernetes
can't
do
and
then
the
destination
rule
type
is
so
it's
right
when
I've
chosen,
where
I've
chosen,
where
I'm
going
to
root
it
to
when
we're
going
to
I'm
going
to
go
to
say
service
two.
How
do
I
talk
to
that
thing?
So
how
do
I
load
balance
across
all
the
pods
in
service?
A
So
we've
expanded
a
blown
up
the
sort
of
service
chain
here
right.
This
is
this
is
the
previous.
This
is
service
two.
This
is
service
four
and
then
for
service.
Three,
like
the
naive
kubernetes
way
right
as
I
make
a
deployment
and
I
put
a
service
in
front
of
it
and
it
selects
you
know
app
equals
through
and
I
just
select
Foo
and
in
this
case
I've
labeled,
the
deployment
version
one.
A
If
I
deploy
a
fubita
alongside
food
stable,
then
you
know
I
label
it
V2,
but
kubernetes
doesn't
know
any
better.
It's
just
going
to
send
the
traffic
to
both
right,
the
kubernetes
101.,
sorry
to
teach
you
for
sucking
eggs,
and
it's
going
to
do
that
relative
to
the
you
know
proportional
to
the
number
of
pods
in
each
one.
A
So
I
can
come
along
with,
and
these
could
be
replica
sets.
Rather
than
deployments
right
and
that's
how
rolling
update
in
kubernetes
works,
so
I
can
come
along
with
the
existing
OC
IDs
instead
and
I
can
slot
that
virtual
service
in
front
I
can
make
these
two
destination
rules
and
I
can
say
right.
There's
a
subset
of
this
Foo
thing.
That's
V1
and
that's
that's
this
deployment
and
there's
a
subset
of
this
food
thing.
That's
V2
and
that's
this
deployment.
A
I
can
then
slot
the
virtual
service
fulfill
in
front
and,
as
we'll
see,
it'll
it'll
say
under
some
circumstances.
I
want
to
go
to
this
V1
part
of
it
and
under
some
circumstances,
I
want
to
get
a
V2,
and
my
I
can
tell
my
virtual
service
right.
Your
default
config,
your
default
mode
is
to
say
in
all
the
traffic,
so
all
of
this
is
going
through
V1,
right
and
so
I
can
deploy.
V2
I
can
deploy
it
into
prod,
but
I
haven't
released
it
because
it
won't
get
any
user
traffic.
A
If
you've
got
a
header
that
says
hey
I'd
like
to
override
food
V2,
we
can
send
it
through
V2
and
then
back
into
the
chain
as
normal.
So
this
is
this
is
how
we
do
you
know
what
I'm
talking
about
this
sort
of
testing
in
prod
this,
this
override
testing.
This
is
how
we
do
what
recreate
what
lifted,
and
this
is
how
we
can
figure
it
with
istio.
A
So
what
do
those
resources
look
like
if
you've
seen
you
know
in
our
physio
before
to
follow
this?
We've
got
a
destination
rule
for
food
right,
so
we
say
that
this
is
talking
about
the
host
Foo,
and
this
is
the
identity
function
right.
This
is
istio
101,
we're
saying
well,
there's
two
subsets
in
this
case
version
one
and
you
identify
that
by
a
label
on
it
that
says
version
one
and
version
two
which
is
identified
by
label
that
says
version
two.
A
We
then
get
the
virtual
service
which
slots
in
front
and
does
a
little
bit
more
routing.
So
this
is
going
to
be
I'm
going
to
call
this
few
overrides
this
thing.
It's
looking
again
for
any
requests.
You
can
actually
override
more
than
one
service,
but
it's
saying
any
request
to
go
to
Foo
and
then
it's
got
this
HTTP
block
and
that's
versus
and
again
pretty
simple
stuff
saying
right
match
the
header,
so
Mash
X
override.
A
If
it
says
through
V1,
then
it's
going
to
go
to
Foo
subset
version,
one
if
it
says
3v2
it's
going
to
go
to
subset
version
two
and
then
our
last
stanza
at
the
end.
So
these
these
routing
rules.
This
is
like
nginx
config
right.
These
things
are
matched
and
applied
in
order.
We
then
have
like
a
default
route.
Basically,
it
doesn't
have
a
match
section
in
it
which
says
anything
that
didn't
match
either
the
previous
two.
A
So
people
who
you
know
people
who
aren't
setting
these
override
headers
they're
just
going
to
go
to
V1,
so
normal
traffic
flow
V1.
But
if
you
opt
in
with
the
header,
you
can
go
after
V2
or
of
course,
you
know
that
can
be
as
long
as
you
want
any
any
other
version.
There's
one
little
caveat
with
this.
This
isn't
quite
what
they
look
like.
You
actually
need
to
match.
A
Like
this,
you
actually
need
to
use
a
regex
to
match
the
middle
of
you
to
match
this
x
override
header,
so
it
gets
a
little
bit
messy
just
to
go
through
the
Practical
details
right
in
case
anybody
goes
home
and
tries
to
re-implement
this.
The
reason
for
that
is,
we
might,
we
might
have
more
than
one
override
right.
I
might
want
to
go
to
Food
version
two
and
then
bar
version
three
and
something
else
version
seven.
A
A
If
you,
if
you
use
Curl,
to
make
this
request,
you
win
multiple
X
override
headers
Cole
puts
them
on
the
wire
as
multiple
keys
right.
So
you'll
get
two
x
override
through
V1
X
over
Y
Bar
V2
you'll
get
those
separate
headers
that
used
to
not
be
allowed
by
the
spec,
the
spec
used
to
say
you
had
to
condense
them
like
one
key,
several
values
with
a
comma
between
them,
the
spec,
the
the
new
spec
I,
spent
a
long
time.
A
A
What
Envoy
presents
you
when
you're
trying
to
match
these
things?
Is
the
single
combined
thing
again
not
against
the
spec,
but
a
little
bit
of
an
old
version?
So
you
get
you
get
presented,
X
override
precisely
once,
no
matter
how
many
times
you
specified
it
and
you
get
the
values
collapse
with
with
commas.
A
So
we
have
to
match
a
substring,
because
we
we
might
have
presented
X
override
more
than
once
Envoy
if
you're
configuring
Envoy
manually,
which
you
should
never
do,
gives
you
several
ways
of
matching
headers
I
can
say:
I
want
to
match
an
exact
string,
a
prefix,
a
suffix
I've
got
a
regex
or
I
can
use
the
contains
thing
to
match.
The
substring
istio
only
exposes
part
of
envoy's,
API
and
istio
does
not
give
you
contains
helpfully.
It
still
gives
you
exact,
prefix
and
regex.
It
doesn't
give
you
contains.
A
It
doesn't
give
you
suffix
so
regex
it
is.
The
registers
in
Envoy
are
Google's,
re2
syntax,
which
took
me
a
while,
because
it's
a
little
different
to
other
things
and
that
you
have
to
and
you
have
to
match
the
whole
string.
So
we
end
up
with
I
think
this
is
the
tightest
you
can
get
I,
don't
think
we
can
be
any
tighter
than
that,
but
basically
saying
well,
you
might
be
the
carrot,
you
might
be
the
start
of
the
string.
You
might
be
the
end.
A
You
might
be
both
because
it
might
be
the
only
value
for
this
header,
but
there
might
be
something
before
it
and
then
a
comma.
There
might
be
something
after
it
anyway.
Just
so,
you
understand
like
why
on
Earth,
that
horrible
regex
was
for
and
this
this
was
the
longest
part
I
was
like
I
can
I
left
that
conference
thinking
I
can
implement
this
in
history.
Oh,
this
is
fine.
This
was
honestly
the
longest
part,
so
the
ammo's
actually
pretty
simple
right.
A
It's
almost
kind
of
basic
istio
usage,
but
you're
going
to
need
one
of
each
of
those
resources.
One
destination
rule
one
virtual
service
for
every.
You
know
every
service,
every
workload
you've
got
you're
going
to
need
a
match
for
every
version,
and
it's
going
to
need
to
be
updated
every
time
a
new
version
is
deployed.
A
So
that's
a
pretty
big
combinatorial
explosion
and
it's
ongoing
work
as
well
right
if
you're
in
a
CS,
automated
CI,
CD
environment,
where
multiple
versions
are
going
out
all
the
time,
you're
going
to
need
to
be
updating
those
things
quite
a
lot.
So
it
sounds
like
a
great
Target
for
Automation
and
that's
exactly
what
I
did
and
it
was
writing
this.
You
know
I
sort
of
went
home
and
thought
I
can
do
this
and
I
wrote
this
and
then
that's
what
made
me
think.
A
I
should
give
this
talk
to
some
tell
folks
about
it.
So
I've
only
got
five
minutes
left.
I'm
gonna!
Try
to
give
you
a
quick
demo.
If
anybody
saw
my
tweet
earlier,
doesn't
the
latest
version
it
doesn't
work,
but
I
was
able
to
roll
back
and
the
previous
version
does
but
anyways
good
enough.
Hopefully
everybody
well,
that's
got
a
bit
smaller
than
I
thought.
A
A
A
A
A
A
A
A
Obviously
there
are
security
issues
with
that
right,
you're,
definitely
going
to
need
to
filter
that
out
at
Ingress,
so
a
user
can't
set
it.
You
I
will
have
trust
issues
inside
the
mesh
for
that
header,
because
anything,
that's
compromised
can
start
heading
it
setting
it.
You
know
if
it's
a
massive
security
risk
to
be
redirected
to
a
beta
version.
I
don't
know,
but
it's
definitely
something
you
don't
want
to
make
this
into
the
wild
west.
So
you
need
to
trust
that
header.
A
A
That's
something
that
I
I
just
honestly
didn't
have
enough
time
to
add
to
this
operator.
Yet
that's
definitely
a
way
to
do
it.
That
also
gives
the
possibility
for
some
more
interesting
stuff
like
we
might
want
to
do
like
a
conditional
override.
So
one
of
the
things
tetrate's
working
on
in
his
product
is
like
conditional,
auth,
so
service
a
can
talk
to
service
B.
Only
if,
like
service
Z
was
further
down
the
chain
right,
so
you
have
to
go
Z
to
A
and
then
you're
allowed
to
go
to
B.
A
Imagine
if
we
could
I'm.
You
know.
One
solution
to
that
is
is
embedding
nesting
jots.
So
imagine
if
we
could
do
that
with
this
right.
I
want
the
override
version
of
the
database
shim
only
if
I've
been
through
the
override
version
of
service
one
and
the
right
version
of
service
two,
so
that's
something
I
could
start
to
to
work
on.
I
have
no
idea,
even
if
this
was
downloading,
it
should
be
quicker
than
this
I.
Don't
know.
What's
going
on,
I'm,
probably
not
going
to
show
you
this
sorry
about
that.
A
Well,
there
you
go
honestly
I've
loaded,
the
I've
been
here
before,
and
I've
got
a
couple
of
scripts,
one
that
pulls
all
the
images
out
of
mini
Cube
and
one
that
pushes
them
back
in.
So
they
should
that's.
Why
istio
camera
is
so
quick.
They
should
be
there.
Oh
well
I'm
not
going
to
take
your
time
with
this.
Honestly,
it
works.
A
What
we
might
have
got
enough
if
we've
got
the
services,
that's
actually
enough
to
generate
the
metadata.
Let
me
just
push
push
all
these
in.
Where
is
yeah,
not
one
of
the
top
HTTP
log?
That's
my
little
thing
like
that's.
The
thing
that
should
be
should
be
happening.
We
haven't
got
any
of
them,
but
we've
got
okay.
We've
got
the
services,
so
I
think
yeah.
If
we
were
to
make
a
call
down
that
chain,
we
would
get
primary
and
beta
versions
at
random
standard,
kubernetes
load.
A
Is
the
generation?
Oh,
this
is
just
going
to
apply
them,
so
this
went
in
and
it
ran
so
my
code
has
got
a
CLI
mode
and
an
operator
mode,
so
it
can
run
as
an
operator
watch
the
services,
the
pods
the
deployments
and
continually
emit
the
virtual
services
and
the
destination
rules
or
there's
a
CLI
mode
right.
So
you
can
try
it
out
see
what
it's
going
to
do,
integrate
it
into
a
git
Ops
pipeline.
Do
that
kind
of
stuff,
because
all
of
you
know
what
I'm
doing
is
derivable
from
other
permanent
resources.
A
A
So
I
just
run
it
I'm
running
CLI
mode
locally.
It's
going
to
connect
to
the
cluster
right
and
emit
all
of
this
stuff,
and
this
this
looks
as
you
would
expect
it
to
look.
There's
also
an
operator
mode
that
uses
exactly
the
same
logic
and
just
just
does
a
watch
I
promise.
So
that's
all
the
demo
we're
getting
thanks,
Docker
hub,
correct,
Emoji,
I!
Guess
what
else
right!
A
So
there's
a
few
caveats
this
that
you
know
this
was
kind
of
a
proof
of
concept
and
I
thought
I'll
talk
about
it
and
it
does
work,
but
I
got
a
bit
busy.
So
it
does
need
a
bit
more
work
the
moment.
The
thing
basically
needs
a
namespace
to
itself
right.
It's
going
to
look
at
the
services
and
emit
vs
and
DRS.
A
If
you've
got
any
other
virtual
Services
trying
to
do
any
more
routing,
they're
just
going
to
clash,
it
does
use,
the
off-roader
does
use
server
side
apply,
but
obviously
it
doesn't
own
those
resources.
So
it
gets
a
little
bit.
Tricky
I
think
there's
a
way
around
that
with
you
can
delegate
from
one
virtual
service
to
another,
so
I
think
I.
Might
use
an
emission
controller
to
patch
anything
I
find
to
delegate
and
delegate
to
myself
and
then
do
this
thing.
A
I
need
to
think
about
it,
but
at
the
moment
it
kind
of
needs
a
namespace
for
itself.
It
is
Alpha.
You
know,
suggestions
and
PR's.
Welcome.
Yes,
what
other
stuff
you
want
me
to
do.
I
talked
about
the
header,
while
we
were
waiting,
yeah
get
Ops.
So
it's
it's
kind
of
like
transient
state
right.
These
things
that
we
emit,
but
you
can
just
you
can
run
the
CLI
mode
as
part
of
some
kind
of
you
know,
generation
pipeline
that
that
produces
yaml,
so
you'll
get
Ops
repo
yeah.
A
It's
going
to
clash
with
anything
that
exists,
but
I
think
I've
got
a
couple
of
tricks.
I
can
use
to
fix
that.
I.
Probably
don't
have
time
to
talk
about
that,
but
it's
I.
Imagine
if
you
could
set
up
a
route
where
I'm
talking
to
remember
I
talked
about
the
different
types
of
microservice.
If
one
of
them
is
a
database,
shim
I
might
actually
want
to
send,
gets
to
the
stable
one.
A
So
I'm
reading
from
the
prod
database
but
posts
database
rights
go
to
the
the
V2
shim
the
test
shim,
which
is
maybe
just
going
to
Black
Hole
them
right
because
of
a
test
order.
I
don't
actually
want
it
to
go
in
the
database.
I
don't
want
Logistics
to
do
anything.
So
imagine
if
we
could
do
something
like
that
yeah.
So
this
thing's,
a
database
sharing
right,
I,
might
say:
hey
I've,
been
through
a
test
V2,
so
I
actually
want
to
go
to
a
shim.
A
A
A
Why
we
might
want
to
do
this
so
that
it's
possible
show
that
it's
possible
in
istio
and
then
talk
to
show
you
how
to
do
it
and
then
talk
about
the
the
sort
of
automation
I
started
using
if
anybody
wants
to
go
home
and
and
do
this
themselves,
the
one
last
thing
I
wanted
to
show
is
a
sort
of
if
we're
coming
through.
You
know,
as
as
a
service
goes
through
like
a
cicd
lifecycle
right,
it
gets
built,
it
gets
linted,
it
gets
unit
tested,
you
know,
and
then
we
get
to
right.
A
Let's
test
it
in
prod,
so
we're
not
sending
user
traffic
to
it.
The
basic
way
of
doing
that
is
by
using
a
staging
environment
or
an
environmental
environment.
I've,
hopefully
persuaded
you
with
the
problems
with
those
I
think
overrides.
What
I
just
showed
is
the
sophisticated
way
of
doing
that.
Are
you,
then
that's
really
as
much
testing
as
we
can
do
in
isolation.
We
then
get
on
to
you
know
releasing
it
separate
from
deploying
it,
but
we
get
on
to
releasing
it.
A
A
The
sophisticated
way
of
doing
that
is
is
Flagger,
so
I
think
a
combination
of
this
thing.
You
know
you
deploy
your
tester
comes
along
sets.
The
header
gets
to
isolate
things
and
tests
them,
and
then
you
can
let
you
know
when
you're
happy
with
that
and
you
press
the
button
and
Flagger
starts
to
do
a
role.
That's
a
real
users,
that's
what
I
was
going
to
show
you.
Thank
you.
A
I
shouldn't
have
told
you:
the
code
was
broken
because
if
we
couldn't
I
could
have
just
been
our
Docker
Hub.
Sorry
I
can't
show
you
anything,
but
I
will
say
the
the
version.
0.2
works
and
I
will
fix.
It.
I
think
that's
the
end
of
the
slot
I'll
be
around.
If
anybody
asks
any
questions,
but
I
won't
I
won't
keep
you
here
thanks
a
lot.