►
From YouTube: Unblocking the Release Train with Istio Traffic Management - Dave Shepherd & Pierre Meunier
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Unblocking the Release Train with Istio Traffic Management - Dave Shepherd & Pierre Meunier, Wealth Wizards
Wealth Wizards employs a microservice architecture, with each service being actively developed. As soon as a change is in the mainline it’s on the release train and any subsequent changes traditionally ended up in a queue. If that change has a delay, then all subsequent changes get stuck and then have to be released together. There are many ways to try and avoid these problems, but one way is to stop items getting on the release train until it's ready. Istio does many things, but the traffic management functionality is the key to this solution. It allows traffic to be routed to different versions of the same service. The Wealth Wizards implementation will route all traffic to the mainline or master version of a service by default. However, if the hostname includes a version prefix then it will route traffic to the corresponding version of the service, if it exists.
https://sched.co/MPfY
A
We're
ready
right
good
morning,
everyone
thanks
very
much
for
coming
so
I'm
here.
Money
and
I've
got
Dave
Shepard
with
me
today.
So
we
posed
platform
engineer
at
once.
Wizard
today,
we
here
to
talk
to
you
about
how
we
use
this
to
your
traffic
management
into
our
release
pipeline.
So
we'll
go
through
a
bit
of
a
background
of
who
we
are
what
what
we
do.
We
look
at
the
problem.
A
So
we
are
once
with
it.
We
are
little
FinTech
in
Leamington
Spa
in
the
centre
of
England
in
the
UK
we
are
regulating,
robe
advises,
robe
advice
just
means
that
we
use
technology
an
algorithm
to
automate
the
generation
of
financial
advice.
We
focus
primarily
on
on
pensions
and
retirement
advice.
We
also
have
some
financial
well-being
products
available,
direct
to
consumer
or
through
employee
benefits.
Our
vision
is
financial
advice
for
everyone.
We
want
to
make
it
affordable
and
accessible
to
everyone,
we're
not
just
an
engineering
company.
A
We
do
have
practicing
financial
advisers,
the
AdWords
wizard
as
well,
which
helped
greatly
when
we
when
we
come
to
write
our
product.
A
key
element
is
that
we
are
FCA
regulated
FCA
is
the
financial
conduct
authority.
That's
the
UK
body
that
controls
everything
related
to
financial
services.
So
it
means
that
we
are
in
a
regulated
industry.
A
A
Wiki
bamako
service
is
small.
We
want
them
to
solve
just
a
single
purpose
or
a
single
concern,
whether
it's
a
business
functionality
or
technical
functionality
like
generating
a
PDF
or
a
docx
file
suit.
So
size
varies,
but
we
try
to
keep
them
as
simple
as
possible
and
we
want
to
avoid
duplication
of
code
between
services
and
instead
we
have
services
talking
to
each
other.
A
B
So
we
choice
key.
Ladies,
probably
the
same
reasons,
everyone
else.
It
gives
us
a
lot
of
flexibility
whilst
providing
us
with
a
way
of
running
would
have
much
services
reliably
and
we
started.
Probably
over
three
years
ago
we
use
the
good
old
cube
up
script,
set
up
two
clusters,
one
for
our
production
systems
and
one
for
our
non
production
systems
and
we
kind
of
went
and
upgraded
from
there
from
about
version
1.2.
We
just
kept
going
up
and
up
to
the
later
versions,
we're
really
keen
on
having
all
our
infrastructure
as
code.
B
We
introduced
this
before
how
long
kind
of
came
about,
so
we
haven't
really
moved
away
from
it
yet,
and
we
wanted
to
keep
that
automation
in
place
and
all
that
infrastructure
zko,
because
it
gives
us
confidence,
gives
us
a
really
like
the
ability
to
repeatedly
bring
up
clustered
if
we
need
to,
and
that
kind
of
worst
case
scenario.
If
we've
really
had
and
we
lost
a
kubernetes
cluster,
we
could
just
rebuild
it
within
a
couple
of
hours.
Luckily,
we've
only
ever
done
that
as
an
exercise
rather
than
because
our
production
system
went
down.
A
So
our
platform
has
evolved
over
time,
as,
as
Dave
said,
there
are
a
lot
of
new
tools
coming
out,
so
a
lot
of
the
time
with
with
kubernetes
and
some
of
the
tools
that
some
of
the
tools-
sorry
that
are
available
now,
one
of
available
them.
So
we
have
to
write
some
of
those
ourselves.
We
still
use
some
of
them
now.
Our
deployment
processes
is
pretty
much
a
mix
of
bash
script,
console
vault
and
some
JSON
configuration
and
it
still
works
quite
well.
So
we're
quite
happy
with
that.
A
We
have
a
fairly
standard,
CIC
D
pipeline.
So
any
all
the
commits
are
built,
tested,
unit
tested
component
tested.
We
do
some
security
checks
as
well
and
when
we
build
onto
master,
we
also
create
a
docker
image
and
we
push
the
automatically
to
our
dev
service
for
our
staging
and
production
Lac
environment.
We
still
have
a
push
of
a
button
and
we
were
quite
happy
with
with
that
process.
A
Self-Service
is
also
something
that's
called
important
for
us.
We
don't
want
to
have
a
engineer
to
have
to
rely
on
us
to
do
some
deployment
or
anything
ready
to
production
environment.
So,
instead
we
try
to
provide
them
the
tool
so
that
they
can
do
the
all
their
work
without
having
to
require
on
on
our
time.
A
We
have
also.
The
configuration
is
like
a
a
bunch
of
JSON
files
and
engineer
can
can
control.
The
versions
of
the
service
that
goes
to
different
environments,
can
control
the
configuration
of
those
services
and
where
to
find
the
secrets
for
the
services
as
well,
but
we
are
in
a
slow-moving
industry.
That's
one
of
the
reason
why
we
got
into
into
this.
In
the
first
place.
A
We
want
to
make
it
a
lot
quicker,
and
it
means
that
sometimes
we
can't
push
the
production
as
often
as
we
like
and
often
is
due
to
compliance,
jack
and
regulation
checks.
To
give
you
a
simple
example
just
to
drive,
the
point
is,
if
you
take
the
simplest
change
that
you
can
think
of
so
say,
just
changing
a
bit
of
text
in
a
document
that
you're
going
to
send
your
customer
from
an
engineering
point
of
view.
It
is
very
simple:
you
change
some
text,
it
goes
through
your
pipeline.
You
have
some
test.
A
It
goes
to
production,
typically
within
hours,
maybe
less
you
can
get
that
on
to
your
production,
but
from
the
compliant
compliance
side
of
things.
Changing
a
sentence
might
mean
that
you
will
change
the
advice
that
you're
giving
to
your
come
to
your
consumer
and
point
in
potentially
we
know
we
might
break
some
regulations,
so
there's
a
lot
of
testing
that
needs
to
happen
on
that
side
of
the
business.
So
we
want
to
deploy
quickly,
but
at
the
same
times
we
have
to
to
comply
with
regulation
and
there's
a
bit
of
a
friction.
There.
A
So,
as
I
just
say,
we
we
must
comply
with
regulations.
So
our
pipeline
looks
like
this.
When
a
feature
is
they've
completed.
We
have
a
standard
code
review
process
which
is
just
put
request
system
really
like
as
much
into
map
master.
We've
got
automated
bill
that
go
to
our
dev
environment
and
then
we've
got
a
pretty
standard,
QA
review
some
automated
test.
That
runs
on
top
of
that.
We
then
have
the
very
important,
important
compliance
check
and
then
we
can
push
down
into
our
production
environment.
A
So
it's
a
very
linear
process
and
when
everything
is
fine,
just
a
tick
in
the
box
and
it's
in
production.
However,
if
we
have
any
problem
with
any
of
those
steps,
especially
the
compliance
check,
then
we
we
now
we're
not
blocking
all
all
the
release
that
are
coming
to
our
production
environment.
We
now
have
a
master,
build
that
is
corrupted
and
we
find
sometimes
that
compliance
check
could
mean
that
we
needed
a
couple
of
days
to
fix
those.
So
it's
a
very
risky
position.
A
B
For
the
first
way
around
the
problem
was
20
G's,
the
idea
of
feature
flags.
We
did
experiment
with
this
little
bit
and
the
problem
we
mainly
found
was
that
the
over
hadn't
doing
this
introducing
code
like
this
into
each
of
the
Micra
service,
a
change
potentially
effects
started
to
become
more
work
than
the
change.
So
the
example
that
Pierre
just
said
a
one-line
change
which
to
an
engineering
team,
is
very
simple
to
check
to
do
but
adding
the
feature
flag.
B
On
top
of
that
makes
it
more
complicated,
and
but
it's
that
sort
of
change
that
is
most
likely
to
to
cause
a
problem
in
terms
of
compliance
and
therefore
things
would
start
not
getting
the
feature
flags
around
them,
and
then
we
still
end
up
with
the
same
problem
with
the
other.
Another
option
is
around
or
BT
users,
where
it's
a
very
similar
argument
that
the
application
itself
needs
to
know
about
whether
a
user
or
a
feature
flag
is
enabled
or
a
user's
allowed
to
use.
B
B
So
one
of
one
of
the
things
we
do
anyway,
our
engineering
team
will
have
the
ability
to
spin
up
parts
of
our
system
locally
and
they'll,
either
interact
with
mock,
end
point
or
they'll
interact
with
some
of
our
development
clusters
and
that
kind
of
works
well
for
the
small
small
iterations,
the
engineering
of
doing
that,
once
you
start
handing
stuff
over
to
our
compliance
teams
to
check
they
don't
really
want
something.
That's
very
tech
heavy
to
to
deal
with.
So
we
start
being
asked
to
spin
up
environments
because
we
want
to
test
a
feature.
A
So
in
a
monolithic
world,
when
you
want
one,
you
get
asked
to
get
your
environment,
you
tend
to
have
like
a
a
single
being
big
artifact.
So
if
you
want
you
to
duplicate,
it
is
fairly
easy.
You
just
spin
up
a
new
instance
whether
it's
on
the
laptop
on
the
server
somewhere.
It
doesn't
really
matter,
and
if
you
ask
for
some
more,
you
just
add
more
and
you
ask
for
some
more
you
are.
You
are
known
in
our
microservices
world.
While
we
could
do
the
same
really.
A
We
know
that
is
very
simple:
to
deploy
services
with
with
kubernetes,
and
if
we
ask
for
some
more
environment
we
could
also
duplicate
them.
There
is,
however,
cost
into
infrastructure.
Obviously
it
get.
It
gets
a
little
bit
more
complex
as
well,
but
we
also
find
that
there
was
a
cost
in
time
in
managing
all
the
configurations
for
those
environments
and
also
there
was
a
lot
of
duplications
between
rope,
all
of
those,
so
that
wasn't
really
quite
the
right
way
to
fix
that.
A
So
instead
we
thought
why
not
just
spin
up
the
bits
that
have
changed
so
I've
got
my
macro
services
world
and
if
I've
changed
only
a
couple
of
services,
why
could
I
just
not
run
alternative
versions
of
those
of
those
services
alongside
the
master
version
and
find
a
nice
way
to
route
traffic
to
the
services?
And
that's
where
we
started
to
look
into
Ecco.
B
So
we've
been
to
quite
a
few
toxicity-
has
come
up
an
awful
lot
this
week,
it's
just
one
of
a
number
of
service
meshes
they're,
very
much
higher
on
the
radio
out
on
the
radar
and
we've
been
aware
of
SDO
for
probably
a
couple
of
years.
In
fact,
we
played
around
with
it
a
few
years
ago
and
just
found
it.
It
had
so
much
potential
and
a
lot
of
features,
but
we
didn't
really
have
a
problem
to
solve
at
that
time
that
we
wanted
to
introduce
the
complexity
in
learning
it
and
running
it.
B
B
It
just
gets
routed
to
any
of
the
pods
within
that
service.
When
you're
introduced
this
do
you
can
start
doing
some
rules
and
we're
going
to
talk
about
the
HTTP
headers.
So
in
this
example,
we
might
we
would
look
at
user
agent
set
to
Android
which
would
get
route
to
the
current
version
of
service
B
and
then
an
user
agent
of
iPhone
would
get
to
so
routed
to
the
Canaria
version
of
service
B,
all
the
while
there,
those
services
running,
don't
know
anything
about
it.
B
So
I'm
going
to
run
for
a
little
demonstration
of
how
how
we've
implemented
this,
how
this
works
from
a
developer's
point
of
view.
So
this
is
our
Robo
power,
planner
application.
This
is
what
our
smooth
financial
advisors
will
use
for
general
financial
advice
and
with
going
to
imagine
that
we've
picked
up
a
piece
of
work
to
rebrand
our
application.
B
So
this
is
running
in
a
development
environment.
We
called
it
green
and
that's
the
name
of
the
brand
of
the
application
we're
going
to
rename
it
as
part
of
our
a
new
brand
and
as
a
good
fun
bit
of
code
that
we're
going
to
change
it's
just
there,
we're
gonna,
rename
it
and
we're
going
to
use
Dave,
because
all
good
engineers
use
their
name
when
doing
demonstrations
so
I'm
going
to
commit
that
to
our
source
repository,
we're
gonna
use
the
ticket
number
that
we're
doing
this
work
on,
which
is
going
to
be
ops.
B
B
So
this
I'm,
just
gonna
bring
up
a
new
window
and
Annie's
little
plugin
for
Chrome,
and
this
plug-in
will
allow
us
to
attach
HTTP
headers
for
the
browser
requests,
so
I'm
setting
X
variant
ID
to
the
ticket
number
we
used
in
the
branch
name,
which
is
ops
one
two,
three
four.
So
if
I
enable
that-
and
I
refresh
this
page-
we'll
see
that
my
change
has
now
been
deployed
and
I'm
being
routed
to
it.
So
you
can
see
my
my
name's
there
in
the
middle
and
we
thought
this
is
great.
B
Our
engineers
thought
this
was
great,
but
one
of
the
things
that
was
missing
was
being
able
to
give
this
to
our
compliance
teams,
who
are
not
necessarily
technical
so,
rather
than
getting
them
to
install
a
plug-in,
we
introduced
the
ability
to
do
this
through
the
hostname.
So
we
also
allow
prefixing
the
host
thing
there
with
the
same
ticket
number,
and
you
can
see
that
comes
up
with
the
same
variant.
We
even
with
the
headers
disabled
and
just
improve
our
mainline
version.
B
I'm
a
master
version
still
running
I've
refreshed
that
page
and
we
still
got
the
existing
application
running
in
parallel.
I'm
gonna
talk
a
little
bit
about
the
sto
configuration
that
makes
that
possible.
The
first
one
is
a
destination
rule.
The
destination
rule
is
basically
defining
subsets
of
a
Cuban
80
service
that
we
we
want
to
be
out
of
really
traffic
to.
So
in
this
case,
our
our
service
is
fact.
B
And
if
that
matches
first
thing,
we
do
as
an
appendix
parent
ID
header,
with
the
same
ticket
number,
which
I'll
talk
about
in
a
second
and
then
we
route
it
to
the
destination
service,
giving
it
the
subset
that
we
defined
on
the
previous
slide
of
ops.
One
two
three
four
now
a
second
rule
which
is
basically
doing
the
same
thing,
but
only
matching
the
HTTP
header.
B
B
So
we
had
to
introduce
automation
to
to
manage
this,
so
the
first
one
is
how
they're
deployed-
and
this
was
quite
a
simple
change
for
us,
because
we
already
had
CI
on
on
our
branches
anyway,
we
just
had
as
a
step
to
the
end
to
deploy
that
variant,
all
that
branch
to
a
Cuban,
80s
development,
cluster
and
worldly,
deploying
on
whenever
we
build
master
anyway.
So
this
was
a
small
variation
on
that
at
the
other
end
of
the
process.
B
We
wanted
to
tidy
stuff
up,
because
we
didn't
want
these
branch
deployments
or
variant
deployments
being
left
lying
around.
So
we
just
introduced
something
we
called
cube
housekeeper,
which
is
a
effectively
just
a
cube,
cron
job
that
goes
and
cleans
up
branches
that
have
been
left
lying
around
for
a
few
days
so
that
we
don't
don't
use
too
much
resource.
The
big
piece
we
added
was
something
we
call
Auto
cube
and
we
had
we
one
of
the
problems
we
had
with
this
queue
in
the
declarative
way.
B
It
works
is
that
we
needed
to
now
get
about
all
the
subsets
or
all
the
deployed
versions
of
a
particular
service
in
order
to
generate
that
that
destination
willam
or
the
virtual
service
Djamel.
So
we
we
wrote
a
little
tool
which
is
kind
of
based
on
console
template
and
primarily
because
we
already
stored
a
lot
of
information
about
what's
running
on
Kievan
ATS
within
console
to
generate
those
yeah
more
file
for
every
variant
that's
currently
deployed.
B
A
The
next
one
is
edge,
propagate
h,
propagate
is
actually
a
node.js
module
that
we
had
to
write
most
of
our
services
run
running
in
node.js
and
fairly
early
on.
We
recognized
that
we,
we
would
have
a
requirement
to
propagate
headers
through
our
services.
So
if
you
got
services
a
B
and
C,
it
talks
to
b
and
b
talks.
You
see
we
do
new
features
and
you
have
changes
in
a
and
C
when
we
send
requests
with
our
x
value
ID
from
from
to
service
a
sorry.
A
We
now
need
service
a
to
propagate
the
header
to
B
and
we
need
B
to
propagate
the
header
to
C
as
well.
So
a
simple
change
would
be
to
go
through
all
our
services,
all
our
endpoints
and
start
to
handle
that
manually,
but
the
obviously
that
didn't
feel
like
the
right
way
to
use
to
solve
this.
And
then
we
thought
of
how
like
library,
like
new,
relic
and
elastic
APM,
managed
to
do
that
stuff.
So
I'm,
not
sure.
A
If
any
of
you
have
used
neuron
ik
before
in
the
node.js
world,
you
import
or
require
the
neuron
ik
module
in
it
instruments
your
code,
and
then
you
start
to
receive
data
in
new
relic.
So
we
thought
that's
quite
a
nice
way
to
solve
that
sort
of
problem,
so
we're
actually
looking
in
in
the
code
a
bit
more.
It
turns
out
that
it's
a
bit
messy
and
quite
difficult
to
get
right.
We
look
a
bit
further
and
I.
A
Think
some
of
the
committees
in
the
elastic
APM
project
have
released
some
nice
videos
and
presentation
about
how
to
do
this,
and
it
turns
out
with
no
J's
night,
so
it'll
be
easier
to
to
do.
They've
got
a
core
library
called
a
acing
hoops
and
that
lets
you
get
get
notification
into
a
synchronous
event.
So
what
that
lets
us
do
is
effectively
in
our
service.
When
an
HTTP
request
comes
in,
we
get
a
little
notification,
we
can
go
and
look
at
the
headers
to
grab
the
value
of
the
X
variant
ID.
A
Similarly,
when
we
have
outbound
HTTP
requests
going
out,
we
get.
We
also
get
a
notification
and
we
can
inject
a
header
into
it.
Another
cool
stuff
with
that
library
is
that
you
can
link
the
outbound
requests
to
the
incoming
requests,
so
you
can
map
which
headers
needs
to
go
to
which
requests
and
then,
with
that
we
started
to
have
so
our
service
propagate
the
header
automatically.
A
It
does
need
a
little
change
in
your
code,
so
that's
the
first
line
there,
so
you
just
require
edge
propagate
and
then
you
start
to
have
your
headers
being
propagated.
We
also
added
a
decoration
idea
which
we
to
do
debugging
across
multiple
services
and
now
that
we
used
SEO,
we
also
looked
into
the
East.
A
Your
observability
and
yoga
you've
got
Jaeger
and
request
tracing
and
East
he'll
handle
some
of
those
headers
for
you
already,
but
there
is
one
requirement
in
your
service
is
to
propagate
some
headers,
so
we
just
we
just
wrote
edge
propagate
at
the
time.
So
we
we
added
the
six
letters
I,
think
that
you
need
to
have
your
tracing
and
we
started
to
receive
Tracy's
almost
for
free,
so
that
was
quite
cool.
A
Okay.
So
let's
look
at
the
gains.
So
six
months
later,
and
that
feature
has
become
really
indispensable,
especially
for
one
of
our
team
who
works
a
lot
with
regulation
and
compliance.
So
they
don't
have
to
worry
about
dependencies
between
their
features
anymore.
They
can
just
have
a
feature
deployed
in
the
branch
and
they
can
queue
a
test
in
isolation.
They
can
compliance
test
in
isolation
as
well.
A
So
we
also
brought
the
QA
team
and
the
compliance
team
a
bit
earlier
in
AI
pipeline,
which
I
think
is
actually
an
improvement
and
a
nice
added
bonus
as
well
nice,
because
we
do
so
much
testing
on
the
branches
that
once
we
decide
to
merge,
the
merging
is
really
smooth
now
and
we
just
go
through
a
few
regression
tests
and
compliance
tests
again.
Obviously,
but
we
we
are
now
able
to
release
to
production
a
lot
quicker
than
we
used
to.
A
As
a
another
example.
Very
recently
we
introduced
a
new
user
authentication
mechanism
into
that
project,
and
that
was
there
was
qualitative
work
involved
and
we
managed
to
isolate
that
in
the
branch
we
were
able
to
evaluate
the
product.
We
were
using
quite
well
without
affecting
any
other
features
going
into
production
and
again
it
was
well
tested,
and
once
we
were
ready
to
use
it,
we
merge
and
very
quickly
within
a
day
we
had
that
through
to
our
customers,
so
very
successful
for
us.
A
A
The
traffic
management
effect
the
networking
so
the
core
thing
into
your
infrastructure.
So
when
something
goes
wrong,
you
can
have
some
quite
important
impact.
There
was
a
steep
learning
curve.
We
we
have
struggled
to
find
out
how
to
configure
the
pilot
and
the
control
plane
properly
to
support
our
configuration,
and
we
had.
We
had
quality
of
stability
issues
in
high
insight.
I
think
we
were
under
provision
for
what
we
are
trying
to
do
number
of
services
that
we
were
trying
to
support
and
when
the
control
plane
was
under
load.
A
We
started
to
receive
at
some
point
Corot
of
five
or
three
errors,
and
that
was
only
impacting
our
dev
environment.
But
we
did
impact
our
engineer
at
the
time
and
we
were
not
very
popular
at
that
time
and
we
almost
backed
out
of
obvious
to
you
at
that
point,
but
the
the
deploying
into
a
branch
was
was
so
powerful
for
one,
our
team,
that
we
stuck
with
it.
A
We
we
got
to
know
SEO
a
little
bit
more
and
a
little
bit
better.
We
we've
also
upgraded,
as
they
were
caught
of
bug,
fixes
committed
by
the
SEO
team
to
the
point
now
that
for
the
last
six
months
has
been
very
stable
and
we
were
really
happy
with
it,
and
we
actually
are
thinking
of
pushing
missed
you
into
staging
and
production
and
start
solving
different
problem
with
it.
B
So
if
sto
has
been
a,
you
know
huge
improvement
in
terms
of
unblocking
our
release,
train,
which
we
think
our
engineering
teams
that
find
it
indispensable
and
and
they're
using
it
an
awful
lot,
because
it
means
we
can
do
all
the
stuff
that
would
previously
be
in
our
pipeline
after
we
committed
or
merge
to
our
changes
into
our
mainline.
We
can
do
that
before
that,
and
now
we
can
release
into
mass
into
production
within
a
couple
of
hours
of
doing
massacres.
B
All
we
do
is
a
few
regression
testing
at
that
point,
whereas
in
the
past
we
do
a
change
and
then
any
impact,
any
problems
would
block
any
other
changes
potentially
and
the
really
fantastic
thing
about
it
is
the
micro
services
themselves.
Don't
know
anything
about
this,
we
other
than
the
edge
propagate
stuff.
We
haven't
had
to
introduce
anything
at
all
into
our
micro
services,
so
they
they
are
nice
and
simple
still.
B
So
we
think
it's
chairs
got
an
awful
lot
potential
and,
as
I
said,
we're
gonna
extend
its
usage
now
I
think
we're
much
more
comfortable
with
it
and
it's
a
little
bit
more
stable,
but
it
does
come
with
a
complexity,
particularly
when
it
doesn't
do
what
you're
thinking
it's.
It
should
do
so
I'd
like
to
thank
you
very
much.
The
some
of
the
projects
we've
been
doing
are
on
github.
B
D
D
B
C
C
E
B
F
B
This
is
a
problem
we're
currently
tackling
where
we
introduce
this.
It
wasn't
long
before
someone
wanted
to
do
a
change
for
the
make
the
schemas
that
we're
using
and
that
started
causing
his
headaches.
We've
got
a
couple
of
ideas.
One
of
them
involves
Sto
in
terms
of
migrating
through
to
production
and
running
multiple
versions
of
the
same
service
with
different
schemas,
but
within
our
development
environment.
B
We're
looking
at
actually
just
pushing
to
collections
or
databases
that
have
a
different
ticket
number
or
something
the
branch
variant
ID
in
it.
So
we
can.
We
can
split
up
effectively
a
database
quite
easily,
because
a
database
is
just
when
you
persister
it.
It
exists.
So
we
can
just
do
that
quite
quite
straightforward.
In.
B
We
automate
too
much
testing
as
we
can.
We
do
some
exploratory
testing,
which
is
manual
and
but
we
we
have
a
quite
a
good,
a
QA
team
that
will
work
with
our
engineers
to
come
up
with,
with
the
test
Suites
to
make
sure
that
we're
meeting
the
Quartier
because
we're
a
regulated
industry.
We
need
to
be
very
sure
that
that
the
test
results
reflect
what
the
requirements
are,
what
the
compliance
requirements
are.
B
C
B
Not
explicitly
so
the
issues
we
had,
but
we
first
started
trying
to
get
this
to
work
was
around
the
control
plane
and
actually
it
was
a
lot
more
in
resource
intensive.
Given
the
number
of
pods
and
services
we
were
running
in
terms
of
the
traffic,
we're
not
seeing
that
particular
slowdown,
but
we're
not
a
performant
we're,
not
performances
in
their
highest
requirement
at
the
moment,
particularly
in
our
engineering
environment.
There
was
some
talks
that
I've
been
to
over
the
width
over
this
week.
B
G
B
We
nginx
ingress
initially
that
was
we
had
that
pride
for
our
terrestrial,
then
anything
we
won
the
user
share
with
gets
routed
to
the
sto
gateway
which
handles
at
the
booting.
The
reason
we've
kept
the
nginx
ingress
there
is
because
it
provides
a
lot
more
functionality.
We
had
a
lot
of
rules.
You
know
we
talked
about
white
listing
a
second
ago.
We've
got
rules
around
allowing
our
monitoring
tools
into
our
network,
things
like
that
and
a
lot
of
the
security
requirements
that
we've
introduced,
like
cross-site,
scripting,
headers
and
stuff,
like
that.