►
Description
Confident Canary Deployment to Production With Istio - Raju Dawadi, Oyster
The session covers covers the production use case of Oyster Financial on using Istio service mesh for handling traffic. The testing in non-production environment and rolling out to live users was not effective for fintech product where the usage is critical. Also, due to the inconsistent in third party, there was need to test traffic in live environment for internal user and that has to be for selective or all services. The usage of Istio feature on routing traffic based on header as well as percentage rollout was used effectively which has made deployment to Prod0 seamless. Also measuring the performance as well as real use case test of newer version helped in providing a good end user experience for evolving fintech startup in Mexico. But the management complexity rises when number of services increases and there are too may configs to be managed. Combination of helm helped a lot throughout the process.
A
A
So,
starting
from
the
early
days
it
was
late
2018.
We
started
with
five
services
how's
it
going
up
to
80
in
in
the
span
of
mode
of
one
and
a
half
years,
with
both
grpc
and
the
http
services
and
from
the
early
days
from
the
day,
one.
We
started
adopting
ht
services,
who
said
a
lot
for
managing
the
traffic
routes
between
those
applications,
keeping
us
our
cluster
and
the
services.
A
But
they
can
communicate,
but
we
still
keep
it
secure
by
using
the
policies
between
them
and
a
very
good
health
section
mechanism
for
taking
the
liveness
and
the
readiness
props.
And
there
are
two
cases
like
for
serving
the
internal
users
and
for
the
live
users
for
getting
us
more
confident
during
the
release
time
and
for
that,
in
the
early
days,
we
built
two
stacks
of
our
application.
The
live
one
for
the
live
users,
which
is
called
green
and
another
one
for
the
internal
users,
which
is
called
gray.
A
A
So
for
that
we
started
applying
the
studio
virtual
service
routing,
based
on
header
headers,
that
before
releasing
the
app
to
the
green
users
or
the
live
users,
then
the
internal
user
can
test
a
rest
of
the
version
rest
of
the
green
version
of
the
services,
but
can
also
use
the
specific
gray
version.
So
in
this
case
we
can
see
the
internal
user
can
test
the
gray
version
of
the
user
service,
along
with
the
green
version
of
the
rest
of
the
services.
A
So
we
can
directly
plug
in
any
of
the
test
version
of
our
application
without
impacting
the
live
users,
and
when
we
are
confident,
then
we
release
that
and
also
further
the
integration
of
the
third
parties
or
our
partners
that
used
to
work
mostly
with
the
dive
and
the
staging
environment.
But
but
that
was
not
too
expected
for
in
that
way
to
run
inside
the
production
environment.
So
we
have
to
deal
do
a
lot
of
digging
to
get
the
proper
response
from
the
third
parties
or
the
partners.
A
So
in
that
scenario,
also
by
testing
the
endpoints
for
the
internal
users
before
reaching
to
the
live
users
that
give
us
very
much
flexibility
and
confidence
in
us.
So
for
that
we
use
the
history,
routing
rule
of
the
virtual
service
and
the
destination
rule
the
personal
service,
basically
with
the
mobile
app
sending
the
header.
A
So
in
this
case
the
tenant
have
gray
b1,
so
the
header
will
be
sent
from
the
mobile
to
the
api
and
to
all
of
the
services
and
based
on
that,
the
routing
will
be
done
with
that
to
be
sent
to
the
gray
version
or
the
green
version
and
for
the
destination
rule.
We
have
the
two
deployment
with
a
different
level
with
the
instance
that
will
separate.
That
would
be
the
green
or
the
great
release
of
the
app.
A
So,
during
this
transformation
we
faced
few
of
the
challenges
because
the
header
has
to
be
sent
from
the
mobile
to
the
api
and
to
rest
of
all
of
the
services.
But
in
some
cases,
while
doing
the
grpc
call
in
some
of
the
asynchronous
calls.
Some
services
didn't
send
the
header
to
that
service
and
we
handled
that
very
nicely
and
we
started
adopting
the
event-based
services.
A
So
also,
let's
say
there
was
a
case
like
the
gray
version
of
the
event-based
service
has
to
call
to
the
grade
or
the
green
version
of
another
service.
So
in
that
case
the
event-based
service
inject
the
header
based
on
the
environment
variable
presenting
it
also
the
syncing
with
the
mobile
app
team,
because
the
mobile
has
to
say
in
the
header-
and
we
need
to
be
see
in
sync
with
the
mobile
app
development
team,
as
well
as
the
backend
development
and
the
operations.
A
So
with
the
use
of
chiali,
we
are
able
to
scan
our
companies
if
they
are
outdated
or
if
they
are
not
good
or
if
there
is
any
problem
in
those
configuration
and
also
the
routing
things
like
how
we
services
calling
the
we
service
and
what
is
the
latency.
What
is
the
all
of
those
traffic
handling
between
those
services,
though,
that
is
for
a
small
duration
of
the
time
offered
by
the
kiali
and
also
for
keeping
traces
of
our
application.
A
We
used
jaeger
and
by
using
the
jaeger
of,
as
is
to
add
on
we
didn't
have
to
implement,
does
all
of
those
stress
inclined
to
each
of
our
micro
services,
which
are
very,
very
good
overhead,
and
so
that
that
took
us.
That
would
took
us
a
lot
of
engineering
resource,
also
the
time
and
resource
both
so
and
for
the
monitoring
part.
A
We
built
a
central
monitoring
dashboard
with
a
use
the
of
graphene
and
pulling
the
prometheus
metric
from
the
cluster
yourself,
but
also
from
the
service
mesh
or
from
the
seo
service
mesh,
and
by
setting
the
by
setting
the
threshold
of
the
of
the
monitoring.
We
get
a
very
good
allotting
things
to
our
on-call
system,
page
duty
as
well
as
the
slack,
so
that
will
be
proactive
in
terms
of
the
error
response
or
the
latencies
and
those
golden
metrics
of
the
business
logics
as
well.
A
So
that
was
a
very
good
path
towards
a
devops
model,
so
that's
all
obviously
centralized
all
of
the
resource
control
for
the
kubernetes
part
or
for
handling
the
resources
of
the
given
80s
so
by
the
use
of
istio.
We're
really
fast
enough
for
solving
our
business
requirement
and
getting
it
done
at
a
very
small
span
of
time
and
without
impacting
a
lot
of
customer
at
once.
So
that
is
a
very
cheering
point
for
us,
so
that's
it
for
now.
Okay,
bye.