►
From YouTube: Performance and Scale @ Istio Service Mesh - Fawad Khaliq, Laurent Demailly & Surya V Duggirala
Description
Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18)
Performance and Scale @ Istio Service Mesh - Fawad Khaliq, VMware Inc, Laurent Demailly, Google & Surya V Duggirala, IBM (Intermediate Skill Level)
In this session, members from Istio Performance and Scalability working group will talk about the work they have done over the past few months on performance and scalability using a four pronged approach: microservices applications across multiple industry domains, synthetic end to end per component load generation using Fortio, code level micro benchmarks and integration with the test and release. In this session, we will discuss best practices and tuning guidelines for effectively using Istio service mesh to obtain best performance and scalability. This session also talks about Istio performance characteristics on multiple cloud platforms including Kubernetes with few pointers on what to expect in the coming releases.
About Surya
Surya Duggirala is IBM STSM responsible for Architecture and Performance Engineering of IBM Watson and Cloud Platform. He directs a globally distributed team and chairs IBM Cloud performance engineering Guild. He also works closely with various Open Technology Communities and jointly leads various industry Workgroups on Performance and Optimization with other Cloud Providers in the industry. His special interests include designing microservices applications using scripting and cognitive technologies targeted for cloud. As a Global Technical Ambassador (GTA), he works with many customers, partners and ISVs across the world on cloud, application integration, performance and architecture. He has over 20 years of experience in various fields of the computer industry including application architecture and performance.
About Laurent
Laurent is one of the core contributors to Istio. He spent 20 years
so far in top companies and small startups alike in Silicon Valley, currently works at Google after almost 6 years at Facebook's core infrastructure. He has been an open source advocate and contributor since the dawn of the web and is a large scale distributed systems expert. Laurent holds a PhD in Distributed Systems.
About Fawad
Fawad is a senior software engineer at VMware working on Istio, APIs and microservices. Previously, he was a technical lead for the Cloud Group at PLUMgrid, a network virtualization and security startup. He has worked in the networking and distributed systems domain for several years and has been an open source contributor to various OSS projects. His current work is focused on service mesh and Istio.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
C
A
A
So
we
have
a
multi-level
approach
that
we'll
go
over
and
then
we'll
go
over
forward.
We'll
go
over
the
environments
that
we
are
using
we're
testing
in
the
multiple
multitude
of
environments
and
there's
many
scenarios
that
we're
looking
at
then
I'll
take
back
the
mic
and
we'll
look
at
the
characterization
and
probably
more
more
interestingly
to
you
guys
the
issues
that
we
found
out.
We
found
them
how
we
fix
them
so
far
and
then
Surya
will
take
on
the
mic
and
talk
to
you
guys
about
how
we
study
in
a
realistic
application.
A
Oh
the
previous
talk
was
about
a
steel,
so
our
working
group
is
meeting
regularly
and
since
December
we
studied
the
default
and
I
want
I
want
to
say
a
few
words
about
our
parts.
So
there's
many
levels
in
which
we
can
look
at
the
performance
characterization
of
a
system
and
we
do
it
at
those
at
multiple
levels.
First,
a
lot
of
ECU
is
written
in
the
go
language.
Go
makes
it
fairly
easy
to
write
benchmarks.
A
So
we
do
have
a
lot
of
benchmarks
at
the
component
level
to
try
to
make
sure
that,
for
instance,
mixers
library
as
efficient
as
possible
and
processing
the
rules,
and
we've
made
significant
improvements
in
the
last
few
months
in
mixer,
for
instance.
The
second
level
is,
we
do
synthetic
benchmarks,
so
we
simulate
the
setup,
and
for
that
we
wrote
a
tool
called
for
tio,
that
of
Osseo
that
lets
you
basically
be
a
next
generation
echo
server,
alot
generation
and
lets
you
visualize
the
results.
A
We
also,
then,
at
the
next
level,
use
something
that
more
realistic
for
applications,
which
is
the
blue
proof.
It's
also
open
source
like
for
to
you,
so
you
can
reproduce
the
results
that
we
produce
in
your
in
your
environment,
and
this
is
what
we
are
trying
to
do.
We're
trying
to
set
up
some
base
tools,
some
set
of
libraries
that
anyone
can
use
in
order
to
test
in
their
particular
cloud
environment
or
their
particular
setup,
because
it
still
works
on
QA.
B
Oh
thanks
all
right,
packed
room
here,
I'm
pretty
sure
a
lot
of
you
guys
are
users
of
different
type
of
cloud
environments,
AWS
Google,
IBM,
on-prem,
OpenStack,
maybe
cloud
foundry
and
looking
at
service
mesh
and
must
be
wondering
if
these
guys
are
actually
testing
or
looking
at
scenarios
from
your
angle.
So
I'll
try
to
cover
most
of
these
if
there's
something
which
we
are
not
covering.
Please
come
and
talk
to
us
after
the
presentation
and
we'll
be
happy
to
talk
about
this
or
you
know
more
than
that.
B
Just
come
in
contribute
as
well
to
the
group.
So
the
what
I'm
going
to
cover
is
the
different
environments
and
scenarios,
some
of
those
that
we
have
done
so
far
as
part
of
the
working
group
and
those
include
GE
IBM,
a
deep
us
as
your
and
some
on-prem
data
that
we
got
from
one
of
the
customers
was
using
these
tools.
So
let's
go
over
how
it
looks
like
in
GK.
B
This
is
part
of
the
East,
your
repository,
where
there's
there's
a
an
environment,
set
up
a
framework
that
you
can
use
to
bring
up
the
entire
end-to-end
cubed
as
a
deployment
with
sto
40.
Oh,
and
there
are
few
test
cases
that
you
can
run.
For
example,
one
that
you
see
here
is
where
you
can
deploy
40
you
as
a
client
server,
both
doing
part
about
performance
testing,
and
you
can
just,
for
example,
what
you
see
over
here
on
top
right
is
a
pod
active
server,
1
and
X
over
2.
B
If
both
of
them
have
sidecar,
so
we
will
check
kind
of
data
plane
and
also
some
mixer
rule
checks
and
check
performance
on
that
topic.
The
other
one
is
where
you
can
also
include
ingress
to
see
the
results
for
your
performance
and
that
you
know
traffic
comes
from
40,
to
instance,
running
on
a
GC
EVM
outside
the
cluster,
goodness
cluster,
and
the
third
one
is
where
you
wanna,
run
test
cases
without
sto
to
see
the
comparison
of
how
it
looks
like
where
they
steer
and
without
sto
wedding
dress
without
Angus
etcetera.
B
That's
one
of
the
things
which
is
there
Laurent
is
gonna,
go
into
details
of
some
of
the
desserts
we
have
captured
from
this
scenario
in
this
environment.
Another
one
is
an
IBM
cloud.
What
Lauren
mentioned
was
regression
Patrol.
This
is
part
of
the
automation
and
CIA
automation
which
covers
which
provides
us
data
on.
You
know
how
daily
builds
are
looking,
and
this
is
the
sample
app,
which
is
there.
It's
kind
of
looks
like
a
real
deployment
app,
which
has
a
flight
authentication
piece.
B
A
customer
booking
has
a
way
to
publish
the
radiograph
Anna
that
you
can
go
and
look
at
lot
of.
Surya
is
going
to
cover
this
and
we'll
give
you
a
small
demo
of
this
one
as
well
and
in
this
one
instead
of
using
4000,
it
uses
geometer,
which
runs
on
a
on
a
vm
outside
the
cluster
and
generates
traffic,
and
most
of
it
goes
to
the
ingress.
So
it
kind
of
looks
at
the
holistic
view
or
real
industry
use
case
point
of
view.
B
Then
vmware
we've
been
looking
at
from
AWS
perspective
and
what
it
does
is
that
you
could
bring
up
a
kubernetes
cluster
using
key
ups
and
there's
a
jupiter
notebook
which
allows
you
to
bring
up
your
cluster.
You
can
define
what
kind
of
test
case
you
want
to
run
or
scenario
you
want
to
run,
and
you
know
it's
super
configurable,
so
you
can
justify
just
defined
like
number
of
40
instances.
It's
40
s
being
used
over
here
for
load
generation,
but
on
the
server
side
were
using
l1,
which
can
handle
100,
100
thousand
plus
requests
per.
B
Second,
it's
supposed
to
be
super
robust,
since
working
well
for
us
and
the
kind
of
scenarios
that
we
started
was
with
was
without
hto,
where
this
teo
and
with
auth
and
then
also
move
to
without
mixer,
see
the
comparison
of
how
numbers
look
like
if
we
go
and
talk
to
anyone,
if
you
know
someone
comes,
is
like
I'm
gonna,
adopt
service
mesh
and
does
it
like?
What's
the
cost
that
I'm
trying
to
cover?
So
there
are
some
numbers
that
we
captured
from
here,
our
latency,
etc
and
soui
I
will
talk
about
some
CPUs
numbers.
B
Will
be
use
over
there
so
just
to
give
an
example
here.
This
is
some
sample
deployment
that
you
can
use
the
Jupiter
notebook
and
you
know
just
enable
a
steal
and
then
run
the
scenario
game
to
capture
the
results,
and
then
you
can
also
enable
authentication
to
capture
the
same
scenarios
for
that
same
results.
B
Another
topic,
another
cloud
environment
where
one
of
our
colleagues
from
the
community
Kari,
has
been
running
these
scenarios
inside
Azure
and
what
he
has
is
he's
been
running
with
geometer,
which
talks
to
there's
your
load.
Balancer-
and
it
looks
like,
as
is
to
your
ingress
configured
for
the
you-
know,
traffic
path
and
there's
a
target
service
which
has
two
nested
services.
One
is
an
HTTP
back-end,
the
other
one
is
a
RPC
back-end
and
it's
running
on
aqueous
pressure
and
how
it
looks
like
is
some
results
that
we've
captured
over
here.
B
Is
you
see
on
the
left
side?
It's
first
15
minutes
captured
the
service
target
service,
is
sending
traffic
to
the
HTTP
back-end
and
afterwards
is
sending
the
GOP
sea
back
in,
and
you
see
some
difference
over
there
there.
Of
course,
you
may
think
there
are
many
variables
here.
It
could
be
application
code
as
well,
but
there
is
some
you
know:
numbers
we
have
captured
over.
B
Therefore,
in
terms
of
differences
between
different
application,
different
out
paths
and
the
last
one
I
want
to
cover
is
the
on
Prem,
where
for
tu
is
being
used
to
generate
traffic
and
is
a
hardware
load
balancer?
This
is
a
completely
arm
deployment
and
the
summary
service
looks
very
similar.
We
had
to
anonymize
it
because
of
the
customer
and
there's
an
entry
service
which
has
two
nested
G
RPC
backends,
and
there
are
some
results
being
captured
over
there.
B
A
You
so,
as
I
mentioned,
we
were
you
as
I
mentioned
earlier.
Sorry,
one
of
the
one
of
the
way
we
we
characterize
the
performance
of
sto
is
through
40.
So
the
idea
for
to
you
is
kind
to
be
a
modern
micro
service
area,
Swiss
Army
knife
of
network
testing.
So
it's
doing
a
lot
of
a
lot
of
little
function,
all
that
in
a
very
small
image
of
a
few
megabytes.
A
So
you
can
deploy
hundreds
of
it
if
you
wanted
to
in
order
to
simulate
a
fairly
deep
or
wide
cluster
and
and
still
be
very,
very
efficient.
So
it's
also
a
go
library.
So
we
use
it
inside
is
to
you
to
do
functional
testing,
not
just
performance
testing.
So,
for
instance,
we
send
exactly
200
requests
and
we
make
sure
there's.
No.
You
can
read
the
map
of
the
result
codes
and
we
make
sure
that
there's
no
errors
or
that
the
metrics
actually
add
up
exactly
and
there's
no
no
errors.
A
So
it
has
a
component
which
is
a
command
line
and,
as
a
client
has
a
load
generator
and
also
a
server-side
like
an
echo
server
bit
like
HTTP
bin,
where
you
can
customize
the
result
code
that
it
replies
the
payload
size,
many
parameters,
so
the
the
main,
the
key
element
there
is
to
run
the
system
at
a
fixed
QPS.
So
if
you,
what
you
want
to
do
when
you're
trying
to
characterize
performance,
is
you
want
to
push
your
system
higher
and
higher?
And
then
you
find
the
point
where
it
starts
to
saturate?
A
Maybe
the
CPU
is
saturated
or
the
iOS
is
saturated,
and
then
you
want
to
walk
back
from
that
about
80%
and
and
run
the
test
at
that
fixed
ups
in
order
to
get
the
latency
that
you
expect
to
get
the
distribution
of
latencies
that
you
expect
to
get
in
in
your
production
system.
So
this
is
one
of
the
main
features
it
can
run
at
a
set
capias
and
there
is
a
bit
of
a
a
way
to
visualize
the
data.
A
It's
also
fairly
high
performance
because
it
it
can
easily
do
400
QPS
on
self-testing,
so
I'm
gonna.
Do
a
bit
of
a
small
demo
of
the
actual
system,
so
some
of
its
study-
this
is
the
it's
big
enough.
So,
as
you
can
tell
it's
built
by
engineers,
so
the
UI
is
not
it's
not
supposed
to
be
pretty,
but
it
has
a
bunch
of
features
you
can
set
up
whether
you're
gonna
do
a
gr,
PC
or
an
HTTP
load.
A
This
is
the
UI
to
actually
create
a
lot,
so
it's
going
to
run
against
itself
and
the
eco
server
that's
running.
You
can
tell
it,
for
instance,
in
this
case
or
in
five
percent
of
the
requests
introduced,
a
five
millisecond
delay
and
also
in
the
oh
five
percent
of
the
requests.
Please
please
make
a
five
five
or
three
instead
of
a
two
hundred,
so
we
click
here.
It
runs
against
itself
and
then
you
get
you
get
the
result
and,
as
you
can
see,
there
is
indeed
like
a
bump
there.
A
So
this
is
the
scenario
one
the
service
one
to
service
to
in
the
picture
earlier
there
was
two
services,
and
then
we
have
a
what
I
call
a
cache,
busting
rule.
So
there's
a
rule
that
makes
it
that
in
one
direction
you
never.
The
mixer
results
are
never
cashed
and
the
other
direction.
They
are
always
cashed.
A
So
that
gives
you
a
zero
percent,
which
is
the
worst
case
scenario
and
hundred
percent,
which
is
the
best
case
scenario,
and
then
you
can
adjust
those
two
numbers
to
get
an
idea
of
what
it's
going
to
be
for
your
actual
cash.
It
may
show
that
you
expect
the
mixer
dashboard
also
shows
the
cash
ratio,
so
you
can
see
in
your
production
deployment
what
cash
it
may
show
you
actually
get.
A
So
here
you
can
see
the
evolution.
This
is
the
zero
five
released
in
the
middle,
the
zero
six
release
and
latest
release,
because
your
8
is
not
out
yet,
which
is
zero,
seven
one.
So
in
that
setup
the
black
line
is
the
QPS.
You
can
see
that
we
went
from
about
700
QPS
from
2,000
to
PS
and
now
to
above
1700
QPS
at
at
the
max.
The
latency,
obviously
is
not
is
not
great.
The
p90
line
is
very
high
because
we
are
pushing
the
system
there.
It's
like
cpu-bound.
A
So
here
we
can
compare
to
$2
point
where
this
is
your
six.
This
is
your
seven
and
if
you,
if
you
put
the
two
together,
you
can
see,
you
can
exactly
compare
exactly
the
distribution
of
data
and
the
curve
of
the
percentage
cumulative
percent.
You
I
think
I'm
gonna
go
back
to
the
presentation,
but
if
you
have
any
more
questions,
you
can
look
up.
The
song
github
is
tu,
/
patio
there's
a
lot
of
dogs
and
examples,
and
those
data
is
on
photo.
That
is
tutorials.
You
can
go
look
up.
A
Okay,
so
this
is
what
I
showed
you
so
now
for
the
interesting
things
like:
how
did
we
get?
How
was
it
like
six
hundred
QPS
and
now
it's
1700
gps
in
in
three
months?
So
what
happened?
Well,
a
lot
of
it
is
what
happens
when
you
write
production
software?
Is
you
write
it
and
it
seems
fine,
and
then
you
load
it
and
unexpected
things
happen.
One
easy
way
to
to
get
into
trouble
is,
is
logging
part
exactly
so
to
do
a
good
job?
A
So
we
we
found
that
in
mixer
we
were
logging
things
for
every
single
check
and
that's
way
way
too
much
logging
for
outside
of
troubleshooting
another
good
one
that
happens
is
when
you,
when
you
load
test,
or
you
run
a
long-running
test.
You,
you
might
discover
a
memory
leak,
so
in
this
case
we
had
in
the
Zipkin
library
there
was.
It
was
spawning
goroutines,
unbounded,
number
of
collisions
and
yet
all
accessing
a
single
lock
to
try
to
extract
that
out
of
that
lock,
so
it
was
was
basically
consuming
memory
by
consuming
goroutines.
A
Thanks
to
the
the
right
patch
hole
that
Sarah
is
going
to
talk
about.
We
found
that
at
one
release
we
actually
were
doing
twice.
The
number
of
mixer
calls.
There
was
a
mix
up
in
the
and
void
configuration
of
the
filter,
and
there
was
two
entries
for
for
the
mixer
filters.
We
were
doing
twice
as
many
calls,
which
obviously
doesn't
improve
the
throughput.
A
Likewise
for
TCP
and
alpha
closed
issues
that
were
causing
both
functional
and
performance
problems.
If
you
recall
I
mentioned,
we
do
have
that
cache
busting
rule
right.
The
way
that
cache
busting
rule
is
you
can
check
it
on
on
github.
Is
it
doesn't
end
between
something
that's
either,
always
true
or
always
false,
and
then
evaluates
something
that
reads
from
the
request:
ID,
which
is
always
different,
so
cannot
be
cached.
Well,
there
was
a
release
where
the
the
short-circuiting
wasn't
actually
working,
so
the
zero-percent
cache
and
100%
cache
were
actually
always
the
zero
percent
caching.
A
A
So
we
look,
we
look
at
performance.
We
will
also
look
at
scalability
right.
So
when
you
add
a
hundreds
of
services,
you
have
to
make
sure
that
pilot
and
mixer
and
each
company
is,
you
can
handle
that
scam,
and
there
was
a
bit
of
a
problem
with
our
we
we're
using
the
the
API
server,
go-go,
client
library
and
we
ended
up
crashing
pilot
fairly
early
with
excessive
memory,
so
that
was
also
fixed.
So
these
two
items
that
were
still
working
on.
A
Okay,
so
so
these
two
items
that
we're
still
working
on
I'll
talk
a
bit
about
the
last
one,
because
it's
a
low-hanging
fruit
right,
the
cash.
The
way
the
cash
works
today
is
that
it's
going
to
cash,
the
mixer
result
for
up
to
100
requests
or
one
second,
and
this
is
just
what
happens,
what
we
have
decided
like
a
year
ago.
We
just
put
that
value
as
the
default
value
and
we
haven't
really
looked
at
it
until
recently.
A
Obviously,
it's
a
bit
excessive,
because
if
you
make
a
policy
change
that
it
that
it
applies
exactly
within
one
second
of
your
policy
change
or
one
hundred
requests,
this
is
too
much,
so
we
could
easily
put
that
to
be
a
thousand
requests
or
or
five
second
and
and
we
would
get
a
much
better
cash
it
cash
performance.
So
we're
going
to
make
that
change
and
make
it
configurable.
A
So
you
can
change
it
to
whichever
you
want
and
the
other
item
we're
still
working
on
is
which
we're
trying
to
explain
why
the
mixer
client
performance
doesn't
seem
to
be
what
we
expect,
because
when
we
test
with
the
internal
mixer
client
in
goal,
we
get
much
better
performance
than
what
we
get
end
to
end
with
the
invoice.
So
we're
gonna
fix
that
too.
A
So
those
numbers
are
not
the
absolute
numbers
that
we
want
to
have
right.
We
want
to
have
single
millisecond
overhead
for
foresty,
oh,
but
the
trend
is
in
the
right
direction
and
by
end
of
here
we're
committed
to
to
work
toward
that.
Do
those
better
codes,
that's
another
view
of
the
improvements.
0.4
is
even
older
and
that's
that
data
and
now
I
will
hand
it
to
Surya.
C
Thank
you,
Thank
You
Lauren,
as
you
can
see
as
Laurent,
and
for
what
mentioned
from
a
community
point
of
view
from
a
steel
community
point
of
view,
we
had
three
goals.
We
want
to
make
sure
that
there
is
a
regression
mechanism
built
in
as
we
develop
the
service
mesh
and
the
second
one
is
actually
we
want
to
make
sure
not
only
the
specific
primitives
so
that
you
can
go
deep
and
fix
any
design
issues,
but
also
go
in
a
like
a
breadthwise.
C
You
want
to
go
to
the
multiple
industry,
use
cases
and
actually
work
right
at
the
beginning
itself
to
understand
and
fix
any
kind
of
performance
and
scalability
issues
towards
that.
As
we
have
introduced,
the
regression
patrol
one
of
the
benchmarks
that
we
have
used
is
called
the
AK
near
blueproof.
It's
actually
open
sourced,
so
all
of
you
can
actually
take
take
a
look
at
that.
It
is
basically
a
flight
reservation
system.
It
has
five
or
six
specific
micro
services.
So
you
can
actually
see
that
you
know
when
you
deploy
this.
C
It's
actually
being
now
run
on
IBM
cloud,
but
it's
actually
a
steer
that
IO.
You
can
actually
go
and
see.
So
what
will
happen
is
if
I'm,
the
mix
of
developer
or
the
pilot
developer
I
go,
and
you
know
just
integrate
my
code
changes
and
every
night
there
is.
This
is
do
regression
bucket.
It
runs
in
the
midnight
and
you
can
actually
see
the
results
and
then
see
it's
actually
color
coded.
If
you
see
a
much
bigger
change
in
performance,
if
it
breaches
the
threshold,
you
can
actually
see
that.
C
So
what
will
happen
once
you
see
a
regression
right,
so
you
can
actually
see
this,
so
people
will
come
in
interest
of
time.
Actually
we
videoed
it
this
one,
so
they
will
go
and
check
it
out.
Okay,
what
is
the
latest
bill
like
we
have
three
scenarios
running
so
use
to
your
full?
That
is
the
default
configuration
he'll
go
on
check.
These
are
all
automatically
generated
data,
so
they'll
go
and
look
at
it.
Whether
pilot,
if
I'm,
a
pilot
developer
I'll
go
on
check
out.
C
Is
there
something
that
I
have
done
wrong
here
that
created
the
regression?
Here
we
have
an
integrated
dashboard.
You
can
see
the
East
EO
and
our
own
custom
dashboards
put
together
and
you
can
see
in
this
the
worker
node
all
the
metrics,
that
you
have,
whether
you
have
a
disk
issue
or
a
io
issue,
or
you
have
any
other
specific
issues
that
you
know
crept
in
that
caused
the
regression
and
you
can
actually
look
at
the
East
your
dashboard
and
then
see
whether
you
have
any
specific
service
level
problems
that
you
have.
C
Maybe
you
have
a
sidecar
issue,
it's
taking
more
CPU
or
memory.
You
know
all
those
things
that
actually
automate
it
and
since
we
rolled
out,
we
identified
many
many
regressions
and
actually
this
is
improving
the
quality
of
the
sto
service
mesh
itself,
as
we
are
developing
and
I
would
like
to
go
and
show
you
how
the
whole
network
traffic
flows
through
this
is
the
one
that's
running
on
a
three
node
cluster.
You
can
see.
This
is
the
service
mesh
deployed
on
a
cluster,
and
this
is
like
without
any
load.
C
So
you
can
see
the
pilot
and
the
sidecar
interaction,
the
packet,
that's
going
there
network
traffic,
and
let
me
now
we're
actually
going
to
drive
some
load
here
from
the
from
J
mirror.
Now,
once
you
send
the
load
from
there
that
load,
the
traffic
will
go,
the
HTTP
requests
will
go
through
the
east,
you
ingress
controller
and
you
can
actually
see
once
the
load
kicks
in.
C
You
can
see
the
network
flow,
how
this
whole
traffic
is
actually
going
through
and
we
can
identify
any
kind
of
bottlenecks
in
the
whole
system
and
you
can
see
that
traffic
going
through
from
the
client
in
Stephen
Chris
controller.
Now
it
will
go,
it
will
get
into
the
flight
service
and
then
yeah.
The
flight
service
is
the
is
the
main
service
that
takes
most
of
these
transactions,
and
then
it
will
get
into
art,
service,
booking
service
and
others
too
right
you
can.
So.
C
This
is
one
way
to
clearly
see
whether
you
have
any
kind
of
an
issue
the
in
the
system
and
if
you
see
some
kind
of
a
red
dots
that
shows
some
kind
of
an
errors.
You
can
see
the
incoming
connections
and
outgoing
connections
here,
how
many
transactions
each
of
those
services
are
taking.
What's
the
throughput
that
you're
getting?
What
is
the
latency?
All
that?
Can
we
clearly
see
this
will
give
a
pretty
good
overview
of
you
know.
How
is
tio
is
performing
and
scaling
in
a
realistic
customer
application.
C
As
you
can
see,
there
are
config
a
config
GU,
the
way
we
develop.
This
is
actually
we
have
a
default
configuration
we
look
at
and
then
we
disable
the
mixer.
We
disable
all
the
other
components
and
then
look
at
the
performance
of
the
East.
You
ingress
controller
right.
Just
we
want
to
make
sure
that
which
component
is
actually
having
issues
and
what
are
the
different
industry
use
cases
that
we
are
using
right?
You
can
see.
This
is
the
polyglot
micro
service.
C
Each
service
is
developed
in
a
different
language,
one
in
Java
spring
boat
go,
and
you
know
you
can
see
that
in
node
right.
So
it's
available
it's
an
open
source.
You
can
actually
go
and
take
a
look
at
a
blooper
and
another
thing
that,
because
this
tio
there's
a
lot
of
interest
from
the
community
from
the
industry,
we
want
to
make
sure
that
the
real
customer
use
cases
are
being
looked
at
as
we
are
developing
this
framework.
One
of
them
is
the
online
retail
online
banking
application.
We
have
enabled
sto
with
that.
C
This
is
the
application,
that's
actually
in
production
in
one
of
the
North
American
banks,
we
have
what
we
have
done
with
this
is
actually
we
have
applied
sto
service
mesh
and
we
drove
this
to
almost
4.5
billion
API
calls
per
day.
We
want
to
really
push
that
to
to
its
limits.
That's
without
is
to
you
we
could
reach
4.5
billion,
but
we
want
to
understand
what
is
the
additional
capacity
that
you
need.
What
is
the
additional
overhead
you
have
to
pay
when
you
enable
issed
you
right?
C
Those
are
the
things
that
we
are
looking
at
another
in
the
Street
benchmark
that
we
are
looking
at
is
in
the
healthcare
industry.
That
is
the
the
typical
back-end
for
front-end
micro-services
pattern
is
another
thing
that
we
are
actually
looking
at
for
is
tio,
so
this
is
the
online
banking
application,
so
you
can
see
with
and
without
sto.
What
is
that
additional
overhead
right
now,
you
can
see
is
around
56%,
so
you
need
another
50%
additional
capacity
to
get
the
same
kind
of
throughput
as
a
community
we
are
trying
to.
As
a
working
group.
C
We
are
working
hard,
as
Lauren
mentioned,
we
have
a
pipeline
of
things
to
optimize.
One
of
the
main
things
that
we
are
trying
to
look
at
is
the
mixer
overhead,
but
right
now
you
can
see
a
typical
industrial
use
case.
You
need
this
is
the
distribution
without
this
do,
if
you
distribute
where
you're
actually
spending
the
resources
to
get
that
158
million
API
calls
per
day
at
around
3:30
millisecond
response
time,
you
can
see
the
the
olb
Orchestrator
is
taking
around
150
percent
and
stub
is
doing
120
and
ingress
is
taking
65%.
C
So
what
will
happen
if
you
apply
is
to
your
service
mesh?
You
can
see
this.
You
are
paying
almost
like
two
hundred.
Eighty
percent
with
these
to
your
mixer
is
teo.
Ingress
is
a
one
hundred
and
forty
percent,
so
you
can
see
why
this
additional
overhead
right
now
the
main
focus
is
on
the
mixer.
We
are
working
on
that
at
this
point,
so
another
area
is
because
sto,
if
you
see
there
is
a
sidecar
pattern,
so
you
have
for
each
service.
C
You
have
a
proxy
that
on
by
proxy
attached
to
it,
so
our
as
part
of
our
analysis,
we
are
also
looking
at
what
is
the
impact
of
the
sidecar?
What
is
the
impact
when
you
have
an
egress
traffic?
That's
going
out
to
a
public
versus
the
same
traffic
going
to
an
on-premise
service.
There
is
a
big
difference
in
both
of
those
two.
So
all
those
things
actually,
we
are
working
in
the
community
and
one
of
the
other
things
that
we
really
want
help
from
you
is,
as
we
are
continuing
to
optimize
the
sole
service
mesh.
C
The
things
that
we
are
working
right
now,
as
I
mentioned
the
regression
Patrol
right
now
daily,
it
runs
so
our
goal
is:
we
are
trying
to
build
a
kind
of
a
bot
as
well,
so
that
automatically,
if
you
find
a
specific
issue,
it
can
actually
log
the
issue
and
send
a
you
know
like
a
slack
message
to
a
specific
channel,
maybe
for
pilot
or
mixer.
So
those
are
the
things
in
the
works
right
now,
so
we
need
your
help
in
continuing
to
push
this
and
optimize.