►
From YouTube: Istio Day 2 Operations
Description
SoloCon 2022:
Istio Day 2 Operations
Speakers:
Ram Vennam
Field Engineer, Solo.io
Nick Nellis
Field Engineer, Solo.io
Session Abstract:
In this session, two Field Engineers who work with production Istio users everyday, share what they learned in the field with a focus on running it at scale. Istio is powerful, but managing it in a large environment with HA and stringent security requirements poses some challenges. Some of the topics that we will cover in this session include:
- No downtime Istio lifecycle management in production
- Establish security and configuration boundaries
- Leveraging existing observability systems to monitor health
- Debugging tools when things aren't working as expected
Track:
Community and Open Source
A
B
Yep
thanks,
I
also
have
been
working
on
istio
for
a
number
of
years,
implementing
it
also
contributing
to
the
istio
co-base.
Now
I
spend
a
lot
of
my
time,
helping
our
customers
implement
and
architect
and
adopt
istio
and
service
mesh
best
practices
production
support.
So
I
spend
a
lot
of
time
helping
our
customers
walk
through
the
life
cycle
of
using
istio
and
helping
with
the
gotchas
and
then
also
keeping
their
environments
healthy.
A
One
of
the
biggest
challenges
with
istio
that
our
customers
face
is
around
the
life
cycle.
Management
of
it,
but
really
the
the
zero
downtime
upgrades
is
where
a
lot
of
our
users
might
get
stuck
on.
You
know
how
to
go
from
one
version
to
another
when
the
versions
are
coming
so
fast,
whether
it's
a
patch
update
or
or
a
minor
update.
A
How
quickly
do
you
have
to
keep
up
and
and
how
much
time
you
want
to
invest
in
doing
that
and
when
you're
talking
about
zero
downtime
updates,
you
know
istio
is
just
not
just
you
know
one
single
monolith
right:
there's
the
control,
plane
component,
there's
a
side
card
next
to
each
one
of
your
applications
and
then
there's
the
one
or
many
gateways
so
you're
talking
about
you
know
several
different
components
here
and
you
can't
just
update
all
of
them
at
once.
You
kind
of
have
to
go.
A
We
kind
of
have
to
build
a
like
a
custom
solution
for
for
our
clients,
and
we've
been
working
with
the
community
to
kind
of
improve
this
approach
over
time.
So
you
know
we've
written
blogs
on
istio.io,
as
well
as
on
solo.I
o
for
the
various
versions.
These
are
just
three
blogs
that
we
wrote
over
the
last.
I
think
these
are
all
three
blogs
were
written
last
year
and
they
go
into
the
various
ways
you
can
go
about
updating,
depending
on
what
your
requirements
are,
the
the
one.
A
That's
all
the
way
on
the
right
upgrading
is
here
to
downtime
nick
actually
wrote
that
at
the
end
of
last
year,
for
a
customer
that
required
absolutely
that
had
no
tolerance
for
a
potential
downtime.
So
there
was
duplication
of
we're
doing
like
a
blue
green
approach
where
you
deployed
the
old
version
of
istio.
The
new
version
of
istio
and
you're
kind
of
deploying
multiple
components
and
that
block
post
will
kind
of
go
talk
about
in
more
detail.
A
Thankfully,
the
upstream
istio
project
itself
has
has
come
huge
strides
in
in
terms
of
improvement
on
how
to
do
update
from
from
one
to
another,
before
the
only
update
was
kind
of
you
know
directly
like
in
place
where
you
just
update
all
the
components
you
know
and
then
just
kind
of
hope
that
everything
works,
but
then
gradually
there
were
canary
style
updates
where
you
can
just
do
the
control
plane
separately.
But
then
you
update
your
data
plane
in
place
or
you
upload
your
gateways
in
place.
A
A
A
So
you
have
your
old
control
plane
and
the
new
control
plane.
So
it's
a
new
deployment
that
comes
with
its
own
istio
d
service
and
it
also
comes
with
its
own
version,
injector
as
well,
and
then
to
update
your
your
data
planes,
which
should
be
your.
Your
next
step
is
instead
of
using
the
the
normal
name
space
label.
Where
you
just
you
know,
say:
istio
injection
enabled
equals
to
true.
You
set
the
that
label,
istio.io
rev
to
the
revision
name,
and
that
basically
tells
you
know
istio
which
injector
to
use.
A
A
You
label
all
your
name
spaces
to
use
that
revision
and
then,
when
a
new
revision
comes,
you
just
update
that
label
and
then
restart
the
workloads
that
are
in
that
namespace.
You
do
a
rolling
deploy
of
the
workloads
in
that
namespace
and
then
the
new
injector
comes
in
and
and
updates
the
sidecars
to
the
new
one.
So
you
can
so
this
diagram
is
showing
a
transient
state
where
you
know
some
of
your
name.
A
So
to
talk
a
little
bit
about
that,
injected
gateways
are
basically
just
they
work
like
the
sidecar
does
meaning
when
you
first
deploy
a
gateway,
it's
just
a
stub
and
the
injector,
the
istio
sidecar
injector
will
go
fill
in
the
the
correct
proxy
image
and
all
the
configurations.
So
it
works
very
much
like
a
sidecar
where
it's
just
a
templated
gateway.
So
this
allows
you
to
deploy
a
second
gateway
that
runs
next
to
your
original
gateway,
but
this
time
the
the
second
gateway
was
injected
by
the
112
sidecar
injector.
A
So
now
you
have
two
ingress
gateways
deployed.
Both
of
them
will
have
be
exposed
and
they're
using
their
own
kubernetes
service,
so
they'll
have
their
each
to
have
their
own.
Endpoint
gives
you
a
chance
to
test
out
the
canary
version
of
the
gateway,
and
then,
if
all
is
well
with
the
canary
version,
then
you
can
just
do
a
straight
in
place.
Update
of
your
original
gateway
or
just
you
know,
switch
your
external
load
balancer
to
the
endpoint
of
the
canary
ingress
gateway.
B
Yeah,
I
think
you
know
a
lot
of
customers
that
try
out
sdo,
don't
actually
aren't
privy
to
the
revision
tags
up
front.
It's
all
the
seo.
Examples,
don't
really
mention
them
until
you
get
to
the
day
two
type
operations,
and
so
we
see
a
lot
of
our
customers
today,
not
using
revisions
in
production
and
and
when
they're
looking
for
safer
upgrades.
One
of
the
first
suggestions
that
we
do
make
is
that
they
actually
migrate
to
a
revision
based
deployment
and
can
future
upgrades
after
that
go
a
lot
smoother
and
a
lot
safer.
A
A
You
know
the
center
of
our
customers
world
right,
like
it's
meant
to
be
yet
another
tool
to
solve
a
particular.
You
know
challenge
in
the
kubernetes
stack,
so
they
don't
have
like
that.
Much
amount
of
time
to
orchestrate
this.
You
know
you
know
revision
istio
with
like
a
second
revision
and
then
switching
from
one
name
space
to
another
like
this.
This
is
a
very
simple
environment
and
sometimes
the
istio
documentation
makes
it
you
get
the
feeling
that
you
know.
They
think
that
the
whole
world
revolves
around
istio
so
like
they.
A
This
orchestration
is
not
something
that
our
customers
typically
want
to
deal
with,
at
least
most
of
them.
You
know,
because,
because
most
of
our
customer
environments
don't
look
like
that,
but
instead
they
look
something
like
this.
Where
there's
application,
there's
lots
of
different
name
spaces.
Each
namespace
is
owned
by
a
different
team.
There's
applications
talking
to
one
another,
so
you
can't
you
can't
just
set
aside
a
time
that
says
here's
a
new
revision.
A
A
A
Yes,
by
all
means,
you
know,
adopt
revisions,
do
everything
correctly,
but
for
some
of
our
customers
you
know
that
that's
too
much
overhead
and
they
would
rather
just
spin
up
a
new
cluster
with
a
new
new
istio
version
and
move
things
over
or
they
have
like
failover
backup
clusters
or
active
active
clusters
where
they
can
partition
one
of
the
clusters
off
for
construction
construction
and
then
update
it.
So
we'll
talk
a
little
bit
about
that.
A
So,
in
order
for
you
to
to
to
do
things
like
that,
where
you
have
multiple
clusters-
and
you
can
just
kind
of
you
know-
pick
one
and
say
like
okay-
this
is
under
maintenance
mode
and
update
in
place
and
then
pick
another
cluster.
A
management
plane
would
really.
You
know
help
in
that.
In
that
orchestration,
the
glue
mesh
our
glue
mesh
product
comes
with
a
management
plane,
meaning
that
this
management
plane
is
typically
runs
in
its
own,
separate
cluster
and
then
each
one
of
your
workload
clusters.
A
They
have
a
management
plane
agent,
the
glue
mesh
agent
that
reaches
out
to
this
management
plane
to
get
its
config.
That
allows
you
to
have
a
single
place.
The
management
plane
where
you
apply
all
of
your
service
mesh
config
and
that
management
plane
will
kind
of
distribute
that
that
config
to
each
one
of
your
worker
clusters
that
also
simplifies
east-west
cross-cluster
traffic.
A
So
one
istio
cluster
knows
about
the
end
points
of
your
applications
in
a
separate
istio
clusters,
and
it
can
fail
over
across
clusters
just
as
easily
as
it's
sending
traffic
to
a
local
cluster.
So
it
really
like
helps
you
kind
of
group
all
of
your
clusters
together
and
build
one.
You
know
essentially
one
larger
cluster,
which
we
call
you
know
a
virtual
mesh,
which
is
a
grouping
of
multiple
meshes
together
and
you
can
do
traffic
routing
between
them.
B
That
that's
a
really
important
concept,
like
that's
something
that
doesn't
exist
in
istio
today,
is
a
global
representation
of
your
environment.
Istio
is
really
isolated
kind
of
to
your
local
clusters,
and
so
you
need
a
way
to
tie
those
all
together
so
that
they
know
about
the
wider
environment
that
they
run
in.
A
Exactly
so
you
can
you
know
you
can
use,
you
might
be
thinking.
Oh,
I
could
just
use
like
some
sort
of
ci
cd
solution
to
like
stamp
out
multiple
clusters
for
like
an
active,
active
setup,
but
when
you
do
that
you're,
not
enabling
cross-cluster
routing.
So
if
in
this
example,
if
the
recommendation
app
on
your
left
were
to
go
down
for
whatever
reason,
whether
it's
you
know
crashed
out
of
memory,
the
node
went
down
or
whatever
you
want.
A
The
web
api
service
to
just
automatically
fail
over
to
the
recommendation
service
running
in
a
different
cluster,
and
you
want
to
be
able
to
do
that
without
without
intervention
right
without
definitely
not
a
code
change.
But
you
know
you
don't
want
any
sort
of
explicit
policy
to
enable
you
to
do
that.
A
So
what
that
one
then
allows
you
to
do
is
very
easily
block
a
cluster
from
from
your
ingress
traffic,
your
main
north-south
traffic,
and
then
let
this
failover
you
know,
work
its
magic
so
that
all
the
cluster,
all
the
endpoints
are
propagated
to
the
to
your
other
clusters
and
all
the
traffic
continues
to
work
and
from
an
external
client
perspective.
They
don't
even
know
that
anything
happens.
B
Alternate,
alternatively,
you
could
also
bring
up
a
new
cluster
with
those
applications
and
with
this
management
plane
I
can
automatically
detect
those
applications
and
those
gateways
and
then
add
them
to
the
pool
of
resources
that
you're
already
routing
to,
and
so
you
can
also
expand
as
well.
As
you
know,
take
things
down
for
construction,
yep.
A
So
from
the
management
plane,
you
can
apply
a
custom
resource
like
let's
say
in
this
example,
we're
applying
this
istio
installation
custom
resource
which
tells
the
management
plane
what
istio
installation
you
want
to
install
and
on
what
clusters,
what
profile
and
any
overrides.
So
it's
like
an
abstraction
on
top
of
the
istio's
istio
operator
api
that
is
multi-cluster
aware
and
then
the
management
plane
will
take
care
of
figuring
out
what
it
wants
to
name
the
revision
and
how
it
wants
to
do
the
update
and
the
installation
in
each
one
of
those
clusters.
A
The
next
portion
I
want
to
talk
about
is
observability,
so
istio
does
somewhat
come
with
comes
with
like
out
of
the
box
observability
tools,
you
know
they
don't
come
with
like
the
default
profile,
but
if
you
install
the
the
demo
profile
or
just
go
to,
I
think
there's
the
add-ons
documentation
page.
It
tells
you
how
to
deploy
a
demo
version
of
you
know,
grafana.
You
know
prometheus
and
kiali,
etc.
A
The
the
reason
why
it's
it's
just
like
a
demo
instance
is
because,
for
a
production
instance
of
you
know
these
observability
tools,
you
would
need
to
have
a
true
like
persistence,
layer
where
it
can
save
all
of
this
information,
all
these
metrics
traces
et
cetera,
so
that
you
have
a
historical
record
and
you
can
go
back
and
observe
patterns,
but,
and
most
of
our
customers
do
have
things
like
that,
or
at
least
like
enterprise
prometheus
or
they
have.
You
know
things
like
datadog.
A
The
point
I'm
trying
to
make
is
that
the
add-ons
that
come
out
of
the
box
with
istio
are
good
for
you
to
get
started,
but
you
should
immediately
like
move
on
off
of
them
and
integrate
with
your
existing
observability
systems
that
allows
you
to
to.
You
know,
maintain
things
like
his
much
longer
historical
records
not
have
to
do
like
an
istio
specific
dashboards
instead
you're
integrating
it
with
your
you
know,
existing
infrastructure
and
kind
of
just
make
it
more
pervasive
and
part
of
your
your
normal
kubernetes
application
deployment
life
cycle.
A
Another
thing
that
you
should
not
be
scared
to
do
is
create
your
own
custom,
grafana
dashboards.
This
one
is
a
this
one
is
a
great
example,
and
I'm
going
to
let
like
nick
talk
about
this,
because
he
actually
helped
one
of
our
large
customers
identify
performance
issues
with
with
their
environment
by
creating
these
dashboard.
B
Yeah,
so
we
we
had
a
customer
that
very
experienced
with
istio
in
a
large
environment,
doing
some
artificial
intelligence,
but
they
were
having
some
performance
segregation
in
a
number
of
different
areas
of
istio
and
when
they
looked
at
those
dashboards,
that
issue
gave
them
out
of
the
out
of
the
box.
They
were,
they
were
quite
overwhelming
in
terms
of
what
do
I
need
to
look
at
and
what
do
I
care
about,
and
so
we
hooked
into
their
existing
metrics,
which
they
were
using
grifana.
B
We
created
some
custom
dashboards
for
them
that
highlighted
the
things
that
they
needed
to
care
about
in
terms
of
debugging
and
solving
some
of
their
istio
related
problems,
and
this
was
at
a
pretty
large
scale.
So
we
were
looking
at
thousands
of
pods
and
thousands
of
services
and
some
pretty
big
environments,
and
then
they
also
had
k
native
on
top
of
it.
B
And
so
you
know
this
dashboard
just
really
helped
them
dive
deep
into
the
things
that
they
cared
about
and
were
trying
to
tune
and
specifically
memory
usage
with
each
sidecar
and
then
the
response
times
of
the
istio
control
plane.
And
so
we
put
a
lot
of
metrics
on
around
those
areas
to
see
if
we
could,
you
know
bring
down
that
memory
usage
and
also
bring
down
the
translation
times,
and
so
definitely
don't
be
afraid
to
isolate
down
to
the
things
that
you
care
about
when
you're.
Looking
at
sdm.
A
As
you
grow
to
to
multiple
clusters,
one
of
the
benefits
of
a
management
plan
is
that
it
can
act
as
a
metrics
aggregator
in
in
the
glue
mesh
scenario
on
each
one
of
the
workload
clusters.
We
run
this
enterprise
agent
component
that
acts
as
a
metric
sink,
which
goes
and
scrapes
all
the
metrics
from
each
one
of
your
application
pods
and
then
sends
that
to
the
management
plane.
A
You
know
it
decorates
those
metrics
with
information
like
the
cluster
metadata,
for
example,
and
then
it
sends
it
to
the
management
plane
and
the
management
plane
could
be
a
place.
The
management
cluster
can
be
a
place
where
you
deploy
your
prometheus
server.
Your
production
prometheus
server
that
aggregates
istio
metrics
across
all
of
your
your
workload
applications,
and
then
you
know
you
can
use
your
own
observability
tools
that
sit
on
top
of
that
those
metrics
or
you
know,
use
you
know
the
glue
mesh
dashboard.
A
B
Great
yep,
so
as
you
mature
through
your
life
cycle
of
istio,
you
eventually
need
to
start
looking
at
some
of
the
internals.
You
need
to
get
familiar
with
the
tooling
the
architectures
to
help
you
be
successful
with
istio.
It
isn't
something
you
can
just
drop
in
and
hope
it
never
has
problems,
because
there
ultimately
will
be
problems,
but
with
the
proper,
tooling
and
proper
observability
you
can
actually,
you
know,
isolate
down
and
hopefully
remedy
them
relatively
quickly.
If
you
empower
yourself.
B
But
debugging
istio
is
hard,
and
the
reason
for
that
is
that
you
have
a
number
of
different
roles
and
responsibilities
that
all
tie
into
istio.
You
have
the
developer,
who
uses
those
sidecars
to
abstract,
away
its
logic
and
then
handle
security,
routing
outlier
detection
circuit
breaking
stuff,
like
that,
so
a
lot
of
the
functionality
that
developers
are
kind
of
used
to
has
now
moved
out
into
the
istio
landscape.
B
And
finally,
you
also
will
have
a
big.
You
know.
A
lot
of
reasons
that
istio
is
used
today
is
for
security,
and
so
most
often
our
larger
customers
have
security
teams
involved
within
their
istio
deployments
and
that's
making
sure
there's
uniform
policy
enforcement,
such
as
like
an
mtls
policy
that
all
communication
is
encrypted
and
authenticated.
A
A
We
didn't
quite
have
these
like
personas
laid
out
quite
as
clearly,
but
you
know
now,
as
istio
becomes
more
and
more
mainstream
we're
noticing
that
you
know
these.
These
developer
operations
and
security
personas
be
more
clearly
delineated
where
each
one
of
them
has
a
specific
use
case
that
they
want
out
of
istio.
B
But
thankfully
there's
a
tool
for
everyone,
and
that
is
provided
by
the
istio
community,
known
as
istio
ctl.
It
is
our
go-to
tool
for
debugging
most
of
our
istio
problems
within
our
customers.
Environments
and
the
reason
being
is
that
it
it
tries
to
visualize
or
make
known
more
internals
of
what
istio
is
doing
from
the
control
plane
or
even
debugging
the
sidecars
themselves.
B
And
then
that
goes
from
secrets
to
the
envoy
proxy
configuration
to
the
health
of
your
sdod
control
plane.
And
so
all
those
personas
should
be
get
should
get
really
comfortable
with
using
istio
ctl
and
we're
going
to
talk
about
some
of
the
different
ways
that
we
use
it
for
some
different
use.
Cases
that
are
pretty
common
in
our
customer
environments.
B
Istio
ctl
kind
of
enables
you
to
get
a
pretty
good
view
of
those
decisions
that
are
being
made
by
istio
to
program
the
envoy
proxy,
whether
it's
setting
up
the
listeners
for
the
inbound
ports
and
protocols
or
where
traffic
is
routed
or
how
and
how
it's
manipulated,
such
as
you
know,
request
header,
if
you're
adding
request,
headers
or
envoy
filters
and
then
finally,
the
android
clusters,
which
helps
determine
where
the
the
traffic
is
going
to
be
routed
and
how
it's
going
to
be
routed.
And
so
you
can
see
multiple
roles
coming
in
here.
B
For
example,
the
security
role
may
come
and
take
a
look
to
make
sure
that
mtls
is
being
enforced
and
they
just
want
to
check.
They
may
need
to
check
that
the
configuration
meets
the
security
requirements
of
their
organization
and
we've
had
certain
organizations
actually
do
an
audit
of
their
configuration
to
make
sure
that
it
is
to
what
you
know.
Their
standards
expect.
A
This
is
a
good
way
for
you,
like
nick,
is
saying
this
is
a
good
way
for
you
to
to
audit
to
verify
to
debug
the
applied
configuration
at
the
end
of
the
day.
All
istio
is
doing.
Is
it's
converting
your
istio
apis
to
thousands
of
lines
of
onway
config
so
using
the
istio
ctl
tools,
you
can
actually
go
look
at
a
subset
of
that
envoy
config
to
make
sure
that
it
istio's
done
what
you
expected
to
do
and
and
if
there's
not,
then,
then
you
know
where
to
look
exactly.
B
A
B
B
B
So
we're
going
to
start
with
a
little
bit
of
a
simple
architecture
here
of
how
would
you
join
a
virtual
machine
that
doesn't
exist
in
a
kubernetes
cluster
or
a
service
mesh?
How
would
you
join
that
to
istio
and
the
sidecar
communicates
over
a
protocol
called
xds,
and
so
what
we
do
is
we
we
install.
You
can
install
an
envoy
proxy
for
an
istio
proxy
in
that
virtual
machine
and
then
have
it
communicate
back
with
an
istio
control
plan
via
this
xds
protocol,
and
we
do
that
through
typically
an
east-west
gateway.
B
So
you
get
secure
encrypted
communication
between
the
virtual
machine
proxy,
all
the
way
to
istio,
and
so
that's
a
typical
architecture
of
joining
a
virtual
machine
to
initial
mesh.
But
we've
actually
worked
with
some
other
customers
to
come
up
with
some
alternatives
because
a
lot
of
times
this
isn't
feasible
in
everyone's
environment.
B
B
One
final
thing
I
did
want
to
mention
too
is
that
designing
these
architectures
based
on
our
customers,
environments
and
how
they,
how
they
are
in
real
life,
has
allowed
us
to
make
some
strides
and
customers
who
wouldn't
typically
adopt
virtual
machines
joining
surface
meshes.
But
we've
now
been
able
to
connect
data,
centers
and
clouds,
and
so
we've
been
designing
architectures
that
make
it
easier
for
you
to
like
migrate,
your
virtual
machines
to
the
cloud
in
an
iterative
pattern
and
we'll
walk
through
that
next
yeah.
A
B
So
today
the
default
istio
installation
only
works
for
abuse,
few
select
environments,
but
we've
been
finding
that
our
customers
are
asking
for
it
in
a
lot
of
different
environments
that
they've
run
in
today,
and
so
the
biggest
ones
are
docker
and
potentially
windows
and
so
they're.
Looking
for
solutions
that
issue
just
doesn't
offer
today,
and
so
we've
been
actually
working
to
make
that
a
lot
easier
as
well
and
a
side
effect
of
that
is,
we've
actually
been
able
to
connect
local
machines.
So
we've
a
lot.
B
Another
another
architecture
pattern
that
actually
we
saw
is
that,
instead
of
installing
istio
proxies
on
all
of
your
virtual
machines,
it
would
it
made
more
sense
to
install
a
forwarding
proxy
that
gave
those
virtual
machines
access
to
service
meshes.
And
so
we
helped
a
customer
design,
a
virtual
machine
that
just
had
an
istio
proxy
on
it
that
we
manually
configured
to
allow
virtual
machines
to
securely
communicate
with
applications
in
the
service
mesh.
B
B
This
solution
also
is
enhanced,
if
you
put
it
behind
something
like
glue
mesh
and
so
with
the
power
of
blue
mesh.
In
that
global
view,
with
the
management
plane
you
now
can
be
able
to
actually
route
to
any
east-west
gateway
within
your
mesh
and
glue
mesh
will
help
that
request
figure
out
where
it
needs
to
land,
and
so
this
is
offering
a
high
availability
now
to
your
virtual
machine
requests
in
my
example.
B
Here
you
can
actually
call
any
east-west
gateway
within
blue
mesh
and
it
the
request
will
be
intelligently
routed
to
the
nearest
application
and
so
by
adding
a
management
plan.
On
top
of
all
of
your
seo
clusters,
you
can
actually
enhance
the
uptime
of
your
virtual
machines,
communicating
with
applications
in
the
mesh.