►
From YouTube: The good, the bad, and the meshy a journey adopting Istio across 100 clusters at T Mobile
Description
#IstioCon2021
Presented at IstioCon 2021 by Joe Searcy.
This is a story of struggle, tradeoffs, and triumphs. Istio, as you may know, is a mission-critical piece of software for securing and connecting microservices across platforms. However, it can be daunting to introduce, operationalize, or adopt it successfully.
In this talk, we dig into T-Mobile’s journey of adopting Istio across 100+ clusters to support microservices for fraud detection, billing, sales and APIs across many teams. The journey was not all rainbows and unicorns. We cover things such as tenancy, install/upgrade, feature adoption, CI/CD integration, and architecture tradeoffs. We see first-hand lessons learned around:
The importance of iteration
Structured boundary and isolation
Approaches to multi-cluster
Stateful workloads in Istio
Team processes
Attendees of this talk will leave with a better understanding of how a large organization runs Istio to secure, and observe microservices in a large-scale deployment with all of its pros and cons.
A
A
Let
me
start
by
introducing
myself
a
bit,
I'm
a
technology
enthusiast
at
heart,
but
I
get
paid
money
to
be
a
staff
engineer
at
t-mobile,
with
a
specific
focus
around
distributed
systems
and
all
things
kubernetes
related
I've
been
working
with
the
kubernetes
and
istio
communities
for
a
while.
Now
I
try
to
stay
involved
in
the
upstream
communities
in
various
ways.
I'm
also
the
maintainer
of
the
open
source,
magtape
project
that
focuses
on
kubernetes
policy
as
code.
A
We're
not
here
to
just
talk
about
me,
though,
at
t-mobile,
our
platforms
and
services
are
the
product
of
a
team
of
rock
stars.
I'm
just
the
one
here
talking
about
it
today.
So
I
want
to
take
a
moment
to
shout
out
to
all
the
amazing
folks
that
work
with
me
to
make
all
of
what
I'm
about
to
cover
possible.
A
Now,
before
we
jump
into
the
service
mesh
story,
let's
do
some
quick
coverage
of
where
we
were.
When
we
came
to
consider
adopting
service
mesh
technology.
We
have
been
doing
the
kubernetes
thing
for
a
while.
We
started
to
get
to
the
point
that
most
people
do
where
one
cluster
turns
into
two
turns
into
ten
and
so
on.
A
We
still
had
a
pretty
diverse
environment
with
other
container
platforms
like
cloud
foundry
and
a
lot
of
applications
that
didn't
make
sense
to
drive
the
kubernetes
from
virtual
machines
and
bare
metal
systems.
Yet
security
was
a
big
concern.
As
our
platform
grew
and
new
risk
surfaces
emerged,
there
was
a
definite
learning
curve
in
the
container
space
for
our
developers,
and
we
wanted
to
simplify
that
as
much
as
possible.
A
A
A
A
A
We
package
our
own
internal
releases
from
a
combination
of
upstream
istio
releases
and
our
environment,
specific
configurations,
and
with
this
we
can
pinpoint
changes
to
specific
clusters,
entire
regions,
life
cycle
environments
or
the
global
feet
all
at
once.
This
model
greatly
simplifies
the
operational
overhead
of
managing
so
many
mesh
instances
with
a
small
staff
of
engineers.
A
A
This
is
part
of
the
process
where
we
learned
a
ton,
as
it
was
sort
of
a
new
model
with
respect
to
how
things
could
impact
consumers
of
the
platform,
with
the
data
plane,
sort
of
being
user
facing
to
some
degree,
to
help
ease
a
lot
of
this.
We
came
up
with
a
pretty
formal
process
for
promoting
istio
changes
and
upgrades
in
our
environment.
A
A
We
verify
things
through
established,
functional
tests
that
allow
us
to
assess
functionality
against
the
known
baseline
and
in
cases
where
we're
adopting
new
features.
We
may
have
specific
customer
teams
install
an
application
specifically
designed
to
take
advantage
of
those
and
test
those
features.
A
A
For
this
we
have
the
load,
balancers
dns,
the
tls
sort
of
pre-plumbed,
so
things
just
work
and
our
mesh
consumers
don't
have
to
worry
about
any
of
that
users
opt
in
to
consuming
the
mesh
via
namespace
labels,
again
with
the
goal
that
mesh
and
non-mesh
consumers
can
coexist
without
issues
altogether.
This
helps
to
give
mature
development
teams
immediate
access
to
service
mesh
features
and
also
provides
a
clear
runway
for
other
teams
to
adopt
over
time.
A
This
takes
the
concept
that
most
folks
are
used
to
with
an
ingress
controller
for
http
services
and
makes
it
possible
for
tcp
based
services
that
support
s.
I
we
get
the
same
one
to
many
name-based
routing
functionality
as
an
in-gas
controller
instead
of
mapping,
node
ports
or
burning
through
load
balancers.
A
A
We
started
to
play
around
with
integrating
non-kubernetes
services
into
the
mesh
things
like
virtual
machines,
we're
following
along
with
the
enhancements
coming
from
the
istio
project,
but
aren't
far
left
along
to
have
developed
any
real
opinions
so
far,
another
one
of
the
big
things
in
our
service
mesh
roadmap
is
moving
to
a
multi-cluster
mesh
topology,
while
we're
definitely
solving
problems
with
the
multi-mesh
topology.
We
have
today,
overall,
the
more
inclusive
the
mesh
is
the
more
useful
it
can
be.
A
There's
a
lot
of
cool
work
going
on
in
the
community
to
help
solve
the
soil
problem,
in
addition
to
the
multitude
of
topology
options
and
vm
integration
istio
offers.
Now
there
are
things
like
the
service
mesh
interface
or
smi.
That
provides
a
common
way
to
configure
mesh
functionality
and
also
things
like
solo.ios
glue,
mesh
and
blue
federation.
That
sort
of
show
this
stuff
in
practice.
A
A
There's
been
a
lot
of
work
over
the
past
few
releases
focusing
on
stability,
but
it
has
been
a
real
problem
in
the
past.
We
keep
a
trust
but
verify
attitude
these
days,
going
into
upgrades
and
sort
of
tying
into
the
release.
Stability,
we've
seen
default,
values
change
here
and
there
and
in
general,
the
api
changes
came
fast
and
furious
for
a
while
most
of
the
apis
are
mature
at
this
point
definitely
pay
attention
to
the
release.
Notes.
A
We've
been
bitten
more
than
once
with
things
outside
the
mesh
being
invasive,
some
in
slightly
annoying
ways
and
some
in
catastrophic
ways.
I
don't
have
time
to
dive
into
full
details
of
those
right
now
but
feel
free
to
follow
up
with
me
offline,
and
I
can
share
specifics
about
those
scenarios
releases
come
often
it
can
be
really
hard
to
keep
up
the
running
joke.
We
have
is
anytime
we're
in
a
meeting
talking
about
a
specific
istio
release.
There's
probably
a
new
seo
release
being
made.
A
Now
I
want
to
take
some
time
and
share
some
of
our
lessons
learned.
I've
tried
to
keep
these
focused
on
things
that
are
not
specific
to
our
environment,
but,
as
usual,
your
mileage
may
vary,
and
some
of
these
things
may
not
be
applicable
for
you,
we'll
start
by
addressing
some
things
around
complexity.
A
A
Here
are
a
few
things
to
keep
in
mind
that
we
found
helpful.
Do
not
let
buzzwords
or
the
fear
of
missing
out
drive
your
adoption.
I
know
everybody
likes
to
hear
what's
going
on
in
the
community
and
they
don't
want
to
be
left
out,
but
talk
through
your
needs
and
lead
by
features
that
are
important
to
you.
It's
also
entirely
okay,
if
you're
not
ready
or
don't
have
a
need
for
a
service
mesh.
Yet
next,
finding
real
world
examples
can
be
tough.
A
A
A
Next
make
sure
you
have
a
functional
test
for
every
mesh
feature
you
adopt
it's
easier
to
have
the
automation.
Remember
all
the
little
things
and
verify
that
everything
works
than
to
have
your
people
do
it.
This
is
also
helpful
when
things
change
upstream
and
can
allow
you
to
catch
those
sort
of
things
sooner
rather
than
later.
A
A
Next,
let's
take
a
brief
moment
to
talk
about
usability.
We
saw
a
huge
advantage
in
embedding
our
platform
team
with
application
development
teams.
This
boundary
is
closing
more
and
more
every
day
and
decisions
on
how
the
mesh
should
be
installed
and
configured
need
to
be
made
with
awareness
of
applications
in
your
environment,
meet
with
teams
regularly
to
identify
what
does
and
does
not
work.
A
A
That
being
said,
as
we're
testing
out
how
we
want
to
handle
our
multi-cluster
mesh
strategy,
we
found
that
there's
still
a
lot
of
edges
that
can
come
during
upgrades,
config
changes
and
environment
specific
oddities,
while
the
istio
project
supports
multiple
topologies
for
how
to
tackle
multi-cluster
meshes
and
some
even
offer
more
operational
savings
for
us.
We're
sticking
with
multiple
control
planes
to
reduce
our
overall
risk
of
impact.
A
On
more
than
one
occasion,
we
found
that
noisy
neighbors
can
be
a
real
problem.
Within
a
mesh
instance,
we've
had
mesh
users
affect
other
mesh
users
across
namespace
boundaries,
as
well
as
non-mesh
users
affecting
mesh
users.
The
isolation
boundaries
are
not
very
rigid
right
now,
and
this
was,
albeit
possibly
naive,
a
sort
of
surprise
to
us.
In
the
beginning,
I'm
happy
to
chat
afterwards
and
provide
some
additional
detail
around
these
problems.