►
From YouTube: Organize Your Mesh - How to Run a Multi-Tenant Service Mesh in Production- Christian Posta, Solo.io
Description
Organize Your Mesh - How to Run a Multi-Tenant Service Mesh in Production- Christian Posta, Solo.io
Service meshes offer a breadth of benefits from securing to adding reliability to gaining visibility into your applications. However, as you start to scale your environment and start onboarding different teams or applications into the mesh you run into challenges of tenant isolation in terms of configuration management, resource consumption and security. In this session, Christian will present how to securely operate and run a multi-tenant mesh in production using the primitives available from service mesh like Istio. You will also learn how to take these concepts from a single cluster to multi cluster environment and successfully run applications across different clusters in a multi tenant unified service mesh.
A
Like
lin
said,
I'm
going
to
talk
today
about
service,
mesh
and
and
and
tenancy
and
some
of
the
problems
that
we've
run
into
we.
You
know
where
I
work
here
at
solo,
where
we
work
with
people
adopting
service
mesh,
and
you
know
we'll
look
at
some
practical
examples,
I'll
try
to
mix
in
some
demos
and
then
we'll
look
at
some
of
the
things
that
we've
done
as
a
solutions
that
may
be
interesting
to
to
you
all.
A
A
There's,
I
think,
close
to
500
pages
of
istio
knowledge
in
there
accumulated
over
that
time,
and
you
know
it
went
to
print
back
in
march.
We
are
doing
a
meet
the
author.
I
thought
it
was
a.
It
was
a
book
signing,
but
it's
a
well
we'll
give
away
digital
copies
of
of
the
book,
but
you
can
also
request
physical
copies
and
we
can
send
them
out
afterward.
A
So
at
the
con,
the
solo
servicemesh
con
booth
at
510
come
stop
by.
So
I
work
at
solo.
You've
probably
seen
a
few
of
us
speaking
here
today.
We
work
on
solving
application,
networking
problems.
How
do
services
connect
and
how
do
we
drive
policy
for
those
services
across
you
know
anywhere
where
workloads
might
be
deployed
in
kubernetes
on
vms,
across
kubernetes
clusters,
across
private
and
public
clouds,
and
we
bring
a
lot
of
interesting,
open
source
technology
to
those
solutions.
A
You
may
recognize
some
of
us
not
only
having
spoken
here
but
being
leaders
and
contributors
maintainers
in
the
open
source
projects.
This
deal
specifically,
you
know,
and
we
bring
a
lot
of
that
expertise
to
our
the
people
that
we
hire.
So
it's
a
good
opportunity
to
learn,
as
well
as
to
our
our
our
customers
I'll
speed
through
some
of
these.
I
know
some
of
the
other
sessions
showed
that,
but
getting
to
the
the
main
point,
the
main
reason
for
this.
A
This
talk,
as
I
was
saying,
we
work
with
a
lot
of
organizations
that
are
adopting
service
mesh
and
some
of
these
organizations
are
the
largest
deployments
in
the
world
or
the
most
complicated
or
approaching
the
most
complicated
because
of
the
types
of
organization
organizations
that
are
adopting
them.
These
are
you
know,
financial
services,
companies
or
insurance
companies,
retail
companies,
a
lot
of
them
that
have
been
around
for
a
really
long
time.
A
They
have
ways
of
you
know
how
they
run
their
business.
They
have
organizational
policies
that
you
would
have
never
guessed
would
make
sense
just
by
working
in
the
open
source.
You
have
to
actually
go
see
and
experience
it
for
yourself
really,
and
so
one
of
the
one
of
the
things
that
you
see
that
comes
up
almost
every
single
time
is
when
you
are
adopting
this
new
technology
when
you're,
when
you're
modernized
you're
bringing
in
containers
you're
bringing
in
these
some
of
these
cloud
solutions.
A
A
I
I
was
having
a
an
interesting
discussion
with
with
a
customer,
not
that
long
ago,
and
they
said
they
don't
have.
This
particular
one
which
I
would
have
assumed
of
any
of
the
customers
we
talked
with,
would
have
a
big
tenancy
problem.
They
said,
no,
we
don't
have
a
tenancy
problem
because
we
run
a
single
cluster
per
application
and
I
was
thinking
well.
They
must
have
had
some
tendency
problem,
otherwise
they
wouldn't
have
been
running
the
the
in
that
mode.
A
But
then
I
just
threw
this
out
there
on
twitter
and
got
a
really
really
good
response
to
it,
and
you
could
see
people.
People
were
adopting
and
deploying
kubernetes
clusters
specifically
and
the
way
they
looked
at
tenancy
in
in
all
of
the
different
extremes
that
that
you
could
possibly
imagine
one.
You
know
some
people
responded.
Oh
yeah,
we
run
massive
massive
clusters,
openshift
clusters
or
just
whatever
kubernetes
clusters,
and
you
know
they
went
into
some
of
the
details
about
the
the
tendency
problems
there.
A
Someone
even
responded
that
they
run
a
single
pod
per
cluster,
which
I
thought
hopefully
was
not
real
but
but
nevertheless
go
go
through
the
you
know
go
through
this.
This
thread
there's
some
interesting
things
there,
but,
but
really
you
know
the
things
that
stood
out
not
only
in
that
thread,
but
generally
when,
when
we're
talking
about
tenancy
and
and
why
it
gets
hard
is
it
involves
a
lot
of
different
facets.
It
involves
the
infrastructure,
of
course.
A
How
do
you
do
that
across
these
various
teams?
What
mechanisms
might
already
exist
in
in
in
the
mesh
that
you
might
be
looking
at
now?
Of
course,
I've
been
involved
with
this
deal
for
a
long
time,
I'll
be
looking
at
this,
the
lens
of
istio,
but
but
the
questions
and
the
scenarios
aren't
aren't
tied
to
to
istio.
Specifically.
A
Istio
has
an
ingress
gateway
which
not
every
service
mesh
has,
but
this
is
actually
pretty
pretty
useful
as
a
way
to
get
started
with
a
service
mesh
if
you're
not
comfortable,
deploying
sidecar
proxies
everywhere,
but
starting
with
a
single
proxy
getting
traffic
into
the
system
and
incrementally
adding
from
there.
It's
a
good
place
to
start
now,
when
you
have
multiple
teams
that
you
might
be
servicing
or
or
enabling,
with
with
this
with
a
mesh
or
with
with
technology
like
this,
you
have
to
take
into
account.
A
Well,
I
need
to
expose
the
service
a
certain
way
for
this
team
and
this
team
actually
wants
to
own
more
control
or
have
more
control
over
how
it
exposes
its
services
in
istio.
There's
already
a
split
in
the
api
between
you
know
what
what
what
happens
on
the
gateway?
What
happens
when
we
open
a
port?
What
protocol
is
it
speaking?
What
security
we
might
want
to
associate
it
with
with
it
some
simple
security
and
then
another
part
of
the
api
that
specifies
traffic,
routing
and
matching
and
splitting
and
and
and
so
on.
A
A
A
A
All
right
we'll
go
here
and
we'll
go
here
all
right,
so
the
first
thing
we're
going
to
do
is
we're
going
to
take
a
look
at
like
I
mentioned
the
gateway
api
and
the
virtual
service
api.
I'm
also,
this
is
a
live
demo.
I
I
clearly
didn't
type
that
out.
I
had
it
scripted,
but
this
is
a
live
demo.
I'm
usually
terrible
at
typing
live
so
you're
gonna
see
the
the
script
here,
but
the
first
thing
to
notice
is
that
we
specify
the
gateway.
A
It's
pretty
straightforward,
specified
some
security
properties
about
it,
the
port
we
want
to
open
and
and
the
protocol
we've
also
specified
a
hostname
where
we
want
to
match
on
on
traffic,
and
then
you
know
eventually
delegate
that
to
to
something
else
that
will
specify
the
routing
rules,
which
is
the
virtual
service,
all
right.
So
if
you're
familiar
with
seos,
it's
pretty
pretty
pretty
straightforward.
A
The
second
thing,
we'll
notice
is
we
can
from
that
matching
we'll
say
well
for
these
rules.
I
want
these
to
be
applied
to
a
certain
gateway.
So
there's
already
things,
there's
already
mechanisms
in
istio
to
allow
you
to
kind
of
split
up
and
isolate
exactly
where
the
configuration
should
be
coming
from.
So
this
is
already
applied.
Let's,
let's
actually
call
it,
you
should
be
able
to
see
it.
A
A
So
if
you
were
to
look
in
the
virtual
service
here,
you
know
the
reference
docs
here
goes
into
the
details
of
that.
So,
let's,
let's
actually
try
that
we're
going
to
expose
a
service,
a
new
service
that
hasn't
been
exposed
on
the
gateway
yet
using
the
the
gateway
api
and
and
we're
going
to
delegate
traffic
routing
rules
to
the
ratings
team
and
the
way
we
do
that,
let's
apply
the
gateway,
real,
quick.
A
The
way
we
do,
that
is
by
specifying
that
we're
going
to
we're
gonna
delegate
the
rules
right,
we'll
control
the
top
level
domain
matching.
We
can
add
additional
capabilities
there,
that
the
end
user
team
shouldn't
be
configuring,
and
then
we
can
delegate
the
traffic
routing
rules
to
to
the
to
the
team.
A
A
A
But
unfortunately
we
do
not
so
istio
does
have
the
ability
to
delegate
the
routing
table
to
other
teams,
but
there's
some
nuance
here.
So
if
we
take
a
look
at
the
the
gateway
itself,
we
see
that,
like.
In
the
previous
example,
we
were
we're
actually
telling
istio
to
pull
the
routing
rules
from
a
specific
name
space.
A
In
this
case,
we
don't
want
to
let
the
ratings
team
decide
what
all
of
the
routing
rules
are.
So
we're
only
delegating
part
of
it.
The
routing
rule
still
the
matching
rule,
the
for
the
host
name,
actually
lives
in
the
in
the
same
name
space
as
as
this
gateway
resource,
not
with
the
the
team,
so
we
gotta,
we
gotta,
understand
that
and
make
sure
that
we
unders
that
we
configure
things
correctly
once
we
start
going
down
the
path
of
of
of
delegation.
A
A
That's
one
component
to
now.
If
we
think
about
splitting
out
traffic
rules,
we
split
out,
you
know
what
configuration
belongs
to
what
team
this
is.
This
is
kind
of
one
of
the
first
things
that
you
can
start
to
to
look
at
now.
If
we
take
and
expand
this
a
little
bit
more
to
you
know,
we
looked
at
sort
of
traffic
coming
in
how
you
start
to
think
about
tenancy
at
the
at
the
edge
level.
A
You
know,
let's
say
you,
you
own
the
service.
So
in
this
example,
this
is
a
recommendation
service
which
is
calling
another
service,
the
purchase
history
service,
all
right,
so
maybe
as
the
as
the
developer
owner
of
the
purchase
history
service,
you
care
about.
You
know
things
like
how
how
how
the
host
name
gets,
matched
or
cores
or
traffic
splitting,
maybe
you're,
introducing
a
new
version
or
something
all
right,
so
istio's
apis
for
doing
that.
Are
you
know
the
virtual
service,
the
destination
rules,
potentially
the
sidecar
resource.
A
Should
the
team
that
provides
the
service
on
that
resource
or
should
it
be
shared
somehow
and
they
should
run
through
some?
You
know:
pull
request
mechanism
to
try
to
get
everybody
merged
and
get
all
the
configs
on
the
same
page
right,
and
so
we
see
some
of
the
con,
the
contention
that
can
come
up
around
this
api-
and
you
know,
we've
seen
in
various
ways
for
teams
trying
to
trying
to
mitigate
this.
But
what
I'm
gonna
go
into.
Real
quick
is
a
little
bit
more
detail.
A
A
Ingress
traffic
comes
in
and
hits
the
web
api
service.
That
service
then
calls
another
service,
the
recommendation
service
and
that
service
calls
yet
another
one.
The
purchase
history
service
now
by
default
when
we've
deployed
this
in
our
in
our
cluster
istio
configures
the
web
api
service,
to
know
about
everything,
and
that
might
not
be
what
what
we
want,
especially
if
we're
thinking
about
tenancy
and
what
should
it
know
or
what
should
be
it
be
able
to
connect
to.
A
If
we
look
at
the
the
clusters
that
the
web
api
service
knows
about,
if
we
come
over
here
and
take
a
look
at
the
command
line,
real
quick
we're
just
asking.
If
so
just
tell
me
about
the
clusters
or
the
upstream
services
that
the
web
api
service
knows
about.
We
see
it
knows
about
a
bunch
of
different
things,
including
purchase
history.
Ratings
recommendation
all
this
stuff,
but
web
api
only
calls
recommendation.
A
In
this
case
I
think,
and
and
that's
it
right
so
we'll
we'll
trim
down
the
configuration
there.
So
let's
apply
this
that
got
created
now,
let's
go
back
and
check
what
are
the
services
upstream
that
the
web
api
service
knows
about?
It
knows
about
recommendation,
and
it
knows
about
some
other.
You
know
helper
services
in
the
control,
plane
and
so
on.
It
doesn't
know
about
the
whole
list
and
it
doesn't
need
to
so.
We've
started
to
kind
of
trim
down
and
and
focus
what
the
configuration
should
be
for
a
particular
tenant.
A
Okay,
now,
the
last
little
bit
here
that
we're
going
to
focus
on
is
when
we
apply
a
configuration,
a
virtual
service
or
a
destination
rule,
or
you
know
any
any
of
the
configs
where
we
might
need
to
control
the
connectivity
between
services.
You
know.
One
thing
I
I
pointed
out
was
the
contention
that
we
see
between
two
different
teams
trying
to
share
a
virtual
service,
for
example,
but
we
don't
have
to
share
a
virtual
service
so
in
seo
we
can
take
a
virtual
service
and
we
can
explicitly
say
that
this
configuration
this.
A
These
routing
rules
only
apply
for
services
that
live
here
in
this
namespace.
So
maybe
this
is
where
the
services
are
actually
running
like
in
this
case.
This
would
be
the
web
api
virtual
service.
Maybe
we
can
expose
this
directly
into
where
the
you
know
the
web
api
service
actually
runs.
We
can
put
that
virtual
service
there
for
clients
that
care
about
timeouts
retries,
all
this
other
stuff
that
might
be
different
from
what
the
server
you
know
is
is
providing.
A
A
So
let's
apply
this
and
we'll
apply
it
to
some
of
the
other
services
here.
Real,
quick
and
we're
also
going
to
take
a
look.
Oh,
I
guess
I
didn't
reset
the
demo
and
we're
also
going
to
take
a
look
at
the
another
configuration
in
istio
called
the
service
entry
and
the
service
entry
allows
you
to
configure
access
to
other
services
that
either
live
outside
of
the
mesh
or
to
build
a
a
a
globally
aware
name
that
can
be
then
used
to
resolve
workloads
that
might
live
in
in
the
master
across
clusters
right.
A
But
if
you
were
following
along
with
me
on
the
demos,
it's
it's
pretty
tedious
to
try
to
think
at
least
for
a
single
team.
Where
does
where
does
this
configuration
need
to
live?
Where
does
it
need
to
go?
You
know
you
need
to
think
about
it
a
little
bit,
and
now
you
start
to
expand
that
you
mix
delegation,
you
mix
well,
the
gateway
host
matching
can
deal
with
namespaces
and
is
that
virtual
service?
Really
there
is
it
exported
correctly?
And
now
you
start
to
take
this
across
multiple
services.
A
Hundreds
of
services
across
clusters
now
trying
to
manage
the
tenancy
model
in
your
head
is
not
ideal,
but
this
this
can
get
complicated
very
quickly,
and
so
what?
What
we
work
on
with
our
customers
at
solo
and
what
we've
seen
people
adopt
in
the
community
is
building
blocks
workflows.
On
top
of
the
constructs
that
istio
gives
istio's
api
was
intended
to
be
sort
of
a
lower
level
api
with
workflows
built
on
on
top
of
it,
and
the
the
tenancy
construct
is
one
that
people
have
have
built.
A
You
know
if
you,
if
you
look
at
the
talks
from
istio
con
this
year
and
last
year,
you
know
you
see
you
see,
organizations
that
have
talked
about
their
journey
build
these
these
types
of
constructs,
so
basically
it
comes
down
to
who
who
who
decides
what
this
workflow
is?
Where
does
this
live?
How
do
you
make
it
consistent?
How
do
you
make
sure
that
there
are
no
errors?
A
You
know
this
like.
I
said
this
can
be
pretty
tedious
and
error
prone
and
and
lead
to
lead
to
issues
what
we've
worked
on
at
solar,
and
I
think,
you've
seen
some
of
the
learnings
that
we've
we've
shared
earlier
today
is
that
we've
sort
of
combined
git
ops
with
the
smart
controller
that
understands
higher
level
policies
and
it
can
build
a
tendency
model.
A
A
little
bit
more
made
a
little
bit
higher
level
and
then
built
a
tenancy
model
around
that.
So
we
call
it
workspaces,
it's
basically
a
bucket
or
a
grouping
of
service
mesh
policies
that
then
get
translated.
Come
back
here,
get
translated
to
the
istio
resources
that
correctly
handle.
You
know
those
lower
level
tenancy
constructs
and
all
of
this
can
be
driven
by
git
ops.