►
Description
SoloCon 2022:
Scaling Gloo Mesh in Production for Performance and Reliability
Speakers:
Neeraj Poddar
Director of Engineering, Solo.io
Session Abstract:
Scaling a single service mesh is hard! Now when you add to it the complexity of a unified mesh deployed across clusters running on different clouds/on-premises all driven by a single management plane that provides a single pane of glass for visibility and configuration management, the challenge can be daunting. In this session, we will explore how Gloo Mesh was architected to handle scaling of this magnitude while meeting the performance and high availability expectations of your complex environment. We will share improvements made in API v2 and primitives available to you to scale Gloo Mesh securely in production.
Track:
Service Mesh and Application Networking
A
Welcome
everyone,
thanks
for
joining
in
on
this
session,
on
scaling
blue
mesh
to
achieve
performance
and
reliability
at
scale
and
production.
So
today
I'm
going
to
talk
about
how
can
you
scale
your
risk
to
your
service
mesh
in
a
single
cluster
and
multi-cluster
environment?
Some
of
the
challenges
you
might
face
and
how
glue
mesh
is
fundamentally
architected
and
designed
to
solve
those
challenges
and
meet
the
requirements
for
production
at
scale.
A
I
am
niraj
podar.
I
am
head
of
engineering
here
at
solo.
I
have
been
a
long
time.
Istio
maintainer
have
been
working.
Group
leads
for
several
working
groups
like
networking
and
park
security
working
group,
I'm
a
former
steering
committee
member
and
currently
honest
your
toc.
So
let's
go
over
today's
agenda.
A
First,
I'm
going
to
talk
about
what
scaling
means
for
stu
service
mesh
in
a
single
cluster
environment.
Then
talk
about
multi-cluster
scale,
challenges,
but
focus
on
the
requirements
that
most
of
our
customers
have
for
multi-cluster,
multi-cluster
mesh,
what
architecture
and
what
solutions
steve
provides
and
then
some
of
the
challenges
you
might
face
with
it.
A
So
let's
get
started
so
first
off,
let's
look
at
the
istio
architecture
and
so
that
we
can
understand
what
scaling
means
here.
So
istio
primarily
is
composed
of
two
components:
the
data
plane
and
the
control
plane.
A
data
plane
is
comprised
of
sidecar
proxies.
These
proxies
are
onward
proxies,
which
are
deployed
in
a
site
proxy
architecture.
A
By
cycle
proxy
architecture.
I
mean
these
are
injected
in
the
same
pod
as
the
application
container
and
we
add
some
iptable
magic
to
make
sure
all
the
traffic
from
the
application
container
is
injected
and
sent
to
the
on
web
proxy.
Similarly,
all
the
traffic
coming
into
the
cont
into
the
application
part
is
sent
is
intercepted
and
routed
through
the
proxy,
because
these
are
in
the
same
pod.
From
an
outside
point
of
view.
They
look
as
one
logical
entity
because
they
share
the
same
networking
name
space.
A
So
basically,
after
all,
the
data
plot
injection
has
been
done.
What
it
means
is
when
app
a
talks
to
app
b.
The
request
will
go
through
the
envoy
on
the
app
is
talked
about,
and
then
it
will
be
intercepted
by
the
envoy
on
app
b
and
then.
Similarly,
the
response
will
go
through
the
on
voice
on
the
reverse
direction.
So
that's
the
data
plane
component.
The
second
component
is
the
control
plane,
which
is
called
istio
discovery
or
short.
A
In
short,
it's
called
stod,
it's
responsible
for
looking
at
your
platform
configuration
so,
for
example,
if
you're
running
in
kubernetes
to
look
at
all
the
services
you
have
and
the
user
provided
mess
configuration
and
translate
that
to
all
the
data
plane
proxies.
So
basically,
study
is
an
abstraction
layer
that
lets
the
operators
configure
sdo
in
the
language
of
the
platform
and
basically
in
kubernetes,
and
then
hides
away
all
the
complexity
of
the
onward
configuration.
A
So
next,
let's
look
at
what
scaling
means
right,
we'll
talk
in
terms
of
each
component
so
for
data
plane,
since
the
onward
proxies
are
injected
in
the
pod
as
side
cars,
when
you
scale
out
your
port
using
kubernetes
primitives
the
data
plane
scales
with
it,
so
that's
very
beneficial.
So,
for
example,
if
you
have
two
replicas
of
app
a
and
now
three
replicas
of
app
b,
you
have,
in
total
five
onward
proxies
running
matching
the
five
instances
of
your
application.
So
this
scaling
is
very
beneficial.
You
don't
have
to
do
extra
work.
A
You
automatically
get
it
by
just
using
the
kubernetes
primitives
of
deployments
or
demons
whatever
you
have.
So
that's
great.
The
second
scaling
aspect
is
how
do
you
scale
the
control
plane,
so
istio
allows
you
to
run
multiple
instances
of
sdod,
so
the
primary
benefit
is
when
you
have
multiple
instances,
the
proxies
that
connect
to
the
steel
control
plane
for
getting
there
are
getting
the
configuration.
It
gets
load
balance
to
those
instances.
A
So,
for
example,
here
the
app
a
sidecars
proxy
can
go
to
the
studio
on
the
left
and
then
the
app
b
might
go
to
the
s2d
on
the
right.
This
is
great
in
terms
of
dividing
the
load
for
the
connections
that
are
being
handled
by
one
instance.
At
the
same
time,
there
is
a
limitation
here
that
all
the
student
instances
end
up
looking
at
the
entire
state
of
your
kubernetes
cluster
of
your
platform
and
the
configuration
and
compute
the
output
results
for
all
the
envoys.
A
So
this
leads
to
limitations
around
both
the
sqd
instances
will
more
or
less
consume
the
same
amount
of
cpu
and
ram
when
they
are
working
at
scale
and
looking
at
large
number
of
services.
Large
number
of
configurations
right,
so
you
get
the
benefit
of
the
onward
proxies
getting
load
balance
for
trying
to
get
the
connection,
but
the
limitation
is
you?
Don't
really
have
some
efficient
sharding
here
right?
So
that's
what
is
summarized
in
this
slide.
A
Another
aspect
of
this
scaling
is
in
sto.
By
default,
all
the
proxies
are
configured
with
the
entire
state
of
the
system.
That
means,
if
you
have
large
number
of
services,
each
of
the
proxies
will
have
connectivity
information
and
which
means
clusters
and
end
points
associated
with
all
your
services
that
can
be
reduced.
If
you
are
configuring
side,
car
resources,
but
istio
cannot
automatically
do
that
and
it's
very
tedious
and
very
onerous
to
configure
the
site
query.
A
You
need
to
isolate
them
and
you
have
multiple
clusters
for
that
requirement
or
or
you
want
some
reliability,
so
you
want
multiple
clusters
deployed
so
that,
if
one
is,
you
know
having
issues
you
can
fail
over
to
the
other,
irrespective
for
most
of
our
large
customers.
Multiple
clusters
is
basically
a
requirement
right,
so
you
need
to
have
applications
deployed
in
one
cluster,
which
is
which
are
able
to
communicate
effectively
to
applications
deployed
in
another
cluster,
all
while
preserving
end-to-end
identity
and
that's
what
the
first
requirement
for
a
multi-cluster
mesh
at
scale
is.
A
You
need
to
have
inter-cluster
traffic.
You
need
to
have
seamless
traffic
that
communicates
that
allows
communication
between
applications
and
they
can
be
deployed
across
clusters.
You
preserve
the
identity
for
end
to
end
and
you
can
have
failover
support.
So,
if
you're
going
across
regions,
you
want
to
only
fail
over
when
it's
when
the
when
the
instances
of
services
in
your
own
region
are
failing,
so
you
have
some
cost
optimization
done
there
right.
You
want
fault
tolerance
between
these
clusters.
A
What
I
mean
by
that
is
each
cluster
and
mesh
should
be
functioning
independently.
You
don't
want
to
have
the
mesh
or
the
kubernetes
server
on
one
cluster
going
down
affecting
the
functionality
and
the
traffic
behavior
for
the
clusters.
Essentially
many
of
these
many
of
the
times
the
customers
are
moving
to
different
clusters.
Like
I
said
for
reliability,
and
if
you
have
these
dependencies,
you
lose
that
aspect
of
you
do
lose
that
benefit
from
it.
A
And,
lastly,
we
want
services
that
are
deployed
in
each
cluster
to
have
overlapping
ip
addresses.
Basically,
we
don't
want
to
have
requirements
around
networking
being
flat.
You
want
to
make
sure
your
clusters
can
be
deployed
across
different
clouds.
You
don't
want
to
make
any
assumptions
around
how
the
topology
of
the
clusters
is.
It
might
be
simpler
for
to
come
up
with
solutions
when
you
have
flat
networking
or
all
the
parts
are
reachable
from
different
clusters
directly.
A
A
So
stu
has
a
lot
of
different
solutions
when
it
comes
to
multi-cluster,
but
on
these
requirements,
the
architecture
that
works
is
you
need
to
give
stods
running
in
each
of
these
clusters,
access
to
the
kubernetes
server
and
that's
needed,
because
students
will
automatically
discover
endpoints
so
that
services
from
one
cluster
can
transparently
talk
to
another
cluster.
While
this
looks
very
attractive
in
terms
of
simplicity,
there
are
some
severe
limitations
of
this
architecture
right,
so
the
main
restrict
the
main
limitation
from
a
security
point
of
view
that
I've
often
seen
customers
mention.
A
A
You
want
to
lock
down
access
for
kubernetes
api
servers
to
a
small
group
of
services,
probably
which
are
living
inside
your
own
cluster.
You
don't
want
an
actively
running
service
outside
the
cluster
which
can
reach
into
the
kubernetes
api
server.
Secondly,
since
all
the
sqd
instances
are
talking
to
each
other's
kubernetes,
api
servers
you're,
basically
limited
by
the
scalability
of
single
apis.
A
Additionally,
all
the
configuration
overhead
is
basically
on
the
users
who
are
running
this
architecture,
so
you
need
to
replicate
the
configuration
across
different
clusters
to
make
sure
you
have
the
same
policy
and
the
resources
running
across.
So
you
get
consistency
only
if
you
do
this
manual
replication.
A
Are
you
having
failures
when
that
is
happening
right,
so
you
can
have
prometheus
instances
deployed
in
each
cluster,
but
then,
once
the
traffic
goes
from
one
cluster
to
the
next,
how
do
you
actually
get
the
metrics
reported
in
a
way,
so
you
can
have
a
good
connectivity
graph.
So
these
are
the
problems
that
the
glue
mesh
solution
solves
so
in
glue
mesh.
A
A
What
we
do
is
we
are
adding
an
agent
that
runs
in
each
cluster.
This
agent
is
responsible
for
looking
at
the
kubernetes
services.
Looking
at
what
history
resources
are
running
on
this
cluster
looking
and
discovering
the
various
input
objects
and
providing
them
to
the
management
plane,
which
runs
in
one
location.
So,
basically
all
the
agents
are
going
to
discover
these
objects
from
each
of
these
clusters,
send
them
to
the
management
plane
and
then
the
management
plane
is
responsible
for
taking
all
of
the
runtime
information
for
various
clusters.
A
Having
this
architecture
fundamentally
gives
you
better
flexibility
and
scalability
compared
to
the
architecture
where
you
have
to
have
connectivity
across
each
of
the
api
servers
right.
So
if
you
look
at
this
architecture,
the
main
benefit
that
you're
providing
is
we
don't
have
the
security
issue
of
cross
cluster
kubernetes
api
server
access
right?
We
have
agent
which
is
going
to
do
the
discovery
and
send
the
information
out
and
the
same
agent
will
do
the
application
of
the
kubernetes
istio
resources
on
each
cluster.
So
there's
no
there's
a
nice
security,
isolation
foundry.
A
We
also
give
our
users
a
higher
level
apis,
which
are
persona
mapped
instead
of
lower
level
constructs,
which
you
have
to
deal
with
when
you're
configuring
sql
directly,
it's
more
intuitive,
it's
more
applicable
to
the
persona.
That
is
configuring.
The
system,
rather
than
all
lower
level
networking
apis
and
the
interesting
thing
here
is
istio-
is
powerful
enough
that
it
gives
us
the
right
primitives
to
build
on
and
create
high
level
use
cases
and
high
level
high
level
management,
plane
and
products
like
blue
mesh.
A
Right
and
recently
we
are
automating
the
hto
con
istio
configuration
creation
in
each
cluster
and
management.
So
you
don't
have
to
do
that
manually.
You
don't
have
to
do
that
application
or
whatever
sto
config
is
needed.
You
can
all
drive
it
just
by
creating
blue
resources.
We
also
provide
automated
press
bootstrapping
and
a
single
pane
of
glass
for
visibility.
A
So
this
is
what
we
call
the
glue
mesh
relay
architecture
right
so
in
glue
mesh
like
I
was
saying
we
have
the
agent
which
is
going
to
look
at
the
kubernetes
api
server,
get
all
the
information
about
the
services
and
end
points,
send
it
securely
over
the
management
plane,
which
we
call
the
enterprise,
networking
port
and
the
enterprise.
Networking
part
looks
at
the
glue
resources
combines
it
with
the
stuff
agent
has
given
converts
to
the
history
resources
sends
it
back
down
to
the
agent
which
applies
it.
A
So
this
fundamentally
is
very
scalable,
because
each
of
those
clusters
you
can
see
will
have
the
agent
running
the
as
the
workload
cluster
scale
you
can.
You
will
have
each
independent
agents
doing
this
work
without
any
coordination
between
them.
So
whenever
you
see
terminologies
like
relay
architecture
and
relay
in
our
documentation,
this
is
the
this
is
basically
the
communication
part
we
are
referring
to
so
building
on
this.
Let's
look
at
how
gloomesh
handles
trust.
A
So
in
gloomesh
there
are
multiple
kinds
of
certificates
and,
depending
on
how
you
have
configured
it,
we
can
automate
some
of
it
or
all
of
it
right.
So
two
primary
cert
certification-
I
should
just
say
two
primary
certificate
systems
and
blue
mesh-
are
one
that
is
being
used
for
relay,
that
is,
for
communication
between
the
agent
and
the
management
plan
for
for
getting
all
the
config
back
and
forth,
and
then
another
certificate
management
system
is
the
data
path.
A
A
In
this
case,
the
intermediate
ca
will
be
used
by
the
management
cluster
to
create
this,
create
the
relay
server
certificate
and
then
also
to
create
the
agent
certificate,
and
then
the
intermediate
htoc
will
be
used
for
all
the
stod
signing
search
and
from
there
creating
the
workload
search.
So
this
in
this
mode,
you
are
going
to
have
one
root
ca,
maybe
for
your
organization,
but
then
the
intermediate
cas
have
glute
delineation,
where
they
can
be
used
for
kind
of
control,
plane,
traffic
between
the
agent
and
the
management
plan
and
all
the
data
plane.
A
Where
you
have
another
ca
system
going
on
in
terms
of
management
of
these
certificates,
you
have
again
a
bunch
of
different
options.
This
is
one
of
the
options
that
I
like
to
show
because
this
is
most
commonly
what's
used
by
our
customers
right,
so
you
can
have
the
root
ca
configured
so
that
it
is
provided
externally.
A
It's
handled
externally,
like
a
vault
or
aws
certified
management
system.
So
in
that
case
the
root
ca
will
be
configured
manually
right.
You
have
to
provide
the
root
c
and
the
intermediate
relay
cas
and
the
istio
cs
to
us
in
the
management
plane
and
the
release
server
certificate
and
but
from
there
on
all,
the
certificates
will
be
automatically
generated
and
rotated
and
managed
by
the
glue
management
plan.
A
So
from
there
on
the
agent
certificate
and
the
sdo
certificate
will
be
created
by
us,
the
sdod
will
look
at
those
certificates
that
we
have
manually
provisioned
from
there.
They
will
create
the
workload
certificates,
so
this
is
like
a
hybrid
mode
in
which
some
of
the
work
for
management
is
on
the
user,
whereas
the
rest
of
the
trust
bootstrapping
is
done
by
written
by
the
glue
mesh
management
plane
moving
forward.
A
At
the
same
time,
it
also
acts
as
a
matrix
and
a
log
server.
So
what
I
mean
by
that,
when
you're
running
glue
mesh,
we
configure
the
sidecar
proxies
running
in
each
part
in
each
pod
to
send
the
metrics
and
logs
to
to
the
enterprise
agent
at
a
specific
port,
where
it
is
when
I
receive
those
metrics
and
logs
and
send
them
via
mutual
tls
again
to
the
enterprise,
networking
port
or
the
management
plan,
where
the
management
plane
acts
accessing,
because
these
are
running
in
each
clusters.
A
What
that
and
we
add
specific
labels
or
tags
for
traffic
coming
for
each
cluster.
So
basically,
what
you
end
up,
having
is
all
the
metrics
and
logs
for
all.
Your
clusters
are
sent
by
agent,
the
networking
part
running
in
the
management
plane
and
then
in
the
management
plane.
You
will
have
prometheus
running,
which
can
scrape
this
metric,
and
we
can
help
you
that
can
help
us
create
visualization
for
traffic
going
from
one
cluster
to
another,
because
we
are
not
losing
any
of
the
context.
A
You
can
have
prometheus
running
securely
in
one
place
where
you
can
scale
it
up,
and
then
you
can
have
all
the
telemetry
for
all
the
clusters
in
one
spot.
Additionally,
our
dashboard
has
the
visualizations
to
make
sure
you
can
actually
look
at
the
graph
from
one
going
from
one
cluster
to
the
other.
The
dashboard
itself
coordinates
with
bunch
of
different
things,
so
the
dashboard
in
the
version
1.2
of
the
product
is
going
to
get
the
metrics
from
prometheus.
A
It
also
communicates
the
enterprise
networking
part
to
get
some
intermediate
objects,
and
then
it
communicates
to
the
kubernetes
server,
where
the
enterprise
networking
part
has
persisted
some
intermediate
objects.
It
combines
all
of
them
together.
Basically,
and
the
ui
gives
you
a
view
of
holistic
view
of
your
configuration
and
the
runtime
state
with
all
the
graph
and
showing
you,
the
connectivity
between
services
differed
across
clusters.
A
So
now,
let's
talk
about
what
scaling
means
for
blue
mesh
even
after
this
right.
So
we
have
a
fundamentally
secure
architecture
where
the
agents
are
doing
the
work
securely
in
one
in
each
of
the
clusters.
And
then
the
information
comes
up
to
the
management
plane,
but
you
can
see
we
want
to
make
sure
we
can
scale
out
each
of
these
components
so
that,
depending
on
the
size
of
your
cluster,
none
of
them
are
having
issues
transporting
data.
None
of
them
are
actually
acting
as
a
choke
point
right.
A
That
means,
when
you
have
multiple
instances
of
when
you
have
a
lot
of
different
workloads
running,
they
can
connect
to
different
agent
to
send
to
make
sure
you
can
get
all
the
metrics
and
log
exported
to
your
networking
to
the
management
plan
right.
So
this
gives
you
the
first
level
of
scaling,
so
your
relay
is
handled
by
the
lead
leader
and
all
the
other
instances
are
able
to
send
the
metrics
and
logs
up.
So,
if
you're,
if
you
are
growing
your
applications
in
one
cluster,
this
can
still
handle
the
load.
A
But,
as
you
can
imagine,
just
scaling
the
glue
mesh
agent
is
not
going
to
be
enough
right.
You
can
still
be
bottlenecked
at
the
glue
management
plane.
What
we
found
in
the
version
1.2
was
there
a
lot
of
efficient
inefficiencies
in
translation
that
were
actually
linked
to
propagation
delay
at
scale
and
and
and
that
propagation
delay
can
sometimes
you
know,
can
get
worse
and
worse.
As
you
add
more
configuration
in
your
in
your
cluster
or
you
are
adding
more
as
you're
adding
more
clusters
and
scaling
out
right.
A
So
you
wanted
to
address
that
in
a
new
release.
Additionally,
an
interesting
aspect
of
the
design
we
had
in
version
1.2
was
the
blue
mesh
management
plane
or
the
enterprise.
Networking
part
was
doing
a
lot
of
rides
to
the
kubernetes
api
server
to
persist.
Intermediate
state
now
that's
beneficial
in
general,
because
you
can
query
the
kubernetes
api
server
and
get
a
lot
of
debug
information.
A
At
the
same
time,
it
was
causing
a
lot
of
issues
and
actually
trying
to
scale
that
out
when
you
are
having
a
lot
of
clusters
and
lot
of
data
flowing
through
your
system
configuration
data
flowing
through
a
system,
another
import.
Another
interesting
thing
was:
we
only
allowed
the
glue,
custom
resources
or
api
objects
to
be
written
into
management
cluster.
You
want
to
make
sure
your
redundancy,
and
you
want
to.
A
You-
want
to
be
able
to
write
those
objects
in
all
the
workload
clusters
not
be
restricted
to
just
the
management
cluster
and
then
the
key
thing
was
that
was
missing
in
our
earlier
architecture
was
there
was
no
easy
way
to
organize
data
in
the
dashboard
or
in
our
translation
at
scale.
We
didn't
have
a
good
unit
to
say
this
is
how
we
should
be
processing
data.
This
is
how
we
should
be.
A
A
In
regards
of
isolating
your
configuration
and
also
isolating
traffic
between
between
services
that
are
part
of
workspaces,
or
that
you
have
imported
right
in
histo,
multi-tenancy
in
general,
is
very
difficult,
especially
in
terms
of
configuration
in
history.
It's
very
easy
when
you
can
have
one
one
stereo
resource
created
in
one
name,
space
that
can
hijack
traffic
in
another
name,
space
right.
A
This
is
not
saying
that
the
istio
apis
are
not
capable
of
doing
a
lot
of
other
things
it's
just
by
design
because
they
are
so
powerful.
They
also
have
this
limitation,
so
we
wanted
to
create
a
logical
boundary
so
that
whatever
is
to
your
resources,
whatever
whatever
glue
resources
you
create
in
that
boundary,
will
never
affect
outside
that
boundary.
Unless
you
explicitly
tell
it
to
and
that's
what
a
workspace
is.
A
So
you
can
see
in
this
diagram,
we
are
showing
how
workspaces
are
drawn
right,
so
the
dotted
blue
lines
are
showing
workspaces
that
are
sometimes
go
just
spanning
one
cluster.
The
other
workspaces
are
spanning
two
namespaces
across
different
clusters
and
the
third
one
is
actually
composed
of
four
clusters
right
and
you
can
compose
them.
However,
you
want
and
we
have
primitives
in
workspaces
that
allows
you
to
export
at
import
at
various
resolutions.
A
So
basically,
you
can
say
I'm
importing
a
particular
workspace
or
a
resource
or
service
within
that
workspace.
That
allows
you
to
basically
start
consuming
it,
but
without
doing
those
imports
and
exports.
You
are
basically
in
an
isolated
environment
and
you
will
never
affect
anything
from
a
security
point
of
view,
something
that
is
in
a
different
workspace.
A
A
Additionally,
we
can
automatically
create
the
sidecar
resources,
because
we
know
your
service
dependencies,
as
you
have
described
in
the
workspace.
Now
we
can
reduce
your
memory
and
cpu
consumption
in
on
y
for
processing
a
lot
of
configuration.
This
is
a
big
benefit
at
scale
now,
as
part
of
this
rework
and
creating
this
2.0
product.
A
A
So
this
is
very
interesting
right,
so
the
multiple
benefits
you
get
out
of
it.
So
so,
as
the
different
enterprise
agents
across
different
clusters
connect
to
the
networking
pods
on
the
management
plane,
you
get
an
automa.
You
get
an
you,
get
a
reliable
way
of
sharding
your
data,
so
as
the
leader
in
each
cluster,
as
the
leader
agent
in
each
cluster,
connects
to
one
of
the
instances
of
the
met.
The
management
plane
part
that
management
plane
plot
only
needs
to
worry
about
the
configuration
for
that
particular
cluster.
A
It
doesn't
have
to
worry
about
cluster
2,
for
example,
if
the
the
leader
for
cluster
one
connected
to
it.
So
this
sharding
is
very
useful
and
it's
automatic
because,
as
the
leader
agent
comes
and
picks
up,
one
of
the
it
connects
to
the
networking
port,
it
is
going
to
connect
to
one
of
them.
It
will
automatically
get
load
balanced,
so
that
gives
us
a
unit
of
division
of
work
that
helps
us
scale
better.
A
A
To
summarize,
you
can
scale
out
the
management
server
pod
we
get
efficient
sharding,
we
have
reduced
the
number
of
intermediate
objects
that
we
are
creating.
More
importantly,
that
we
are
persisting
that
helps
us
make
sure
we
can
do
translation
quickly
at
scale.
We
are
reducing
the
relay
of
objects
to
the
point.
Whatever
is
needed.
Those
are
the
objects
that
are
transported
from
each
cluster.
That
saves
some
bandwidth
and
also
churn
time
in
terms
of
processing.
A
So
with
these
enhancements,
I'm
very
excited
and
looking
forward
to
what
our
customers
can
do
in
production
to
get
better
scale.
I'll
highly
recommend
that
you
scale
out
the
agent
part
and
the
management
server
part
on
the
management
cluster
make
sure
you
have
anti-affinity
configured
for
control,
plane
parts.
So,
basically,
all
the
different
instances
of
the
agent
and
the
management
server
don't
end
up
on
the
same
node.
In
case
a
node
goes
down.