►
From YouTube: How to Improve Your Kubernetes Experience with Service Mesh... Maksim Chudnovskii & Igor Gustomyasov
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
How to Improve Your Kubernetes Experience with Service Mesh and MLOps - Maksim Chudnovskii & Igor Gustomyasov, Sber
In this session, speakers will talk about using a machine learning approach to optimize application performance in Kubernetes clusters in a large private cloud (50+ On-Premise Kubernetes Clusters in a Private Cloud, 500+ Compute Nodes, 10+ Istio Meshes. The speech from Sberbank will cover concrete practical cases and tell in detail about the experience of using machine learning models, consider in detail the architecture of the models, as well as the process of preparing training data, which is based on service mesh telemetry.
A
A
A
A
A
A
A
B
B
We
have
our
own
private
cloud,
that's
where
in
which
we
host
many
kubernetes
clusters
and
in
general,
we
can
say
that
each
of
these
clusters
is
something
like
an
on-premise
hosted
cluster,
and
each
of
these
clusters
can
have
several
hundreds
of
nodes
for
launching
ports.
We
use
a
standard.
Scheduler
workloads
are
completely
independent
in
terms
of
deployment
and
can
belong
to
different
teams.
So
here
is
an
assumption.
We
can
reduce
network
resource
consumption
as
well
as
optimize
overall
latency
by
combining
workloads
in
the
thing
which
is
known
as
schedule
group.
B
B
B
The
edges
of
the
graph
are
are
the
effects
of
network
interactions,
and
this
is
also
quite
simple.
You
can
notice
that
the
graph
is
weighted,
and,
yes,
you
are
right.
As
an
example,
I
added
the
average
latency
of
request
processing
as
a
weight.
The
values
are
taken
from
ps3
request,
duration
metric
respectively.
B
B
A
B
The
second
pair
of
services,
since
s5
is
already
in
group
1,
then
s3
is
added
to
it.
Next
s1
and
s6
create
a
new
group,
since
no
related
services
have
been
found
yet,
and
here
there
is
an
important
point.
The
size
of
the
groups
acts
as
a
parameter
of
this
method,
and
this
allows
us
to
split
the
connectivity
of
services
and
not
combine
the
entire
chain
into
one
group.
B
In
this
case
the
size
is
three
and
when
we
process
in
pages
of
four
and
five
nothing
happens,
the
services
are
already
allocated
and
since
the
limit
for
group
1
has
been
reached,
the
second
and
the
first
groups
will
not
merge
at
the
end
of
the
method.
Service
has
two
assigns
to
the
second
group,
and
here
the
work
is
done
and
let's
see
what
we
got
here.
B
We
have
two
group
of
booking
for
services
in
our
cluster:
yellow
and
green
teams,
services,
product,
page
revision
version,
1
and
details
were
included
in
group,
1
and
services,
reviews
version,
2
and
3
and
ratings
fell
into
the
second
group.
Great
the
approach
works
and,
let's
see
what
we
can
do
about
it.
B
In
the
simplest
case,
we
can
use
node
selectors
to
place
a
group
of
applications
in
a
specific
zone
of
the
cluster,
which
provides
the
most
comfortable
conditions
for
the
grouping
criteria
and
we
have
chosen,
for
example,
the
network
between
hosts
is
very
fast
in
this
zone.
We
can
also
use
service
topology
to
localize
traffic
within
the
selected
zones,
but
I
have
to
say
that
now
this
mechanism
is
duplicated
and
the
more
correctly
to
use
the
topology
versions.
B
However,
we
have
to
be
careful
here,
because
this
affinity
can
lead
to
performance,
can
lead
to
performance
issues
in
large
clusters
and
we
should
not
forget
about
fault,
tolerance
and
other
stuff
for
he
and
zr
processes.
Finally,
you
can
use
this
grouping
mechanism
for
the,
I
would
say:
smart
port
topology
spread
constraints.
B
As
for
us,
it's
bare
in
our
clusters,
there
are
a
large
number
of
independent
applications
that
can
interact
between
this
between
each
other.
We
use
collocation
in
a
recommendation
mode.
It
automatically
investigate
the
relationships
between
workloads
and
clusters
and
then
manually,
and
then
we
use
manually
this
information
to
mark
up
the
cluster
infra
and
configure
the
cluster
scheduler.
B
Okay,
that's
all
about
workloads,
collocation!
Let's
talk
about
normally
detection.
We
have
a
large
set
of
metrics
here,
and
this
metrics
can
be
analyzed
to
find
the
normal
assigned
them,
and
here
is
an
assumption.
We
can
recognize
abnormal
values
of
a
system
and
determine
the
root
cause
of
these
anomalies
through
the
correlation
of
network
and
system
metrics.
Let's
see
how
it
can
be
done
here.
B
B
B
Okay,
now
we
know
how
to
automatically
find
a
normal
listing
metrics,
let's
see
what
we
can
do
with
this
information
for
coupon.
For
for
the
we
checked
the
micro
rc
algorithm
on
the
example
of
our
favorite
booking
fabrication,
and
let's
see
what
came
of
it
anomaly,
detector,
notices,
time
the
deviation,
deviations
in
requests
and
a
root
cause
detection
method
can
start
with
this
data.
B
The
first
thing
we
need
to
do
in
this
approach
is
to
build
a
directed
application
graph,
where
the
nodes,
services
and
arrows
between
them
are
the
corresponding
requests
from
one
service
to
another.
We
already
seen
it
in
the
previous
task
from
the
complete
graph
of
the
application,
as
a
graph
is
allocated
which
contains
the
abnormal
nodes
plus
adjustment,
normal
ones,.
A
B
This
method
produce
produces
a
ring,
at
least
very
where
in
the
first
place
in
the
root
cause
of
the
anomalies,
we
can
find
the
reviews
version
two
service,
the
most
related
anomalous
response.
Time
metric
is
a
rate
container
memory
working
sub
bytes,
which
reflects
to
the
essence
of
this
anomaly,
hip
overflow,.
B
So
we
looked
at
anomaly,
detection
methods
and
an
algorithm
for
finding
through
to
finding
out
causes.
How
can
this
information
be
useful
at
first,
I
want
to
speak
about
alerting
in
modern
enterprise
systems.
There
are
thousands
of
alerting
rules
and
a
single
problem
can
cause
thousands
of
notifications,
while
detecting
anomalies
will
automatically
automatically
respond
to
suspicious
events
in
the
infrastructure
method.
Methods
of
finding
root
causes
can
significantly
reduce
the
flow
of
notifications,
which
is
a
perfect
solution
for
the
system
administration.
B
It
is
important
to
note
that
the
anomaly
detector
can
be
implied
to
establish
monitoring
thresholds.
I
would
say,
since
the
threshold
values
are
most
often
selected
individually,
based
on
experimental
data,
anomaly
detections
allows
you
to
automate
this
process
here.
We
got
the
third
important
aspect
based
on
the
automatic
detection
of
anomalies,
so
we
can
form
flexible
rules
for
automatic
traffic
management.
B
We
achieve
deep
understanding,
authorized
aspects
of
the
systems
and
then
use
this
information
to
manually,
configure
our
monitoring
solutions
and
what,
unfortunately
concrete
models
of
root
cause
analysis
in
our
infrastructure
is
not
open
sources
yet,
but
so
I
will
not
cover
them
in
this
talk,
but
maybe
in
future
we
will
talk
about
it
and
the
last
but
not
least,
predictive
auto
scaling
when
migrating
to
kubernetes.
Many
of
our
colleagues,
especially.
B
Those
who
write
code
in
java
faced
the
problem
of
slow
start
of
applications,
given
the
fact
that
they
put
some
water
and
the
elasticity
of
applications
in
our
environment
is
achieved
through
automatic
scaling.
A
slow,
slow
start
of
an
application
became
an
unpleasant
problem
that
we
really
had
to
find.
B
The
method
is
designed
to
predict
the
required
number
of
service
ports.
Two
key
methods
are
used
to
predict
the
number
of
ports
in
service
standard
hpa
and
predictive
autoscaler
as
an
input.
The
method
uses
the
time
series
of
a
specific
metric
called
value,
for
example
cpu,
as
well
as
data
on
board
resource
limits
for
limit
and
it
its
desired
ratio
on
the
resources.
B
B
A
limitation
is
also
imposed
in
terms
of
the
minimum
and
maximum
number
of
pots.
Predictive,
auto
scalia
calculates
the
number
of
poles
in
the
same
way,
but
instead
of
the
last
actual
value
of
the
our
metric,
the
predicted
value
is
used
for
training
and
prediction.
The
time
series
data
is
supplemented
with
some
features
at
the
moment
legs
for
the
last
20
minutes
and
an
hour
in
a
day
I
used
as
a
science
as
features,
but
this
list
is
no
is
not
exhaustive
and
can
be
supplemented
on.
B
The
slide
is
an
example
of
data
transformation
and
feature
generation.
Predictive,
auto
scaler
uses
two
models
to
predict,
values,
cut
boost
model
cpm
and
the
moving
average
with
a
window
of
20
values.
Cut
boost
model
is
gradient,
boosting
model
or
decision
trees.
The
model
allows
you
to
track
the
growth
of
the
load
with
a
small
time
delay.
B
B
A
Now,
thank
you
maxine.
Thank
you
for
your
brilliant
speech
and,
as
maxim
said,
we
use
our
machine
learning
cluster
now
as
a
in
recommendation
mode
and,
of
course,
after
collecting
enough
amount
of
statistics,
we
are
going
to
switch
in
operation
mode
and
implement
all
our
features
in
real-time
manner
and
the
sec
in
the
second.