►
From YouTube: Practice of Fine-grained Cgroups Resources Scheduling in Kubernetes - Xianlu Chen & Qingcan Wang
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2021 Virtual from May 4–7, 2021. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Practice of Fine-grained Cgroups Resources Scheduling in Kubernetes - Xianlu Chen & Qingcan Wang, Alibaba
Alibaba supports resource scheduling for hundreds of thousands of nodes, millions of containers, and tens of thousands of applications. Many online services need to dynamically increase the resource limit during operation, and cannot accept the impact of restart.Other applications may require NUMA awareness, CPU Core binding, reduce data copy between CPU caches, and speed up data processing tasks. We have developed a combined scheduling system based on Kubernetes Scheduler framework and Cgroups controller. The scheduler perceives cgroups level resources, such as numa, cpu core, memory limit, etc., and applies scheduler dynamic scheduling to specified nodes, while allowing certain Pod is bound to the specified cpu core. The cgroups controller can also dynamically adjust the pod resources limit without causing the Pod to restart.
https://sched.co/f3cp
A
B
Hello:
everyone,
my
name
is
kanibal
from
alibaba
cloud
today,
my
colleague
ching
chang,
and
I
will
talk
about
the
practice
of
seagroup
resource
scheduling
in
kubernetes
environment.
In
our
business
scenario,
such
as
database
service,
many
airlines
tends
need
to
dynamic,
increase
the
resource
limitation
during
operations
and
can't
accept
the
impact
of
polystar.
B
Then
we
need
to
develop
a
service
to
configure
c
group
for
the
payload,
also
multiplication
stress
on
the
parameter
distance
in
a
bar
cloud
and
limit
the
multi-numeral
performance
problem.
The
applications
may
require
new
awareness,
cpu
compounding
reduce
the
discovery
between
cpu
cache
and
speed
up
the
data
processing
tasks.
B
We
have
developed
a
combined
scheduling
system
based
on
kubernetes
scheduler
framework
and
c
group
controllers.
The
scheduler
perceives
the
group's
neighbor
resource
such
as
pneuma
cpu
cores
memory
limit
and
applied
scheduler
dynamic
scheduling
to
specific
news,
while
allowing
certain
ports
is
bound
to
the
specified
cpu
core.
The
sequel
controller
can
also
dynamically
adjust
the
power
source
limit
without
causing
the
power
to
start
okay.
This
is
the
agenda
of
today's
topic.
In
the
first
part,
I
will
introduce
our
practice
of
sea
group
resource
means
management.
B
Then
my
colleague,
ching
chang,
will
talk
about
the
cpu
scheduling
business
schedule
also
about
some
other
related
works,
as
we
know
that
sql
is
an
implementation
to
limited
resource
for
the
process
and
is
the
basic
ability
in
those
containers.
We
think
we
can
set
the
limitation
for
number
of
resources,
then
the
process
group
can
use
and
also
it
can
set
the
priority
control
of
the
process
group.
B
Also,
it
can
record
the
number
of
resources
used
by
the
process
group
in
our
production
coordinates.
Environment
database,
in
instance,
is
deployed
in
pulse,
so
we
use
the
source
limitation
configurations
in
container
level
for
the
resource
limitation,
because
it
is
not
enough.
First,
we
can
only
configure
the
cpu
or
memory
in
the
continuous
spec
limitation,
for
the
storage
is
not
supported.
B
Secondly,
the
limitation
is
set
with
the
container
starting
and
cannot
be
changed
when
the
client
is
in
one
state.
Also,
the
limitation
is
only
configured
for
the
containers
and
not
for
the
pulse.
So
when
you
develop
a
service
that
can
dynamic,
update
the
sql
implementation,
both
for
a
container
and
post
without
a
payload
restart
and
with
a
single
smart
controller,
resource
limitation
can
automatically
configure
by
some
sql
policy.
B
Sql
controller
is
developed
just
for
this
purpose.
From
the
perspective
of
functions
with
sql
controller,
we
can
dynamic,
configure
sql
limitation
with
creator
equivalent
resource
and
you
not
need
to
care
about
the
pro
location
and
running
environment.
From
the
perspective
of
realization.
The
sql
controller
is
a
realization
of
the
ideas,
foster
groups
and
sql
policy.
We
secure
the
ide,
the
controller,
dynamic
configuration
limitation
for
target
resource
and
payload,
and
the
sql
policy
makes
the
policy
to
automatically
adjust
the
gross
resource
limitations
with
the
group.
Smart
controller
communities
can
increase
resource
usage
with
smart
resource
allocations.
B
Sql
controller
also
supports
both
all
support
to
configure
different
payloads
include
development.
Server
set
limit
set
pause.
If
the
limitations
say
to
the
deployment,
all
posts
will
be
configured
with
single
controller.
We
can
configure
the
different
type
of
resource
for
the
limitation
in
closed
cpu
cpu
set
memory
and
build
io
communities
can
only
set
a
limitation
for
container
level.
Here's
your
controller
suppose
to
set
the
limitation
both
for
port
and
containers
with
positive
resource
limitation.
B
All
continuous
in
the
pod
will
share
the
resource
and
share
the
limitations
in
some
sharing
business.
The
salination
in
the
ports
level
can
improve
the
source
usage.
Vlkl
can
be
supported
in
stable,
sql
controller,
but
the
block
device
is
compressed
because
there
are
be
more
than
one
device
you
might
know.
So
how
defined
the
block
device
is
important?
Civil
sequel
controller
supposed
to
define
the
block
device
with
multiple,
such
as
interface
volumes
device
id
such
as
vdx
the
dv
path?
B
B
The
study
also
defines
the
target
payload,
including
deployment,
still
says,
give
me
set
on
post
and
also
need
to
point
out
the
target
poll
and
target
control
name
also,
assignee
can
record
the
source
limitation
status
with
it.
The
controller
can
check
the
limitations.
Setting
is
complete,
complete
completed
or
not.
Next,
let's
talk
about
the
main
process.
B
The
controller
will
check
the
payload,
for
example
the
development
of
post
and
get
the
targeted
node
id
and
contain
information,
and
the
third
step
is
the
controller
will
create
a
job
by
the
executor
which
will
run
in
the
target
mode
target
node.
The
created
job
is
configured
with
target
payload
and
limitation
calculations.
The
job
job
part
will
get
the
signal
path
and
the
target
continues
output
and
then
it
sets
the
limitation
value
to
the
target
pass
file.
B
B
We
should
configure
the
control,
name,
belongs
to
the
port
and
also
we
can
configure
the
cpu
and
memory
limitation
at
the
same
time.
The
second
way
and
third
one
is
setting
the
cpu
set
for
the
development
or
online
special
port.
We
should
configure
the
target
content
name
for
that.
The
first
yam
template
is
setting
the
I
blko
for
the
block
device.
B
B
B
The
sql
policy
should
record
the
power
limitation
on
the
targeted
source
in
the
status
field,
which
can
show
you
the
limitation
details
on
wine
resource.
Next.
Let's
talk
about
the
main
process.
The
sql
policy
in
place
implemented
the
cid
logic
for
the
sql
policy.
The
controller
was
the
post.
That
knows
the
serial
policy
object
from
the
pass
server.
B
In
the
controller
you
find
a
port
is
added
or
removed
from
the
targeted
sequel
policies.
It
will
create
us
secure
objects
there
to
adjust
the
source
limitation
for
the
different
payloads.
For
example,
if
a
satellite
releases
one's
group
policy
increased
from
the
number
of
tools
to
sleep,
the
new
resource
limitation
is
rebalanced
further
for
all
those
reports
and
the
single
sql
object
is
created
to
configure
the
new
limitation
for
the
three
parts
step.
B
Also
we
define
a
resource
type
resource
id.
Then
we
can
identify
the
unicode
resource
in
class
when
the
first
type
is
defined
as
blk
io.
The
source
rps
limit
can
be
defined
to
the
target
device,
total
rps
limitation
value,
and
if
the
resource
type
is
cpu
or
memory,
the
resource,
cpu
percentage
or
resource
memory
percentage
can
be
configured
and
the
policy
field
is
defined.
The
resource
limitation
policy
you
can
configure
as
the
speed,
which
means
the
same
limitation
will
be
configured
between
given
pillars
as
soon
as
possible.
B
B
The
payload
field
shows
the
source
limitation
settings
and
you
can
find
the
post
works
on
that
as
limitations.
Details
of
the,
if
you
want
to,
if
you
want
to
enable
the
resource
immigration
policy
for
the
special
special
payload,
the
sql
policy
defined
defined
definition,
should
be
added
to
the
payload
annotations.
Just
like
the
example,
pause,
dot,
sql
dot,
alpha
cloud.com
as
one
policy
object,
then
the
silver
controller
will
get
the
configurations
and
do
the
source
limit
balance.
B
B
A
A
At
present,
a
topology
is
divided
into
three
layers
circuit,
human
node
and
chrome.
Each
core
has
a
dedicated
level,
one
or
level
two
cache.
This
is
a
referral
through
a
private
cache.
No
other
call
can
overwrite
the
level
3
is
shared
between
cores
cpus
belonging
to
different
human
nodes,
to
access
different
paths
of
memory
at
different
speeds.
A
Based
on
these
characteristics,
we
think
it
is
necessary
to
find
cpu
scheduling.
Our
goal
is
one
to
reduce
the
program
loss
caused
by
frequent
switching
between
multiple
cores,
two
to
reduce
the
loss
caused
by
frequent
switching
between
different
human
nodes.
Third,
to
select
the
best
cpu
scheduling
result
in
cloud
dimension.
A
This
is
our
complete
architecture.
We
have
three
extension
components:
first,
cpu
scheduling,
plugins
based
on
scaling
framework;
second
c
group
controller.
It
is
responsible
for
c
group
operation.
This
is
mentioned
in
previous
part
third
c
grove
agent.
It
is
responsible
for
reporting
the
cpu
topology
under
real
cpu
real
load
of
cpu.
A
This
will
be
used
in
scheduling
decisions.
A
process
is
like
this
subtype
one.
When
we
add
to
a
new
nodes
in
kubernetes
cluster
sql
region
will
report
the
topology
to
api
server
step.
2,
quick
schedule,
watches
the
port
creation
event
step.
3
scheduling
decision
is
made
according
to
the
algorithm
step.
4
qubit
schedule
will
create
sql
ci
for
cpu
set
step,
5
c
group
controller,
which
watches
the
c
group
ci
creation
event
step
6
seek
group
controller,
creates
job
4c
group
cpu,
set
operation,
step
7
the
job
will
complete
the
cpu
bonding
operation.
A
A
Theoretically,
filter
will
fill
out
the
nodes
that
contribute
the
requirements
of
the
portfolios
score
cigar
will
select
optimal
cpu
scheduling
result
by
algorithm
result
result
will
first
resolve
the
optimal
cpu
scheduling,
result
to
prevent
reallocation
to
other
posts,
and
it
will
clean
the
cpu
scheduling
result
if
failure
occurs
in
bonding
cycle
post
by
postbid
will
create
supposed
sql
cr
to
store
the
cpu
scheduling.
Result.
A
A
A
A
A
A
A
Because
the
result
of
scheduling
is
optimal
result
at
the
moment
of
poult
creation,
as
shown
in
the
first
picture,
due
to
limited
resources.
Some
poles
are
allocated
cpus
across
new
manuals.
After
a
period
of
time,
the
cpus
allocated
by
port
what
port
a
are
released
part
b
can
be
allocated
from
the
single
newman
node.