►
From YouTube: Using Kubernetes to Offer Scalable Deep Learning on Alibaba Cloud - Kai Zhang & Yang Che, Alibaba
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Using Kubernetes to Offer Scalable Deep Learning on Alibaba Cloud - Kai Zhang & Yang Che, Alibaba
Running deep learning (DL) jobs requires end to end workflow to accelerate model training iteratively. It must be scalable on massive data and computational resources, and be framework agnostic to relieve the pain of managing diverse dependencies. In Alibaba cloud, we use Kubernetes to build elastic DL platform for continuous model training and optimization. It manages heterogeneous cluster including CPU/GPU/FPGA. Jobs are automatically scheduled to the best-fit resources. Kubeflow, which is a great machine learning scaffold on Kubernetes, is used to setup training pipeline. Project Arena is created to manage and instrument jobs with friendly user experience. In this talk, we will discuss how the platform is designed, and how it facilitates users to focus on DL tasks instead of managing underlying complexity. A demo shows how to run distributed neural network training in a minute.
To Learn More: https://sched.co/GrVk
A
B
A
A
Okay,
yeah
before
here
is
today's
our
session
talk
about
how
we
using
kubernetes
children,
deep
learning,
jobs,
Alibaba
cloud
and
before
we
get
started.
I
just
want
to
quick
sorry
to
say
how
many
of
you
guys
have
already
been
running,
machine
learning
or
deep
learning
jobs
on
your
kubernetes
already.
Could
you
raise
your
hands?
Okay.
Also.
Many
people
already
do
that.
Okay,
so
how
many
of
you
already
have
some
experience
about?
The
project
could
be
flow
or
some
equivalent
services,
such
makers,
something
oh
yeah.
Okay,
so
that's
great!
Thank
you.
Thank
you
for
our
response.
A
So
today,
I
post
of
us
coming
from
Beijing
from
China,
we
are
my
name-
is
Kai
Jeong
and
I'm
working
for
Alibaba
cloud,
container
service
team-
and
here
is
my
colleague,
Jung
Cho-
he's
also
from
the
container
team.
Post
offers
working
on
the
album
cloud,
kubernetes
service
post
for
the
public
cloud
or
private
cloud
or
hybrid
cloud,
and
we
also
build
our
people
heterogeneous
computing
solution
and
also
deep
learning
solutions
on
top
of
containers
and
kubernetes.
A
A
Secondly,
I
will
give
some
quick
introduce
introduction
through
the
the
flow
and
the
arena,
those
two
open-source
projects
we
are
using
to
have
our
clients
to
accelerate
their
deep
learning
works
on
their
cluster,
and
so
certainly,
we
will
also
share
some
interesting
real
customer
requirements
for
our
real
clients
scenarios,
and
there
are
some
demos
for
above
features
or
scenarios.
Sorry,
it's
jumping,
ok
and
finally,
I
will
wrap
up
with
some
future
works
in
our
plan.
Ok,
let's
get
started.
A
The
first
part
is
about
let's
talk
about
how
our
design
still
works
on
their
data,
algorithms
or
Amato's
day-by-day.
Basically,
these
spend
a
lot
of
time
to
do
the
to
prepare
your
data
right.
Actually,
there
is
a
saying
that
a
scientist
spend
17
percent
of
his
time
of
his
life
to
clean
the
his
data
and
only
in
the
study
percent
of
the
time
is
spending
on
the
models
right,
and
so
they
do
a
lot
of
teeth.
A
Cleaning
did
labeling
data,
augmentation
pre-processing,
a
bunch
of
things
and
then
z2
some
coding
to
build
their
models
based
on
using
some
statistic,
a
Messer's
or
some
even
sometimes
using
your
networks
right
and
after
the
park
and
the
welfare
verify
their
code.
Cam
works
to
be
some
more
decides,
maybe
with
a
single
machine
with
well-to-do
people
versus
the
jewelry
to
submit
a
job
to
their
cluster,
a
cluster
of
a
GPU
or
XP
you,
whatever
yeah.
A
Then
the
during
their
training,
they
do
a
lot
of
doula
a
lot
of
parameters
tuning
and
the
searching
works
in
order
to
cut
their
model
with
good
accuracy
and
the
current
disease
as
a
possible
memoir.
They
also
need
to
care
about
how
to
distribute
his
job
and
tasks
to
multiple
machines
in
order
to
consume
more
computation
resources
in
parallel
and
to
speed
up
his
training
right,
and
although
they
did
a
lot
of
job
for
that,
but
normally
a
training
job
is
still
can
run
for
hours
days
even
weeks.
A
That's
that's
normal
for
the
tip
learning
engineering
right
and
and
after
that,
when
they
got
a
satisfied
model,
the
model
will
be
released
online,
to
provide
some
kind
of
specific
inference.
Services
for
some
specific
domain
right,
and
sometimes
this
inference
service,
may
also
collect
new
data
and
approve
that
new
data
back
to
the
to
kick
off
a
new
iteration
for
training
to
keep
improve
his
models
performance.
So
that's
kind
of
what
a
workflow
cycle
right
from
this
workflow.
A
We
find
that
basically,
the
deep
learning
jobs
have
some
characteristics
like
sorry,
okay,
the
first
one
is
a
it's
a
kind
of
entrance
job,
it's
a
entrant
life
cycle
with
rotating
and
the
executable
motor
out,
and
some
Muslim
and
the
most
officer
job
will
take
a
long
time
to
run
and
unfortunately,
most
important
and
meanwhile
the
most
importantly,
the
a
lot
of
deploying
jobs
need
massive
data
and
massive
competition
resource
resources
to
process
all
the
tasks.
Okay.
A
So
here
we
summarize
the
of
course,
all
those
characteristics
Perrine.
Some
key
challenges
for
running
active
learning
at
askew.
The
first
one
is
a
need
efficient,
heterogeneous.
The
resource
management,
just
like
previous
session
mentioned
that
nowadays
we
are
not
used
just
the
using
CPU
right.
We
lot,
we
use
a
lot
of
GPU
will
tip
you.
A
Also
hers
am
I
a
MP
you
hp-ux,
pu,
whatever
and
and
also
a
PGA,
and
we
sometimes
we
need
our
InfiniBand
are
Rema
technologies
to
provide
the
high
bandwidth
networks.
For
that
all
of
this
we
call
it
accelerator
technology
or
accelerated
devices.
So
we
need
the
unified
cluster
management
and
resource
scheduling,
algorithm
for
different
types
of
competition
or
networking
resources.
A
Ok,
secretary
I
just
mentioned
it's
a
kind
of
end
to
end
lifecycle,
deep
learning,
experiment.
We
need
provide
a
full
lifecycle
solution
to
help
prepare
data,
prepare
your
model,
train
your
model
and
then
release
your
model
online
right
and
just
like
a
kind
of
experiment
and
all
the
all
the
experiment.
A
History
should
be
recorded
persisted
and
you
can
search
it
and
look
for
it
and
you
repeat
it
again,
because
sometimes
you
tell
you
scientists
that
may
remember
that
he
do
a
just
a
training
job
a
month
ago,
but
he
cannot
find
anything
SS
related
to
that
job.
So
history,
the
environment
management,
is
a
key
to
the
entrant
life
cycle
and
suddenly
the
challenges
it's
continuously
tree
and
serves
a
model
of
skill
at
large.
Skew
that
that
means
we
need
to
make
sure
that
all
the
training
jobs
can
skew
out
and
mount
just
on
the
mound.
A
A
A
A
Actually,
you
can
say
that
in
a
under
layer
is
the
infrastructure
layer
right
is
providing
all
kinds
of
computing
resource
or
networks
and
give
different
types
of
storage
services
and
in
the
middle
we
create
different
services
for
different
life
cycle
stage.
For
that
your
element,
Jupiter
another
box
and
the
gate,
and
to
support
all
those
stuffs-
are
very
integrated
together
and
for
the
developers
to
appeal
to
coding
to
build
your
model
initially
and
then
for
the
training
part
all
the
popular
open
source
frameworks.
A
The
planning
frameworks
are
already
supported,
including
tensorflow,
MX
knowledge
cafe
and
punch
office
at,
and
actually
it's
also
very
easy
for
customers
to
extend
to
use
your
own
own
frameworks
as
well:
okay,
you're
in
training,
yeah
and
then
for
inference.
We
now
support
is
flow
serving
40
by
default,
and
we
also
support
each
create
some
other
generic
inference.
Services
like
southern
core
or
immediate,
is
our
team
first
server,
which
may
provide
a
more
generic
inference
capabilities
for
different
frameworks
right
and
meanwhile,
we
use
a
for
the
inference.
A
Server
we
already
used
is
to
use
a
service
mash
to
control
the
request
traffic
coming
traffic.
With
that
help,
we
can
also
do
the
online
model.
A
B
version
test
can
erase
and
also
the
traffic
control
with
wait.
So
that's
all
already
integrated
or
for
the
inference
services
and
for
the
operational
part
we
are
together
with
each
coach
with
Ali
Baba
cloud
infrastructure.
We
already
provides
a
load
balancing,
auto
scaling
monitoring
logging.
A
All
those
operational
services
are
already
building
to
every
jobs
of
your
training
and
serving
so
you
needn't
care
how
to
configure
that
it's
already
integrated
and
you
can
see
following
we
have
a
demo
to
show
you
just
to
use
a
single
command
line
to
do
all
of
things
and
over
the
top
you
can
see
we
create
an
orchestration
layer
and
some
tooling
to
cover
all
those
underlying
sources.
So,
as
a
result,
the
end
user
only
need
care
use
of
single
command.
A
True
to
do
all
his
jobs
kubernetes
and
he
you
know
and
don't
need
and
don't
need
to
care
about
how
to
configure
set
up
all
those
underlying
services.
Okay.
So
let's
move
on
to
the
second
part,
I
will
do
some
introduction
to
the
cube
of
Law,
Project
and
arena
project.
I.
Think
most
of
us
already
have
some
knowledge
or
experience
about
cool
flow.
So
in
the
past
I
see
60
months
or
past
a
year.
Goofily
has
already
got
a
lot
of
interest
in
the
community.
A
People
want
to
repeatedly
run
his
machine
learning
jobs
on
kubernetes
easily
right,
so
Cooper
flow
is
the
first
choice
for
country
its
first
trials
for
to
doing
that.
Basically,
cool
flow
is
to
build
pot,
pour
motion,
learning
solutions
using
an
October
Natives.
Actually,
its
goal
is
to
make
skating
machine
learning
models
and
the
polar
but
deploying
the
models
to
the
production
as
simple
as
possible,
so
actually
aligned
with
the
kubernetes
concept
to
flow,
provides
some
different
operators
to
support
to
manage
the
different
deployment
frameworks.
A
As
you
can
see
on
the
diagram,
it's
already
support,
tensorflow,
a
patterned
coffee,
coffee
max
net.
Also,
it's
created
another
job
ampere
job
to
support
her
around
or
MPI
works
as
well
and
and
the
cookie
flow
also
has
a
sub
project
named
car
tip,
which
is
used
to
do
some
kind
of
automatically
hyper
parameters.
Searching
and
attuning
to
accelerate
and
simplify
the
training.
A
Work
works
for
humans
and
in
this
November
another
new
subject,
project
is
a
quarter.
Pipe
lines
is
already
announced
and
the
pipeline
project
is
to
support,
develop
and
Ching
up
all
the
training
stuff
steps
as
a
workflow
and
and
using
people
can
using
Python
based
DSL
can
of
a
Python
script,
pasang
SDK
to
to
describe
your
workflow,
how
to
train
a
model
and
then
the
workflow
can
be
exported
as
a
script,
and
it
can
be
persisted
and
share
and
the
repeat
to
run
in
the
pipeline.
A
So
that's
basically
is
a
background
report
could
be
flow
and,
however,
actually
we
create
this
project
named
arena,
which
is
already
integrated
and
a
contributor
to
crew,
follow
community
and
why
we
create
this.
Even
we
already
have
could
flow
that
be
course
for
our
a
lot
of
our
clients,
defendants.
They
don't
don't
they're
not
familiar
with
kubernetes.
At
all-
and
they
don't
care
about
that-
they
don't
want
to
take
any
spend
any
time
on
the
underlying
infrastructure.
They
don't
want
to
use
to
control
those
specific
command
lines
and
tools
they
just
actually.
A
They
prefer
to
use
MPI
or
spark
or
yon
such
kind
of
choose
to
stop,
submit
a
job
directly
and
use
the
same
tool
to
manage
all
the
job
all
the
time.
So
too
easy
their
life.
We
create
arena,
it's
a
simple
command
line,
it's
hidden,
all
the
complete,
complicated,
a
concept
from
kubernetes
and
the
you,
even
you
even
may
not
see
the
concept
from
could
flow
as
well.
Okay,
so,
basically,
the
arena
provides
unified
cluster
management
and
scheduling
for
heterogeneous,
reduce
resources.
A
We
already
sport,
CPU,
immediate
CPU,
VGA
and
some
and
hours
a
year
I
mean
our
DMA
in
some
same
old
and
we
are
working
on
as
a
relay
mode
to
our
DMA
support
with
some
kind
of
oscillation
and
arena
also
orchestrated
and
run
deep
learning
applications
in
contract.
So
all
the
script,
all
the
packages
are
income
charts
from
act
with
building
support.
A
We
also
we
already,
as
I
mentioned
before,
I
already
integrates
a
monitoring
and
login
for
all
your
jobs,
so
you
can
cut
your
real-time
training
logs
during
your
training
and
also
you
can
hurry
and
search
your
own
floor
lock
after
your
training
and
you
can
support
and
and
some
equivalent
we
realized
service
services
already
integrated
with
your
with
your
training.
Job
depends
on
which
frameworks
you
are
using,
and
okay
and
the
the
fun
thing
is
that
for
arena
because
we
create,
we
were
it's
born
from
the
Alibaba
cloud
customers.
A
A
Firstly,
you
specify
the
job
type,
is
ampere
job
or
a
tensor
fellow
job
or
a
max
net
job
and
then
give
some
name,
and
then
you
specify
how
many
workers
and
the
cheapest
you
want
to
use
to
run
this
job
and
give
some
your
special
hyper
parameters
as
the
environment
variables
Anderson's
best
for
your
commands
and
our
image.
You
want
to
use
to
train
this
job.
So
after
that,
you
just
assume
is
this
job.
A
It
will
automatically
deploys
all
the
workers
and
set
up
all
the
GPUs,
seeing
that
for
each
workers
and
keep
kick
off
running
and
meanwhile
all
the
logging
and
monitoring
and
the
checkpoints
are
presented
automatically
and
you
can
use
keyboard
as
the
meanwhile.
So,
as
I
said,
you
can
use
arena
to
just
a
list
or
get
details
of
each
of
the
job.
You
can
see
the
real-time
all
the
full
locks
very
easily,
and
also
you
can
okay.
This
is
a
very
important
feature.
A
Our
clients
are
required
that
they
want
to
use
a
simple
want
to
say:
holder,
GPUs
are
located
from
different
angles,
for
example,
for
the
note
level
how
many
GPUs
are
available
and
who
take,
which
GPUs
and
how
many
memories
already
be
captured
by
specific
jobs.
Okay,
there
are
also
many
other
commands.
A
You
can
check
it
out
on
the
arena.
Github
and
two
of
them
are
very
interesting.
Why
is
arena
Teta?
We
do
some
initial
works
to
use
arena
to
management.
Your
training
status
adds
volumes
and
we
also
support
arena,
serve
that's
kind
of
to
bring
your
your
model,
your
model
to
allowing
inference
service,
and
you
can
even
use
that
to
control
your
surveys,
version
control,
versioning
or
traffic
control.
Ok,
ok,
I,
think
I
will
stop
here
and
let
my
colleague
young
to
give
you
some
quick
demo
how
to
use
arena.
B
B
B
Ok,
let's
submit
a
untie
job
to
run
our
reduce
style
distributed,
distributed
training
you
can
see.
The
comment
is
simple.
You
just
need
to
specify
the
job.
Tab
is
impaired,
job
and
in
the
job
name
and
how
many
workers
you
are
going
to
use
and
how
many
GPUs
for
each
worker
and
the
CPU
memory
site
in
forest,
and
if
you
are
going
to
use
tents
about
you,
can
also
enable
tents
about
and
which
image
which
docker
image
you
are
going
to
use
and
the
data
you
are
going
to
import
here.
B
B
B
B
B
Okay,
let's
go,
and
we
know
the
training
usually
takes
hours
days
even
weeks
in
such
situation,
the
user
needed
to
check
the
historical
data
of
the
GPU
usage.
Then
they
can
use
grana
to
visualize
it
in
the
graph
Anna
you
can
check
the
GPU
GPU
memory
and
the
GPU
usage
and
the
temperature
and
the
power
usage.
B
Okay,
let's
go
back
to
the
terminal
game.
Let's
run
arena
top
job
again
we
can
see
here
it
is
the
GPU
usage.
When
the
GPU
usage
is
going
down
to
zero.
It
means
the
training
is
finished
and
we
can
control
it
and
we
can
use
arena
list
to
check
the
consider
training
is
done.
We
can
use
arena
list
to
check
the
job
status.
B
A
B
The
first
requirement
is
for
GPU
sharing,
sharing
NVIDIA
GPUs
among
multiple
pause
and
multiple
containers
to
increase
the
utilization
of
the
kubernetes
cluster
is
the
real
requirement,
especially
for
the
big
business
organization.
Let
how
many
models
and
most
of
the
models
are
not
large
enough
to
fit
the
whole
GPU.
They
want
to
share
their
jib
dirt
review
to
reduce
code
to
reduce
cost,
but
it's
difficult
to
implement
it
in
kubernetes,
because
there
are
some
challenges
in
schedule
soon
in
schedule
and
isolation,
perspective,
kubernetes
schedule,
scheduler
enforces
exclusive
GPU
requests,
even
in
isolation
level.
B
There
are
more
challenge
Bowl,
as
we
know,
MPs
is
only
for
water
and
no
to
vary
production
right
already
in
in
in
in
very
ready
for
the
in
the
production,
and
the
Nvidia
great
grid
is
only
for
the
virtual
machine.
Now
our
solution
is
not
it's
not
to
change.
The
kubernetes
core
code
will
be
leveraged
in
extended
resource
to
claim
GPU
memory
request
and
the
request
can't
be
more
than
one
GPU
and
as
the
end
user,
it
just
need
to
specify
how
many
GPU
memories
they
are
going
to
use.
B
Scheduler
is
responsible
for
determining
which
node
transmitted
the
GPU
request
and
the
rich
GPU
devised
can
be
put
in
the
port,
and
we
also
provide
a
device
plug-in
to
allocate
the
device
to
the
containers
by
passing
some
specific
environmental
variables
to
the
cupola,
cooperate
and
the
for
the
isolation
level
and
for
the
isolation
level.
We
also
reuse
the
oh
sorry
for
the
isolation
level.
We
will
reuse
the
MPs
and
some
alpha
technology
to
to
concrete
and
we
will
open
source
less
solution
later.
Okay,
let's
go
to
see
the
demo
first.
B
First,
let's
make
sure
the
GPU
sharing,
scheduler
extender
and
the
device
plug-in
up
and
running,
and
we
also
provide
a
control
extension
tools
which
named
Cooper
control,
inspect
to
check
the
GPO
sharing
of
the
of
agent
of
the
cluster
we
can
send.
There
is
only
one
node
for
GPU
sharing
and
it
has
two
GPUs.
Each
of
them
has
about
16
Gabi
beta
cap.
In
my
memory
now
we
are
going
to
create
a
deployment
named
pimple
pimple,
one
with
request
8
about
8,000
Matt,
maybe
badge
and
means
half
of
a
GPU.
B
B
B
B
B
B
A
Okay,
thank
you
and
for
the
demo,
and
let
me
see
okay.
Well,
let's
get
it
quick.
We
still
have
so
many
interesting
scenarios
for
our
clients
to
teach
you
another.
One
is
about
DEFCON
scheduling,
I.
Think
most
of
us
already
got
some
knowledge
of
Alken
scheduling,
but
basically
it's
being
scheduled,
your
port
with
a
policy
we
required
or
NASA.
That
means
you
just.
A
We
just
find
your
node
to
a
job
only
if
the
all
the
sub
task
of
this
job
can
cut
or
enough
resources,
otherwise
the
job
is
itself
or
be
pending
and
the
norm
of
no
tree
resources
will
be
allocated
to
and
in
some
parts
of
this
job.
So
if
you
look
at
the
top
of
the
simple
sample
scenario
that
we
have
to
note,
each
of
them
have
two
GPUs
and
while
one
GPU
has
already
be
used,
that
means
each
other.
Each
node
there
will
be
only
one
ship,
you'll,
add
all-staters,
and
there
are
we.
A
Now
we
have.
Two
jobs
need
to
be
scheduled.
Supposing,
in
the
first
one
job,
one
will
be
submitted
first
job
one
have
has
two
reports
task
force:
each
each
one
will
need
a
one
GPU
car
and
the
job
to
just
have
one
port
and
just
once
one
chip
you
in
that
case,
because
job
y
is
submitted
first
by
the
default
kubernetes
scheduler.
It
will
allocated
to
a
dull
GPU
card
through
the
two
of
the
three
ports
for
GPU
the
job
one.
However,
job
is
doing
pending
status
because
it's
Union
one
more
GPU
for
the
third.
A
So
the
car
scheduling
is
to
make
sure
that
the
job
one
cannot
be
allocated
with
earning
resource,
and
it's
always
in
handy
in
that
case
the
job
to
can
have
some
chance
to
keep
on
running
and
the
job
was
still
waiting
for
the
enough
resources.
So
we
basically
we
implement
scheduling
with
the
help
of
capacities
that
project,
which
will
be
our
two
summer,
refactor
works
and
we
will
call
comfortably
the
community
to
make
it
more
stable
and
more
production
reality.
Actually,
we
already
commit
our
GPU
support
to
the
batch
TV
projects.
A
Okay,
the
next
one
is
in
the
demo.
We
already
mentioned
that
our
chippy
on
mark
train
is
a
little
bit
different
from
the
other
normal
device
monitoring.
It's
some
interesting
that
or
one
some
interesting
metrics
is
a
temperature,
especially
when
you
use
a
g-force,
for
example,
wow.
It's
a
tie,
kind
of
g-force
card
for
your
training.
Of
course
it's
not
an
enterprise
level
card.
Sometimes
it's
not
that
stable
right.
The
temperature
is
a
very
important
metrics.
To
give
you
some
hints,
that's
your
GPU
card
may
in
a
dangerous
situation.
A
Sometimes
in
the
fine
instance
of
work
is
not
in
the
housing
status.
We
can
also
monitor,
defines
health
status
as
well,
and
the
other
one
I
won't
spend
too
much
time
on
this
complete
scenarios,
but
I
just
mention
that
we
use
the
arena
serve
to
integrate
with
is
to
to
to
the
traffic
control
for
the
different
versions
of
the
different
server.
To
do
sorry,
who
do
the
ad
test
and
cannibal
release,
and
you
can
control
every
single
trav
request
traffic
to
different
versions
of
your
model.
A
A
There
are
AI
lab
user
communities
and
arenas
to
delay,
treating
jobs
so
far
they
can
are
treating
their
job
against
one
ten
millions
features
with
ten
millions
features
and
1
billion
sample
data
so
day.
So
basically,
they're
training
clusters
has
already
managed
over
200
GPU
nodes
and
with
help
with
arena
they
can
create
such
kind
of
for
GPU
clustered
within
10
minutes
and
a
kickoff
large
skill
tip
learning
jobs
in
one
minute
and
the
other
one
is
a.
A
We
are
corporate
with
our
Baba
internal
AI
teams
to
do
some
chip
learning
framework,
optimization
works,
we
miss
contest
flow
or
pet
watch.
We
do
some
tuning
and
I'm
here
class
ring
or
radius
all
those
kind
of
works
with
other
readouts.
So
far,
we
can
enable
16
for
GPU
cards
to
get
19%
speed-up
and
that
that
will
be
training
on
the
rest
net
15
against
the
image
net
data
set
and
that
will
be
14
5
percent
higher
than
the
native
tensorflow
speed.
A
A
We
will
focus
on
to
do
some
works
to
treat
moves
or
training
were
close
integrated
with
our
some
lace
kubernetes
and
using
the
support
instance
of
GPU,
considering
the
cost
effect
effective
and
also
the
design
is
scared
about
cluster
concept
at
all,
and
we
also
count
out
a
no
topology
about
scheduling.
That
means
a
CPU,
GPU,
Numa,
PCIe,
connector
or
emmy-winning
connect
all
those
specific
requirement
to
the
fine-grained
scheduling
on
your
note
on
your
single
note
and
we
will
integrate
with
different
model,
helps
to
to
provide
the
protein
models
and
repeatable
workflows.