►
From YouTube: Measuring and Optimizing Kubeflow Clusters at Lyft - Konstantin Gizdarski & Richard Liu
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Measuring and Optimizing Kubeflow Clusters at Lyft - Konstantin Gizdarski, Lyft & Richard Liu, Google
Machine learning workloads are often resource-intensive operations. As companies adopt more of these workloads, tracking resource consumption and optimizing spending becomes more challenging.
At Lyft, we developed a system which scrapes metrics from Kubernetes clusters and persists them in data warehouses. We then built a pipeline that transforms snapshots into cluster utilization metrics along the dimensions of CPU, memory, and GPU. Finally we join these metrics into our cost and usage dataset, so teams can budget resources accordingly and reduce spending. In this talk, we will give an overview of Infraspend - our infrastructure for tracking Kubernetes usage. Attendees will learn how the data we collected helped Lyft reduce spending for Kubeflow clusters. The audience will also gain insights into how Kubernetes clusters can be optimized without performance or stability compromises.
https://sched.co/UabJ
A
A
Thank
you
appreciate
that
hi
I'm,
Priyanka
and
I'm
your
track
host
I
am
director
of
evangelism
at
gitlab
and
also
member
of
the
governing
board
of
the
cloud
native
computing
foundation
or
the
CNC
F,
the
org.
That
brings
you
coop
con
so
well.
Thank
you.
So
much
for
coming
and
being
troopers
throughout
the
day.
I
hope
you've
learned
a
lot
of
awesome
stuff.
I
definitely
did
this
track.
A
A
The
knowledge
that
you
gained
so
please
do
go
to
scat
calm
after
and
rate
the
session
I'm
on
the
program
committee
and
I
can't
tell
you
how
helpful
it
is
because
every
year
we
get
hundreds
and
thousands
of
submissions
at
this
point
and
having
some
direction
from
your
ratings
is
really
helpful.
So
with
that
housekeeping
out
of
the
way
I'd
like
to
introduce
the
last
speakers
of
this
in
this
room,
so
Derek
from
Google
and
Konstantin
from
lyft
are
going
to
talk
about
Richard.
Oh.
A
A
There's
kubernetes,
there's
control
planes
this
that
it's
hard
to
know
exactly
where
your
resources
are
being
allocated
where
the
spend
is
going,
and
that
is
very
important
to
the
longevity
of
any
project
that
you're
working
on
so
they're
gonna
talk
about
something
they
built
called
infraspinatus
set
and
dashboards
to
analyze,
where
your
spends
going
and
talk
about
how
to
think
through
these
things
for
resources,
intensive
ml
workloads.
So
with
that
Richard
and
Konstantin,
please
take
it
away.
Thank.
C
B
I
am
Constantine
I
sort
of
put
a
few
logos
to
make
myself
seem
more
legit,
so
I
recently
graduated
from
Northeastern
kind
of
last
year,
and
before
that
I
interned
at
Instagram
and
stripe,
working
on
product
and
machine
learning,
and
then
at
lyft
I
joined
a
team
called
capacity.
And
basically
the
point
of
the
capacity
team
was
that
our
infrastructure
costs
were
increasing
a
lot
and
we
needed
to
build
visibility
and
tooling
to
control
that
so
I
want
to
give
you
guys
a
brief
overview
but
you're
going
to
learn.
B
Basically
we're
gonna
try
to
cover
a
lot
of
ground
and
I
want
you
to
understand
sort
of
like
our
journey
with
cloud
infrastructure
spending
at
lift
a
little
bit
about
sort
of
the
way
that
we
use
cube
flow.
Some
of
the
machine
learning
workloads,
how
those
complicate
our
cloud
and
for
spending
story
and
then,
finally,
how
we
built
a
product
called
infra
spend
and
how
we
extended
it
to
track
platforms
like
kubernetes
and
also
other
multi-tenant
platforms
and
I
kind
of
want.
B
You
guys
to
keep
in
mind
why
this
is
important,
what
matters
and
then,
hopefully,
maybe
if
you
don't
have
something
similar
at
your
organization,
can
inspire
you
to
try
to
take
on
this
journey
cool.
So,
like
I,
said
around
the
time
that
I
joined
our
infrastructure,
bill
was
increasing
quite
a
bit,
so
at
lift
our
philosophy
with
infra
is
that
we
want
to
give
teams
autonomy
to
spend
up
whatever
resources
they
need.
B
But
this
was
a
problem
because
it
also
meant
that
we
pretty
much
had
no
idea
where,
like
our
spend,
was
going
until
we
built
a
spreadsheet.
So
my
manager,
who
I
guess
what,
by
the
time,
was
not
a
manager
sort
of
this
initiative
at
lift,
and
he
was
just
scraping,
AWS,
api's
and
uploading
that
data
into
Google
sheets
and
then
around
the
times
I
joined.
We
started
working
on
something
called
inter
spun
2.0,
which
is
much
more
sophisticated
technologies
to
pull.
B
This
thing
called
the
cost
and
usage
report
from
AWS
and
process
it
using
sort
of
like
standard
infrastructure
like
lifts
data
platform
and
create
this
like
nice
data
set
that
we
can
build
on
top
of.
So
don't
want
to
dive
into
it
too
much
right
now,
but
there
are
two
features
that
we
should
keep
in
mind.
One
of
them
is
that
we
wanted
to
derive
something
called
a
lift
label
which
lets
us
track
spending
and
resources
in
a
way.
B
D
B
Go
through
these
real
quick,
so
this
is
kind
of
like
one
of
the
entry
points
to
in
for
spend
it
lets
you
choose
or
for
dashboards.
So
there's
like
the
deep
dive,
dashboard
executive,
dashboard
stuff
like
that,
and
this
is
a
view
that
people
tend
to
like
a
lot.
So
you
can
see
that
the
time
series
with
like
the
stacked
bar
chart,
so
that
kind
of
shows
like
each
of
those
is
like.
B
It
really
helps
get
people
to
like
look
into
it
and
like
dig
into
it,
because
it's
so
easy
to
digest
another
interest,
one
demo
trends,
executive,
dashboards,
so
I
guess
the
the
complication
here
is
that
in
the
olden
days,
when
you
had
a
single
resource
like
an
auto
scaling
group
or
like
a
cluster,
you
would
have
like
one
service
running
on
it.
So
you
might
have
like
the
pricing
service,
which
runs
on
the
pricing
cluster,
but
obviously
with
multi
hit
on
platforms.
B
So,
for
example,
this
is
like
the
this
is
like
the
number
of
easy
to
see.
518
X
large
instances
that
we
were
running
in
our
core
kubernetes
clusters
at
one
point,
and
then
this
is
further
complicated
by
the
fact
that
we've
had
dozens
of
different
clusters
that
lift
that
we
needed
to
attribute.
So
there's
the
core
kubernetes
cluster
is
in
flight
and
machine
learning
clusters.
So
Richard
is
going
to
speak
a
bit
about
cube
flow
and
some
of
the
features
that
we've
used.
C
Okay,
thank
you.
So,
as
a
constantine
mentioned
at
left,
we
I
mean
they.
So
they
have
this
platform
called
lips
learned
for
a
machine
learning
platform
and
it
uses
components
from
coop
flow.
So
that's
that's
what
I
work
on
at
Google,
so
one
of
the
problems
that
consoles
invention
is
like
criminalities,
reduces
visibility
into
your
usage
right,
and
this
is
like
is
a
sir
baited
by
the
fact
that
machine
learning
workloads
are
quite
complex.
So
we're
going
to
take
a
tour
of
Google
and
we're
going
to
look
at
how
different
machine
learning
were
closed.
C
C
You
start
a
container
instance
and
you
can
open
up
the
web
portal
and
you
use
your
you
can
type
your
your
training
code
here
and
write
everything
in
notebook
all
right.
So,
what's
the
use,
this
pattern
like
you're,
most
likely
using
it
as
individual
users
all
right,
so
anybody
any
data
scientists
or
any
machine
learning
engineering
can
just
spin
up
a
notebook.
Okay
and
you
won't
happy
to
be
a
high-availability
right.
If
you
want
to
start
a
notebook
session,
you
wanted
to
start
out
right
away.
C
You
don't
want
to
wait,
30
minutes
or
so
for
it
to
start
up.
So
most
likely,
you
have
multiple
users
using
multiple
notebook
instances
and
the
users
pattern
kind
of
grows
like
a
linear
fashion,
and
but
you
could
have
problems
like
potentially
I.
Don't
notebooks
right,
someone
starts
a
notebook
session,
used
it
for
cover
hours
and
just
I
forget
about
it,
and
you
know
you
have
these
idle
resources
sitting
around
right.
So
that's
for
notebook
instances.
C
Another
possibility
is
like
k
of
serving
which
is
there
were.
There
was
a
talk
earlier
on
this,
but
basically
this
is
the
scalable
kubernetes
native
enforcing
service.
For
this
you
want
the
usage
to
be
most
likely.
It's
also
high
availability,
but
you
also
want
the
ability
to
quickly
at
capacity
right,
let's
say
you're
you're
calling
for
for
the
enforcing
calls
right.
You
have
a
certain
spike.
This
requires
you
to
quickly
adjust
to
the
additional
level
of
demand.
C
Right
and
potentially,
you
need
a
lot
more
GPUs,
so
you
can
see
that
this
is
quite
different
from
the
notebook
instance
where
it
grows
fairly
linearly
and
for
notebooks.
Your
usage
is
most
likely
a
tribute
to
individual
users,
but
in
this
case,
is
more
likely
to
be
attributed
to
a
service
or
a
team.
C
Okay,
we
also
have
a
code
for
pipelines,
so
this
is
a
component
component
that
allows
you
to
build
into
in
machine
learning
workflows.
So
we
can
see
an
example.
On
the
right
hand,
side,
the
writing
is
little
small,
but
you
can
see
it's
just
like
creating
a
class
or
there's
a
transform
training
predict.
So
it
takes
these
machine
learning
steps
and
build
the
dependencies
into
this
dependency
graph,
and
you
can
run
these
workflows
regularly
on
a
schedule
right.
So
what's
that
like
in
the
real
production
environment?
C
So
you
could
have
these
bottlenecks
because
some
steps,
like
the
training
sets,
may
be
very
resource
intensive,
and
so
you
could
have
several
of
these
pipelines
competing
for
the
same
resources,
and
this
is
my
favorite
hyper
parameter
optimization,
because
this
is
a
component
that
I
work
on
the
most
so
a
little
bit
about
hyper
parameter,
optimization
right,
so
hyper
parameters
are
parameters
that
are
external
to
the
model.
Unlike
the
model
parameters
which
is
in
a
training
process,
you
learn
the
values
of
the
model
parameters
right
for
hybrid
parameter.
C
You
set
the
values
of
these
before
you
started
training
process,
so
examples
are
like
the
learning
rate
number
of
layers,
a
batch
size,
kernel
type
and
so
on.
So
the
process
of
hyper
parameter.
Optimization
is
finding
the
best
values
such
that
the
amount
of
performance
is
maximized,
so
in
qu
flow
there
is
a
component
called
ketchup,
that's
basically
the
open
source
implementation
of
hyper
around
or
tuning
on,
Chrome
or
Nettie's,
and
what
it
does
is
you
just
initialize
your
search
space
and
your
model,
and
there
are
two
main
constraints
for
a
tuning
experiment.
C
There
is
a
objective
value,
which
is,
let's
say,
you're,
trying
to
optimize
your
prediction:
accuracy
right,
so
you
can
set
a
goal
like
I
want
my
prediction:
accuracy
be
90
percent
or
something
there
is
also
a
budget.
So
this
is
like
how
many
trials
I
want
to
run
right.
So
these
are
the
constraints
for
my
experiment,
and
each
initial
iteration
only
does
a
guess.
C
The
next
set
of
parameter
suggestions
from
saturation
service
through
this
call
called
guess,
suggestions,
and
that
gives
you
the
next
set
of
your
hyper
parameters,
use
and
then
this
gets
plugged
into
a
trial.
I
try
our
just
a
running
instance
of
the
of
your
water
raters
are
trying
to
optimize
your
experiment.
So
it
runs
that,
while
using
the
parameter
values-
and
it
produces
some
metrics
values-
and
these
metrics
could
report
it
back
to
the
reporting
service
and
it
continues
right.
C
So
how
does
this
affect
our
our
spanning
story?
Well,
so
a
tuning
job
is
can
be
very
CPU
intensive
right
because
you
could
have
potentially
many
of
these
actually
explained.
Each
trial
is
actually
a
training
job
right,
because
you're
launching
a
job
and
you're
training
for
like
some
number
of
epochs,
and
it
produces
some
set
of
metrics
and
give
me
this
metrics.
We
determine
if
this
is
a
a
good
set
of
hyper
parameters
or
not
right.
C
So
what
does
it
all
mean?
So
we've
seen
a
few
examples
like
we
didn't
see:
distributive
training,
high
preparator
tuning
serving
and
your
notebook
instances
right.
So
all
these
have
very
different
usage
patterns
right.
Some
of
them
are
very
high
volume,
high
demand,
someone
require
very
high
availability
and
some
are
very
resource
intensive
right.
C
So-
and
this
is
a
further
complicated
by
the
fact
that
your
concert
could
be
multi-tenant.
You
could
have
several
different
teams
and
each
team
could
be
doing
different
things.
Some
might
be
more
doing
more
training.
Some
might
be
more
experimenting
with
notebooks
right
so
because
we
are
nation
earlier
the
kubernetes
high
as
a
way
that
are
these
details.
So
if
everything
is
just
aggregated
at
cluster
level,
it
is
very
difficult
to
attribute
what
your
your
actual
spending
is
right.
C
So
multi-tenancy
is
some
a
problem
that
could
flow
itself
is
trying
to
address
and
in
our
upcoming
1.0
release.
We
want
you
to
have
a
complete
story
for
what
desi,
but
we
are
more
working.
Tourists,
like
the
user
isolation
right
to
being
able
to
create
a
a
profile
where
each
user
can
have
a
isolate
a
set
of
resources
to
use.
C
B
So
essentially,
we
wanted
to
build
a
system
that
is
modular
and
extensible,
so
we
wanted
to
start
with
kubernetes,
but
we
wanted
the
concepts
that
we
use
for
multi-tenancy
to
extend
to
other
platforms
like
ETL
experimentation
and
so
on
and
so
forth
and,
finally,
or
and
we
wanted
to
make
this
a
platform
for
platforms.
So
we
basically
wanted
to
provide
a
clear
way
that
platform
owners
can
provide
their
attribution
data
to
us
and
subdivide
their
resources,
because
partially
our
philosophy
is
that
platform
owners
understand
how
they
should
attribute
their
resources
best.
B
B
There
must
be
data
for
every
single
hour
and
then
the
usage
has
to
add
up
to
exactly
one.
So
this
is
essentially
it's
a
hive
table
that
breaks
down
the
hourly
cost
of
a
certain
platform
according
to
different
attribution
labels
as
a
percent
and
in
terms
of
how
a
tribution
schedules
can
be
generated.
We
essentially
support
three
different
methods
for
attribution
schedule
generation.
B
If
that
doesn't
work
for
platform
owners,
there
are
sort
of
two
other
options
that
I'm
just
going
to
touch
on
right
here,
but
not
necessarily
go
into
like
the
details
of
so
basically
either
platform
owners
can
stream
their
usage
to
us
they're
like
lift
standard
analytics
infrastructure,
or
they
can
support
their
own
attribution
schedule.
That
has
the
properties
that
I
brought
up
earlier
and
we
can
consume
that
as
well.
So
they
sort
of
go
from
least
involvement
for
the
platform
writer
and
to
most
involvement.
The
platform
owner.
B
B
So
I
guess
without
further
ado,
I'll
talk
a
little
bit
about
how
we
provide
a
namespace
level
attribution
to
our
kubernetes
workloads
first,
so
here's
a
practical
example
of
how
this
works
up
top.
You
see
the
cost
of
different
resources
in
our
raw
enforcement
data,
so
you
have,
for
example,
at
1:00
p.m.
cluster
0
cost
$1,
and
on
the
attribution
schedule
down
below
on
the
left
side,
you
can
see
that
the
usage
of
the
cluster
0
platform
should
be
subdivided
like
50%
of
it
should
go
to
the
free
namespace.
B
Another
thing
that
we
had
a
lot
of
discussion
around
and
are
actively
working
to
build
more
support
on
our
different
cost
models
for
platforms.
So
in
the
case
of
kubernetes,
we
have
like
entertained
a
bunch
of
options,
so
we
started
a
half
by
building
the
CPU
allocation,
cost
model
and
I
guess
some
of
the
other
ones
are
fairly
intuitive
in
terms
of
what
they
might
look
like
under
the
hood.
B
Another
aside
here
is
that
we
think
of
enforcement
as
being
about
allocation
rather
than
utilization
or
efficiency,
so
I
guess
like
in
the
non
kubernetes
world.
This
means
that,
like
if
your
team
spends
up
a
node,
even
if
they
don't
use
it
at
all,
it
goes
to
your
team
and
similarly
with
kubernetes.
If
you
request
some
resources,
even
if
you're
not
using
them
effect,
they
should
be
attributed
to
your
team.
B
I
guess
one
other
bit
is,
although
we
don't
really
think
of
enforcement
as
being
about
efficiency,
we
kind
of
have
a
feature
that
allows
us
to
monitor
the
efficiency
of
our
kubernetes
platforms,
which
is
our
unallocated
capacity.
So,
according
to
our
cost
model,
with
like
the
CPU
allocation
things
that
I
discussed,
we
have
a
balance
which
is
the
the
leftover
cores
after
you
have
accounted
for
the
namespaces
that
have
requested
CPU
cores
in
the
cluster,
and
so
we
surface
our
unallocated
CPU
capacity
in
our
data.
B
In
terms
of
how
we
collected
our
kubernetes
metrics,
we
spun
up
a
very
lightweight
micro
service,
which
scrapes
metrics
for
each
of
our
crew
bodies
clusters.
So
two
bodies,
cluster
owners-
can
register
their
cluster,
which
essentially
means
that
our
service
will
scrape
that
clusters
Prometheus
and
at
issue
is
a
couple
of
different
requests.
So
one
is
that
we
we
take
cube
state
metrics
from
Prometheus
for
CPU,
core
requests
for
running
pods,
as
well
as
the
CPU
capacity
memory
capacity
and,
more
recently,
pod
labels.
B
So
of
going
to
a
little
bit
of
detail
about
how
we
use
pod
labels,
but
essentially
it's
to
do
custom
tag,
attribution
on
things
other
than
namespaces
for
annotations,
that
users
might
add
to
their
pods
so
downstream.
Our
service
omits
our
kubernetes
usage
data
so
like
the
requests
and
capacity
and
so
on,
and
to
our
hive
infrastructure,
and
there
are
two
steps
essentially
and
I
kind
of
illustrate
the
purposes
put
screenshots
of
what
that
code
looks
like,
but
compressed
it.
So
you
can't
read
it.
Hopefully
that's
kind
of
entertaining.
B
So
on
the
left
side
we
take
the.
We
essentially
do
this
computation
that
I
discussed
where
we
group
all
our
requests
by
namespace,
and
then
we
divide
those
by
the
total
cluster
capacity
and
by
instance,
type
as
well.
So
did
that
in
kind
of
hide,
QL
I
definitely
would
not
recommend
that
and
then
the
the
step
afterwards
is.
We
run
interpolation
so
because
we
have
a
bunch
of
different
instance
types
and
a
bunch
of
different
kubernetes
clusters.
B
So
it's
sort
of
sort
of
the
point
I
just
brought
up,
because
there
are
lots
of
different
kubernetes
clusters
and
within
those
clusters,
there's
lots
of
different
dimensions
that
we
want
to
support.
The
operational
load
has
actually
been,
or
at
the
beginning
of
the
project,
was
actually
a
fairly
high,
and
so
some
of
the
more
challenging
things
were
just
getting
the
right
visibility
in
our
service
to
scrape
those
metrics
so
that
we're
notified
essentially
as
soon
as
possible.
B
Similarly,
with,
like
roll-ups
and
different
dimensions
being
present
on
the
data,
like
the
distinction
between
Odin's
instance
type
in
our
denote
an
instance
and
like
keeps
a
metrics.
Basically,
it
was
really
important
to
test
assumptions
about
our
data
at
every
single
step,
so
kind
of
seems
like
a
fairly
simple
pipeline,
but
in
reality,
there's
probably
like
seven
or
eight
sequential
components
and
we've
kind
of
experienced
like
different,
unexpected
behaviors.
At
most
of
them,
so
it's
really
important
to
do
sanity
checks
at
every
step
cool.
B
So
that's
a
little
bit
of
details
on
how
we
implemented
this
under
the
hood.
I
guess
up
next
I
wanted
to
just
go
through
a
few
screenshots
and
might
talk
through
what
the
product
experience
was
with
then
respond
2.0
with
kubernetes.
So
this
is
another
screenshot
from
our
product.
So
one
of
the
things
you
can
see
on
the
left
side
and
like
the
green
is
our
either
daily
or
hourly
usage
on
just
like
bear
instances
and
on
the
right
side,
you
can
see
our
hourly
usage
on
kubernetes.
B
B
B
So
raise
awareness
of
inefficiencies
from
an
allocation
standpoint,
and
so
there
there
have
been
a
few
things
that
we've
done
to
make
our
crew
buddies
deployments
more
efficient.
I
think
the
biggest
impact
was
tuning
cluster
scaling
policies
so
like
removing
nodes
when
they're
CP
utilization
is
higher
than
previously,
and
also
for
tuning
plots
getting
policies.
B
So,
ok,
so
quick
aside,
yeah,
so
a
lot
of
the
stuff
going
forward
is
kind
of
like
additional
features
that
we've
been
adding
to
our
Kubrat
ease
for
enforcement
infrastructure,
so
we're
working
on
supporting
more
allocation
schedules
and
adding
different
dimensions.
So
we've
had
a
few
teams
experiment
with
like
looking
at
their
attribution
data
based
on
a
container
or
based
on
custom
pod
labels
that
they
emit
through
our
usage
tracking
service
and
also
working
with
teams
to
build
custom
views
into
the
data.
B
B
Do
the
same
data
set
and
that
sort
of
goes
with
the
point
about
being
able
to
build
custom
dashboards
on
top
of
the
data
set
that
we
built
so
I,
guess
almost
everything
that
we
use
is
open
source
and
it
should
generalize
to
like
different
vendors
I
think
we
have
some
stuff
and
it's
like
essentially
to
integrate
things
like
wavefront
and
so
on,
but
the
technologies
that
we
use
are
essentially
Prometheus
and
kubernetes
and
CubeSat
metrics,
and
just
that
data
into
hive.
We
make
it
usable
through
druid
so
Joe.
B
It
allows
us
to
like
query
the
data
much
more
quickly
and
we
display
it
through
another
lift
open
source
project
called
super
Sun,
which
is
essentially
how
we
built
all
the
dashboards
that
I
shouldn't
screenshots
of
throughout
the
presentation,
cool,
so
yeah
I
mean
going
forward.
We
want
to
on
board
more
platforms
to
enforce
pend,
provide
finer
granularity,
talk
about
efficiency,
more
somebody
that
cool
Thanks.
D
So,
prior
to
using
enforcement,
you
mentioned
that
kubernetes
kind
of
clouded
your
visibility
into
cost
compared
to
the
you
know,
cost
and
usage
data
that
AWS
provides.
But
now
that
you
have
infrared,
would
you
say
that
you
have
more
visibility
into
your
costs
than
you
did
with
AWS
costs
and
usage
reports.
B
B
F
It's
a
question
you
mentioned
that
you
have
different
type
of
allocation
based
on
the
different
team,
so
one
question
I
was
wondering
is
if
you
have
different
team
that
are
like,
let's
say:
I've
got
a
workload
that
is
CPU
driven
while
another
one
that
is
memory
driven
and
they
both
share
within
the
same
cluster.
How
do
you
split
them
apart?.
B
So
when
we
were
first
building,
the
multi-platform
attribution
with
kubernetes
GPU
wasn't
really
supported
through
CubeSat
metrics,
not
sure,
if
that's
still
the
case,
but
we
basically
could
collect
attribution
schedules
for
either
CPU
data
or
for
memory
data,
and
so
we
did
some
like
ad-hoc
analysis
for
our
platforms
to
find
whether
CPU
or
memory
made
more
sense
in
terms
of
like
how
that
allocation
would
come
out
and
so
for
our
core
platforms.
We
found
that
it
was
CPU
for
other
platforms
like
for
some
of
our
ml
workloads.
B
G
C
Version
of
code
flow,
so
they
may
have
like
for
an
example
of
the
hybrid
parameter
tuning
I
think
that
they
built
around
a
UI
to
simplify
some
of
the
like
getting
started
experience
because
if
you
use
the
Cadi
of
just
out
of
the
box,
you
have
to
configure
to
see
RDS
and
all
that
stuff
right
so
that
they
have
a
pretty
nice
user
interface
sense
and
it's
more
specific
towards
it.
But
it's
not
like
it's
not
a
different
version
of
go
floor.
Okay,.