►
From YouTube: Efficient AutoML with Ludwig, Ray, and Nodeless Kubernetes - Anne Marie Holler + Travis Addair
Description
Efficient AutoML with Ludwig, Ray, and Nodeless Kubernetes - Anne Marie Holler, Elotl + Travis Addair, Predibase
The open-source platforms Ludwig and Ray make Deep Learning (DL) accessible to diverse users, by reducing complexity barriers to training, scaling, deploying, and serving DL models. Recently, Ludwig was extended to support AutoML, for tabular datasets (v0.4.1) and for text classification datasets (v0.5.0), using Ray Tune for hyperparameter search. In this talk, we discuss how Ludwig AutoML exploits heuristics developed using a set of training datasets to efficiently produce models for validation datasets. And we show how running Ludwig AutoML on cloud Kubernetes clusters, using Nodeless K8s to add right-sized GPU resources when they are needed and to remove them when not, reduces cost and operational overhead vs running directly on EC2.
A
A
I
just
wanted
to
give
a
brief
shout
out
to
the
community
of
people
that
contributed
to
content
of
this
talk,
including
people
from
the
ray,
ludwig
and
elodal
communities.
There's
four
kind
of
sources
for
the
material,
our
recent
from
february
blog
on
cncf
about
managing
public
cloud
resources
for
deep
learning
training,
our
medium
blog
from
february
on
ludwig
automl
for
deep
learning.
This
was
focused
on
tabular
data
sets
our
more
recent
medium
blog
on
ludwig
automl
for
text
classification
data
sets.
A
And
finally,
our
cloud
native
rejects
from
this
past
fall
where
we
talked
about
a
poc
of
running
ray
on
public
cloud
kubernetes.
So
we're
going
to
discuss
the
efficiency
of
automl
with
respect
to
two
aspects.
One
is
the
efficiency
of
using
automl.
Rather
than
doing
your
own,
you
know
untied
untuned
search
and
the
second
is
when
you're
using
automl
how
you
can
use
it
efficiently
in
public
cloud
kubernetes
versus
using
it.
You
know
like
by
directly
deploying
on
ec2.
A
B
Hi
everyone
thanks
for
coming
to
our
talk
today.
My
name
is
travis
adair,
I'm,
the
cto
of
a
company
called
predabase,
building
an
enterprise,
low
code,
machine
learning
platform
built
on
top
of
ludwig,
and
today
I'd
like
to
tell
you
a
little
bit
about
the
background
behind
the
ludwig
project
and
how
automl
fits
into
the
vision
of
what
we're
doing
with
the
open
source
lumia
project.
B
To
start.
I
want
to
present
the
background
on
why
we
believe
that
ludwig
is
a
valuable
addition
to
the
ml
ecosystem.
So
our
observation
is
that
if
you
look
at
the
way
and
all
is
done
in
industry
today,
there
are
essentially
two
incomplete
options
that
are
available
to
companies
and
organizations
that
want
to
operationalize
ml.
B
On
the
one
hand,
you
have
low
level
apis
like
tensorflow
and
pytorch,
that
provide
a
great
deal
of
flexibility
and,
on
the
other
hand,
you
have
traditional
automl
systems
that
provide
a
lot
of
simplicity,
but
neither
of
them
end
up
being
ideal,
because
oftentimes
the
low-level
apis
are
difficult
to
get
into
production
for
non-expert
users,
while
the
automl
systems
end
up
being
these
black
boxes.
That
you
end
up
graduating
out
of
because
they
don't
always
solve
the
problem,
the
first
time
around,
and
so
when
we
look
at
what
we're
doing
with
ludwig.
B
This
declare
division
possible
is
similar
to
kind
of
systems
that
provide
infrastructure
as
code,
I'm
sure
people
who,
in
the
kubernetes
community
community
are
very
familiar
with.
We
provide
yaml
configurations
that
declare
declaratively
define
models
that
you
might
wish
to
train
and
so,
for
example,
it's
very
easy
to
get
started
in
ludwig.
You
just
say:
here's
a
yaml
config
saying
what
my
input
features
and
their
types
are,
what
my
output
features
and
their
types
are,
and
then
everything
else
the
kind
of
how
will
get
filled
in
automatically
on
your
behalf.
B
But
at
the
same
time
we
provide
a
lot
of
expert
level
control
as
well.
So
if
you
say,
I
want
to
use
a
specific
type
of
mall
architecture
to
encode
a
particular
feature
if
you
want
to
use
a
particular
learning
rate
or
regularization
or
dropout,
all
those
options
are
available
to
you,
as
well
as
more
advanced
features
like
hyper
parameter,
search
on
any
of
the
different
parameters
within
the
config
and
what
makes
this
all
possible
is
the
ludwig
architecture.
B
So
every
input,
feature
and
output
feature
in
your
data
set
passes
through
an
architecture
we
call
ecd
for
encoder
compiler
decoder.
Every
feature
is
pre-processed
according
to
a
pre-processing
rules
that
you
can
configure
in
the
in
the
yaml
config
and
then
encoded
into
a
vector
which
can
be
a
machine
learning,
model,
pre-trained
or
otherwise
or
learned.
And
then
all
the
different
features
are
combined
into
an
embedding
space
and
then
individual
output
features.
B
Then
pass
through
a
very
similar
decoding
step
where
we
get
the
final
prediction,
and
the
benefit
of
this
architecture
is
that
it
provides
a
great
deal
of
task
flexibility
without
a
lot
of
additional
complexity.
So
if
you
want
to
do
a
regression
problem,
you
can
have
any
types
of
inputs
and
then
just
specify
a
numerical
output.
B
B
So
we
use
raytune,
which
sits
at
a
level
on
top
of
the
training
process
and
can
perturb
different
parts
of
the
config
and
every
one
of
those
config
variants
then
becomes
its
own
trial.
That
goes
through
the
same
training,
pre-processing
training
and
evaluation
step
as
any
other
training
process
in
mudwig.
And
then,
at
the
end
of
the
day,
you
can
get
all
the
different
model
trials
that
were
explored
and
choose
the
one
that
you
would
like
to
use
in
production
and
when
we
started
to
look
at
building
an
automl
layer.
B
B
The
important
thing
that
I
want
to
emphasize
here
is
that
there's
more
to
this
story
than
just
the
automotive
side,
because
there's,
when
you're
running
this
thing
in
production
or
kind
of
in
a
large
distributed
setting,
there's
also
a
component
of
how
you
want
to
do
this
process
efficiently
to
optimize
the
usage
of
these
commodity
resources
like
gpus,
so
that
you're
using
them
judiciously
and
not
wasting
resources.
And
so
this
is
where
a
little
fits
into
the
picture,
particularly
for
running
kubernetes
workloads.
B
A
Okay,
great
thanks
to
travis,
we'll
go
back
to
the
slide,
set
now
skip
forward
to
pick
it
up
where
he
left
off.
So
the
last
piece
of
the
puzzle
of
the
background
information
for
this
talk
is
the
nodeless
kubernetes
luna
functionality.
It's
a
smart
cluster
provisioner
that
runs
in
a
standard,
kubernetes
cluster.
A
It
monitors
the
cluster
for
pod
creation
requests
and
adds
additional
compute
to
the
kubernetes
cluster
to
satisfy
those
requests
that
compute
can
be
in
the
form
of
vms,
either
on
demand
or
spot
vms,
or
it
can
be
in
the
form
of
serverless
compute
like,
for
example,
aws
fargate,
and
it
chooses
that
compute
based
on
current
availability
in
the
cloud
we
already
heard
about.
Sometimes
you
can't
get
the
gpus,
you
want
the
cost
in
the
cloud
and
other
user
requirements.
A
In
fact,
the
user
may
have
specific
gpus
that
work
well
for
their
workloads
or
some
that
do
not
it's.
Luna
is
monitoring
your
cluster
and
if
node
usage
is
becoming
low,
it
can
remove
unneeded
nodes
from
your
kubernetes
cluster.
It's
comparable
to
the
kubernetes
cluster,
auto
scaler,
but
it
provides
more
flexible,
node
selection,
without
the
need
to
maintain
what
can
sometimes
be
hundreds
of
node
groups
to
handle
the
number
of
instance
types
available.
It's
somewhat
like
aws
carpenter,
but
it
works
across
cloud.
Vendors
provides
instance,
family
exclusions
and
provides
deterministic
rule
application.
A
So
now,
let's
go
into
we've
got
this
background
information.
Let's
go
into
first
efficiency
in
lidway.automl
for
tabular
data
sets
and
then
we'll
move
on
to
text.
Classification
data
sets
so
looking
at
the
tabular
data
sets
and
looking
at
what
automl
can
bring
you
versus
doing.
Untuned
search
the
automl
heuristics
for
tabular
data
sets
were
developed
by
analyzing.
Thousands
of
hours
of
model
training
across
12
data
sets
that
we
used
as
the
training
data
sets
for
the
heuristics.
A
A
Those
best
type
of
parameters
will
probably
do
a
pretty
good
job
on
data
set
b
if
it's
somewhat
similar,
and
so
we
can
front
load
the
hyper
parameter
search
with
those
settings
for
the
hyper
parameters
and
get
a
good
model
more
quickly,
and
the
fourth
thing
that
we
did
was
to
set
things
up
so
that
automl
uses
async
hyperband
when
it's
using
raytune,
async
hyperband
just
continues
on
promising
trials,
so
the
resources
aren't
wasted
on
them.
So
creating
this
sort
of
set
of
heuristics,
we
then
ran
the
heuristics
on
the
original.
A
A
You
know
at
a
fixed
amount
of
compute
alternative
one
is
that,
instead
of
doing
that,
deploy
the
ray
cluster
into
a
kubernetes
cluster,
that's
running
lona,
the
nodeless
kubernetes
system,
so
luna
will
be
handling
auto
scaling
when
the
ray
auto
scaler
asks
for
it.
So,
basically
the
again
the
ray
head
is
a
gpu
enabled
node.
A
The
ray
workers
are
spun
up
when
the
ray
auto
scaler
says
that
more
resources
are
needed
and
then
luna
sees
those
prods
pending
and
goes
out
to
the
you
know,
club
cloud
resources
and
adds
resources
to
the
kubernetes
cluster
and
then
removes
them
when
the
ray
auto
scaler
removes
them.
Alternative
two
is
just
like
alternative
one
with
one
difference,
which
is
that
the
head
node
is
a
cpu
only
node.
This
is
kind
of
nice
because
it
means
when
the
ray
cluster
is
idle,
then
you're
not
spending
money
on
a
gpu
node.
A
A
I
wanted
a
standardized
amount
of
compute
for
the
budget
time
budget,
so
if
I
run
automl
for
one
hour
two
hour
four
hours,
I
want
to
know
what
that
means
in
compute,
and
I
knew
that
three
t4
gpu
based
machines
would
give
a
standard
amount
which
would
do
a
pretty
good
job
with
those
time
budgets
for
tabular
data
sets-
and
you
know
the
overall
automl
runs
10
trials.
They
would
give
a
reasonable
result
on
10
trials.
A
So
that
was
one
thing.
Is
I
wanted
standard
amount
of
compute?
The
second
thing
is,
I
wanted
reduced
operational
complexity.
I
didn't
want
to
worry
about
whether
I
got
a
legitimate
run
or
not.
So
that
was
a
big
concern
from
my
time
and
the
third
thing
was,
I
wanted
to
limit
idle
cost,
so
I
didn't
run.
You
know
these
jobs
in
parallel.
A
I
didn't
run
the
three
one
hour,
three
two
hour
and
three
four
hour
in
parallel,
because
I
wanted
to
limit
the
idle
cost,
so
I
was
sensitive
to
idle
cost
because
after
the
job
is
over,
I
would
log
in
make
sure
I
really
you
know
liked
the
results
in
terms
of
that
they
were
credible,
that
they
had
run
properly
and
so
on.
So
I
wasn't
going
to
you
know,
spin
down
the
cluster,
the
minute
that
the
job
completed
and
so
for
all
these
reasons
I
had
you
know
this
fixed
size.
A
A
So
of
course
the
baseline
of
these
three
models
running.
You
know
one
three
data
sets
running
one
hour
two
hour
and
four
hour.
You
know
produced
competitive
models
to
manually
tune
models.
The
elapsed
time
was
22.6
hours
and
you
might
say
well
why
wasn't
it
21
hours
3
times,
1
plus
3
times
2
plus
3
times
4.?
A
The
extra
1.6
hours
was
to
load
up
the
data,
sets
and
pre-process
them
and
also
run
the
evaluation
of
the
best
model
from
each
trial,
which
is
done
on
the
head
node
after
the
hyper
parameter
search
is
completed,
the
cost
for
running
this.
You
know
g4d
and
4x.
Larges
are
1.204
dollars
an
hour,
so
it
cost
me.
Eighty
one
dollars
point
six,
three
one
cents
to
run
this
job
and
the
idle
cost
was
three
dollars
point
six
one
two
per
hour.
A
A
So
that
was
really
the
key
to
being
able
to
move
to
a
more
flexible
system
with
auto
scaling.
Now
you
might
say
well,
I
can
see
how
auto
scaling
is
going
to
save
time
in
the
idle
time.
I
can
see
how
running
in
parallel
is
going
to
reduce
the
elapsed
time.
But
are
you
really
going
to
save
anything
while
the
workload
is
running
well,
you're,
going
to
save
at
least
the
workers?
A
So
you've
got
time
on
the
x-axis
and
you've
got
the
data
sets
and
their
trials
on
the
y-axis.
You
know
the
one
hour
one
hour
one
hour
for
the
first
three
data
sets
then
two
hours
for
each
and
then
so
on,
and
so
you
can
see
that
that
first
data
set
only
really
two
trials
survived
to
the
end
of
the
time.
So
you
didn't
need
three
workers.
While
that
trial
was
running,
you
just
really
mostly
needed
two
same
the
second
data
set.
A
You
only
needed
one,
the
third
one
you
needed
more,
but
that's
what
auto
scaling
is
all
about.
So
there's
a
lot
of
opportunity
here
to
spin
down
workers,
even
during
the
run,
so
that
moves
us
to
the
gpu
head
rake
configuration
with
luna
running
on
kubernetes,
so
deployed
in
an
eks
cluster
with
luna
there
to
bring
things
in
when
they
ray
out
a
scalar
asks
for
them,
and
this
time
we
can
run
the
three
workloads.
A
So
again,
we
didn't
have
any
compromise
on
model
accuracy.
It
was
just
like
the
baseline,
but
the
elapsed.
Time
was
much
shorter,
8.75
hours
instead
of
22.6,
so
a
reduction
of
61
percent,
the
idle
cost,
of
course
down
by
66,
because
only
the
head
is
running,
not
the
ed
workers
and
the
workload
cost
was
54
lower
than
the
baseline
just
to
break
down
that
54.
A
More
than
half
of
it
was
coming
from
the
auto
scaling
we
just
talked
about
during
the
the
run,
but
then
some
percentage
of
it
was
talk
was
obtained
because
luna
was
able
to
get
cheaper
instances
that
I
was
able
to.
You
know,
mess
around
and
try
to
get
when
I
would
bring
things
up
annually.
So
luna
put
the
head
node
on
the
4x
large,
but
it
put
the
workers
on
2x
large
based
on
the
resource
requirements
so
now
moving
to
the
cpu
only
head.
A
A
The
elapsed
time
was
a
little
bit
longer
because
the
cpu
head
wasn't
as
efficient
and
running
the
evaluation
at
the
end
of
each
you
know
of
a
set
of
trials
as
a
gpu
node
on
the
head,
but
not
too
much
of
a
compromise,
an
extra
15
minutes.
The
idle
cost
was
significantly
lower
than
the
gpu-based
machine,
so
0.45
2
and
the
workload
cost
was
essentially
the
same.
A
So
that's
the
story.
Around
tabular
data
sets
look
at.
Let's
look
at
text
classification
data
sets.
Can
you
get
an
advantage
running
automl
with
text
classification
data
sets
versus
your
own
untuned
search?
Well,
I
actually
thought
this
wouldn't
be
interesting.
I
mean
I
was
just
naive
or
something
but
I'm
like.
Oh
you
know
I
can
read
a
bunch
of
papers.
I
can
just
use
bird
base
bert
base.
Modeling
is
is
widely
applied
to
text
classification.
A
You
know
here's
the
batch
size
ranges.
I
should
use
here's.
The
learning
rate
ranges.
I
should
use
here's
the
optimizer.
I
should
use
I'll.
Just
do
that
and
it'll
just
work,
I
mean
there's
automl
won't
really
be
doing
anything
other
than
stamping
out
what
people
in
the
literature
tell
you
to
do,
and
so
I
had
10
data
sets.
I
was
going
to
use
to
create
these
heuristics
and
I
ran
this
set
of
parameters
to
to
train
those
models
to
see
what
would
happen.
Well.
A
Three
of
them
got
great
results
within
in
one
hour
and
the
other
seven
crashed
and
they
crashed
because
you
know
bert
bass
is
a
transformer
model.
It's
quadratic
in
the
input
text
token
length
and
I
was
trying
to
run
on
commodity.
You
know
t4
gpus
and
they
wouldn't
fit
in
memory,
and
so
spen
actually
spent
the
next
two
months,
injecting
a
whole
bunch
of
heuristics
into
automl,
so
that
they
could
be
trained
on
this
kind
of
commodity
gpu.
So
there
really
is
a
value
to
automl
even
for
pre-trained
models
that
you're
fine-tuning.
A
So
this
included
adding
a
tune
for
memory
option
to
automl
that
says:
hey,
you
know,
change
the
hyperparameter
search
strategy
and
other
things
if
the
model
is
projected
not
to
fit
in
memory
during
training,
also
a
lot
of
stuff
there,
including
the
fact
that
limiting
the
input
token
length
is
kind
of
a
good
trade-off.
First
before
you
reduce
batch
size
and
so
on.
So
a
bunch
of
stuff
here
I'll
highlight
that
particular
choice
also
limit
the
number
of
trials
needed.
You
remember
in
tabular
data
sets.
We
were
running
trend
10
trials
by
default.
A
We
run
here
but
five
here,
but
if
we
do
look
at
the
combinatorics
and
there's
less
than
five
possible
combinations,
we
reduce
the
number
of
trials
further
setting
the
max
epoch
rate
max
epoch
count
to
six,
because
these
pre-trained
models
when
you're
fine-tuning
them.
Once
you
get
past
about
six
you're
moving
into
the
range
of
catastrophic,
forgetting
it's
not
interesting,
and
so
on.
So
with
these
heuristics,
the
original
10
data
sets.
A
So
the
training
set
worked
well
and
the
other
thing
we
did
here
is
because
it's
combinatoric,
we
were
sort
of
we
sort
of
needed
to
move
into
regimen
where
you
had
a
particular
time
a
budget
set
uniquely
per
data
set.
So
we
couldn't
stick
with
the
kind
of
fixed
time
budget.
So
anyway,
that
worked
well
for
the
10.
We
then
tried
it
on
the
additional
five
data
sets
that
we
hadn't
seen
before
when
we
formed
these
heuristics
and
got
accuracy
competitive
with
publicly
reported
models.
A
So
again,
I
did
the
same
experiment
that
I
did
for
tabular
data
sets,
which
was
choose
three
of
the
data,
sets
from
the
validation
and
see
how
they
would
have
worked
on.
My
baseline
configuration
versus
nodeless
kubernetes
luna
running
on
a
kubernetes
cluster,
and
so
doing
that
I
these
are
the
three
data
sets
I
chose.
A
I
was
also
able
to
save
a
bunch
of
resources
during
runtime,
but
it's
kind
of
interesting
where
those
resource
savings
came
from.
Remember
in
tabular
state
of
data
sets.
It
came
from
the
idea
that
async
hyperband
was
discontinuing
certain
parts
of
the
run
here.
It
was
coming
from
the
fact
that
we
didn't
need
as
many
trials
all
three
of
these
data
sets
because
they
used
tuned
for
memory
and
because
it
reduced
the
hyper
parameter.
A
So
if
you
look,
this
is
a
picture
of
the
runtime
for
these
three
data
sets
being
validated
so
again,
a
big
savings
and
compute
cost
and
idle
cluster
costs
as
the
gpu
head,
not
so
much
savings
and
elapsed
time
this
time
because
elapsed
time
was
dominated
by
that
one
big
data
set
so
anyway.
This
is
the
sort
of
in
this
talk.
A
We've
talked
about
the
savings
you
can
get
in
your
own
efficiency
by
using
automl,
rather
than
your
untuned
search,
and
then
we've
talked
about
how,
once
you
are
using
automl,
there's
a
lot
of
leverage
in
using
automl
on
top
of
a
kubernetes
cluster
with
nodeless
luna
technology.
So
in
the
future,
we're
intending
to
continue
to
extend
ludwig,
auto
and
melting
new
domains
and
we're
continuing
to
enhance
luna
as
needed
to
handle
efficient
scaling
of
all
sorts
of
workloads,
including
machine
learning
across
public
cloud
vendors.
C
C
So
you
were
talking
about
the
idle
cost
savings,
but
you
seem
to
be
assuming
that
you
had
basically
a
single
user
on
the
kubernetes
cluster.
It
seems
like
you
could
also
get
a
lot
of
idle
cost
savings
if
you
were
able
to
segment
things
by
namespace
and
have
multiple
users
share
the
same
cluster.
Oh.
A
Yeah,
that's
a
really
good
point.
Yeah.
I
think
I
was.
This
is
kind
of
a
selfish
view
of
the
world.
I
think
I
was
sort
of
spinning
up
the
kubernetes
cluster
just
for
myself
and
then
deploying
ray
there,
but
I
think
it's
a
very
good
point.
Yeah.
I
think
that's
a
that's
something
we
should
think
about
more
yep.