►
From YouTube: Human-Friendly, Production-Ready Data Science Stack With Metaflow &... - Savin Goyal & Saravanan
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Human-Friendly, Production-Ready Data Science Stack With Metaflow & Kubernetes - Savin Goyal, Outerbounds & Saravanan Balasubramanian, Intuit
Speakers: Saravanan Balasubramanian, Savin Goyal
There is a pressing need for tools and workflows that meet data scientists where they are. This is also a serious business need: How to enable an organization of data scientists, who are not software engineers by training, to build and deploy end-to-end machine learning workflows and applications independently.
In this talk, we discuss the problem space and the approach we took to solving it with Metaflow, the open-source framework we developed at Netflix, which now powers hundreds of business-critical ML projects at Netflix and other companies from bioinformatics and drones to real estate. We wanted to provide the best possible user experience for data scientists, allowing them to focus on parts they like (modeling using their favorite off-the-shelf libraries) while providing robust built-in solutions for the foundational infrastructure: data, compute, orchestration, and versioning.
In this talk, we will demo our latest work that builds on top of Kubernetes.
You will learn about - What to expect from a modern ML infrastructure stack. - Using tools such as Metaflow & Kubernetes to boost the productivity of your data science organization, based on lessons learned from Netflix and many other companies.
A
B
My
name
is
seven
and
it
in
today's
talk,
I'm
going
to
talk
about
my
experience,
building
data
science
infrastructure
for
well
over
a
decade
now
and
some
of
the
work
that
we
have
done
very
recently
within
the
kubernetes
community.
As
part
of
this
talk,
we
also
have
a
quick
demo
that
my
collaborator
Bala
from
Intuit
will
be
handling
if
you
guys
are
interested.
You
can
also
follow
along
this
demo
within
your
own
browser
as
well,
we'll
be
sharing
a
handy
link
to
access
the
demo
too.
B
Maybe
your
data
is
stored
in
some
snowflake
data
warehouse.
Some
data
breaks
data
Lake-
maybe
you
have
sort
of
like
you
know,
stitched
together
your
own
data
platform
on
top
of
S3,
but
there's
usually
sort
of
like
some
mechanism,
that's
needed
for
this
data
scientist
to
now
sort
of
like
understand
what
data
is
available.
What
is
the
quality
of
that
data?
How
to
access
that
data
so
that
all
the
security
and
data
governance
concerns
are
addressed
and
how
to
quickly
and
efficiently
access
that
data?
B
B
You
know
it
could
very
well
be
the
case
that
the
kind
of
data
analysis
that
you
want
to
do,
or
the
kind
of
models
that
you
want
to
train-
that's
not
possible
on
your
local
workstation,
not
possible,
maybe
perhaps
on
the
notebook
instance
that
you
have
in
the
cloud,
maybe
you're
doing
some
feature
engineering
step.
That
requires
huge
amount
of
memory.
That's
just
not
available
on
your
laptop.
Maybe
you
are
training
a
model
that
requires
GPU.
B
Maybe
you
decided
that
maybe
you
want
to
train
a
model
for
every
single
country
that
your
service
operates
in,
so
you
certainly
need
to
train
200,
odd
models,
and
if
you
have
hyper
parameters
as
well,
that
number
can
sort
of
like
grow
rather
quickly
and
then
there's
sort
of
like
you
know,
comes
this
question
that
okay,
how
do
we
handle
this
large-scale
compute?
It's
likely
you
know,
given
that
we
are
at
kubecon
as
well,
that
your
organization
may
have
already
invested
in
some
kubernetes
clusters,
and
then
the
first
question
comes.
Is
that
okay?
B
How
do
you
give
access
to
this
data
scientist
to
these
kubernetes
clusters?
Now,
of
course,
you
know
like
this
data
scientist
very
technically
sophisticated,
but
in
data
science,
not
necessarily
in
software
engineering
now,
is
there
an
expectation
that
they'll
sort
of
like
suddenly
become
a
wizard
in
the
concepts
of
kubernetes
and
understand
how
to
sort
of
like
launch
these
large-scale
workloads
really
efficiently?
B
Is
that
really
sort
of
like
you
know
what
they
should
be
doing
day
in
and
day
out
now,
let's
assume
somehow
magically
they
are
able
to
sort
of
like
you
know,
breach
that,
then
you
know.
Now
we
have
this
data
scientist
who
is
able
to
sort
of
like
access
this
data
not
only
locally
but
also
on
the
kubernetes
cluster.
They
have
somehow
been
able
to
figure
out
a
story
around
like
how
to
ship
their
code
to
the
cluster,
how
to
ship
their
dependencies,
Library
dependencies
data,
dependencies
and
whatnot
to
that
cluster.
B
If
there
are
some
errors
that
have
crept
up,
if
jobs
are
failing
for
whatever
reason,
maybe
perhaps
they
have
some
mechanism,
some
solution
to
sort
of,
like
you
know,
figure
that
out
as
well.
The
next
question
comes
is
how
do
we
sort
of
like
productionize
this
work
right
up
until
now,
the
data
scientist
is
still
sort
of
like
actively
iterating
through
a
bunch
of
hypothesis,
so
they
need
to
sort
of.
Like
now
run
this
model
training
workflow
on
top
of
some
kind
of
workflow
orchestrator.
B
The
next
big
barrier
is
the
expectation
that
now
somebody
is
going
to
sort
of
like
take
all
of
the
work
that
the
data
scientist
has
done
and
rewrite
it
or
re-implement
it
in
the
SDK
that
many
of
these
workflow
orchestrators
expose
and
then
once
you
sort
of
like
you
know,
have
that
the
story
doesn't
really
sort
of
like
stop.
There
you'll
have
many
many
different
versions
of
these
workflows.
People
will
or
your
data
scientists
will
want
to
try
with
different
algorithms
different
features,
different
hyper
parameters.
B
Maybe
they
are
sort
of
like
new
data
scientists
around
the
team
who
want
to
sort
of
like
try
out
different
set
of
experiments,
so
many
versions
that
are
running
either
concurrently
or
over
a
longer
period
of
time,
and
you
need
to
sort
of,
like
you
know,
really
make
sense
of
like
what's
really
happening
around
that.
There's
also
sort
of
like
this
big
question
in
machine
learning
about
trust.
B
Can
you
actually
sort
of
like
replicate
the
results
that
you
have
seen
in
the
past,
because
if
you
can't
do
that,
then
it's
really
difficult
to
trust
the
model
that
has
been
shipped.
So
this
question
of
reproducibility
also
becomes
really
important.
It
is
closely
coupled
with
the
story
of
versioning
as
well
that
it's
not
only
just
important
to
sort
of
like
keep
track
of
what
has
really
happened,
but
it's
also
really
important
sort
of
like
ensure
that
any
past
State
can
be
reliably
sort
of
like
built
up
again
now.
B
You
know
we
have,
let's
say
a
bunch
of
different
models
that
are
being
trained.
The
next
big
step
comes.
How
do
we
sort
of
like
deploy
this
model
now,
depending
on
your
business
context?
Model
deployment
could
mean
many
different
things
you
may
want
to
sort
of
like
deploy
the
model
within
a
cell
phone
device,
or
maybe
it's
a
microservice
that
has
a
really
high
expectations
on
SLA.
You
know
if,
let's
say
it's
Netflix's
recommendation
systems,
then
the
kind
of
slas
that
are
established
around
that
microservice
they
are
like
really
stringent.
B
On
the
other
end
of
the
spectrum,
you
may
have
this
model
servicing
an
internal
dashboard
where
the
QPS
may
not
be
as
high.
If
the
service
goes
down
for
a
few
minutes
or
even
a
few
hours,
nothing
will
sort
of
like
break,
so
that
could
be
an
entirely
different
pattern
as
well.
Another
common
scenario
is
where
people
sort
of
like
just
generate
a
bunch
of
scores
offline.
They
do
some
analysis
write
a
memo
share
it
with
the
execs
that
can
sort
of
like
influence
a
business
decision.
B
So,
on
the
deployment
side
of
the
house,
there
are
many
many
different
ways
to
deploy
any
sort
of
like
given
machine
learning
model
that
was
like
can
be
sort
of
like
quite
specific
to
the
business
case
at
hand,
and
then
you
know
only
at
the
top
of
the
stack
we
sort
of
like
now
start
to
worry
about
the
actual
modeling.
What
is
the
kind
of
data
that
we
are
dealing
with?
Is
it
tabular
data
or
is
it
sort
of
like
you
know,
unstructured
data?
B
What
kind
of
algorithms
do
we
need
to
use
and
what
not?
And
if
you
look
at
it
any
machine
learning
project
within
any
organization
independent
of
the
state
of
its
maturity,
has
to
sort
of
like
deal
with
these
tax
right,
like
you
need
to
sort
of
like
understand.
How
are
you
accessing
your
data?
How
that
data
is
stored?
You
need
to
understand
where
that
compute
is
happening,
even
if
it's
happening
locally
on
your
laptop.
What
is
the
story
around
workflow
orchestration?
Have
you
already
invested
in
a
workflow
orchestrator?
B
Do
you
want
to
sort
of
like
make
a
new
investment?
How
are
you
thinking
about
sort
of
like
you
know
many
of
these
concerns
that
these
days
are
lumped
into
sort
of
like
this
word
ml
Ops,
around
sort
of
like
versioning,
reproducibility,
experimentation,
management
and
so
on
so
forth,
and
then,
of
course,
you
know
like
how
do
you
actually
make
use
of
that
model?
How
do
you
make
sense
of
that?
B
How
do
you
sort
of
like
rinse
and
repeat
and
I
trade
on
that
entire
machine
learning
process
and
then,
at
the
top
end
of
the
stack?
The
concerns
are
all
like:
okay,
what
is
the
actual
machine
learning
problem?
How
are
you
taking
that
business
problem
and
going
all
the
way
and
translating
that
into
a
mathematical
model?
B
And
this
this
was
basically
sort
of
like
my
team's
Charter
at
Netflix,
where
sort
of
like
you
know,
many
of
these
learnings
came
from,
and
our
goal
was
that
you
know
we
had
sort
of
like
a
really
large
data
science
team
that
numbered
upwards
of
300
and
our
goal
was
that,
okay,
how
do
we
sort
of
like
make
these
talented
folks
productive
so
that
they're
not
sort
of
like
spending
a
bulk
of
their
time?
Fighting
with
infrastructure
I
mean,
of
course,
you
know
like.
B
If
they
wanted
to
be
experts
in
kubernetes,
then
they
would
have
chosen
to
be
Engineers,
but
then
they
really
sort
of
like
like
the
craft
of
data
science,
and
we
need
to
sort
of
like
meet
them
where
they
are,
and
one
of
the
biggest
hypotheses
that
we
had
was
that
the
problem
that
we
were
sort
of,
like
you
know
staring
at,
was
the
subproductivity.
If
you
look
at
any
organization,
you
know
they
have
an
immense
desire
to
invest
in
machine
learning.
There's
a
lot
of
data.
B
Every
single
day
we
are
sort
of,
like
you
know,
seeing
sort
of
like
the
specs
that
sort
of
like
public
Cloud
can
offer
sort
of,
like
you
know,
increasing
the
advancements
on
the
machine
learning
side
as
well.
You
know
newer
kinds
of
models
that
are
sort
of
like
coming
up,
but
then
the
big
problem
is
that
these
sort
of
like
two
curves
they
haven't
intersected
yet
and
the
reason
is
not
really
lack
of
Desire.
B
But
it's
sort
of
like
this
lack
of
productivity,
and
that
needs
to
be
bridged
by
any
kind
of
ml
Ops
tool
that
makes
us
out
there
in
the
market
and
then
the
other
sort
of
like
learning
here
was
if
we
sort
of
like
look
at
the
stack
and
look
at
sort
of
like
the
order
of
concerns
from
a
software
engineering
point
of
view.
From
a
builder
point
of
view.
From
our
point
of
view,
it's
it's
bottom
heavy.
B
We
need
to
sort
of
like
make
a
lot
of
Investments
on
the
data
side
of
the
house
on
the
compute
stack
on
the
orchestration
stack.
But
then,
if
you
look
at
it
from
the
top
most
stack,
that's
where
the
data
scientist
really
wants
to
exercise
a
strong
degree
of
autonomy.
They
really
do
care
what
kind
of
training
framework
they
are
using,
how
they
want
to
sort
of
like
build
the
model,
what
kind
of
features
they
want
to
use
most
data
scientists.
B
They
don't
really
have
explicit
opinions
around
how
the
data
is
stored,
whether
they
are
using
kubernetes
for
large
scale
compute,
what
kind
of
workflow
orchestrator
they
are
using
as
long
as
those
concerns
are
well
satisfied
and
that's
where
we
started
this
project
called
metaflow
at
Netflix.
We
which
we
open
sourced
in
December
2019.
B
at
AWS,
reinvent
if
you're
interested
in
metaflow.
Very
recently,
we
also
launched
this
sandbox,
where
you
can
sort
of
like
without
having
to
install
anything.
You
can
go
to
this
URL
and
you
can
sort
of
like
play
with
better
flow,
and
the
entire
infrastructure
is
backed
by
kubernetes
cluster
and
harbor
workflow
instance.
So
you
can
sort
of
like
try
out
all
the
promises
that
we
make
in
our
documentation.
B
Now,
very
simply,
what
metaflow
enables
is
a
programming
model
that
keeps
dags
front
right
and
Center,
so
defining
a
dag
is
very
simple
and
straightforward,
so
you
can
use
some
python
decorators,
so
in
this
case
the
step
decorator
to
denote
nodes
in
and
then
the
self.next
invocations
to
denote
the
edges
in
your
graph.
And
then
you
can
execute
this
as
you
would
execute
any
other
python
file
and
then
metaflow
will
go
ahead
and
execute
this,
and
you
can
sort
of
like
have
a
bunch
of
different
patterns,
including
sort
of
like
branches
for
reaches
and
whatnot.
B
There
are
a
lot
of
Primitives
that
metaflow
comes
with
so
of
course,
you're
like
on
the
data
side
of
the
house,
it
ships
with
a
high
throughput,
S3
client
that
provides
you
higher
throughput
than
what's
possible,
with
a
local
disk
attached
to
your
instance.
So
you
can
get
like
upwards
of
25
gbps
of
throughput
very
easily
using
metaflow,
and
then
this
also
sort
of
like
ties
into
your
internal
data
layers
as
well.
B
As
you
know,
whatever
Primitives
you
have
for
data
catalog,
so
that
data
Discovery
becomes
rather
simple
and
straightforward,
and
then
everything
gets
versioned
by
default.
So
every
single
time
you
execute
any
metaflow
flow.
A
unique
run.
Id
is
generated
the
entire
internal
status
versioned
in
perpetuity
for
you,
so
that
I
can
go
back
and
I
can
ask
that
hey
Bala
executed
this
workflow
six
months
ago
can
I
see
sort
of
like
what
was
the
internal
State.
What
was
the
model
that
was
generated?
What
were
the
internal
attributes
that
were
generated?
B
Now
we
are
at
cubecon,
so
we
definitely
need
to
talk
about
kubernetes
and
that's
where
it
sort
of
like
comes
in
now.
Imagine
you
know,
like
you,
you're
a
data
scientist
and
you
want
to
sort
of,
like,
let's
say,
train
a
model
over
a
list
of
hyper
parameters
that
is
sort
of
like
100
long
with
meta
flow.
B
You
can
sort
of
like
Drop
in
this
decorator
at
resources,
and
let's
say
you
know
in
this
example:
I
specified
memory
as
128
gigs
could
very
easily
also
specify
GPU,
CPUs
and
whatnot,
and
what
metaflow
will
do
is
it
will
execute
your
start
step
locally
on
your
laptop.
It
will
execute
the
train
step,
100
instances
of
that
on
kubernetes,
so
it'll
generate
100,
pods
and
those
steps,
and
it
will
also
pipe
all
the
logs
back
to
your
local
workstation
so
to
the
end
user.
B
It
would
seem
that
you
know,
like
somebody
came
in
and
just
added
a
whole
bunch
of
ram
to
your
laptop
and
went
to
winter
and
that
sort
of
like
makes
it
quite
trivial
for
people
to
sort
of
like
scale
out
their
workflows.
You
know
you
know,
and
then
of
course
you
know
like
once.
Let's
say
the
user
is
Happy
with
what
they
have,
then
it
could
be
the
case
that
you
know
they
want
to
sort
of
like
integrate
their
work
with
their
favorite
workflow
orchestrator.
That
could
be
airflow
that
could
be
Argo
workflows.
B
Step
functions
like
a
whole
bunch
of
options
that
are
available
out
there.
Now
the
good
thing
is
that
all
of
these
workflow
orchestrators
they
execute
a
dag
with
already
written
a
dags.
So
we
can
sort
of
like
compile
down
this
workflow
into
something
that
these
workflow
orchestrators
can
easily
understand,
and
at
that
point
you
basically
get
the
best
of
both
worlds.
So
this
workflow
would
be
sort
of
like
a
native
Argo
workflow
template
as
well
as
a
native
metaflow
flow.
You
can
imagine
let's
say
you
know,
for
whatever
reason.
Let's
say
this
work
here.
B
B
Oops
there
are
a
bunch
of
other
functionalities
as
well
that
metaflow
provides
that
are
sort
of,
like
you
know,
more
geared
towards
larger
scale
teams
like
functionalities
around
like
how
do
you
generate
visual
reports
as
part
of
every
execution?
How
do
you
sort
of
like
manage
larger
projects?
How
do
you
sort
of,
like
you
know,
think
about
dependency
management
and
so
on
so
forth?
B
Given
that
we
are
short
on
time,
and
we
want
to
sort
of
like
give
a
quick
demo,
so
I'll
sort
of
like
skip
on
that
now,
the
one
sort
of
like
message
that
I
want
to
sort
of
like
leave
you
with
before
sort
of
like
we
start
off
the
demo,
is
that
at
the
end
of
the
day,
for
any
organization
to
be
successful
and
sort
of
like
introducing
any
kind
of
intelligence
into
their
applications,
they
need
infrastructure
that
can
sort
of
like
go.
End-To-End
provide
solutions
to
the
entire
life
cycle
of
machine
learning.
B
That
was
basically
our
agenda
at
Netflix
as
well,
which
is
basically
what
we
have
tried
to
build
with
meta
flow.
So
if
projects
of
this
nature
sort
of
like
interest,
you
excite,
you
I
highly
recommend
going
to
this
URL
and
giving
sort
of
like
metaflow
a
try,
but
now
I'll
sort
of
like
pass
it
on
to
Bala
for
a
quick
demo.
C
Hey
everyone
I'm
going
to
demo
the
metaflow
and
Argo
Flow
side,
so
mainly
I'm,
going
to
focus
on
the
three
areas.
One
is
like
a
very
simple
metaphor:
application
how
you
can
develop
and
run
it
under
the
second
one
like
how
you
can
run
your
meta
flow
application
at
scale
using
a
kubernetes,
the
third
one
like
how
you
can
debug
back
your
production
issue
on
your
local,
so
I'm,
using
that
the
same
sandbox,
the
metaphor
of
playground.
C
So
if
you
see
that
it's
a
hollow
flow,
I
have
like
a
start
function,
which
is
which
is
calling
the
heat
function
next
and
end
function.
It's
only.
It
has
like
a
three
functions
if
I
click
the
Run
current
flow,
which
will
run
on
the
local
as
a
python
application,
and
you
can
see
the
nice
outputs
are
coming
like
all
the
logs,
so
starting
eating
and
ending,
and
this
sandbox
is
already
having
like
a
cord
decorator,
so
that
you
can
see
the
TC
output
of
your
flow.
C
And
one
more
thing
like
every
every
run,
the
meta
flow
will
give
like
a
unique
run
ID
for
every
flow,
so
that
in
future
you
can,
if
you
want
to
refer
it
back
or
you
want
to
resume
it
back,
you
can
use
the
serenity
for
The
Meta
flow.
So
you
can
see
the
nice
dog
is
printed
in
the
right
hand,
side.
So
the
given
the
time
I
am
going
to
skip
like
a
3
and
4,
which
is
like
a
artifact
and
versioning.
C
The
second
one
is
like
a
parallel
person,
I'm
going
to
show
that
like
how
you
can
run
your
metaphor,
application
at
scale.
So
in
the
metaphor,
application
I
have
like
a
three
steps
like
a
process,
memory
hog
and
join
mainly
like
a
memory
hog
needed
like
a
more
memory
for
to
process
that,
but
I
cannot
run
it
on
my
local,
so
I
want
like
some
remote
system
like
a
kubernetes
or
Cloud
somewhere.
We
need
to
run
it
so
metaflow
will
support
D
for
like
a
kubernetes.
C
If
you
want
to
test
it,
and
here
before,
going
to
that,
I
am
I'm.
Having
like
the
array
of
data,
I
want
to
run
like
a
these
three
steps
for
every
element
in
that
array.
Like
a
if
you
have
like
a
list
of
hyper
parameters,
then
if
you
want
to
train
that
model
for
each
hyper
parameter,
you
can
have
like
a
for
each
one.
C
C
C
C
So
as
in
that
now
you
have
a
production,
application
ready
and
you
tested
with
the
kubernetes
job,
and
you
are
ready
to
push
it
to
the
production
so
to
pushing
to
the
production.
You
need
like
a
production
grade
like
workflow
orchestration
engine,
which
will
need
to
give
like
a
high
fall
tolerance
and
make
sure,
like
your
application,
is
running
properly.
So
in
that
one,
you
cannot
only
rely
on
like
a
kubernetes
job.
You
need
like
a.
C
Is
coming
user
doesn't
know
that
Argo
workflow,
that
Argo
workflow
is
like
a
production
grade
open
source
container
native
orchestration
engine,
which
will
run
like
a
high
parallel
jobs
on
the
kubernetes
so
to
convert
into
that
or
go
workflow.
The
data
scientist
no
need
to
learn
about
that.
All
the
EML
files
and
kubernetes
The
Meta
flow
will
automatically
convert
your
application
into
that
org
overflow
template
and
submit
it
for
you
to
to
that.
Just
you
can
give
like
cargo
workflow
create,
which
will
create
like
a
workflow
template
on
a
Argo
workflow.
C
C
It
may
take
like
a
little
bit
time
so
I,
given
the
time.
I
am
like
to
show
that
the
previous
run,
which
I
handled
that
so
you
can
see
that
the
algorithm
was
nicely
orchestrated.
You
are
all
the
steps
with
all
the
the
parameters
in
the
array.
It
will
create
a
like
a
five
parallel
tasks
and
it's
executed
like
a
process
and
memory
hog,
and
it's
joined
back
to
the
Joint
step.
C
If
you
see
that
all
your
resource
configured
in
the
Pod
suppose,
if
you
are
declaring
for
a
eight
eight
gig,
it
will
automatically
config
in
your
power
resource
quota,
and
if
you
see
the
nargo
workflow
UI,
you
can
see
that
lot
of
options
are
coming
like
in
the
running
UI.
So
you
can
see
that
you
can
suspend
it
the
Argo
workflow
when
it's
running
and
you
can
resume
it
back
where
it
stopped.
Then
another
thing
like
suppose.
If
you
want
to
completely
stop
it,
you
can
go
out
and
stop.
C
C
Whatever
your
steps
running
like
you
can
click
that
log,
then
it
will
print
like
all
your
logs.
Okay,
let's
go
back
to
things.
Okay.
So
now
now
you
have
like
application
ready
in
the
production,
but
you
don't
want
every
time
submit
manually.
You
want
to
schedule
it
and
run
it
so
so
that
is
The
Decorator
metaphors
providing
like
you,
can
have
like
a
schedule
which
will
basically
converting
to
that
Argo
or
Crone
or
flow.
The
arbor
flow
is
already
supporting
like
a
Chrono
flow.
C
So
when
you
have
like
this
decorator
and
calling
like
Argo
or
procreate
meta
flow
will
automatically
convert
your
application
into
that
Chrome
or
plane
or
the
workflow,
then
it
will
run
it
every
hour.
So
you
can
see
that
here
there
is
another
tab
called
Chrono
flow.
You
can
see
that
which
is
created
now
and
it
will
going
to
run
like
every
one
hour
in
this
case.
Suppose
a
I
want
to
run
your
application
on
demand.
I,
don't
want
to
wait
for
one
hour.
C
C
C
C
Equal
to
because
the
meta
flow
is
keep
tracking
like
all
the
your
run,
so
you
can
just
give
your
run
ID.
Then
it
will
run
it
on
your
local,
where
that
the
workflow
is
left
in
that
kubernetes
cluster,
so
you
can
see
that
it's
failed
in
the
local
also,
so
the
engineer
can
directly
go
and
fix
it
and
run
it
on
the
local.
With
the
given
data
to
make
sure
it
is
working
fine,
then
he
can
push
it
back,
that
the
fix
into
the
production
so
now
he's
going
to
make
sure
like
a.
C
C
C
B
B
Thank
you
what
it
provides.
You
is
a
solution
for
different
layers
of
the
stack
and,
of
course,
what
we
demoed
today
was
our
Integrations
with
kubernetes
and
that
sandbox
environment
by
the
way
was
running
on
AWS.
But
now
metaflow
supports
all
three
major
public
clouds,
as
well
as
on-prem
deployments
for
your
data
storage,
you
can
use
S3,
Azure
blob
store,
GCS
or
you
can
sort
of
electronic
video,
snowflake,
warehouses
or
databricks
Delta
Lake
for
compute.
Of
course,
kubernetes
is
an
option.
B
B
B
B
So,
basically,
the
step
that
required
8
gigs
of
RAM
that
ran
on
the
kubernetes
cluster
as
a
kubernetes
pod
I,
see
because
the
local
instance
that's
running
that
vs
code
instance
that
has
roughly
four
gigs
of
RAM.
So
that
will
fail
and
then
not
only.
We
need
sort
of
like
eight
gigs
of
RAM.
We
were
running
five
concurrent
steps,
so
we'll
need
eight
times
five.
Forty
gigs
of
RAM,
which
is
not
available
in
at
least
sort
of
like
a
normal
laptop
okay.
B
So
the
debugging
step
that
we
had
so
let's
say
you
know
you
had
a
workflow
that
failed
on
Argo
workflow.
So
you
could
resume
that
from
any
arbitrary
step
and
you
can
resume
while
it's
running
on
kubernetes
as
well.
So
so
the
resume
doesn't
have
to
happen
entirely
locally.
You
can
still
sort
of
like
happen
on
kubernetes.