►
From YouTube: Leveraging Argo Workflows and Argo Events for Cloud Agnostic Enter... Charles Adetiloye & Alex Lerma
Description
Leveraging Argo Workflows and Argo Events for Cloud Agnostic Enterprise Machine Learning Deployments and Model Management - Charles Adetiloye, MavenCode & Alex Lerma, MavenCode
The goal of every organization integrating Machine Learning into their product pipeline is to get it into production as fast as possible using standard software engineering best practices, however, in most cases, this is not often the reality. Even when the models are successfully deployed there is always a need for an efficient workflow to monitor for things like model drift, model overfitting, or constantly changing datasets. To solve this recurring problem in our MLOps practice and create a reusable pipeline and solution framework, we have explored and implemented best practices around Argo Workflows and Argo Events. In this talk, we will discuss how we have implemented a continuous deployment environment stack to containerize and deploy Argo workflows for data ingestion, transformation, and feature store curation. We will also talk about how we trigger model retraining and redeployment with the Argo workflow pipeline. This includes scheduled events, message queue updates, and external storage data changes. We will go through the challenges we encountered and lessons learned with recommended best practices for any MLOps team considering the Argo workflow approach.
A
My
name
is
Charles
and
Alexis
will
be
on
the
stage
right
here.
So
and
my
name
is
engineer
mavenkood.
We
do
a
lot
of
things
around
ml,
Ops,
deploying
application
on
kubernetes
and
we've
been
using
ago
a
lot
in
the
last
few
months
and
years
so
we're
from
Texas.
So
we
drill
all
things
around
prohibition,
insulable
ml
applications
on
cloud
diagnostic
infrastructure.
A
So
I'm
going
to
take
on
the
first
few
sections
of
the
slide,
what
it
takes
to
build:
Cloud,
agnostic
and
model
infrastructure,
so
the
goal
of
any
production
grade
ml
machine
learning
projects
is
to
build
a
statistical
model
using
a
well
curated
data
sets
and
the
main
artifacts
like
we
all
know,
is
the
data,
the
model
and
the
code.
So
the
data
will
keep
changing.
Your
code
will
be
changing
and
you
keep
producing
model
output
artifacts.
So
there's
a
typical
amount.
A
Workflow
looks
like
this:
where
you're
acquiring
the
data
you're
doing
data
prep,
you're
doing
feature
engineering,
then
you
run
through
multiple
iterations
of
your
model.
Training,
then
onto
your
server
model,
and
you
still
mounted
out
the
model
for
drift.
So
typically,
a
lot
of
Mr
projects,
stats
on
cloud
services
like
Google,
AWS
Azure,
or
we
have
some
customers
that
do
on-prem
as
well,
but
ml,
like
you
said,
is
very
difficult.
A
Even
if
you
go
with
the
managed
service
providers
like
on
on
Google
on
Amazon
or
Azure,
you
go
through
the
process
of
identifying
the
data.
You
need
connection
to
the
data
source,
preparing
the
data.
Then
you
go
through
multiple
iterations
of
training,
the
model
onto
Savage.
So
there's
a
lot
more
impact
to
this.
That
makes
it
really
difficult
so
and
one
thing
that
will
send
consultant
in
this
place.
A
A
lot
is
like:
if
you're
trying
to
build
an
ml
model,
you
want
to
make
sure
you
can
reproduce
the
entire
pipeline,
which
is
one
thing
we've
been
able
to
successfully
do
with
devops
process,
where
you
have
a
process
that
you
can
reproduce.
You
have
components
that
you
want
to
make
sure
that
you
have
components
that
you
can
reuse.
So
if
a
data,
scientist
or
ml
engineer
is
working
on
a
particular
components,
you
want
to
be
able
to
like
share
with
other
team
members
and
things
like
that.
A
Then
the
other
thing
is
manageability
being
able
to
manage
all
the
output
artifacts
for
audio
trades
and
things
like
that.
Then
automation
they
will
process
so
that
the
process
runs
by
itself
and
things
like
that.
So
to
get
started.
A
lot
of
companies
use,
managed
services
on
AWS
or
you
can
do
the
same
thing
with
azure.
Oh,
you
can
do
the
same
thing
with
Google
they're
all
managed
services
that
will
get
you
up
and
going
and
things
like
that.
A
But
in
reality,
what
you're
doing
whenever
you're
trying
to
build
any
ml
pipeline
is
the
number
one
thing
is
like
you
always
need
to
solve
the
data
you
ingest
it.
You
do
data
prep,
you
do
future
engineering
and
you
can
now
decide
to
run
on
any
of
these
major
Cloud
platforms.
At
least
there's
three
major
ones,
and
you
now
create
your
trainer
models.
You
do
model
scoring
to
see
if
the
model
is
good
enough
and
you
do
your
inferencing.
A
So
on
the
story
pack,
we
basically
think
about
it
as
a
feature
Store,
where
all
the
things
you
do
to
get
the
data
ready
for
the
ml
engineer,
the
data
scientists
we
curate
it
and
you
we
have
everything
in
a
feature,
store
and
everything
that
has
to
do
with
your
compute,
basically
running
your
pipelines
and
your
workflows.
We
basically
abstract
it
as
a
containerized
application
that
runs
on
kubernetes.
B
But
we
want
to
do
this
in
a
cloud
agnostic
way
such
that
our
customers
don't
need
to
worry
about
if
they
need
to
lift
and
shift
from
gcp
to
AWS
they're
able
to
do
so.
So
we
deploy
with
Argo
and
Argo
CD
to
give
them
flexibility
and
to
basically
bring
customers
up
to
speed.
A
lot
of
our
customers
are
first-time
kubernetes
users
and
we
like
to
be
able
to
introduce
them
in
a
way.
B
B
So
some
of
the
ways
that
we
did
this,
we
basically
allow
different
teams
to
use
Argo
workflows
to
do
whatever
they
need.
Some
teams
are
working
on
feature
extractions.
Some
teams
are
working
on
training
but
they're
all
using
the
same
common
platform,
which
is
Argo,
and
this
has
been
very
successful
for
us.
B
So
in
general,
like
we
need
a
data.
Lake
feature
store,
all
that
good
stuff
and
argo's
been
basically
our
go-to
for
building
these
workflows
and
we've
been
using
the
dags
for
some
of
the
more
complicated
pipelines
that
we
build,
and
it's
worked
very
well
for
us.
Some
customers
are
using
tensorflow.
Some
are
using
other
machine
learning
Frameworks,
but
doesn't
really
matter
as
long
as
you
can
build
an
audio
workflow
and
they
know
how
to
build
workflows.
It
works
for
them.
B
B
B
So
yeah
so,
basically
the
some
of
the
steps
that
we
use
to
do
this.
We
provision
on
the
the
cloud
offering
of
their
choice.
We
deployed
node
pools.
We
use
Argo
CD
application
sets
to
define
the
basically
the
base
of
the
cluster
and
the
serving
and
event
plot
components
of
the
cluster,
and
then
we
deploy
it.
B
B
And
here's
kind
of
what
our
Argos
CD
dashboard
looks
like
after
being
deployed.
We
have
three
application
sets.
In
this
example,
we
got
the
base
which
gives
all
the
necessary
infrastructure
to
kind
of
deploy.
Custom
resources
such
as
k-serv
or
Argo
events-
and
this
has
worked
very
well
for
us
and
application
set-
has
been
nice
moving
away
from
the
app
of
apps
pattern.
Application
set
has
been.
B
You
know
it's
it's
a
lot
better,
so
yeah
again,
here's
a
look
at
like
like
how
we
use
application
set
to
deploy
the
all
the
namespaces
needed.
We
keep
our
namespaces
in
one
General
application
so
that
any
any
of
the
apps
that
need
to
reuse
these
namespaces
are
easily
able
to
use
them
without
needing
to
worry
about
the
name.
Space
is
created
or
not.
A
Alex,
so
basically,
we
use
the
ago
City
to
provision
all
the
things
that
we
need
in
a
cluster
to
get
a
ml
process
started
whenever
we
have
a
new
project.
So
the
next
thing
we
do
is
a
data
preparation
of
making
sure
we're
curating
the
right
data
set
for
data
scientists
to
get
started
whenever
they
need
to
train
a
model
and
make
it
easy
for
them
to
do
it
in
a
very
consistent
and
repeatable
way.
A
You
just
you
get
all
your
data
from
your
Jupiter
notebook
and
things
like
that,
and
you
try
to
use
it
to
train
your
model,
but
the
challenge
with
that
is
like,
if
you're
trying
to
basically
do
everything
at
scale,
it's
going
to
work
because
you're
not
going
to
easily
transition
from
what
you're
doing
on
your
local
to
a
production
environment.
So
in
our
case
for
for
what
we
do
and
I
would
go
about
it,
we
basically
create
kind
of
like
a
data
Lake
where
you
can
land
all
your
raw
data
and
we
have
it.
A
I
go
workflow
pipeline
that
transforms
the
data
into
a
feature
store,
so
we
have
a
feature
store.
We
have
a
skimmer
with
all
the
feature,
vectors
and
all
the
attributes
that
you
need
to
train
your
model
and
the
I
go
see
the
basically
runs
on
that
on
on
the
kubernetes
infrastructure
and
because
we
try
to
be
Cloud
agnostic.
Any
solution
we
build
should
be
able
to
run
on
any
of
the
major
Cloud
providers.
So
from
there
the
data
scientists
can
now
go
in
and
start
pulling
all
the
data
they
need
and
start
training
their
models.
A
So
so,
just
to
zoom
in
on
the
data
Lake
path,
which
is
where
we
keep
all
the
raw
data.
So
every
time
we
inject
data,
we
have
new
Delta.
So
we
need
to
keep
ingesting
the
data.
It
could
be
streaming,
it
could
be
batch
and
once
we
ingest
the
data,
we
trigger
a
workflow
to
basically
convert
it
to
the
Future
store
requirements.
So
it
looks
for
the
batch
work
stream.
A
A
Now
we
have
a
pipeline
that
basically
converts
and
transform
this
into
future
groups,
and
the
future
groups
can
now
be
used
to
train
the
model
in
the
offline
mode
or
you
can
have
the
online
mode
where
we
create
a
key
value
pair,
that
maps
to
the
greater
digital
to
that
maps,
to
Greater
details
about
the
data
set
that
you
can
now
use
whenever
you're
serving
the
model.
So
basically,
in
all
these
places
we're
leveraging
the
ago
workflow.
A
So
all
the
way
from
Landing
to
dinner,
I
think
Alex
talked
about
it
earlier,
where
we
have
the
Argo
event
that
whenever
the
file
drops
we
trigger
workflow
so
but
we're
not
going
to
trigger
for
every
file
in
the
batch
process.
So
we
have
a
way
of
basically
having
a
trigger
file
that
whenever
that
trigger
for
lands,
we
just
trigger
a
batch
of
the
files
and
we
process
it
and
we
curate
the
future.
A
So
this
way
the
data
scientists
can
always
have
near
data,
they
can
always
go
back
and
retrain
their
model
and
if
we
notice
any
model,
Drift
We
have
a
pipeline
that
goes
back
and
we
train
the
model
and
basically
checks.
What's
going
on
so
we're
using
aggro
workflowing
a
lot
of
all
these
use
cases-
and
these
are
some
of
the
pipelines
that
we're
working
on
I
think
Alex
has
some
more
details
on
that.
B
Yes,
so
we
use
batch,
workflows,
streaming,
feature,
store,
creation
and
online
feature
store
and
offline
feature
store,
all
using
Argo
workflows
to
kind
of
build
out
these
components.
Here's
an
example
of
how
we
built
a
small
data
Lake.
We
basically
did
you
know
data
ingestion
like
once
the
message
is
fired
off.
We
consume
the
message.
We
do
schema
validation
on
it
make
sure
the
message
looks
like
the
correct
shape
and
then
we
write
it
to
the
data
store
same
example,
but
like
in
a
streaming
situation,
we
consume
a
message
from
Kafka.
B
A
Sure
so,
to
basically
operationalize
all
these
things,
we
need
to
react
to
events
and
the
event
could
be
things
like
code
change.
Whenever
you
come
in
your
code
to
the
git
repo
data
change,
whenever
there's
a
change
in
your
data
sets,
you
want
to
create
a
retraining
if
it's
necessary
and
if
you
notice
that
your
model
is
drifting.
Basically,
you
want
to
recruit
the
pipeline.
A
So
when
we
want
to
operationalize
our
workflow
in
production,
we're
basically
leveraging
our
goal
events
so
ago,
events
is
kind
of
like,
like
The
Listener,
for
us
to
automatically
trigger
all
these
workflows
based
on
code
change,
feature
updates
or
model
drift,
or
everyone
or
beta
Pipeline,
and
things
like
that.
We
also
triggered
this
events
based
on
kubernetes
resource
deployment,
basically
monitoring
the
states
of
a
resource,
especially
with
case
K7
Alex,
is
going
to
talk
about
that
in
a
little
bit.
A
So
a
typical
workflow
looks
like
this.
Where
you
have
the
data
engineer
or
ml
engineer
working
on
a
patch
class
stuff,
you
check
it
in
if
you're
using
Git,
it
triggers
a
good
action
and
we
build
a
container
and
we
push
it
to
the
container
registry.
Then
we
have
Argo
CD
check-in
to
see.
If
there's
a
new
container,
then
it
should
be
deployed.
If
you
need
to
return
the
pipeline,
then
it
basically
retrains
and
deployed
the
models
and
creates
a
version
type
for
it.
A
So
once
the
model
training
is
done,
we
basically
create
another
event.
We
have
a
metrics
table
where
we're
basically
storing
all
the
metrics
it
could
be
it
will.
It
could
be
the
accuration,
the
Precision
of
the
model
and
we
save
the
model
output
of
artifact
the
Frozen
model
in
the
container
registry.
A
So
once
the
model
drops
in
this
container
in
this
sorry
in
this
bucket,
we
can
trigger
another
workflow
as
well,
but
before
we
deploy
the
workflow,
the
pipeline
I
want
to
check
to
see
if
the
metrics,
if
there's
any
Improvement
in
the
mod,
also
but
I'll
go
event.
Workflow
processor
check
to
see
if
there's
any
significant
Improvement
in
the
model
that
we
trained
and
determine
if
we
need
to
redeploy
or
not
and
the
same
thing
with
the
feature
store.
Every
time
we
curate
a
new
data
set
to
the
feature
store.
A
A
So
some
of
the
workflows
that
we're
currently
working
on
we're
walking
on
as
a
feature
stock
at
the
future
stock
creation,
workflow
the
model,
training
status,
dutification
workflow,
the
metrics
and
the
model
drop
workflow
and
the
model
deployment
workflow,
so
I
think
Alex.
You
have
an
example.
You
want
to
show
yeah.
B
So
here's
an
example
of
the
bucket
drop
where
you
drop.
Basically,
this
allows
us
to
drop
files
into
a
cloud
storage
bucket.
After
a
certain
number
of
files
have
been
reached,
the
threshold
is
met
and
a
Argo
event
is
triggered
that
basically
spins
up
a
workflow
that
will
take
all
the
files
from
the
input
directory
and
move
them
over
to
the
training
location.
B
So
what
that
looks
like
and
make
sure
that
there's
the
correct
number
of
files
needed
to
move
to
train
and
if
not
it'll,
it'll
wait
till
another
event
is
fired
to
do
the
counting
we
have.
This
is
one
way
to
do
it
and
sometimes
we
use
a
file
trigger,
but
this
is
just
one
of
the
more
basic
ways
to
get
it
done
once
the
threshold
is
met.
The
the
train
is
the
training
data
set
is
created
and
we
archive
it.
We
zip
it
up
so
that
it
can
be
used
for
a
new
model
version.
B
B
We
do
this
with
the
Argo
workflow.
That
basically
looks
at
the
feature
store
sees
if
the
metrics
value
for
the
current
model
that
got
trained
meets
our
threshold.
If
it
is,
we
use
k-serv
to
deploy
a
new
version
case,
serve
uses
these
inference
Services,
you
basically
just
point
it
to
a
location
in
S3
or
GCS
bucket
with
the
tensorflow
serving
type
of
model,
and
we
have
this
dag
that
we
created
where
we
do
the
validation
looking
into
the
metric
store.
B
If
the
metrics
meet
our
threshold,
we'll
deploy
the
new
model
and
we'll
send
a
notification
on
slack
saying
that
a
new
model
is
being
deployed
for
a
little
toy
example,
we
did.
We
used
the
cars
data
set
to
basically
just
run
a
case,
serve
and
deploy
a
model
that
can
determine
if
the
given
cars
of
a
certain
type.
B
B
A
Yeah
trucks,
so
in
conclusion,
leveraging
ago
for
ML
projects,
we've
been
able
to
improve
our
team
delivery
efficiently.
So
it's
easy
to
ramp
up
new
team
members
because
everybody
is
working
on
the
same
environment
once
you
bootstrap
the
resource,
with
either
CD
on
your
local.
If
you're
running,
mini
Cube
or
if
you're
running
on
the
kubernetes
cluster
in
the
cloud
you
can
get
going,
we
have
a
blueprint
and
a
template.
A
So
if
we
want
to
create
a
new
workflow,
we're
not
starting
from
the
scratch
we're
just
leveraging
the
existing
template
that
we
have
and
let's
talk,
I
mean
cloud
is
not
cheap,
so
we
are
set
up
we're
able
to
reduce
the
cost
for
our
customers
in
terms
of
the
amount
of
money
to
spend
on
cloud
and
things
like
that.
So
that's
it
thanks.
Everyone
for
coming
foreign.
A
Yeah,
so
for
the
future
start
for
the
online
feature
store
we're
using
student
idb,
which
is
it's
basically
kind
of
like
a
sand
grab
nosql,
but
it
allows
us
to
the
memory.
Footprint
is
very
low,
so
we
can
easily
use
that
and
for
the
the
offline
feature
store
where,
basically,
we
use
story
buckets
and
in
some
cases
we're
using
traditional
databases.
A
D
B
B
Take
that
yes,
so
we
just
use
Argo
City
to
essentially
stand
up
the
cluster
for
our
customers.
A
lot
of
them
are
first-time
kubernetes
users
and
we
want
to
introduce
them
to
Gates
in
a
repetitive
or
repeatable
manner.
I
feel
like
our
Argo
CD
is
a
good
way
to
kind
of
introduce
in
the
case
and
keep
them
organized,
and
it
enables
us
to
deploy
like
new
inference
services
and
different
type
of
Argo
events,
so
we
can
create
new
sensors
and
have
them
deployed
in
a
nice
manner
where
you
can
visually
see.
A
Yeah,
and
just
so
after
that,
it's
easier
for
us
to
manage
the
deployment,
the
Manifest
for
all
the
companies
that
we're
running
so
every
time,
there's
a
new
release
of
let's
say
tensorflow
operator
or
case
that
we
can
manage
the
release
through
github's
process.
So
basically
I
go
see.
The
officers
do
that
easily.