►
From YouTube: MLOps Pipelines to Deliver Tf.Lite Models to Distributed Compute Pools at the Edge - Maciej Mazur
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
MLOps Pipelines to Deliver Tf.Lite Models to Distributed Compute Pools at the Edge - Maciej Mazur, Canonical
This session will be a demo showcasing how to create an automated MLOps pipeline, that trains a model for different classes of infrastructure. You will see how with Kubeflow and TensorflowLite you can work on a model for a smart city, which will detect cars, do license plate recognition and tell if plates actually match the car. This model will be then, from the same pipeline, available for a mobile phone app, a small form factor device like RaspberryPi/NVidia Jetson as well as a standard Tensorflow model for the datacenter.
A
Thank
you,
everyone
for
coming
here
and
listening.
My
name
is
Magic
mazur,
I'm,
principal
AIML,
engineering,
canonical
and
I
wanted
to
talk
a
little
bit
about
envelope's
pipelines
and
how
to
actually
build
them
to
produce
models
that
are
usable
at
the
edge
and
also
show
you
a
demo
of
the
keep
flow
platform.
So,
basically,
let
me
start
from
a
recap
about
what
is
envelopes.
How
many
of
you
know
mlops
pipelines?
Well,
okay,
so
let
me
start
from
the
beginning.
A
So,
basically,
whenever
you
are
trying
to
use
some
machine
learning
models
in
production,
there
are
a
couple
of
steps
that
you
need
to
do
before
actually
having
a
ready-made
model.
So
it
all
starts
from
data
collection.
There
are
multiple
data
collection
engines.
You
can
make
an
ETL
process
from
databases
or
collect
some
events
or
any
other
things
that
you
want
to
put
into
your
training
process.
A
Then
you
do
data
labeling
and
Analysis
and
curations
cleaning
and
all
the
process
that
is
necessary
to
make
this
data
actually
ready
to
be
used
in
training
a
model,
because
typically,
data
coming
from
external
systems
would
have
a
lot
of
flows.
A
lot
of
missing
values.
Staff
will
not
be
aligned.
The
keys
and
ideas
to
connect
them
will
not
be
so
easy
to
match,
and
after
this
all
pro
preprocessing
is
done.
A
You
can
actually
go
to
training
a
model
so
either
building
something
simple
like
progression
models
or
training
and
neural
network,
depending
on
what
you
need
and
what
is
your
use
case,
but
in
the
end,
after
your
model
is
trained
and
evaluated
on
a
test
data
set,
you
will
try
to
deploy
it
in
production,
but
it's
not
the
end,
because
whenever
the
model
is
deployed,
you
need
to
fetch
some
sort
of
information.
How
well
it's
doing
and
train
it
again
and
again
and
improve
it.
A
Also,
the
different
setup
is
because
of
the
speed
that
you
want
to
do
it
in
like
there
are
a
lot
of
ml
models
that
are
running,
for
example,
in
the
night,
assessing
the
production
quality
and
that's
around
typically
as
like
a
batch
processing
to
reduce
costs
and
make
things
simpler.
But
there
are
a
lot
of
models
running
on
them,
like
real-time
stream
of
data,
if
you
are
doing
like
Smart
City
monitoring,
which
is
the
use
case
that
we
will
look
into
on
this
demo
or
any
other
use
case.
A
That
requires
like
a
fast
feedback
loop
and
that's
something
that
will
be
important
and
and
together
with
AI
infrastructure,
Alliance,
which
we
are
also
part
of.
We
created
the
workflow
templates,
which
are
basically
describing
an
end-to-end
process
that
is
used
in
order
to
have
a
proper
envelopes
Pipeline
and
what
are
the
different
steps
on
the
way?
So,
obviously
that's
not
enough
like.
If
you
want
to
actually
build
it,
you
will
need
tools
and
the
tools
and
Technologies
are
also
part
of
the
blueprint.
A
So,
typically,
you
will
need
some
sort
of
way
to
get
the
data
from
external
systems,
maybe
enhance
it
with
some
synthetic
data,
generators
or
any
other
like
tokenization
methods,
or
something
to
hide
private
and
important
information.
Then
you
have
a
pipeline
that
will
go
through
the
whole
data,
engineering,
experimentation,
training
and
deployment.
But
what's
the
biggest
issue-
and
what's
the
real
problem
that
we
are
trying
to
solve
in
canonical
is
the
fact
that
you
need
tons
of
different
open
source
projects
connected
together
in
order
to
make
this
pipeline
happen?
A
Because
if
you
want
to
start
fetching
data
from
an
external
systems,
you
will
go
with
an
ETL
tool,
Apache
95,
for
example,
and
the
spark
to
process
it.
Obviously,
you
have
data
sets
that
differ
every
day.
There
are
some
changes,
so
we
would
like
to
Version
Control
it
just
like
you,
Version
Control,
your
code
in
git
repo.
There
are
tools
like
DVC
to
do
Version
Control,
and
then
you
use
different
Frameworks
tensorflow
pytorch
self
models
with
KSL.
So
all
the
setup
is
pretty
complex
and
what's
even
more
tricky,
it's
different
depending
on
the
use
case.
A
So
what
we
use,
what
we
are
using
to
make
this
simpler
is
a
thing
called
charms,
and
this
is
a
construct
of
like
pythonic
kubernetes
operators,
which
we
use
to
deploy
different
sets
of
tools
like
that,
so
the
main
one
that
is
a
baseline
of
every
envelopes
deployment
that
we
are
doing
is
kubeflow.
It's
an
open
source
project
which
was
by
the
way
yesterday
contributed
to
a
cncf.
A
It's
now
an
incubation
project,
so
we
hope
that
the
user
base
will
grow
and
we
are
like
inviting
each
and
every
one
of
you
to
participate,
and
we
in
kubeflow
the
main
part
of
it
is
keep
flow
pipelines.
This
is
something
coming
out
of
Google
where
they
developed
the
pipelines
for
their
own
email
platform
and
open
source
it.
A
So
you
can
run
them
from
a
Jupiter
notebook
like
we'll
do
today.
You
can
hook
it
up
to
your
favorite
IDE,
like
vs
codes,
data
spell
or
whatever
you
are
using,
and
also
it
runs
with
external
tools
like
khatib
for
hyper
parameter
tuning.
You
can
use
ml
flow
if
you
want
to
have
a
proper
model
registry
and
also
compare
different
models
and
trans
and
all
the
other
like
favorite
ml
tools,
and
that
you
might
be
using
it.
Also,
it's
also
pretty
good
in
terms
of
connecting
to
a
serving
pipeline.
A
You
can
use
Seldon
or
ksav
or
any
other
serving
framework,
and
with
Juju
and
charms
you
can
actually
deploy
different
YouTube
bundles,
depending
on
where
you
are
deploying.
So
in
your
main
envelopes
trading
environment,
you
can
deploy
a
full
set
of
tools
like
with
Cube
flow
pipeline,
sample
flow
and
all
your
data
collection.
But
if
you
have
a
serving
environment,
which
is
a
different
case,
cluster
you
can
deploy
just
like
Seldon,
istio
and
feast,
for
example,
for
online
feature
store
and
no
other
components.
A
So
you
minimize
the
footprint
and
can
have
like
a
configurable
deployment
depending
on
where
you
are
doing
it.
So
the
use
case
it
will
look
today
would
be
license
plate
recognition,
it's
just
like
a
sample
to
show
you
how
the
pipeline
looks
like
will
not
be
focused
on
making
like
the
best
model
to
do.
A
It
will
actually
just
download
the
object
detection
model
from
tensorflow
modelz,
but
basically
the
whole
rpr
works
in
a
way
that
you
are
trying
to
detect
the
place
on
the
picture
where
the
license
plate
is
and
just
OCR
it
and
have
the
license
plate
numbers.
So
what's
nice,
if
you
want
to
actually
sell
these
models
at
the
edge,
it's
something
that
was
discussed
in
one
of
the
first
talks
this
morning
is
tensorflow
and
tensorflow
Lite.
A
So
you
actually
have
a
project
called
tensorflow
Model
maker,
which
allows
you
to
export
tensorflow
models
in
different
formats
and
the
light
one
has
an
option
to
export
a
model
in
a
format
that
is
then
digestible
by
iOS
or
Android
devices,
Raspberry,
Pi's
and
so
on.
And
if
you
want
something
a
little
bit
bigger,
you
are
using
like
your
standard.
Tensorflow
model
and
put
it
put
it
on
the
edge
so
yeah.
A
A
So
if
you
look
at
the
standard
way
of
working
of
a
data,
scientist
I
would
try
to
build
basically
first
a
notebook
and
try
on
like
a
single
instance
to
fetch
a
data
set,
try
to
make
a
model,
make
it
better,
do
some
OCR
and
so
on,
but
in
order
to
actually
make
it
repeatable
and
make
the
training
something
that
I
can
do
all
the
time
I
would
build
pipelines.
So
here
is
the
sample
code
to
create
a
pipeline
which
has
several
different
tasks
which,
like
downloads,
the
data
set.
A
A
A
And
start
things
from
one
another,
and
also
you
see
all
the
like
different
parameters.
So
whenever
you
do,
for
example,
a
training,
it's
a
separate
Docker
container,
you
can
get
into
it.
You
can
see
the
logs
see
what
the
output
of
your
like
tensorflow
training
and
each
and
every
artifact.
If
you
go
to
like
export
of
a
model,
you
can
see
that
it
it
went
to
menu.
A
This
is
like
an
open
source,
S3
storage
and
each
and
every
step
on
the
way
gives
you
this
kind
of
outputs,
and
you
can
use
this
zipped
file
in
the
next
steps
of
your
pipeline.
You
can
either
wrap
it
in
a
Docker
container
with
Seldon
and
host
it
on
any
kubernetes
cluster.
You
can
wrap
it
in
a
snap
and
put
it
on
your
Edge
devices
depending
on
what
is
your
use
case,
and
how
do
you
want
to
do
that
so
yeah?
Basically,
that
would
be
all.
A
Thank
you
and
see
you
at
the
booth
p28,
the
quite
a
lot
of
demos.
Around
aim
are
there
as
well,
so
there
is
talking
jellyfish
from
the
Jamie
jellyfish
latest
release
of
Ubuntu.
There
are
a
couple
of
bots
trying
to
play
Starcraft
and
you
can
play
against
them.
If
that's
what
you
enjoy
so
yeah,
you
can
basically
message
me
on
the
social
media
mentioned
here,
happy
to
talk
to
you
later
on
about
any
kind
of
iiml
issues
or
troubles
that
you
have
with
Ubuntu
or
kubernetes
and
try
to
make
it
better.
B
A
Thank
you
for
the
question.
So
basically,
Q
flow
was
an
open
source
project
for
quite
a
long
time.
Google
open
source
it
several
years
ago.
However,
yesterday
it
was
contributed
to
cncf
as
an
incubation
project
and
canonical
the
company
behind
Ubuntu,
where
I
work
was
also
the
contributor
to
kubeflow,
and
there
is
a
distribution
called
Chan
kubeflow,
which
we
automated
and
also
contributed
to
the
community.
So
we
can
find
it
on
like
the
Upstream
project
docs,
and
this
is
something
that
we
are
deploying
here
in
like
wide
areas
of
different
production
deployments.
A
It's
in
automotive
industry
in
life
sciences
and
big
HPC
clusters,
where
it's
training
models
on
like
hundreds
of
a100
Nvidia
gpus,
which
are
connected
together
with
all
the
like
GPU
passthrough,
and
also
more
complicated
GPU,
passthrough
type
features
like
Mig
and
vgpus.
Depending
on
what
do
you
want
to
do
and
in
HPC
setups?
It
even
works
with
like
Infinity
band
and
accelerate
accelerating
the
training
with
multiple
gpus
across
different
data.
Centers.