►
From YouTube: AIoT Ops - Using Kubernetes and ML Ops to Build Edge ML Applications - Asheesh Goja, Cisco
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
AIoT Ops - Using Kubernetes and ML Ops to Build Edge ML Applications - Asheesh Goja, Cisco
Whether you are developing an IIoT application or an IoMT solution that requires closed-loop decisions, bringing machine intelligence closer to the edge offers significant advantages. Energy-efficient ML hardware for tensor operations is driving the move towards Edge ML and the emergence of a new domain of edge computing called AIoT or Artificial Intelligence of Things. This talk will propose a novel approach to build AIoT applications using Kubernetes. Using an illustrative reference architecture, the talk will explain emergent AIoT behaviors and architecture patterns. Such patterns can help in deciding what open source projects to choose for AIoT solutions. You will also see a demo simulating an industrial IoT setting where sensors measure conditions in real-time and feed them to an edge device. The device uses a TPU accelerator to apply logistic regression to predict future equipment breakdown. You will see how ML pipelines measure drift and re-train and re-deploy the model.
A
Hello,
everyone
and
welcome
to
my
talk.
I
hope
you
had
a
good
sessions
in
the
keynotes
this
morning
and
now
I'm
happy
to
take
you
on
a
journey
from
the
world
of
cloud
native
computing
to
the
world
of
edge
computing
and
in
this
journey
we
will
learn
how
kubernetes
can
be
used
to
control
and
manage
aiot
machine
learning
pipelines
on
the
edge
devices.
A
My
talk
today
has
two
sections.
I
will
first
talk
about.
Why
aiot
what
is
important
about
it
and
when
we
bring
a
I
and
ml
together?
What
are
the
emergent
behaviors?
What
are
the
patterns
that
help
us
solve?
Those
behaviors
and
I'll
also
show
you
a
comprehensive
reference
architecture
on
how
to
build
an
aiotml
solution,
along
with
the
reference
implementation
and
then
a
live
demo.
A
A
A
The
solution
lies
in
looking
closely
at
what
do
the
devices
do
when
they
connect
internet
their
primary
reason
to
connect
the
internet
is
to
push
the
data
to
the
cloud
tier
so
that
we
can
generate
insights
and
power,
our
business
applications
and
there
analyze
the
solution.
What,
if
that
the
power
of
making
insights
and
inferences,
is
brought
down
closer
to
these
embedded
devices?
But
that's
easier
said
than
done
once
we
do
that.
A
It
is
a
big
challenge.
Most
machine
learning
frameworks
are
just
too
bulky
to
run
on
embedded
devices,
and
we
also
see
that
there
are
insufficient
metrics
to
measure
the
performance
of
a
machine
learning
algorithm.
The
traditional
flops
on
macs
are
very
generic.
You
know
when
you,
when
you
go
to
a
particular,
let's
say
a
tpu
accelerator
or
a
gpu
device
from
a
particular
vendor.
A
The
performance
and
imaging
performance
is
key
because
these
devices
have
limited
power,
bandwidth,
computational
resources,
and
we
also
find
is
as
machine
learning
is
brought
closer
to
these
devices.
The
optimization
strategies
that
we
frequently
use
on
the
cloud
tier
erode
the
training,
the
the
model
accuracy
and
that
it
leads
overall
to
degradation
of
the
model.
A
But
we'll
also
see
is
that
there
are
multiple
incompatible
competition
architectures
on
the
edge
there.
The
biggest
challenge
I
had
doing.
This
was
a
lot
of
the
stacks.
Don't
work
on
rm64
you.
It
works
fine
on
amd
or
x86
space
machines,
but
on
arm
it
just
just
doesn't,
and
you
also
see
that
when
you
run
categorization
on
on
base
devices,
a
lot
of
things
that
work
in
normal
container
did
does
not
seem
to
work
on
on
devices
and
I'll
talk
about
some
of
those.
In
my
in
my
talk
and
in
the
demo.
A
A
So
let
me
bring
up
this
complexity
chart.
So
I
just
put
a
a
little
table
which,
where
I
show
some
of
the
most
commonly
used
machine
learning,
algorithms
and
the
one
I'm
using
for
this
demo
is
a
logistic
regression
algorithm.
So
if
you
look
at
it
closely,
what
we
see
is
the
columns
here,
the
first
column,
the
size
shows,
space
complexity
and
the
training
inference
is
the
time
complexity
of
the
algorithm.
A
So
what
notice
here
that
the
training
complexity
of
logistic
regression
is
a
polynomial
time,
but
the
inference
is,
is
linear
and
that's
that's
in
where
lie
is
my
primary
hypothesis
and
so
the
demo.
I
will
show
that
so
my
hypothesis
is
that
the
computational
power
on
the
edge
devices
is
sufficient
to
run
inferences,
but
we
also
have
to
bring
the
training
down
to
the
edge
tier
and
to
do
that,
we
have
to
think
about
the
entire
machine
learning
ops
pipelines
that
can
run
on
edge
tier.
A
It
is
not
sufficient
to
run
the
pipelines
on
the
cloud
and
then
wait.
The
downloads
happen
on
the
edge
tier.
The
entire
pipeline
should
be
running
on
the
edge
here
and
to
do
that,
I
uncovered
some
solution
patterns
that
I'm
going
to
talk
about
now.
The
first
and
most
important
pattern
here
is
take
the
machine,
learning
pipelines
and
there's
a
lot
of
complex
dependencies
on
the
in
the
steps
in
the
machine
learning
pipelines
you
express
those
as
dags.
A
So
there
are
a
few
tools
and
I'm
going
to
show
one
of
the
tools
called
argo
to
express
how
to
express
the
dependencies
as
dags.
You
guys,
familiar
with
dags
show
of
hands
great
now
you're
an
expert
on
that.
I
know
that
and
then
use
the
pipelines
for
yes
question
and
use
the
workflow
pipelines
for
continuous
evaluation.
A
What
I
mean
by
evaluation
is
major
drift
and
you
know,
and
the
validations
continuously
on
the
edge
tier
and
then
train
the
models
and
produce
the
actual
models
on
the
hdr
and
then
use-
and
this
is
one
of
the
very
interesting
things
that
I
found
is
and
that's
what
kubernetes
is
is
is
really
helpful.
Is
use
hardware
accelerator,
aware
odd
placement
strategies.
A
What
I
mean
by
that
is
using
the
the
rich
expression
and
syntax
and
kubernetes
to
using
you
know
tense
and
affinities
and
anti-affinities
and
labels.
You
can
basically,
you
know,
change
the
way,
a
particular
workload
that
let's
say
requires
a
gpu
based
acceleration
versus
a
workload
that
requires
a
tpu
tensor
acceleration.
You
can
place
them
and
give
give
the
right
clues
to
the
scheduler
on
where
to
place
them,
because
in
the
edge
world
not
every
device
is
uniform.
A
The
workloads
and-
and
this
is
easier
said
than
done
on
the
on
the
edge
tier
running
containers-
is
really
challenging
and
you
will
see
in
some
some
of
the
demos
and
then
the
code
I'll
show
you
is
the
ip
tables,
the
basic
networking.
It's
all
it
just
doesn't
work
correctly.
The
way
it
works
on
on
the
cloud
here
and
then
another
pattern,
I
think,
is
very
important-
is
using
a
streaming
api,
sidecar
and
I'll
talk
about
more
in
in
the
reference
architecture.
A
This
allows
the
sidecar
allows
decoupling
inference
that
runs
on
the
edge
tier
from
communication.
So
the
sidecar
talks
to
the
kafkas
and
the
enquiries,
and
you
know
all
the
communication
pieces,
but
you
can
keep
the
inference
module
just
focused
on
doing
the
inference.
It
only
has
the
the
the
machine
learning
libraries
in
it
and
nothing
else,
and
that
and
there's
a
reason
for
that,
because
these
libraries,
as
you
put
more
stuff
on
them,
they
become
very
bulky.
And
you
don't
you
cannot.
A
So
now
let
me
bring
all
of
this
home.
There's
a.
There
are
a
lot
of
moving
parts
here,
so
I'm
going
to
talk
about
a
reference
architecture
and
I
spent
a
lot
of
time
painstakingly
putting
all
these
pieces
together
because
it
is
comprehensive.
It
is
there's
a
lot
of
parts,
so
I'll
take
some
time
to
go
over
it.
So
this
architecture
for
running
aiot
ml
ops
on
the
edge
tier,
has
four
hardware
tiers.
A
The
second
tier
is
the
platform
tier.
This
tier
is
what
hosts
various
services,
such
as
a
private
container
registry,
which
is
important
for
edge
tier,
because
you
just
can't
be
waiting
on
docker
hub,
and
you
know,
google
container
industries
to
download
there
is
no
access
to
that
on
the
hdr
or
even
if
it
is,
the
bandwidth
is
limited.
A
Then
there
is
an
ota
ml
code.
Repo
I'll
talk
more
about
ota
is
over
there.
You
know
your
phones
have
a
photo
when
it
gets
new
firmware
on
these
devices,
the
edge
devices,
especially
the
ones
that
are
in
microcontrollers.
They
don't
have
file
system.
So
you
can't
just
download
a
model.
You
have
to
send
the
entire
firmware.
You
have
to
reflash
it
so
the
ota
repo
and
the
services.
A
I
wrote
a
custom
depo
in
golang
to
do
this,
allows
you
to
download
it
to
the
mcu
based
devices
and
reflash
the
device,
and
there
are
a
few
pieces
that
allow
us
to
do.
Protocol
bridging
the
the
mcu
based
microcontroller-based
devices
are
using
mqtt,
but
the
rest
of
the
architecture
on
the
platform
here
is
all
kafka
based.
A
So
there's
a
protocol
bridge
and
various
other
services.
The
third
tier
is
the
inference
and
the
iot
tear.
Actually,
if
you
look
at
this
the
folks
on
on
virtually
connected,
you
won't
be
able
to
see
it,
but
I'll
show
you
a
picture
of
what's
set
up
here
in
in
front
of
the
stage.
You'll
also
see
the
stairs
on
this
on
this
demo
board.
Here,
the
entrance
in
the
iot
tier
has
two
class
of
devices.
A
So
you
can
run
most
of
the
stuff
that
we
are
familiar
with
in
the
world
of
cloud
computing
here.
So
with
the
right
containerization
and
the
right
stripping
down
of
some
some
some
platforms,
they
do
work.
The
second
class
of
devices
is
where
things
get
interested,
and
these
are
the
auto
super
loop
based
devices.
A
Do
folks
know
about.
Do
you
guys
know
about
rtos
r2
stands
for
real-time
operating
system.
That's
what
a
lot
of
embedded
devices
are
on
super
loops.
If
you
are
familiar
with
arduino
programming,
there
is
a
big
setup
and
a
loop
that
runs
forever.
That's
a
super
loop
device
and
these
devices
are
extremely
small.
They
don't
have
file
systems,
and
you
know
gigabytes
of
memory.
They
need
special
attention
and
the
architecture
and
the
reference
solution.
I'll
show
you
addresses
those
concerns.
A
These
devices
run
inference
modules
embedded
and
these
inference
modules
then
communicate
over
mqtt
to
the
rest
of
the
system.
The
analog
there
is
what
you
see
on
in
the
extreme
left-hand
side.
That's
where
I
have
an
induction
motor
connected
with
a
bunch
of
sensors
that
takes
the
analog
world
digitizes
it
and
runs
some
digital
signal,
processing
to
filter
out
inconsequential
data
or
noise,
and
I'll
also
show
you
how?
How
can
you
how
you
can
run
inferences
machine
learning
to
filter
out
noise
in
addition
to
dsp
and
then
so?
These
are
the
four
tiers.
A
Now
there
are
layers
of
services
that
span
these
stairs,
so
the
first
tier,
the
first
layer
is
the
message
broker.
This
is
an
mqtt
based
broker
and
then
there
are
rtos
and
super
low
super
loop
otas.
This
allows
us
to
download
new
firmware
as
a
new
model
shows
up,
and
then
there
are
three
other
layers.
A
A
I
know
there's
a
lot
going
on
here.
So
what
what
I'm
going
to
show
now
is
an
actual
implementation
based
on
this
architecture.
So
I
will
not
implement
all
the
pieces
in
the
reference
implementation,
but
some
of
the
key
ones
are
there
and
in
the
demo
you
will
also
see
how
it
actually
happens
in
the
real
world.
So
it's
not
just
in
the
powerpoint
looking
quick
time
check.
A
A
A
This
device
turned
out
to
be
quite
powerful.
It
has
a
gpu,
coprocessor
and
good
amount
of
computational
power,
and
this
device
hosts
the
entire
training
tier,
so
the
the
ml
jobs
to
extract
training
data
normalize,
it
detect
drift,
train
it
and
then
quantize
it.
You
guys
familiar
with
quantization
and
well
folks,
so
quantization
basically
is
taking
the
the
big
full
frozen
graph
that
comes
out
of
a
machine
learning,
trace
training
job
and
you
basically
convert
it
into
format
that
can
run
on
smaller
devices.
A
The
platform
tier
is
running
on
a
raspberry
pi
and
a
vm
I'm
using
a
vm,
because
I
ran
out
of
raspberry
pi.
So
I
just
couldn't
it's
hard
to
get
hardware
these
days
all
supply
chain
constraints.
So
I
just
couldn't
get
the
right
hardware,
so
the
vm
here
is
an
instance
in
ubuntu
vm
and
its
os
is
very
close
to
the
raspberry
pi.
So
it
was
a
good
drop-in
for
another
raspberry
pi.
The
platform
tier
runs
the.
A
A
It
also
has
a
pre-processing
stage
again
using
tensorflow
and
a
fast
fourier
transform
to
filter
out
noise,
and
you
know
remove
things
that
just
should
not
be
sent
to
the
to
the
rest
of
the
tier.
It
just
increases
so
much
of
computation
that
these
devices
do
not
have
the
capacity
to
run
to
run
messages
that
really
have
no
consequence.
A
A
A
And
the
the
edge
of
the
the
platform
services
or
the
layers
that
span
this,
the
concrete
implementations
for
the
messaging
broker
is
a
mosquito
broker
that
runs
on
the
raspberry
pi,
there's
a
golan
custom,
ota
service
and
then
for
for
this.
The
kafka
streams,
I'm
using
stream
z,
as
I
said,
for
kubernetes,
that's
k3s
and
for
the
ml
workflow
pipelines.
It's
the
argo
workflows.
A
Okay,
now
it's
time
to
demo
this.
So
what
what
you
see
here
is
this
is
primarily
for
the
folks
connected
virtually
is
what
is
what
is
displayed
here
on
the
stage
on
the
stage
you
see
the
the
three
tiers
for
training
platform
and
inference
that
are
that
have
the
an
nvidia
device,
a
raspberry
pi,
a
bunch
of
coral
devices
and
an
mcu
module
and
on
the
far
left
side
is
the
analog
tier
which
has
the
motor,
and
I
have
labeled
all
the
key
components
there.
A
A
A
A
There
are
c,
plus
plus
modules
that
load
the
the
tp
accelerator
drivers
that
work
only
when
it
detects
a
tpu
accelerator
and
then
the
rest
of
the
code
is
to
lo
to
load
the
the
tensorflow
lite
libraries,
which
anybody
here
programmed
with
tensorflow
lite,
no
yeah.
So
this
gets
really
bare
bones.
You
know
it
doesn't
have
all
the
nice
api
and
support
that
you
see
in
keras
or
other
tensorflow
libraries.
So
you
have
to
just
go
down
to
setting
up
the
input.
A
Tensors
and
output,
tensors
understanding
the
data
formats
and
then
and
running
a
prediction
on
it
and
then
getting
the
output
tensors
and
that's
what
you
see
in
the
lot
of
the
code
here
on
the
on
the
right
hand,
side
is
the
the
tensorflow
lite
c,
plus
plus
module
source
code.
So
this
is
the
the
python
counterpart
of
tensorflow
lite,
that
is
purpose
built
to
run
on
on
microcontroller-based
devices.
So
all
these
libraries
are
here
and
then
again
you
set
up
the
input
answers
generally.
A
As
I
said,
the
the
steps
of
the
pipeline
to
extract
detect
drift,
train
and
quantize
are
all
running
on
the
nvidia
device.
So
let
me
show
you
how
this
is
expressed
in
an
argo
dag.
The
extract
step
connects
to
a
google
storage
bucket
and
downloads
download
training
data,
and
this
is
raw
training
data
and
it
then
transforms
and
normalizes
it.
A
A
The
next
step
is
drift
detection.
It
has
a
dependency
and
that's
how
you
use
dags.
It
has
a
dependency
on
the
extract
step.
That's
what
you
see
here
expressed
in
the
in
the
dependencies
section
on
the
tag
and
the
following:
that
is
a
training
step.
Training
has
two
dependencies.
It
requires
the
extract
step
to
complete
and
the
drift
detection
to
be
completed,
and,
finally
is
the
quantized
step
that
takes
the
the
frozen
file
and
then
transforms
it
into
a
tier
flight
module.
A
Here
are
the
four
devices
that
I
have
used:
okay,
so
now
it's
time
to
set
up
the
demo.
So
the
way
I
was
I'm
trying
to
do
it
is
I'll
run
the
demo.
I
will
show
you
a
few
things
how
they
work
and
then,
if
we
have
time
because
I'm
sitting
between
you
and
lunch,
I
I
wanted
to
show
you
how
to
tear
this
whole
thing
down
and
build
it
from
scratch,
but
I
I
doubt
I'll
have
time
for
that.
A
A
B
A
A
A
A
A
A
The
extract
step
just
ran
what
it
did
was
it
downloaded
it
it
connected
to
the
google
cloud,
storage
bucket
and
downloaded
a
training
data
file,
and
you
can
see
the
container
logs
here.
This
is
a
very
good
tool.
It
shows
that
model
download
is
complete.
The
next
step
is
it's
running
a
drift
detection
job
and
if
we
go
back
here.
A
A
A
Registry,
I
wrote
in
golang,
so
it
it
hosts
training
data.
As
it's
downloaded
you'll
see
the
model.
The
model
is
a
zip
file.
So
if
I
open
this
zip
file,
people
who
are
familiar
with
tensorflow
would
see
this
the
pv
files
here
and
then
you
have,
as
the
training
is
running,
the
next
stage
is
to
quantize
it
and,
as
the
quantize
completes,
the
quantized
model
will
be
uploaded
here.
A
A
A
A
That
the
fall
is
always
detected.
You'll
see
this
mcus,
its
leds
should
start.
The
blue
led
should
start
blinking
15
times,
and
then
it
will
send
a
message
to
the
cluster
here
so
that
green
light
there
actually
is
is
showing
that
an
entrance
job
is
running
on
that
that
particular
node
on
the
cluster.
A
So,
folks
with
that,
my
demo
is
complete.
I
wish
I
had
the
time
to
show
you
how
I
built
it,
but
at
this
point
let
me
all
right
so
using
this.
This
message
here
is
important.
You
see
an
inference
level.
Two
message
showed
up
here.
What
that
means
is
the
way
internet
classified
inferences
is
that
if
the
inference
is
running
on
the
analog
tier
close
to
the
mcu
on
the
and
on
the
sensors,
that
is
influence
level,
one
that
is
mostly
pre-processing
inference.
A
Only
then
trans
a
condition
that
is
above
certain
threshold
only
then
transmit
the
message
to
rest
of
the
tier,
so
that
is
inference
level
two,
which
means
that
the
the
conditions
have
been
pre-processed
and
only
the
signals
that
have
consequence
are
being
sent
further
up
the
cluster.
So
that's
the
stage
inferences
I'm
trying
to
show
here
is
the
first
level
of
filtering
happens
on
the
analog
tier
using
dsps.
D
A
B
A
So
the
question
is
of
somebody
repeating
the
question:
okay,
so
the
question
is
what
kind,
what
class
of
problems
can
machine
learning
running
on
the
edge
solve?
It's
a
good
question.
If
foreign
for
an
iot
engineer
like
myself,
the
first
problem
that
I
see
is
closed
loop
decisions
wherein,
when
the
inference
or
the
action
machine
learning
logic
tells
you
something
is
wrong.
The
device
can
can
actually
trigger
what's
called
a
closed
loop
actuation.
A
That
means
it
can
tell
another
set
of
hardware
to
intervene
or
do
something
it
doesn't
have
to
wait
for
this
information
to
go
up
to
some
back
end
in
the
cloud
and
come
back.
There's
too
much
latency
there
that
that's
that's
number
one.
The
second
it
is.
The
second
set
of
of
possibilities
that
opens
up
is,
is
now
being
able
to
keep
the
data
itself
on
the
device.
So
there
are
a
lot
of
concerns
around
privacy
and
you
know
attack
vectors
and
the
devices
connect
the
internet.
We
can
resolve
that.
C
A
A
Difference
as
well
right,
it
can
so
now.
Okay,
so
I
see
your
question.
The
amount
of
power
and
network
and
bandwidth
required
to
run
training
will
be
an
overkill
to
run
it
on
the
entrance
tier,
because
the
inference
devices
are
running
close
to
the
analog
devices
and
the
and
the
sensors
and
some
a
lot
of
those
times.
Those
devices
are
battery-powered,
so
it
makes
sure
that
the
inference
that
the
training
jobs
do
not
consume
all
the
power
and
the
computation,
and
there
is
nothing
left
to
process
the
sensor.
Data.