►
From YouTube: Real-time Vision Processing on Kubernetes: Working with Data Locality - Yisui Hu, Google
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Real-time Vision Processing on Kubernetes: Working with Data Locality - Yisui Hu, Google
Kubernetes as a general purpose cloud native application platform simplifies the operation of applications. Data processing is one of the common of applications that benefit from the power of Kubernetes. However attempting real-time computer vision processing on Kubernetes isn't that straightforward, as the maximum processing time for a single frame is expected to be less than 30ms to reach 30fps or no more than 60ms (15fps, start being laggy). This presentation demonstrates the actual issues, feature gaps encountered and workarounds applied running a real-time vision pipeline on Kubernetes from the presenter's hobby robotics project. The pipeline involves a few image pre-processing tasks, a ML model (SSD mobilenet) for object detection and depth estimation with a single camera. The biggest challenge is how to distribute the tasks across nodes with data locality in consideration.
To Learn More: https://sched.co/GrVe
A
First,
thanks
organizers
and
the
staff
for
setting
all
this
up,
and
thank
you
all
the
audience
for
coming
here
and
all
together.
Let's
witness
moment
when
computer
vision
embraces
communities,
all
let's
say,
communities
embraces
computer
vision
because
which
is
much
older
than
whatever
doesn't
matter.
A
My
name
is
Ray
a
software
engineer
from
Google
today,
I'm
thrilled
to
share
what
I
have
learned
by
running
a
vision,
processing
pipeline
on
Canaries
and
given
this
assertion
is
a
text
sharing
session
based
on
effects
from
my
hobby
project
during
my
spare
time.
Please
don't
take
any
saying
as
a
visual
from
Google
and
I'm,
not
expert
of
computer
vision.
So
please
ask
simple
questions
and
this
session
will
be
mostly
focused
on
Canaries
and
given
the
content
of
this
session
will
be
a
little
tense.
A
Now,
let
me
introduce
a
project.
This
is
my
hobby
project,
so
I
build
a
robot
which
has
a
single
camera.
It
can
move
horizontally
like
this
and
it
can
also
move
vertically
like
this.
It
caused
two
motives
to
move
the
robot
around
and
obviously
it
also
has
two
arms
a
left
one
and
the
right
amount,
of
course,
for
just
being
cute
or
silly
and
with
four
node
tiny
cluster
as
a
base
to
cluster
running
a
kubernetes
cluster
panellists.
A
A
Probably
yes,
but
I
have
a
few
clear
reasons
for
using
whom
Nettie's
the
first
is
canary,
simplifies
operations.
A
lot
I
know
how
I
build
a
code
and
I
know
how
to
pack
all
the
binary
into
containers.
That's
all
and
leave
all
the
rest
to
the
kubernetes
to
perform
all
heavy
lifting
for
deployment
and
very
easy
upgrade,
and
also
done
great
and
the
second
reason
for
Cornelius.
It
brings
up
this
native
support
for
skill
abilities
today.
A
I
have
one
robot
control
and
probably
I
plan
to
build
more,
for
example,
as
his
toys
and
when
there's
more
more
robots,
definitely
the
work
load
will
increase
heavily
as
long
as
I
have
enough
hardware.
Resource
two
could
never
take
care
of
them
seamlessly
and
the
third
reason
about
the
ecosystem.
It's
actually
the
community
I
mean
all
of
the
power
of
the
community
for
endless
creativity,
and
we
can
leverage
so
much
tools
and
to
focus
ourselves
on
our
business
and
running
a
vision.
Process
on
kubernetes
is
definitely
a
non
common
use
of
Canaries
and
I.
A
So,
let's
take
a
look
at
this
fishing
pipeline.
Actually
this
can
be
built
using
cooper-cooper
flow.
I
haven't
tried
if
you
are
interesting,
there's
a
few
Cooper
Floria
sessions
these
days,
so
you
can
attend
so
in
this
pipeline.
The
robot
will
continuously
take
VJ
sized
imaging,
JPEG
and
ascend
over
Wi-Fi
into
the
pipeline.
A
When
the
private
receives
the
image
it
performs
some
pre-processing
tasks
like
decoding
the
image.
Formatting
are
converting
the
image,
pixel
format
and
do
some
resizing
and
cropping
term
stuff.
Then
I
send
to
multiple
our
data
processing
passes
in
parallel,
so
one
dollar
pass
is
going
through
a
SSD
mobile
net
neural
network
which
detects
multiple
objects
from
a
single
image,
and
the
second
pass
goes
through
a
OpenCV
phase
detection.
A
It
attacks
rectangle
regions
of
your
faces
from
image
and
then
send
these
sub
regions
into
a
face
net,
which
extracts
a
feature
from
this
area
of
your
face,
and
if
you
can
map
the
set
of
features
to
label,
basically,
the
robot
has
a
capability
to
tell
your
name
when,
when
it
sees
your
face,
the
output
of
these
beta
passes
are
actually
small,
JSON
objects.
So
we
can
easily
share
that
with
something
like
a
and
Q
DT,
which
is
message
bus
and
to
be
consumed
by
controlling
logic
from
there.
A
And
when
running
this
paper
on
Canaries,
we
do
meet
some
challenges.
The
first
difficult
problem
is
all
about
real-time
processing,
which
requires
a
decision
to
be
made
within
like
30
milliseconds,
and
it's
heavily
impacted
by
some
effects
like
unreliable
Wi-Fi
network
and
also
the
long-running
computer
intensive
tasks
and
because
of
the
tiny
cluster
I.
A
Don't
have
enough
computing
power
on
this
cluster
to
support
neural
networks,
so
I
have
to
use
external
neural
network
accelerators
like
more
videos,
computer
stick
and
we
need
some
additional
work
to
make
that
work
on
Canaries
and
also
the
other
one
is
actually
our
interesting
one.
When
we're
running
who
knows
on
arm
with
a
big
letter
and
architecture,
processor
I
will
talk
that
later.
A
A
The
cluster
is
running
on
a
four
node
system,
each
equipped
with
Samsung
Exynos
5
4
to
2
CPU,
its
octocopter
core
CPU.
You
think
it's
eight
cores,
but
wait.
It's
actually
different.
It
has
four
big
course
running
at
two
gigahertz
and
another
four
little
cores
running
on
124
gigahertz,
that's
called
big.little
architecture
and
we
also
need
corresponding
more
videos.
Computer
stick
for
each
of
the
nodes.
A
A
A
The
stickers
are
controlled
directly
using
the
SDK
from
application,
and
the
SDK
is
using
a
little
USB
to
initialize
the
device
so
initialize
the
process.
We
will
first
always
offloading
of
firmware
and
then
reset
the
device
as
a
result,
error
times
the
stick
and
connects
the
SDK
next
to
the
stick
will
recreate
a
new
device
on
a
host,
so
it's
impossible
to
bind
mounts
that
vice
not
into
a
container.
A
A
A
The
solution
can
be
very
simple
by
offload
preload,
a
few
models
on
twist
ik
and
we
can
selectively
run
one
or
model
at
a
time,
and
if
we
have
many
models
with
every
need
to
distribute
disease
model
across
multiple
sticks
as
they
attach
to
different
nodes,
then
the
pipeline
becomes
distributed,
and
we
also
want
to
build
a
service
around
these
stickers
to
hide
all
the
complexities
for
accessing
these
device.
What
I
was
thinking
about
how
we
want
to
build
these
services,
so
one
big
concern
is
all
about
input
data.
A
The
input
data
is
actually
big
and
especially
for
this
stick
it
had
to
come.
Convert
every
pick,
a
cell
channel
into
1/2,
precision
full
point,
so
basically
a
1,
because
that
would
take
like
six
bytes,
so
usually
for
VJ
size.
Image
would
take
a
few
megabytes
if
we
think,
just
through
all
these
into
a
network
that
will
be
a
very
heavy
traffic
and
can
we
think
a
different
way
how
we
build
a
service
which
doesn't
serve
through
a
network.
What
about
using
share
memory
if
all
the
pods
can
be
co-located
on
the
same
node?
A
Well
think
about
the
temp
of
FS
and
also
M
map
system,
call
I,
think
we
found
a
solution
and
let
me
just
explain
the
tempest
and
MF
in
case
anybody
here
at
doesn't
understand
that
quite
well.
So
a
temp
of
FS
is
a
file
system
on
Linux
which
you're
completing
memory,
so
a
file
content
is
saved
on
this
person
basically
sits
in
a
physical
memory.
A
Page
and
M
Maps
system
call
can
map
content
of
file
into
a
memory
space
of
process
and
the
process
can
purely
use
the
memory
access
instructions
to
access
content
on
file
so
think
about.
If
we
run
an
map
on
top
of
a
tender
file
system,
that
means
we
are
directly
mapping
the
physical
memory
page
into
the
process
space
without
copy
any
data.
Anything
about
multiple
ports
are
using
a
map
on
the
same
shared
temp
FS.
So
basically
they
can
share
the
same
large
amount
of
data
without
copy
any
of
them.
A
So
that's
the
way
we
won't
build.
This
shared
memory
mechanism
and
this
diagram
show
a
more
detailed
now
in
Buckhannon.
So
on
the
left
top
side,
this
is
dvss.
Ehm
no
worry!
This
is
a
10
FS.
One
point
on
most
of
a
new
system
is
on
hoseason
and
we
create
a
folder
on
that
and
in
the
access
service
in
pod,
in
blue
and
also
in
the
app
one
pod
in
green.
We
bind
mount
this
sub
folder
into
these
containers.
A
So
in
fact,
these
two
parts
are
sharing
the
same
temper
FS
when
the
app
won't
receive
the
frame
after
pre-processing.
It
saves
the
image
data
into
the
Tempur
FS
using
N
and
map,
and
it
notifies
Axios
service
using
would
have
mechanisms
because
they
are
sharing
in
same
file
system.
So
the
server
service
is
able
to
see
the
file
on
the
local
file
system
and
use
the
same
and
map
system
call
to
an
F
the
file
into
its
own
memory
space.
A
So
after
that,
we
can
see
the
yellow
block
the
tensor
image,
the
x-ray,
a
single
copy
in
the
physical
memory
press
shared
by
two
ports.
So
they
sharing
the
same
data
without
copy
any
data,
and
the
accessory
service
can
offload
the
image
data
to
the
accelerator
and,
as
with
the
similar
mcadams,
it
can
also
send
back
the
result.
A
Okay,
so
I'm
talking
about
this
simple
problem,
we
are
come
to
the
next
difficult
problem.
It's
all
about
real
time
processing.
First,
let's
take
a
look
at
some
benchmark.
It's
all
about
the
cost
running
are
some
tasks.
The
first
blue
bar
is
a
baseline
of
a
certain
frame
per
second,
the
rate
from
camera,
and
we
can
obviously
find
out
the
neural
network
computation
takes
significantly
longer.
A
As
we
remember,
the
real-time
processing
has
a
requirement
of
the
decision
to
invade
within
30
million
seconds,
but
when
this
data
seems
really
impossible,
but
no
worry,
Tom
Cruise
has
accomplished
important
machines
for
at
least
6
times.
If
I
remain
correctly,
that
there's
a
6,
Mission
Impossible
movies
up
to
2018.
Well,
we
needed
some
small
tricks
to
accomplish
this
impossible
mission.
Please
know
that
this
highlighter
word
effective.
What
does
that
mean
effective
actually
makes
the
result
looks
like
real
time.
A
A
Let's
assume
even
worst
case
the
object.
Detection
task
takes
a
second
to
run
it's
too
terrible
in
real
world,
but
just
for
example,
and
this
task
detects
a
Lego
figure
is
a
person,
that's
awesome
and
it
also
decks.
A
car
is
a
car
of
course,
and
when
it
finishes
actually,
we
already
received
another
cert,
a
frames
based
on
the
s,
so
the
FPS
rate
and
what
we
do
is.
Basically,
we
only
take
the
latest
of
frame
and
compare
the
content
in
the
detected
area
with
a
frame
a
second
ago,
and
we
can
find
out.
A
The
person
area
has
no
change
so
x-ray.
The
result
is
still
effective,
applied
to
the
latest
earth
frame
while
regarding
the
car
or
actually
doesn't
Lightning
McQueen
it.
He
turns
the
face
to
us
and
that
area
changes
significantly.
So
this
detection
is
no
longer
effective
and
we
just
simply
discarded.
A
A
The
use
case
allows
a
little
more
delay,
just
imagining
the
robots
stares
at
you
for
a
while,
as
then
cause
up
out
your
name
isn't
that
cute,
so,
okay,
so
at
one
end
we
are
trying
to
make
the
result,
looks
like
very
quick
now,
I'm
working
on
the
other
end,
to
make
things
work
really
fast
by
trying
to
paralyze
the
data
passes
so
from
the
pipeline.
We
can
obviously
find
out.
A
So,
when
robots
send
the
image
to
any
point,
which
is
the
busiest
service
switch
load
balances
to
one
other
team
parts
based
on
the
input
images,
small,
the
receiving
parts
will
duplicate
the
image
to
other
regime
posts
across
all
nodes
and
all
the
nodes
will
also
run
the
same
pre-processing
tasks
as
they
are
light
weighted
and
they
are
running
across
all
nodes
in
parallel.
So
there's
no
overhead.
A
So,
regarding
the
the
compression
algorithm,
the
h26
for
as
way
better
than
the
g-pack.
Regarding
the
bandwidth,
however,
this
protocol
requires
reliable
connection
like
TCP
and
has
lower
our
lost
torrents.
While
JPEG
has
no
dependence
between
frames,
so
we
can
either
drop
frames.
It
works
better
on
UDP,
okay,
so
we've
talked
about
the
single
problem
and
difficult
problem.
Now
we
come
to
interesting
problem,
it's
very
special
on
ARM
processor,
with
big.little
architecture.
A
So
thinking
about
the
case
when
we
are
scheduling
a
CPU
intensive
task
on
a
little
core
which
runs
slower,
so
this
task
will
finish
much
takes
much
longer
to
run
and
if
we
run
a
accelerate
offload
task
on
a
big
core
that
it
run
faster.
No,
of
course
not
because
it
has
to
wait
until
the
accelerator
completes.
So
this
is
some
optimal
situation
in
a
big
little
architecture
platform.
So,
ideally,
we
want
to
schedule
all
the
CPU
intensive
tasks
on
their
course
today.
The
kubernetes
scheduler
doesn't
support
that
on
this
level.
A
A
So,
first
we
want
to
support
more
videos,
compute
a
stick-on
communities,
so
we
can
use
a
period
container
with
a
host
network
namespace
and
we
want
effectively
shared
memory.
Mccannon
for
sharing
data
across
pause.
I
wish
to
pass
all
opposed
to
be
complicated
on
the
same
notes
with
affinity
rules
according
to
data
locality,
and
we
use
the
simple,
simple
application:
Canada
to
replicate
our
small
input
images
and
also
our
lightweights
pre-processing
tasks
to
further
push
the
pair
isms
and
the
quarter.
Finiti
is
important
for
performance
on
heterogeneous
architecture.
A
B
So,
first
off
this
was
an
absolutely
incredible
talk.
Quite
some
amazing
work
that
you've
been
able
to
do
and
getting
this
system
to
work.
My
question
was
around
load
balancer,
particularly
in
context
of
that
various
neural
compute
stick.
How
do
you
think
about
deciding
what
pot
should
be
running
the
compute
task,
and
how
do
you
know
like
when
the
accelerator
is
active
or
how
long
the
task
is
gonna
run
for
things
like
that.
A
Yes,
that's
very
great
question:
actually
it's
very
complicated
today,
I
even
can
do
that.
I
did
all
the
things
manually
by
manually
label
all
these
stuff
to
make
sure
they
are
running.
On
the
same
note,
I
have
a
few
sorts
may
be
imperfect,
but
because
we
have
service
manager,
all
the
accelerators,
so
Service
knows
what
tasks.
What
models
are
currently
running,
and
today
we
can
only
run
one
model
at
a
time
on
the
on
the
stick.
A
So
basically
the
service
can
report
what's
currently
running
on
this
note
and
which,
as
available
model,
can
be
run
on
this
note
so
based
on
that
some
schedule,
knowledge
can
be
aware
of
the
next
pass
to
be
scheduled
and
which
model
is
going
to
consume.
It
can
find
out
a
available
note,
which
has
this
model
able
able
to
run,
but
on
the
other
hand,
we
also
mentioned
it's
important
to
share
data
across
all
the
tasks
within
the
same
data
flow
and
that
the
schedule
will
also
have
to
take
that
into
consideration.
A
Otherwise
you
have
to
copy
over
the
data,
but
if
the
task
doesn't
consume
the
intermedium
data,
maybe
it's
done
matter
because,
because
of
your
replication,
we
already
have
the
input
data
across
all
notes,
so
you
are
free
to
schedule,
but
there's
definitely
a
lot
of
detail
need
to
be
figured
out.
I
think
that's
a
very
great
question
and
for
the
researching
future.
Yes,
please.
C
D
A
A
Yes,
that's
much
simpler
to
build
everything
into
one
binary,
but
the
thing
about
if
we
won't
make
it
more
easy
to
plug,
and
we
don't
want
to
interrupt
a
running
pipeline
by
just
adding
a
new
path
into
the
system,
it
will
be
much
easier,
especially
with
kubernetes
play
by
dropping
a
new
part
into
a
sister
and
criminality
can
basically
effectively
schedule
applaud
on.
We
have
the
note-
and
that's
reason:
I
came
up
out
this
idea
by
try
to
share
the
memory
across
pause
ina.
A
E
C
A
We
can't
make
sure
so
it's
basically
or
insider
parts,
so
what
Canaries
scheduler
works
is.
Basically,
you
can
specify
the
amount
of
CPU
you
want
to
consume
a
you,
an
FIR.
You
can
specify
number
of
cords,
but
I
I,
never
specified
number
of
courses
if
you
want
to
occupy,
but
just
in
general,
specify
like
megahertz
offer
kind
of
things
of
a
CPU
usage.
A
So
basically
that
opens
all
CPU
to
to
the
container
and
very
likely
if
you
schedule
module
ports
on
same
note,
they
will
share
the
same
CPU,
but
as
long
as
I
want
have
the
court
finitary
inside
a
container,
it's
okay
for
the
Linux
scheduler
to
coordinate
between
the
processes
sharing
the
same
same
core.
This
is
never
not
a
perfect
solution,
but,
as
I
said
today,
I
have
enough
resource
or
not
not
a
problem
for
me,
but
if
there's
restore
resources
start
definitely
probably
to
be
solved.
A
A
You
are
asking
I'm
do
anything
about
is
getting
to
schedule.
Part
to
a
node
is
a
question
of
spirit
so
as
I
attached
to
the
accelerators
to
each
other
of
the
nodes,
so
actually
I'm
running
the
accelerator
service
as
like
a
demon
set
which
can
be
a
Provera
container
and
I
will
label
the
nodes
according
to
the
models
loaded
on
like
sorry,
the
accelerators
and
also
define
your
label
myself
and
to
particularly
specified
port
to
be
scheduled
on
a
note.
I
will
also
put
a
fini
rule
on
the
port.
A
A
G
Sorry
about
that,
have
you
thought
about
making
your
nodes?
The
big
little
nodes
actually
seem
like
two
nodes:
sue
Cooper
Nettie's,
so
you
can
then,
and
kubernetes
would
know
to
schedule
them
and
then
you
can
label
I
need
big
cores
or
I
need
little
cores
on
your
pods
and
then
kubernetes
scheduler
could
do
manage
that.
So
you
it
looks
like
you
have
twice
as
many
nodes
yeah.
A
So
you
are
talking
about.
I
can
split
a
node
effective
it
into
two
virtual
nodes
which
only
contains
because
it
which
contains
little
cause
that's
a
great
idea
and
depending
on
complexity
of
a
couplet,
it
doesn't
work
quite
well
if
two
couplets
running
on
the
same
host
and
try
to
let
it
a
split
on
some
resource,
that's
very
difficult,
based
on
currently
in
Premiere
it.
There
are
a
lot
of
details.
I
cannot
hear
because
I
don't
know
person,
we
can
talk
to
some
people
working
on
a
node,
node
couplet,
but
today
it's
difficult.
A
A
H
Basically,
you
can
also
have
like
in
if
it
was
a
data
center
you'd
have
like
a
GPU
that
Deverill
to
be
a
use
case,
or
you
know
something
something
like
some
of
the
vendors
are
doing
like
Google
TP
use.
So
this
is
the
same
thing
applies.
These
are
accelerators
designed,
for
you
know,
machine
learning,
algorithms
I
can
run
faster
and
yes,.
A
Very
smooth,
so
if
the
services
can
be
abstract
in
that
way,
it
can
support
different
kinds
of
accelerate,
including
GPU
and
CPU
or
kind
of
stuff,
but
GPU
and
TPR
working
in
a
different
communication
model
which
may
be
more
effective
than
being
wrapped
over
this
service.
So
that
part
I,
don't
know
so
maybe
it
was
to
figure
out.
C
I
I
A
That's
a
very
interesting
project,
I
used
to
play
and
put
something
on
this
side,
but
it's
difficult
to
say
so.
You
can
find
out.
There
are
a
lot
of
things.
I
build
myself
so,
for
example
like
the
service
using
a
shared
memory
on
kubernetes,
all
the
service
basically
assumed
to
be
exposing
some
ApS
from
network,
while
sharing
memory
across
ports.
I
built
a
few
mccannon
myself,
probably
just
can
be
something
with
the
canary
provide
a
secondary
thing
about
supporting
this
kind
of
computer
stick.
While
kublai
has
a
device
plug-in
model.
A
I
find
it's
difficult
to
use
that
model
to
support
these
kind
of
devices,
because
it
of
its
own
initializer,
nature
and
sourcing
is
about
the
load.
Balancer
now
I'm
duplicating
this
input
image
across
all
nodes,
while
the
the
schooner
service
only
loaded
been
sitting
to
one
port
to
receive
the
input.
So
if
equipment
have
understand
or
blow
the
parents,
maybes
helps
this
scenario
but
I,
don't
think,
there's
a
common
signal,
but
anyway
I
don't
know
so
also
the
scheduler
arm,
I'm,
not
sure
if
it's
worth
to
support
the
like
core
level
scheduling.
J
A
That's
the
interesting
question
at
the
beginning:
kubernetes
supporting
on
arm
is
very
limited,
so
you
will
get
a
lot
of
problems
compared
to
the
x86
platform.
Well,
today
is
a
quite
a
lot
mature
and
you
can
find
most
most
of
commonly
use
the
image
already
built
supporting
the
arm.
So
you
can
pull
down
like
arm-based
images,
but
you
are
still
missing
lot
of
useful
images,
which
only
supports
the
inter
platform
and
the
other
thing
about
almost
about
the
course.
So
the
little
big,
the
bigger
little
architecture
is
a
very
special
and
I.
A
Don't
think
that
says
where
will
support
increment
is
so
I
think
we
have
reached
the
time
limit.
I
will
be
hanging
out
for
a
while,
especially
around
the
post
area,
so
I'd
like
to
have
a
further
discussion
if
anyone
has
interested
and
I
also
want
to
learn
something
from
you.
If
we
have
some
expertise
on
a
computer
vision
accounts.
Anything.
Thank
you.
Thank
you
for
your
tenancy.