►
From YouTube: Efficient IoT with Protocol Buffers and gRPC - Vladimir Vivien, {code} (Beginner Skill Level)
Description
Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18)
Efficient IoT with Protocol Buffers and gRPC - Vladimir Vivien, {code} (Beginner Skill Level)
Building IoT applications that scales with a variety of devices and capabilities, such as network bandwidth, processor, and power consumption, can be a challenge. Furthermore, this can be complicated with a heterogenous development environment with a multitude of languages, communication frameworks, and protocols. This talk discusses how Protocol Buffers, a language and platform neutral serialization mechanism, can be used as an efficient protocol for small embedded devices with constrained bandwidth and power. We will demonstrate how gRPC, along with Protocol Buffers, provides a performant communication framework that can be used across diverse languages and device platforms to create realtime IoT applications.
About Vladmir
Vladimir Vivien is a software engineer and author of the book ""Learning Go Programming"". Currently he works at VMware as a staff engineer in the Cloud Native Application group where he spends time promoting and contributing to open source including Kubernetes.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
A
Also
I'm
into
go
I
write.
A
lot
of
go
ended
up
writing
a
book
on
go.
So
that's
that
so
today
we're
gonna
explore
the
use
of
protocol
buffers
in
G
RPC
for
efficient
IOT.
So
we'll
see
what
that
means.
It
may
be
something
that
you
find
interesting
and
it
may
be,
may
not
be
so
we'll
see
so
to
get
started
the
internet
of
all
things.
It
said
it
turns
out
that
everything
wants
to
be
connected
to
the
internet.
Now,
not
just
traditional
computers.
A
Everything
is
getting
an
IP.
One
of
the
projections
that
I
found
interesting
is
that
in
2020,
supposedly
there
will
be
about
50
billion
of
these
devices
all
connected
to
the
internet.
Doing
what
God
knows
I
don't
know
so.
There's
been
an
explosion
of
chipsets
and
sensors
and
device
platforms
all
claiming
to
do
IOT
multitude
of
protocols
from
you
name
it
and
cloud
providers
and
other
type
of
platforms
providing
services
for
IOT,
so
IOT
is
all
over
the
place.
Now
modern
IOT
service
stack
probably
will
look
something
like
this.
A
Now
the
problem
is
you
get
to
the
point
where
you
have
a
bunch
of
devices
with
heterogeneous
capabilities,
varied
constraints,
requirements,
diverse
OS
platforms,
all
over
the
place,
many
computing
languages
and
sometimes
you're
going
to
need
a
a
translation
layer
for
these
devices
to
talk
to
each
other.
Now
the
proposition
is
what,
if
we
use
something
like
a
uniform
communication
protocol
that
allows
us
to
do
interoperability
between
those
components
and
the
whole
point
of
this
presentation
is
to
show
you
that
you
can
do
that
with
using
protocol
buffers
and
in
G
RPC.
A
So,
let's
take
a
look
at
protocol
buffers
by
itself.
Anybody
know
what
I'm
talking
about
when
I
say:
protocol
buffers,
alright
awesome
right
group,
so
it's
a
language
and
platform
neutral
mechanism
for
binary,
civilizational
structured
data.
So
what
does
that
mean?
It's
open
source
created
that
Google,
efficient,
binary
format
allows
serialization
of
type
data
structures,
support
many
languages,
which
is
very
interesting
from
C++
Python
Java
Go
is
one
of
the
supported
languages
and
it
turns
out
that
protocol
buffers
is
ideal
for
wire
format
and
and
storage.
A
Specifically,
if
you're
doing
Iowa
team
and
we'll
see
why,
so,
you
may
say:
well
what
about
Jason
I
could
do
the
same
thing
with
Jason
and
you
will
be
right.
Jason
would
be
a
good
solution.
It's
simple,
flexible
university
accepted
well-documented
and
has
a
healthy
ecosystem
of
tools
built
around
it.
However,
it
comes
with
its
own
drawbacks:
jason
has
a
weak
data.
Typing
can
be
inefficient
because
we're
using
text
characters
to
the
mark,
the
surface
of
our
data
when
we
use
Jason.
A
So
that
can
be
an
if
you
have
small
devices
where
you
know
sending
a
byte
can
can
can
be
costly
and
clients
can
be
implemented
and
consistently.
So
if
I
give
you
a
piece
of
JSON,
I
gave
her
a
piece
of
Jason.
We
may
end
up
talking
differently
to
that
same
piece
of
of
encoded
data.
Lastly,
versioning
and
versioning
is
important
to
you
and
I'm
talking
about
versioning
at
the
at
the
field
level
or
backward
compatibility
that
could
be
problematic
with
Jason
now
protocol
buffers
and
IOT.
A
Generally,
when
we're
using
protocol
buffers,
we
do
that
in
three
steps.
So
the
first
thing
we
do
is
to
define
an
ideal
and
the
ideal
is
the
interface
definition
language
and
it
basically
declare
structure
to
represent
the
data
that
we
want
to
one
end
code
and
then
we'll
compile
that
IDL
and
the
compilation
step
takes
that
IDL
and
generate
generate
code
for
a
targeted
language,
and
we
can
take
that
code
and
integrate
it
into
whatever
it
is
we're
building.
A
A
The
first
thing
we
do
is
to
define
the
the
IDL
again.
The
IDL
is
the
interface
definition
language
it
has.
It
shows
the
surface
of
the
data
that
we
want
to
define
that
we
want
to
serialize
over
the
wire
and
first
we
declare
a
version
for
that:
a
protocol
buffer
version.
There
are
two
major
version:
4
protocol
buffers,
one
is
proto
2
and
proto
3.
A
For
this
one,
we'll
use
for
the
2,
like
I
said
we
define
a
message
which
the
message
is
a
container
for
the
data
types
that
we
want
to
send
over
the
wires
and,
as
you
can
see,
each
field
inside
that
message
is
it's
typed.
So,
on
both
end
understanding
and
in
the
receiving
end,
we
know
what
to
expect
when
that
data
arrives.
A
Next,
we
we
use
a
tool
called
proto
C
to
compile
that
data
into
to
compile
the
IDL
into
a
target
language.
So
for
my
example,
the
server
is
written,
go
so
I
use,
go
out
and
it'll
take
that
IDL
and
generate
a
generate
a
piece
of
a
type
that
represents
the
the
data
type
that
was
defined
in
the
IDL.
So
you
could
see.
There's
almost
a
one-to-one
mapping
between
the
between
the
generated
code
and
the
idea
that
was
Jenna.
A
There
was
defined
earlier
so
in
this
piece
of
code
I'm
showing
that
in
the
very
top
line
I'm
reading
into
above
for
the
bytes.
That
represents
the
type
that
was
sent
from
the
from
the
device
and
then
later
on.
We
unmarshal
that
that
series
of,
by
into
the
actual
type
that
we
generated
with
the
proto,
see
that
that
gave
us
that
type
that
to
represent
the
type
coming
from
the
server
from
the
from
the
client.
A
A
A
But
you
have
a
Wi-Fi
radio
and
it
supports
full
tcp/ip
and
you
also
get
TLS.
So
if
you
want
to
go
through
the
trouble,
it
probably
should,
if
you're
doing,
production
stuff
go
to
the
top
of
using
TLS
to
secure
your
connectivity,
you
can
now
you
don't
get
an
operating
system,
but
you
can
use
C
or
C++,
or
do
we
know
to
program
that
device
so
programming.
The
device
consists
of
the
three
steps
that
we
we
talked
earlier.
A
We
we
use
the
same
IDL
that
we
use
to
generate
the
types
on
the
server
side,
because
we're
gonna
share
data
between
the
device
and
the
server
we're
going
to
use
the
same
idea
to
generate
the
same
types.
But
this
time
we're
gonna
use,
proto
C
to
generate
C
code,
we're
gonna
use
a
different
plug-in
called
nano
PB,
and
nano
PB
will
generate
C
NC
C,
that
with
a
small
footprint
that
can
run
on
small
devices
like
these,
and
we
see
the
the
type
that's
generated
again.
A
It's
it's
at
C
and
you
also
get
a
bunch
of
serializer
code
to
go
with
the
generated
type.
So
here
we're
using
the
a
function.
Call
to
generate
to
serialize
the
the
the
type
or
the
value
that's
being
passed
here
and
further
down.
We
use
a
a
tcp/ip
client
to
write
that
data
down
to
the
wire
and
send
the
data
to
the
server,
and
this
happened
over
and
over.
A
A
I,
don't
have
a
setup.
It
would
have
been
kind
of
hard
to
to
set
it
up
here,
so
we'll
just
walk
through
how
it
actually
works.
So
this
is
the
device
is
in
the
actual
picture
of
the
device,
just
a
demo
of
running
the
demo
for
the
device,
and
you
can
see
the
temperature
sensor
sitting
right
there
and
it's
plugged
into
the
data
line
for
the
device
and
as
soon
as
the
device
is
plugged
in
its
going
to
start
reading
the
temperature
from
the
temperature
device
sensor.
A
Next,
we
start
grow
fauna.
Hookah
find
out
is
what
I'm
using
to
to
do
to
show
you
the
the
visualization
for
the
data
again
we're
using
a
flex
TV.
So
we
start
our
in
flux,
DB
server
and
once
that's
online.
We
start
our
actual
goal
code.
Now
again,
the
go
code
is
basically
a
very
simple
server
that
pretty
much
opens
a
tcp/ip
port
that
the
device
will
look
for
as
soon
as
it
comes
on
the
same
network,
connect
to
it
and
immediately
start
sending
data
non-stop
until
you
unplug
it
from
the
wall.
A
Now.
This
is
what
you
end
up
with,
and
this
is
a
go
fine
and
showing
temperature
reading
using
that
device
for,
but
I
think
it
was
about
six
hours,
and
that
was
in
my
office
and
it
you
could
see
the
the
the
thermostat
loop
as
the
temperature
rises,
the
AC
kicks
in
and
it
drops
and
then
the
whole
pattern
keeps
repeating
throughout
the
the
morning
and
it
gets
a
little
hotter
and
then
at
the
bottom
is
another
reading
that
the
device
will
send.
A
So
here
we,
you
know
we're
using
protocol
buffer
on
a
tiny
device
running
see
we
send
that
data
to
a
back-end
server
running
go,
and
the
only
thing
we
had
to
do
is
use
the
protocol
buffer
to
compile
the
to
compile
the
to
generate
the
code
that
we
need
to
create.
Our
serialize
errs:
there's
no
translation
code.
We
didn't
have
to
do
anything
crazy.
Everything
just
works
now
an
option
that
I
could
have
done,
but
I
didn't,
but
I
just
wanted
to
point
that
out.
You
could
have
set
up
the
that
particular
device.
A
There's
a
implementation
for
MQTT
client
out
there,
where
you
can
have
used
the
imputed
client,
if
you
so
desire
to
have
a
message,
bus,
a
message,
bus
with
messaging
capabilities
such
as
durability,
etc.
So
you
could
have
I
could
have
done
that
and
put
that
in
the
middle.
But
this
is
optional
and
then
what
you
do
is
you
take
your
you
dump
your
your
binary
reading
of
your
protocol
buffer
types
onto
the
message
onto
a
message
queue
now.
A
All
right,
so,
let's
talk
about
G
RPC
will
cover
at
a
higher
level,
so
universal
open
source,
our
fec
framework
designed
to
create
efficient,
fast
services.
Now
it
uses
protocol
buffer
underneath
the
covers
for
efficient
binary
encoding.
So
you
don't
get
to
see
the
protocol
buffer
directly,
but
that's
what
it's
using
it
so
earlier
when
we
just
used
protocol
buffer
directly,
we
just
defined
messages,
but
when
we're
using
G
RPC
will
not
only
define
messages
in
our
idea,
we
also
define
service
and
methods
for
those
services.
It
relies
on
HTTP
2
for
fast
multiplex
transport.
A
Think
one
of
the
main
interesting
thing
about
G
RPC
itself
is
the
fact
that
it
support
tons
of
languages
11
last
time,
I
counted
so
chances
are
your
favorite
language
is
supported,
if
not
I,
don't
know
so
gr,
FEC
and
IOT.
So
the
example
we
did
with
protocol
but
protocol
buffer
is
basically
something
where
we're
gathering
data
we
sent
it
over
the
wall
and
later
on
with
batch
it
and
do
analysis.
So
that's
one
type
of
IOT
application,
but
another
type
is
where
you
want
to
do.
A
Real-Time
interactivity,
maybe
you
press
a
button
on
a
wall
and
you
want
something
to
happen
immediately.
Well,
when
you
do
that,
it's
you
know.
How
would
you
do
something
like
that
right?
So
again,
if
you
go
on
the
internet
in
your
search
chances,
are
you
gonna
come
across
this
model
where
you
know
it's
prescribed
to
set
up
to
use
a
messaging
with
with
two
topics?
A
One
topic
is
a
request
topic
and
then
set
up
another
topic
for
a
response
topic
for
for
his
fun
topic
and
then,
when
you
get
a
broker
in
the
middle
and
then
you
send
your
data
on
the
request,
you
listen
to
for
the
response,
and
then
you
correlate
based
on
some
idea,
et
cetera,
et
cetera.
Now
that
works,
but
it's
a
lot
of
moving
parts
right.
A
So
let's
take
all
these
out
and
let's
replace
this
with
GRDC.
So
what
gr
FPC
allow
us
to
do
is
basically
use
idioms
from
from
the
language
we're
using
to
make
our
service
call
look
like
a
local
method
call.
So,
let's
see
again,
gr
Fe
see
how
we
use
it.
Generally,
it's
three
steps,
as
you
would
expect,
so
you
define
an
ID
L,
you
compile
it
ideal
and
during
the
compilation
you
get
clients
stubs,
which
are
methods
that
make
your
service
call.
A
In
this
case
it's
a
Google
speech,
feature
text
service
and
then
the
response
when
it
comes
back,
we're
gonna
take
that
response
and
the
response
will
continue
contain
the
transcription
and
we'll
have
the
device
react
to
that
response,
and
all
of
that
happens
in
you
know
seamlessly
using
the
idioms
idioms
of
Python
as
we'll
see.
So,
let's
look
at
the
service
again,
the
service
is
from
Google,
so
we're
not
responsible
for
creating
that
service
was
already
created.
A
For
us,
it
provides
real-time
speech
transcription
it's
available
as
both
rest
and
as
well
as
an
RPC
API
using
G
RPC,
and
it
supports
several
languages.
Go
Python.
C-Sharp
etc
now
the
device
again
it's
a
raspberry
pi
supports
Wi-Fi
Ethernet,
full-blown
OS
and
on
that
PI
we're
gonna
run
Python
with
that
with
a
gr,
PC
client
and
we're
gonna
talk
to
our
speech
service
in
the
backend.
A
So
the
way
we
do
that,
when
we
program
the
device
we're
not
going
to
create
our
own
idea
of
the
IDL
will
come
from
Google
itself,
because
it's
their
service
all
right
now
to
show
you,
the
flexibility
of
G,
RPC
and
protocol
buffers
is
that
we're
gonna
use
proto
C
to
compile
that
IDL
into
Python.
So
we've
we've
proved,
compile
our
ideal
and
to
C
go
and
now
as
an
example,
we're
doing
it
for
Python
and
then
so
we
generate
the
code.
A
But
the
next
slide
shows
that
this
is
an
optional
steps
are
optional,
because
when
you
download
the
Google
libraries
for
Python,
they
already
have
those
stubs
precompile
for
you.
So
it's
an
optional
stuff,
but
you
can
go
through
that
step
and
do
it
yourself,
but
it's
already
it's
already
there
for
you.
A
So
let's
walk
through
the
code,
real
quick.
So
this
is
the
Python
code.
What
we're
seeing
is
is
we're
importing
PI
audio
as
the
the
library
we're
going
to
use
to
do
voice
capturing
and
then
toward
the
bottom,
we're
also
importing
the
Google
cloud
clients
again.
The
the
speech
package
is
where
our
client
lives.
It
includes
the
the
G
RPC
stubs
that
we're
going
to
call
to
to
talk
to
the
back-end
server,
we're
initializing,
our
audio
library.
A
A
configuration
object
and
the
actual
audio
data
that
we've
captured
and
then
it
sends
it
to
the
to
the
back-end
server
and
wait
and
when
the
result
comes
back,
everything
gets
everything
gets
DC,
realized
and
unmarshal
into
a
type
that
was
defining
the
IDL,
and
then
we
take
the
type
and
we
use
it
down
here
so
that
the
device
can
react
to
what
came
back.
So
what
this
shows
is
we're
basically
making
a
call
to
a
remote
service
and
it
looks
like
we're
making
a
local
call
using
idiomatic
Python
call
to
a
remote
service.
A
A
Hopefully
it
won't.
So
this
is
me
playing
with
ok,
so
Internet
will
fail
me
now.
Oh
there
you
go
so
I'm
just
sure.
Come
on.
Okay,
eventually
we'll
play
because
I
did
that
earlier
and
it
kind
of
played,
but
anyway
I'm
just
showing
the
actual
device-
and
this
is
the
microphone.
It's
a
USB
microphone
connected
to
the
PI
right
here
and.
A
A
So
what
did
we
see?
We
saw
that
we
were
able
to
create
we're
able
to
talk
to
several
devices
several
types
of
devices
and
one
was
a
service
sitting
somewhere.
We
don't
even
know
where,
but
we
were
able
to
seamlessly
talk
to
it
using
using
G
RPC
just
by
defining
the
the
IDL
and
compiling
that
IDL
into
local
code,
and
that
we
can
we
can.
A
So
everything
that
I've
discussed
is
is
on.
Github
probably
need
a
little
polish
as
far
as
documentation,
but
everything
that
that
we
we
talked
about
here
is
on
github
and
if
you're
interested
in
looking
that
into
that
kind
of
activity,
because
to
me
it's
fun:
you're,
welcome
to
and
and
I'm
on,
github-
and
this
is
my
my
handle.
So
if
you
want
to
contact
me,
we
can
you
know
we
can
chat
and
that's
it.
I
think
I
got
five
minutes
to
spare.
If
you
have
questions
question.
A
I,
looked
up
so
I
ran
into
flatbuffers
now
the
thing
about
flatbuffers
I'm,
not
sure
if
the
amount
of
support
as
far
as
language
is
as
widely
you
is
as
wide
as
protocol
buffers
now
flatbuffers
claim
to
fame
is
that
they
it's
even
smaller
than
protocol
buffers
and
they've,
been
more
efficient.
I
think
if
flatbuffers
get
to
a
point
where
it
is
well
supported,
then
it
probably
wouldn't
make
even
more
sense
to
to
bring
it
on
unto
smaller
devices.
A
C
Back-End
services
obviously
have
to
be
a
multi-tenant
and
capable
to
solve.
Each
customer,
of
course,
is
seperated
from
each
other
in
the
state
they
are
data
as
well,
so
and
and
the
typical
MQTT
brokers
have
yeah,
have
utilities
for
that.
I
have
a
device
registry
where
you
can
say:
okay,
this
is
device
right
XYZ.
It
has
a.
It
has
a
pop
key
in
a
fingerprint,
and
this
is
this
device,
and
all
the
stuff
is
missing
for
me
here.
Yeah.
A
A
Secondly,
I
want
to
make
it
clear
that
MQTT
doesn't
go
away.
It
just
solves
the
problem,
one
way,
whereas
this
solves
the
problem
slightly
better
because
it
so
you
can
still
do
so.
Gr
feces
does
support
the
it's.
You
can
put
security
around
it
this
one
now
as
far
as
multi-tenancy,
it
depends
on
how
you
want
to
solve
it,
because
on
the
on
the
server,
all
it
is
is
a
HTTP
endpoint.
So
ideally.
D
A
A
C
A
A
Know
no
one
yeah
I,
don't
like
I,
said
the
the
transport
that
I've
came
across
and
that's
why
I
started
looking
through
this
and
I'm
like
well.
Why
isn't
anybody
doing
GRCC
because
to
me
it
it
kind
of
you
know
it's
shorten
or
make
the
the
surface
the
solution
surface
a
little
smaller,
because
all
I
need
is
just
spare
now.
I
should
also
professor
say
that
this
is
only
for
real-time,
interactive
IOT.
If
you're
doing
something
else,
then
MQTT
makes
sense.
A
If
you
can,
if
you're
gathering
information
and
collecting
information
batching
it
and
do
analysis,
and
you
still
want
something
like
a
messaging
service
and
especially
if
you're
interested
in
you
know,
if
you're
looking
for
guaranteed
delivery,
you
know
that
kind
of
those
kind
of
those
kind
of
a
features
so
but
I
don't
know
of
any
platform
that
coalesce
all
of
this
and
make
it
look
seamless,
and
you
don't
even
care
about
what
what
you
know
what
the
protocol
is.
I,
don't
know
if
any
of
anything
like
that
thanks.
C
A
A
A
There's
no
there's
no
there's
no
project,
so
the
only
reason
why
the
only
reason
why
I
was
able
to
run
protocol
buffer
on
the
ESP
devices,
because
I
found
the
Nano
PB
project.
There
are
a
couple
of
others
out
there,
but
this
one
was
the
better
maintain
project,
but
as
far
as
protocol
G
RPC
itself,
first
of
all,
you
only
get
30
to
kill
a
bit
32
kilobyte
to
run
so
no
and
not
to
say
it's
impossible,
but
I
just
didn't
run
into
it.
A
A
A
B
A
A
Fec
is
a
set
of
libraries,
okay
and
as
part
of
the
library,
you
can
write
extension
points
so
when
and
in
these
extension
points
what
you
could
do
is
insert
code
where
right
before
I'm,
just
giving
you
an
example
right
before
it
makes
say
a
Senate
request
out
to
the
server
you
can
your
code
get
gets
called
as
a
call
back
to
do
something
that
thing
may
be
logging.
It
may
be
security,
there's
a
tons
of
other
stuff.
You
can
do
right.
A
Actually
there
is,
there
is
a
I
forget
the
name,
but
there
is
a
a
project
that
is
a
collection
of
middleware
type
code
that
you
can
that's
already
defined,
that
you
can
use
in
your
own
projects.
I
just
can't
remember
the
name
like
right
now
it
escapes
me,
but
if
you
do
a
search
on
google
for
geo,
RPC,
middleware
yep.