►
From YouTube: Binary Protocols - The Current State of Affairs [I] - Jason Smith, Container Solutions
Description
Binary Protocols - The Current State of Affairs [I] - Jason Smith, Container Solutions
With gRPC becoming increasingly popular, Protobuf is also drawing attention. Binary protocols are making a comeback in popularity. We will looks at the advantages and disadvantages of binary as well as how some of the protocols such as Flatbuffers, Capn'n Proto, Protobuf, etc differ.
About Jason Smith
Jason is a backend engineer who has 10 years of experience in building integrations, and yet he still doesn't hate SOAP. He is a devout student of the K.I.S.S. philosophy. He has become focused on microservices, and the best practices for developing in a distributed environment.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Hello,
I'm
Jason,
Smith
I
work
at
container
solutions,
senior
engineer
there.
That's
my
twitter
handle
I,
don't
really
use
Twitter
that
much
decide
from
retweeting
other
people's
stuff.
I.
Don't
really
feel
like
I
have
anything
that
important
to
say
so
today,
I'm
gonna
be
talking
about
binary
protocols,
but
before
I
get
into
what
you
probably
came
here
to
see
I'm
gonna
start
a
little
earlier
at
the
beginning,
we're
gonna
go
back
to
1844,
so
in
1844
the
Telegraph
was
released.
A
A
Morse
code
is
actually
binary.
It's
broken
down
by
a
time
unit
which
is
a
bit
and
each
dit
da
gap.
Everything
else
is,
is
actually
equatable
to
a
binary
value
gets
being
a
one
plus
zero.
It
actually
has
to
be
defined
by
the
gap
following
it
just
follow
along
some
of
you.
People
look
a
little
confused,
so
that
means
telegraphy
was
the
first
binary
protocol
on
the
wire.
It
was
actually
the
first
time
we
were
transferring
data
on
the
wire
with
binary
data.
A
A
So
is
that
all
we
have
to
learn
from
Morse
code,
not
exactly
so
Morse
was
thinking
ahead.
If
you
notice
some
of
these
letters
have
shorter
notation
than
other
ones,
such
as
e
aí
st,
and
he
was
actually
he
chose
those
because
he
felt
like
those
are
some
of
the
more
common
letters
used
when
communicating
and
it
was
so
what
he
wanted
to
do.
Is
he
wanted
to
make
them
have
the
shortest
notation?
He
wanted
the
actual
message
being
sent
on
the
wire
to
be
as
short
as
possible.
A
He
was
being
frugal
with
the
number
of
time
units
he
wanted
to
make.
Like
I
said,
you
wanted
to
make
the
message
as
small
as
possible,
so
there's
actually
name
for
this.
This
is
called
entropy,
encoding,
I,
don't
know
if
any
of
you
are
familiar
with
Huffman
encoding,
it
uses
the
same
principle
and
the
idea
behind
Huffman,
encoding
or
I'm.
Sorry,
entropy
encoding
is
that
you
take
the
most
common
occurring
symbols
and
you
assign
them
the
smallest
values
when
you're
transferring
that
message.
A
So
this
is
so
Morse
code
wasn't
only
the
first
time
that
we
sent
binary
data
on
a
wire.
It
was
actually
the
first
time
we
encoded
data
for
the
smallest
footprint
on
the
wire
as
possible.
So
that
was
the
history
lesson
for
today,
so
also
nothing,
nothing
we're
doing
is
really
new.
I
mean
we've
been
kind
of
dealing
with
the
same
problems
for
a
long
time,
so
we're
gonna
cover
three
things
today.
A
First,
is
how
we
got
to
where
we're
at
right
now,
what
we're
discussing
binary
protocols,
how
buying
a
binary
protocols
work,
and
why
are
we
adopting
them
at
this
time
so
and
hopefully,
if
you're
gonna
walk
away
with
anything
out
of
this?
If
you
already
know
this
information
you're,
probably
going
to
walk
away
with
a
sense
of
smug,
satisfaction,
and
if
you
don't
hopefully
it'll
help-
you
think
about
what
it
means
to
be
writing
microservices
in
a
cloud
native
environment.
A
So,
let's
start
by
looking
on
how
we
got
here
so
to
start
we're
gonna
fast
forward
to
from
1844
to
the
1960s
in
the
1960s,
pretty
much
all
the
way
through
the
1990s.
Our
PC
became
pretty
much
the
dominant
form
of
inter
service
communication
for
a
distributed
system,
but
it
had
problems.
We
had
stubbing
coupling
complexity
and
it
wasn't
very
portable
across
a
lot
of
different
languages
and
most
in
most
cases
that
was
using
binary
data
to
communicate.
A
One
of
the
bigger
problems
with
RPC,
though,
is,
is
it
had
this
idea
that
the
network
was
supposed
to
be
transparent
to
the
developer,
meaning
that
they
weren't
supposed
to
be
aware
that
you
were
communicating
over
a
network
right
now?
We
know
that's
just
not
true.
We
need
to
know
that
we're
working
on
a
network,
but
aside
from
all
this,
even
with
all
these
caveats,
RPC
is
still
widely
used
today
in
distributed
systems.
A
So
then,
in
the
90s
we
got
HTTP
and
it
was
it's
been
widely
adopted.
It's
it's
adopted
because
it's
open
and
it's
easy
to
tool
for
and
there's
a
robust
number
of
tools.
Lots
of
languages
can
write
interfaces
for
it
so
and
it
that
made
it
easily
multilingual,
but
it
led
to
what
we
have.
What
we're
commonly
seeing
today
is
data
in
text
and
the
first
popular
one
being
XML.
A
A
So
next
we
had
JSON
popularly
probably
the
most
popular
series,
ation
format.
We
have
it's
widely
used
in
rest,
it's,
but
it
has
problems.
It's
not
typesafe.
The
servic
server
client
contracts
have
just
recently
become
popular
API
blueprint,
swagger
things
like
that.
Only
recently
have
they
started
to
be
adopted
and
it
was
still
text.
So
it's
loaded
to
serialize,
but
both
HTTP
with
the
text
formats
has
led
to
what
we
commonly
refer
to
today
as
web
services.
It's
a
lot,
it's
it's
it's
it's!
A
So
that's
a
little
history
on
how
we
got
there
now
I'm
gonna
go
into
a
little
bit
on
how
they
work.
I'm,
just
gonna
go
a
general
overview
and
try
to
get
a
concept
of
how
they're
trying
to
reduce
the
message
message
size
by
using
binary
formats.
So
they
try
to
achieve
one
thing:
they
try
to
try
to
make
things
fast,
they're,
trying
to
make
make
it
fast
in
three
different
ways.
In
one
way
is
they
try
to
make?
This
message
is
small,
so
they
transfer
over
the
wire
quickly.
A
They
also
try
to
make
decoding
and
encoding
faster,
meaning
at
the
expense
of
being
small.
They
may
encode
the
values,
but
they're
gonna
try
it
and
but
some
of
them
don't
and
they
try
to
save
it
as
raw
values
that
you
can
actually
map
into
memory
and
I'll
get
into
that
in
a
minute
and
another
way
they
try
to
make
it
fast
is.
If
you
have
big
data
packages,
they
try
to
make
it
fast
by
being
able
to
traverse
the
message
quickly,
so
I'm
going
to
cover
three
primary
ones:
protocol
buffers
flatbuffers
in
captain
proto.
A
A
So
protocol
buffers
was
developed
at
Google.
If
anybody
attended
the
GRP
see,
you
may
may
have
heard
something
about
it.
It's
developed
for
compact
messages,
traversal,
is
done
by
stepping
and
skipping,
depending
on
the
type
of
field
that
you
have.
There
is
some
decoding
that
has
to
take,
has
to
take
place
to
to
produce
usable
values
and
not
all
values
need
to
be
read,
but
the
entire
message
needs
to
be
med,
read
because
protocol
buffers
supports
duplicate
values,
meaning
if
you
have
the
same
filled
twice
in
the
same
message:
it
either
concatenates
it.
A
So
when
you're,
looking
at
Pro,
so
I'm
gonna
look
ahead
at
the
code
protocol
buffers,
it's
really
simple
format.
It
just
starts
with
a
when
you
first
enter
the
message:
it
starts
with
the
field
label,
a
field
type,
then
oh
yeah,
and
then
any
ancillary
data
that
may
go
along
with
it.
Some
fields
have
more
information.
That
needs
to
be
attached
to
it
and
then
the
data
itself.
A
If
we're
looking
at
a
variable
integer,
they
work
a
little
different
same
but
starts
the
same
exact
way.
It's
a
label,
the
type
which
is
of
our
integer
and
then
the
bytes
that
follow
it.
Now,
with
a
variable
integer,
they
do
encode
it
in
a
certain
way,
but
the
the
way
they
delineate
the
end
of
the
field
is
by
the
last
byte
in
the
data
is
going
to
have
the
first.
The
most
significant
bit
will
have
a
0.
That
means
you
know
you've
reached
the
end
of
your
data,
which
means,
if
you're
traversing,
the
field.
A
I'm.
Sorry,
let
me
back
up
here
a
second
so
using
the
length
of
the
word
and
the
length
of
the
variable
integer
and
by
skipping
over
these
bytes.
You
can
actually
tell
where
one
field
begins,
and
then
you
assume
the
next
field
or
one
field
ends,
and
you
assume,
where
the
next
one
begins
and
this
sort
of
stacks
the
the
fields
on
top
of
each
other
they're
completely
independent.
A
So
now,
I'm
going
to
talk
about
flatbuffers
in
camp
proto
together
and
then
I'm
going
to
tell
about
to
talk
about
some
of
the
differences
in
some
of
the
decoding
processes.
So
flappers
was
also
developed
internal
by
Google,
and
captain
proto
was
developed
by
Kenton
vara,
who
is
a
former
developer
for
proto
pub
at
Google,
so
they
both
use
zero
copy,
meaning
that
the
values
come
straight
out
of
your
message
straight
into
memory,
and
there
is
no
decoding
process
involved.
A
They
also
use
a
root
and
hierarchal
table
structures
that
define
how
to
access
the
data,
so
in
vs.
protocol
buffers
will
you
actually
had
type
and
labels
flatbuffers
and
cap
proto
actually
have
a
table
that
defines
how
to
go
into
your
data
and
they
both
require
a
root
table
as
well,
and
you
can
also
use
em
map
to
do
memory,
mapping
of
your
values
straight
into
into
memory.
A
So
the
key
differences
between
these
are
so
flatbuffers
uses.
Indirection.
It
uses
a
two
tables
actually
to
define
how
to
access
the
table.
One
is
the
the
V
table
and
the
other
one
is
another
table.
It
defines
how
to
actually
access
the
data
in
the
table,
but
and
and
can
proto
uses
64-bit
words
and
it
bounds
the
data
in
these
64-bit
words
and
using
these
two
methods
flatbuffers
actually
has
to
be
built
from
the
bottom
up,
meaning
you
have
to
get
your
your
leaves.
A
If
you're
building
a
tree,
you
have
to
get
your
leaves
before
you
get
to
the
root
cap
and
proto
gets
built
from
the
root
down
to
the
leaves
so
I'm
gonna,
look
at
the
coding
captain,
proto
really
quick.
So
with
Captain
proto,
the
first
64
bits
in
a
message
define
the
root
table
pointer
and
it
is
defined
as
and
as
you're
entering
these
first
64
bits
you
get
the
type.
A
Where
is
the
the
data,
the
data
length
and
the
pointers
length?
So
in
this
example,
you
have
a
you,
get
the
type
which
is
a
structure
pointer,
which
is
going
to
be
your
first
couple.
Bits
is
going
to
always
give
me
a
structure
pointer,
you're
gonna
get
the
word
offset
of
how
many
words
over.
Are
you
going
to
be
traversing
before
you
get
to
your
words
of
data
and
you're?
A
So
looking
at
those
words
of
data
it
going
back
to
this,
your
words
of
data
and
cell
3
and
4,
the
words
of
data
are
are
in
are
in
order
in
there
based
on
the
spec,
meaning
that
if
you
give
it
a
value
of
1
and
then
the
next
field
has
a
value
of
2,
they
have
to
remain
in
that
order.
In
these
words
of
data,
if
you
add
more
fields
that
can
only
be
added
with
a
higher
field
value
than
the
previous
values
have
already
assigned
and
you'll
see
why
that
is
in
a
second.
A
A
I
could
explain
it
in
in
any
reasonable
amount
of
time
here,
but
it's
it
actually
uses
labelled
tables
of
indirection.
So,
as
you
can
see
on
the
top
of
the
screen,
where
you
see
the
twelve,
that's
actually,
where
you
enter
your
message,
it
defines
the
v--
table.
The
v--
table
defines
then
the
route
tables
and
the
fields,
and
then
the
fields
jump
back
and
forth.
And
then
you
finally
get
your
data
I,
don't
think
I
could
really
get
through
this
whole
thing
in
just
thirty
minutes.
A
A
So
flatbuffers
in
Kappa
proto
are
their
tree
structure.
They
require
a
route
data
that
gets
traversed
into
lower
branches.
Now,
where
this
helps
is,
if
you
have
big
data
structures,
you
can
you
only
need
sections
of
your
data,
it
makes
traversal
very
quick.
You
can
go
deep
into
large
data
sets
and
get
data
quick,
whereas
protobuf
you
have
to
traverse
your
entire
message
and
read
each
and
every
field
before
you
actually
get
the
data
that
you
need,
but
this
also
comes
at
the
cost
of
more
padding.
A
You're
gonna
get
bigger
messages
because
you're
padding
your
data,
there's
a
lot
more
options.
I
can't
really
cover
all
of
them.
I
chose
the
three
that
I
did
because
they're
the
most
popular
and
it's
more
I
really
want
to
just
give
an
example
and
just
get
people
to
think
more
about
how
binary
protocols
are
being
encoded,
so
they
can
get
a
better
understanding
of
how
it
all
works.
So
you
may
be
thinking
I'm
gonna
talk
about
benchmarks
right
now,
but
that
is
definitely
not
the
case.
I,
don't
really
think
it's
about
benchmarks.
A
So
why
are
we
adopting
binary
protocols
now?
Why
is
it
a
thing?
Why
are
we
talking
about
it
now?
Why
is
it?
Why
is
it
that
is
it's
raised
its
head
again
from
the
days
gone
by,
so
to
get
a
get,
an
idea
of
why
that
is.
We
have
to
go
back
to
our
PC.
It
had
its
drawbacks,
but
we're
seeing
a
resurgence,
but
it's
not
the
same
RPC.
A
It
also
is
well-defined
and
it's
open
and
it's
widely
available
across
languages
and
I'm
sure
they're,
trying
to
port
it
to
more
and
more
languages
every
day,
and
we
have
to
come
to
terms
the
fact
that
micro
services
are
not
web
services,
which
means
we
don't
want
to
write
our
services
as
web
services
anymore.
We
need
to
be
writing
our
services
to
to
handle
to
be
as
minimal
as
possible,
so
I'm
gonna
give
you
an
example.
A
I
don't
know
if
anybody
here
is
familiar
with
the
sock
shop,
it's
a
it's
an
open
source
project
sponsored
by
weave,
so
it
it's
only
basically
made
up
about
ten
services,
but
this
is
all
the
other
traffic
that
is
required
to
get
it.
That
is
happening
in
the
background,
so
it's
not
just
the
sock
shop.
You
also
have
kubernetes
trap.
You
also
have
Prometheus
traffic.
A
You
also
have
open
tracing
going
on
and
you
can
see
this
isn't
just
making
a
request
to
FedEx
to
figure
out
how
much
the
shipping
cost
is
for
a
product
you
want
to
ship.
This
is
constant,
persistent
communication
going
over
the
network
by
the
way
this
is
weave
works.
They
got
a
booth
upstairs
if
you
haven't
checked
out
their
product.
It's
amazing.
A
So
the
reason
we're
doing
this
is
we're
trying
to
shrink
our
footprint.
We
want
to
shrink
it
on
the
network.
We
want
to
shrink
it
in
the
CPU
and
we
want
to
shrink
it
in
memory
and
I
know.
Binary
protocols
helps
us
on
all
three
fronts,
so
I
was
actually
surprised
by
how
many
Americans
are
actually
at
this
conference,
but
in
1992
there
was
a
campaign
manager
for
Bill
Clinton
and
he
famously
said
the
economy
stupid
and
this
James
Carville
I
love
this
statement,
I
think
it's.
It
cuts
right
to
the
issue.
A
A
So
the
point
is
we're
trying
to
move
to
be
cloud
native
in
that
we
are
trying
to
reduce
our
service
footprint.
So
we
stopped
thinking
about
using
api's
and
web
services
and
things
like
that,
and
we
have
to
think
about
being
more
economical
and
productive
forms
of
inner
surface
communication,
and
that
is
primarily
why
binary
protocols
has
become
the
hot
topic
of
conversation
at
this
point
in
time.
A
A
A
A
A
So
all
three
that
I
showed
they
all
do
provide
tooling
to
actually
do
text
output.
It's
usually
based
on
the
actual
message
package,
like
you
did
to
code
the
whole
message
package.
So
if
you
write
it
to
a
file,
then
you
will
decode
it
like
I,
don't
think
G
RPC
provides
the
tooling
to
actually
decode
from
within
your
code
there.
But
if
you
were
to
write
your
your
message,
Pat
your
the
entire
message
to
a
file
I
think
you
can
decode
it
all
them
from
into
text
format,
which
makes
it
a
little
easier
ooh.
A
Yes,
so
I
think
cat
proto,
I'm
pretty
sure
his
little
Indian
protobuf
uses
VAR
integer,
which
is
encode
encoded
integers
and
they
it's
they
it's
little
endian,
but
they
break
it
into
bytes.
So
and
then
they
it
sits
like
this.
That's
your
that's
your
integer!
They
take
it
break
it
up,
put
it
like
this
and
then
they
put
the
zero
at
the
end
and
then,
when
you
decode
it,
you
got
to
put
it
back
over
here
like
this.
So.