►
Description
Protop: A Package Manager for gRPC and Protocol Buffers - Jeffery Shivers, Toast, Inc
This presentation will cover a new open source tool called protop (pronounced “proto p”) which enables developers to easily publish and share protocol buffers across code bases. Protop was created as a lightweight solution for projects that need access to the same protobufs, such as common libraries that span multiple services, or even external gRPC APIs. The tool can be plugged into most pipelines or projects with a minimal footprint, and aims to encourage projects to keep protobuf schemas small, portable, and ultimately separate from their implementations. We’ll see how protop works through a few real examples and examine how it might help accomplish good development practices in the world of gRPC, especially in collaborative and open-source contexts.
A
Hi,
my
name
is
jeffrey
shivers
and
this
presentation
is
going
to
be
about
a
new
tool
called
proto-p,
which
is
a
package
manager
for
grpc
and
protobufs.
So
a
little
background
on
myself.
First,
I'm
a
software
developer
based
in
boston
and
I've
spent
the
last
few
years,
working
at
a
company
called
toast
building
technology
for
restaurants
and
in
the
evenings.
A
I
study
in
the
engineering
management
program
at
tulsa
university
before
I
was
even
interested
in
software,
I
was
composing
music
studying
and
traveling
around
the
world,
doing
artist,
residencies
and
weird
performances
like
this,
and
then
I
happen
to
have
my
first
experience
in
software
through
a
program
called
google
summer
of
code.
If
you're
not
familiar
with
it,
it's
basically
a
program
where
google
sponsors
students
to
spend
their
summer
working
for
open
source
projects.
In
my
case,
I
had
the
opportunity
to
work
for
the
gnu
project
on
a
music
notation
software
called
lily
pond.
A
So
this
isn't
necessarily
for
a
seasoned
developer
in
this
framework.
So
if
you're
brand
new,
then
this
might
solve
some
of
the
technical
overhead
when
you're
just
getting
set
up
for
the
first
time.
So
I
don't
usually
like
to
work
on
projects
that
add
complexity.
To
other
things,
I
found
that
sometimes
new
tools
like
this
can
be
intimidating
sometimes
so
it
will
add
to
our
add,
to
our
tool
set
just
to
solve
some
issues.
A
So
the
presentation
is
going
to
go
a
little
bit
like
this.
We'll
talk
about
existing
development
patterns
with
protobufs
and
what
that
kind
of
looks
like
and
then
I'll
give
an
introduction
to
the
tool
itself
and
then
a
quick
demo
of
a
distributed,
grpc
app
and
then
we'll
end
with
some
key
takeaways.
A
A
It
happens
to
be
really
convenient
in
the
case
of
protobufs,
because
there's
no
copying
pasting,
no
figuring
out
how
to
distribute
them
across
your
system
if
they're,
just
always
there
most
of
us
are
probably
working
in
distributed
systems,
though
so
our
services
are
broken
up
into
a
few
or
many
different
parts
of
the
system,
and
somehow
we
need
to
get
the
protos
if
we're
using
protobus
from
one
service
to
the
next.
So
the
next
step.
A
We're
passing
them
through
via
the
generated
code,
a
slightly
better
version
of
that
is
proto
repos,
where
basically,
you
just
break
the
protos
out
of
your
repository,
but
still
in
most
cases.
Today.
What
I
see
is
that
services
are,
your
teams
are
still
creating
pipelines
where
the
protos
are
just
immediately
being
generated
into
whatever
implementations
or
stubs
might
be
needed,
and
then
those
libraries
are
pushed
and
out
into
the
other
services.
A
A
A
So
if
you
can
think
of
maven
and
npm
are
pretty
direct
analogs
to
this
tool,
but
it's
literally
a
command
line
tool,
so
you
install
it
to
your
machine
and
you
use
it
with
nearly
synonymous
commands
such
as
init
and
publish
and
sync,
in
order
to
move
protos
around
between
projects
and
then
at
the
highest
level.
It's
also
kind
of
a
pattern
for
protobuf
and
grpc
development.
So
I
keep
talking
about
distributing
protos
and
not
their
implementations.
A
A
A
So
you
would
just
say:
protop
init
or
you
can
manually
create
a
protop.json
file,
and
this
is
just
json
format
with
some
properties
that
give
it
give
us
a
way
to
identify
the
project.
And
then,
if
it
has
any
dependencies
on
other
proto
projects,
then
you
give
it
a
dependency
map
and
you
point
to
them
that
way.
A
If
you're
developing
protos
locally
using
proto-p,
you
can
just
use
proto-link-
and
this
is
an
analog
to
publishing
to
your
local
maven.
Repo
in
your
machine
or
npm
link
is
where
the
inspiration
for
the
name
actually
came,
and
it
really
just
does
that.
It
creates
a
symbolic
link
in
your
file
system
and
really
nothing
more
in
order
to
get
the
protos
across
your
projects.
A
But
you
can
also
use
protop,
publish
which
right
now
is
a
technically
experimental
feature.
So
it's
not
on
the
master
branch.
If
you
install
this
from
homebrew
you're
not
going
to
get
this,
but
it's
something
we're
working
on
developing
right
now,
the
alternative
to
that
is
to
just
simply
publish
the
project
to
a
get
repo.
A
This
is
the
easiest
way
and
probably
what
we're
already
doing,
if
you
have
your
individual
proto
repos,
so
all
you
need
to
do
is
just
drop
in
a
manifest
and
then
technically
it's
a
published
proto
package
and
the
last
thing
you
do
in
order
to
get
your
protos
into
the
next
project
that
needs
them
is
you
run
protop
sync
and
this
basically
resolves
them
in
a
certain
order.
So
if
you
say
that
you
want
to
use
the
linked
projects
on
your
file
system,
then
you
add
a
flag
for
that.
A
A
A
So
I
have
a
little
api
here
and
this
is
for
a
ping
pong
service
and
all
it
does
is
it
creates
a
binary
stream
here.
So
we
have
an
rpc
call
called
ping
pong
and
it's
going
to
take
a
stream
of
these
ping
messages
and
return
a
stream
of
messages,
and
so
this
is
what
I
want
to
use
between
my
services
here.
A
So
in
order
to
actually
get
the
protos
into
those
projects,
I
need
to
give
this
project
a
manifest
and
then
link
it
because
I
haven't
published
it
yet,
and
so
I'm
just
developing
these.
So
in
order
to
get
started
with
that
I'll
say:
protop
init-
and
this
is
going
to
ask
me
a
few
questions.
What
the
organization
name
is
so
I'll
say
that
I'm
with
proto-p
and
for
project
name,
I'm
just
going
to
say
the
project
is
called
ping
pong
and
then
the
revision
is
arbitrary.
A
A
A
That
says
that
I
want
to
use
links
on
my
machine,
so
I
want
to
check
if
there's
any
projects
for
the
dependencies
before
checking
the
external
registries
and
then
just
going
through
what
I
need
to
do
with
the
setup
here.
It's
pretty
minimal.
I
also
need
to
just
add
my
proto
sources
from
the
protop
directory,
so
that
the
protobuf
generator
can
find
them,
and
then
I
want
to
make
sure
that
the
protobuf
generator
only
does
that
after
proto
psync
has
been
run
so
that
we're
not
generating
anything
before
everything
is
pulled
into
the
project.
A
A
So
the
project
that
I'm
going
to
depend
on
is
the
organization
slash
project
name
which
looks
like
that,
and
I
do
need
to
give
it
a
valid
version,
even
though
I'm
not
pulling
it
from
any
external
registry
yet.
But
it
would
look
something
like
this,
so
this
project
was
published
on
github.
If
the
api
protos
were
published
there,
I
could
give
it
a
prefix
like
gh,
to
say
that
they're
on
github
and
then
I
can
give
it
the
registry
path.
A
So
let's
go
ahead
and
try
to
run
proto-p
sync
here
and
we'll
do
that
using
gradle
build
since
it's
built
into
that.
So
I'm
just
gonna,
you
can
see
proto
psync
is
running
and
it
looks
like
it
succeeded.
So
now
I
should
see
a
new
folder
in
my
file
tree
here
called
proto-p,
and
if
I
follow
the
tree
down,
I
can
see
that
my
api
has
been
pulled
into
this
project
and
again
it
didn't
copy
them.
A
A
This
and
I'm
going
to
create
a
new
class
called
ping
pong
service-
okay
and
I'm
going
to
have
this-
extend
the
ping
pong
service
implementation
base
and
there's
just
that
one
rpc
that
I
want
to
override
and
it's
this
guy
right
here,
ping
pong
which
is
going
to
take
a
response
observer
and
it's
going
to
pass
back
this
stream
observer.
So
I
want
to
override
this
default
stuff
here
and
I'm
going
to
create
a
new
stream
observer,
and
so
every
time
I
receive
a
ping
message
from
the
client.
A
A
So
I'm
going
to
do
out
of
a
little
log,
and
I'm
just
going
to
use
this
here
and
let's
say,
received
ping,
and
what
I
want
to
do
is
just
increment
every
time
this
happens,
so
I'm
going
to
create
a
variable
called
count,
and
let's
make
that
here,
let's
create
a
topic,
integer
call
it
count,
and
every
time
we
get
a
call
here,
let's
increment
and
get
okay.
So
this
should
log
out
received
number
one:
two
three,
as
we
go:
okay,
that's
just
about
it!
A
A
A
So
what
I
have
set
up
so
far
is
a
simple
node
cli
and
it
really
it
just
gives
me
a
little
prompt
here
and
what
I
want
to
happen
is
when
I
type
this
message
in
ping,
I
want
to
get
back
a
response
from
the
service.
I
want
it
to
send
a
ping
message
using
that
rpc
and
then,
whenever
it
gets
back
pong
as
a
message,
I
want
to
do
a
log
here
and
then
open
up
another
pump,
so
I
can
keep
doing
that.
A
A
So
I'm
going
to
create
a
new
file.
Call
it
protop.json
and
in
the
interest
of
time,
let's
just
go
ahead
and
copy
this,
since
it's
using
the
same
dependencies
from
the
service.
Okay.
So
now
I
should
be
able
to
run
yarn
and
you
see
it's
running
the
command
right
there
and
it
looks
like
it
worked.
Okay,
so
let's
check
the
same
directory
here
and
let's
see
if
our
api
looks
good
yep.
So
this
is
again
you
can
see
this
little
arrow
here
in
visual
studio
code.
A
This
just
also
just
means
it's
a
symbolic
link,
so
it
seems
like
this
is
working
all
right.
So
what
I
want
to
do
is
load
those
protos
into
the
script
now.
So
I
need
a
few
apis
to
do
that.
So
there's
a
grpc
library
for
node,
and
that
is
a
grpc
js
and
then
I
also
need
the
proto
loader
to
actually
load
them
initially,
and
that's
this
guy
right
here
and
I'm
going
to
create
a
little
constant
for
the
path
to
the
protos.
A
And
let's
say
that
that's
the
definition
and
I
can
do
stuff
with
it
and
what
I
want
to
do
is
use
the
grpc
library
now
to
actually
get
a
meaningful
object,
that
I
can
create
a
service
with.
So
I'm
going
to
pass
the
definition
in
here
and
I'm
going
to
call
this
proto
so
that
when
that
happens,
I
can
start
using
this
before.
I
do
anything
else.
So,
let's
pull
the
rest
of
this
code
in
here
and
make
sure
that
nothing
happens
until
we've
loaded
our
protos.
A
And
I'm
going
to
create
an
instance
of
a
client
now
so
we'll
call
it
client,
and
all
I
need
to
do
is
just
follow
the
tree
down.
I
know
that
the
api
is
under
protop
dot
ping
pong,
so
I
can
just
follow
that
from
the
top
of
the
g
here
and
just
say:
proto
p
ping-pong
and
in
the
javascript
implementation
of
grpc.
The
naming
convention
is
just
it's
not
like
client
or
stub
or
anything.
It
just
is
called
service
like
that
so
you're
creating
a
client
stub
of
the
service.
A
I'm
using
that
name
like
that
and
then
the
first
argument
that
it
needs
is
where
the
service
is
running,
so
that
was
on
port
8080
and
then
it
needs
credentials,
but
in
this
case
we're
not
authenticating.
So,
let's
just
pass
in
insecure
credentials
like
that,
okay,
and
so
now
I
should
be
able
to
create
a
open
up,
a
stream
with
a
server.
So
let's
do
that.
A
Let's
say
that
let's
call
it
ping
pong,
because
that's
what
it
is
and,
let's
just
say,
client.ping
pong
like
this,
so
I
should
be
able
to
use
this
to
start
sending
messages
to
the
server.
Now
so
what
was
happening
was
every
time
I
put
this
phrase
into
the
console.
It
was
just
opening
up
a
new
prompt,
but
I
actually
don't
want
to
do
that.
A
A
So
I
have
a
nifty
little
message.
Sorry
function
here,
that's
going
to
do
that
for
me
and
we're
gonna
use
that
to
just
log
out
pong
and
we're
gonna
have
a
little
cartoon.
Cal
say
that
for
us
and
then
I
want
to
open
up
another
pump
so
that
I
can
keep
sending
messages
to
the
server
and
also
whenever
I
close.
A
A
A
A
Okay,
it
still
didn't
like
something
another
little
thing
here.
I
need
to
pass
this
guy
the
new
keyword
so
we're
creating
an
instance
of
the
class.
Let's
give
it
one
more
shot
there.
We
go
all
right,
so
we
have
a
prompt
here
so
fingers
crossed
when
I
type
enter.
I
want
to
see
a
message
in
the
service
saying
that
it
got
another
king
from
my
client
and
I
want
to
see
a
little
cartoon
cow
saying
that
they
got
a
response
from
the
server.
So
here
goes
nothing
there.
A
We
go
so
the
surface
is
counting
the
messages
it's
getting
from
the
client
and
we
have
our
little
cartoon
cow.
Talking
back
to
us.
Let's
try
it
one
more
time
just
to
see
if
it's
incrementing
and
everything's
happening
as
expected,
and
there
we
go.
Our
cal
is
happy
and
our
service
is
bumping
up
the
number
as
we
go
so
there's
our
example
of
a
distributed
grpc
app
using
proto-p
to
share
the
proto
packages
across
projects.
A
You
still
have
to
decide
how
to
organize
and
architect
your
stack
and
once
you
get
there,
hopefully
leveraging
tools
like
this
to
keep
it
as
simple
as
you
can,
until
you
just
have
to
add,
add
things
like
pipelines
and
and
all
that
and
then
it
also
doesn't
deal
with
code
generation.
So
that's
kind
of
the
biggest
thing
for
this
is
it
just
passes
the
protos
around
and
it
doesn't
become,
it
doesn't
add
any
complication
to
the
actual
code
generation
part
of
protos.
It
doesn't
really
get
involved
at
once.