►
Description
Next Generation Services at Indeed Using gRPC [I] - Jaye Pitzeruse, Indeed.com
At Indeed, we use an internal framework for interprocess communication called Boxcar. Boxcar was developed in 2010 and provides built-in advantages when used with Indeed’s infrastructure. This framework was originally built as a proof of concept and only targeted Java as a supported language. Due to this limitation, it has not scaled with Indeed’s growth and adoption of more and more languages. Recently, Indeed has started to experiment with gRPC as a replacement for the framework. In this talk, we’ll describe our existing service infrastructure and the changes we made in order to support gRPC. We’ll also discuss the strategy we used to migrate existing Boxcar services over to using gRPC. Finally, we’ll compare benchmarks between Boxcar and the new gRPC-based system. Other technologies mentioned in the talk: linkerd for load balancing, opentracing.
About Jaye Pitzeruse
Senior Software Engineer working out of Indeed's Austin tech office for the last 4 years. Today, I own the distributed services framework that drives many of the systems at Indeed. I also work with our Services Infrastructure Group to expand our service capabilities. Such capabilities include supporting HTTP2 based communication, securing existing interprocess communication, and centralizing load balancing between services.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
My
name
is
Jay
I'm,
an
engineer
at
indeed.com
I
work
on
our
service
architecture,
team
and
today
I'm
here
to
talk
to
you
about
how
we're
starting
to
adopt
G
RPC
services
and
facilitate
kind
of
newer
capabilities
inside
of
our
organization
I've
been
with
indeed
for
four-and-a-half
years.
If
you
want
to
reach
out
to
me,
you
can
contact
me
via
email,
Twitter,
github
kind
of
you
name
it
by
and
large
I'm
here
to
talk
to
you
today
about
or
tell
you
the
story
about
migration
in
2009.
A
Indeed,
developed
boxcar
its
proprietary
distributed
services
framework
in
2012,
we
announced
the
performance
improvements
that
offered
to
our
infrastructure
and
then
in
2013
we
did
a
tech
talk
on
the
protocol
and
some
of
the
finer
components
for
the
purpose
of
today's
talk.
You
don't
actually
need
to
know
all
that
much
about
boxcar.
If
you're
interested
in
the
finer
details,
you
can
go
and
check
out
our
tech
talk
on
YouTube
and
with
that
being
said,
you
can
kind
of
keep
a
few
little
details
in
mind.
First
boxcar
was
written
on
top
of
protocol
buffers.
A
It
balances
connections
between
servers
and
not
requests.
There's
one
ongoing
request
per
connection
and
the
load
balancing
scheme
requires
a
fixed
number
of
connections
to
be
pre
established.
In
order
for
things
to
work,
the
scheme
lies
somewhere
between
round-robin
and
in
the
proxy
based
model,
but
by
and
large
it
tends
to
manifest
itself
as
naive,
round-robin.
A
A
Http
and
rest
have
started
to
play
more
and
more
of
a
crucial
role
in
our
infrastructure,
and
so
we
see
about
20
or
so
of
services
in
production.
Today,
these
systems
tend
to
have
a
very
high
latency
involved
when
performing
communication
with
them
and
configuring
these
services
and
getting
them
running
actually
takes
a
fair
amount
of
work
from
our
operation
team,
especially
if
you
require
TLS
in
here,
means
the
communication
in
its
original
implementation.
Boxcar
was
implemented
as
a
library
that
your
team
would
pull
into
your
projects.
A
This
pattern
is
often
referred
to
as
a
thick
client
and
each
web
app
embeds
a
small
load
balancer
inside
of
itself.
There's
one
load
balancer
per
service.
So
if
you
have
an
account
management
service,
it
gives
its
own
dedicated
load
balancer.
If
you
have
a
candidate
data
service,
it
gets
its
own
load
balancer
as
well.
A
A
Additionally,
it
was
hard
to
test
locally
teams
would
have
to
spin
up
local
proxies.
Make
manual
manually
configured
requests
things
that
just
were
very
slow
to
get
going
as
with
any
kind
of
iterative
solution.
We
look
back
that
we
look
back
over
our
original
implementation
and
see
what
to
see
what
improvements
we
could
make.
One
thing
we
could
do
is
the
couple
the
boxcar
implementation
with
our
web
application.
A
This
would
allow
us
to
write
web
apps
in
any
language
that
we
wanted
to,
and
the
load
balancing
would
be
encapsulated
by
the
sidecar
process,
and
so
that's
what
we
started
to
consider.
It
went
and
developed
a
sidecar.
This
is
a
very
popular
pattern,
starting
to
emerge.
Canonically,
indeed,
kind
of
refers
to
these
as
Co
processes,
which
are
a
little
odd,
but
even
companies
like
Microsoft,
boxed
and
Netflix
have
all
talked
about
Co
processes
running
in
their
infrastructure,
as
well
as
a
sidecar.
A
While
indeed
needs
to
continue
to
maintain
backwards
and
forwards
compatibility
on
the
buyer
protocol,
we're
now
able
to
control
the
release
cycles
a
little
bit
better
by
going
and
actually
manually,
deploying
these
sidecar
processes.
This
makes
sure
this
makes
it
so
we
can
have
the
sidecar
pickup
required
features
by
a
certain
date
and
push
that
architecture
a
little
bit
further,
because
this,
because
our
engineers
are
historically
bad
at
naming
things,
we
obviously
named
this
coprocessor
sidecar
in
its
original
implementation.
A
By
introducing
sidecar,
we
were
able
to
solve
some
of
these
problems.
Most
languages
have
native
HTTP
clients,
which
made
it
really
easy
to
communicate
with
clients
no
longer
had
it.
Working.
No
clients
no
longer
had
to
worry
about
the
specific
implementation,
details
of
load,
balancing
and
pipe
languages
like
Python
and
PHP
could
easily
get
started
with
this
system.
The.
A
Additionally,
one
of
the
other
things
that
kind
of
got
introduced
as
we
introduced
the
sidecar
process
was
some
more
development,
toil
custom
library
and
Python
was
written
that
encapsulated
the
logic
for
speaking
with
sidecar.
This
took
a
protocol
buffer
file
and
code
generated
a
small
footprint
for
the
consuming
application.
This
was
yet
another
thing
that
we
needed
to
maintain
and
something
that
we
needed
to
iterate
on
as
time
went
on
and
so
acknowledging
that
indeed,
has
been
growing
and
starting
to
adopt
new
languages
and
new
technologies.
A
We
really
wanted
to
consider
our
survey
reconsider
our
service
architecture.
Over
the
summer,
a
few
of
my
friends
and
I
went
and
went
through
an
innovation
rotation
where
we
started
to
reimagine
some
of
these
components.
An
innovation
rotation
is
just
a
small
three-month
block.
We
were
able
to
work
on
something
that
we
find
valuable
to
the
company
and
then
present
our
findings
at
the
end
of
that
three-month
block
in
our
innovation
rotation.
We
set
out
to
do
three
things.
One
was
improve
restful
services
that
indeed
make
us.
A
A
At
the
end
of
the
summer,
we
had
tested
and
monitored
the
overhead
of
an
HTTP
2
connection,
and
now
all
of
our
Java
processes
support
HTTP
2
as
means
of
communication
out
of
box
the
service
master.
We
wanted
to
establish
criteria
that
we
wanted
in
side
of
the
solution,
evaluated
the
various
options
on
the
market
and
ultimately
wound
up
selecting
one
in
the
end,
and
so,
in
our
considered
view
architecture.
We
wanted
something
to
look
a
little
bit
more
like
this.
A
Now,
as
we
started
to
work
on
this
v2,
one
question
I
started
to
ask
myself:
was
you
know?
How
might
we
migrate
our
existing
infrastructure
to
it
and
sidecar
posed
a
very
interesting
position
in
this
process,
but
before
we
could
leverage
sidecar,
we
needed
to
make
a
few
improvements.
One
we
needed
to
improve
performance.
Having
multiple
network
connections
over
to
a
single
sidecar
process
was
very
crafty
and
using
a
text-based
protocol
was
a
little
non
optimal
for
the
types
of
requests
we
were
performing.
A
We
wanted
to
remove
the
toil
for
adding
new
languages,
so
no
more
of
these
custom
libraries
that
people
would
have
to
go
and
write
or
maintain
no
more
code.
None
of
that
and,
lastly,
we
wanted
to
keep
in
mind
that
we
wanted
to
treat
this
process
as
an
intermediary,
as
we
migrated
so
going
back
and
reevaluating
our
original
solution.
A
quick,
optimization
comes
out
of
box
of
using
HTTP
2.
A
By
using
this,
we
now
have
all
of
our
requests
going
over
a
single
TCP
connection
there
multiplex,
so
we
don't
have
to
worry
about
spawning
up
additional
ones
as
needed,
and
this
makes
much
more
efficient
use
of
our
network
space.
The
other
thing
it
does
is:
it
facilitates
the
use
of
a
binary
protocol
rather
than
the
text-based
one.
So
we
didn't
have
to
go
through
and
basic
c4
encode.
All
of
our
requests
that
we
made
as
we
communicated
with
sidecar.
A
A
Our
sidecar
process
is
written
and
go
and
so
goes
implementation
of
G
RPC.
Has
this
nice
little
feature
of
an
unknown
service
handler
for
those
that
you
that
don't
know
the
unknown
service
handler
is
invoked
whenever
a
service
is
not
discovered
on
the
target
server
from
there?
We
can
take
the
request
and
parse
out
the
components
such
as
the
service
that
we're
calling,
as
well
as
the
method
that
we're
invoking
and
then
relay
that,
according
accordingly
to
the
various
boxcar
services
that
we're
calling
with
that
solution,
we
were
no
longer.
A
We
no
longer
needed
to
maintain
our
custom
side.
Car
implemented
a
sidecar
Python
library,
but
as
I
started
to
go
and
get
more
and
more
client
languages
supported
in
add.
Indeed,
I
found,
adding
all
the
dependencies
to
my
local
box
to
be
a
little
crufty.
Each
language
required
their
own
set
of
dependencies
to
be
able
to
compile
and
generate
source
code
and
indeed
supports
five
out
of
box
Java
Go,
Python,
PHP
and
node.
A
A
Yep
we
do
all
of
our
cogeneration
inside
of
a
managed
build
system.
So
it's
able
to
encapsulate
some
of
those
dependencies,
but
when
you
want
to
add
a
new
support,
you
don't
want
to
go
out
to
every
build
sort,
server,
add
every
dependency
and
go
from
there,
but
yeah
that
Couzens
actually
stands
up.
The
docker
images
can
do
it
in
parallel,
pull
all
the
information
out
and
then
have
all
of
your
client
libraries
ready
for
you
and
available
I'm
going
to
skip
that
video,
like
I,
said
as
we
wanted
to
evolve
and
iterate
on
sidecar.
A
We
wanted
to
keep
in
mind
that
we
wanted
to
move
to
a
service
mission
later
on.
Some
of
the
key
things
to
remember
about
a
service
mesh
is
that
it
helps
encapsulate
a
lot
of
business
logic
that
you
don't
want
to
encumber
your
applications
with
things
like
circuit
breaking
is
really
easy
to
get
configured
in
applications,
but
as
you
go
and
start
to
support
multiple
libraries,
you
find
every
circuit
breaking
library
has
their
own
way
to
configure
different
feature
supports
so
on
and
so
forth.
A
A
We
now
have
smaller
and
dumber
clients,
and
our
pattern
starts
to
look
a
little
bit
more
like
that
service
mesh
that
we
had
targeted
originally
from
here.
We
can
a
be
test.
Both
the
service
mesh
in
our
existing
solution
make
sure
performance
is
on
par,
make
sure
requests
are
being
relayed
completely
certain
services
that
are
read-only.
We
can
actually
do
dark
traffic
tests
where
we
work
a
request
and
just
throughout
its
response.
A
Again
revisiting
some
of
those
core
concepts
and
benefits
of
using
a
service
mesh,
all
of
those
business
logic
and
key
implementation.
Details
such
as
circuit
breaking,
load,
balancing
and
service
discovery
are
all
encapsulated
in
one.
You
don't
have
to
write
that
flavor
library
for
every
service
or
every
client
language
that
you
want
to
support
your
request.
Past
request
path
is
consistent,
so,
whether
you're
writing
an
HTTP
or
restful
base
service.
You
have
knowledge
of
how
the
request
was
performed.
A
A
Lastly,
this
gives
you
the
ability
to
centralize
visibility
into
these
request
flows.
Things
like
open
tracing
and
zip
can
play
very
nicely
with
this
implementation,
and
you
can
see
where
requests
fail
along
the
way
by
and
large.
This
is
a
very
easy
integration,
especially
for
developers
to
understand.
All
of
your
communication
is
pointed
out
localhost
and
when
you're
writing
clients,
you
don't
have
to
think
too
hard
about
how
things
happen.
A
So
where
are
we
today?
Indeed,
currently
has
G
RPC
support
in
sidecar
and
has
three
client
libraries
generating
for
that
one
for
go
Python
and
node.js.
We
have
a
bridge
layer
that
lets
us
continue
to
use
boxcar
generated
code
but
perform
all
of
our
communication
over
G
RPC.
This
will
make
it
easy
for
our
boxcar
services
to
migrate
over
to
using
this
infrastructure
later
on.
They
can
continue
to
use
thick
client
options
or
they
can
even
delegate
all
that
information
to
the
service
mesh
layer.
A
Some
of
the
things
that
are
still
in
progress
is
the
full
adoption
of
a
service
mesh
we're
considering
solutions
like
my
sequel,
Redis
MongoDB
as
immediate
adopters.
We
have
data
teams
and
different
development
team
starting
to
stand
up
their
own
proxies
for
communicating
with
various
some
of
these
various
systems,
and
some
of
the
things
that's
we
got
a
little
blocked
on
was
how
to
handle
gr
piecing
in
Java.
Naturally,
when
you
run
your
own
Hadoop
cluster
internally,
the
question
of
how
do
you
deal
with
the
proto
3
library
version
comes
into
place.
A
There's
a
few
solutions
out
there,
the
GRP
CIO
forums
call
out
to
shading
that
library
and
handling
it.
That
way,
I
did
a
little
bit
of
a
compatibility
analysis
between
the
proto,
3
and
proto
2
serialization,
and
found
that
they
were
roughly
compatible
with
one
another
and
there
weren't
too
many
sharp
edges,
but
the
the
kind
of
way
that
we've
been
looking
at
things
lately
has
been
using
things
like
OSGi
and
shading
at
Build
time,
rather
than
doing
it
during
the
source
code,
compilation.
A
So,
let's
kind
of
recap
over
the
slew
of
features:
that's
we
have
run
through
or
the
slew
of
things
that
we
have
talked
about.
We
talked
about
some
of
the
inefficiencies
inside
of
the
Boxcar
framework,
as
indeed
has
started
to
grow
and
adopt
new
languages.
Additionally,
as
I
called
out,
boxcar
requires
a
fixed
number
of
connections.
So,
as
you
hire
on
400
new
engineers,
you
have
1600
new
connections
to
deal
with.
A
It's
not
that
it
wasn't
compatible.
We
found
serialization
so
long
as
you're
still
generating
code
using
the
proto
2
compiler.
Things
tended
to
play
well
with
one
another
as
long
as
you
didn't
wind
up
with
that
protojet
at
3
there's
some
more
like
extensive,
based
testing
that
I
wanted
to
do,
but
I
haven't
gotten
around
to
doing
it
yet.
But
we've
seen
pretty
good
success
with
these
Python
and
you
know,
clients
that
are
using
Protege
to
our
pro
2
3
generated
code
and
interrupting
with
our
existing
boxcar
services
that
are
using
proto
2.
A
A
A
Obviously,
with
the
initial
connection,
establishment
there's
a
lot
more
overhead,
but
ideally
your
application
isn't
issuing
requests
in
the
critical
path.
On
that
initial
connection
establishment,
we
have
background
dependencies
that
are
constantly
pinging
these
services
and
keeping
these
channels
open.
For
us
that
way,
we
can
ensure
that
when
a
request
comes
in,
we
can
have
that
pre
established
connection
for
us.
A
So
our
labs
team
is,
by
and
large,
had
an
incubator
they're,
making
new
projects
on
a
day
to
day
basis
they
use
Python
by
and
large,
and
the
reason
for
that
is
speed
of
delivery,
they're
able
to
prototype
an
idea
and
get
it
out
in
a
matter
of
time
or
in
a
matter
of
much
smaller
time
than
if
they
were
to
use
Java.
Now
they
need
to
get
access
to
existing
production
data
so
rather
going
and
rolling
their
own
system.
A
They
wanted
to
talk
to
box
hard
so
going
and
using
something
like
G
RPC
to
communicate
with
those
existing
services
proved
to
be
promising,
because
G
RPC
offers
code
generation
across
a
larger
variety
of
languages.
We
don't
need
to
worry
too
much
about
adding
our
own
support
for
it
and
then
trying
to
build
that
into
our
timing.
A
lot
of
this
would
just
come
kind
of
naturally,
and
we
can
offer
it
quickly.
I
think
I
got
the
nodejs
implementation
working
in
a
day
or
so
so,
not
really.
An
overhead
they're.
A
A
The
Spring
Framework,
if
you're
using
Java,
has
an
actual
message
convertor
built
into
it,
for
converting
requests
based
on
either
protobuf,
JSON
or
XML,
and
so
you
can
use
protobuf
to
just
describe
your
request
response
models,
but
then
in
add
that
converter,
both
on
the
client
layer
and
on
the
server
layer
to
use
protocol
buffers,
as
you
mean
it's
communication,
but
also,
then
let
testers
go
and
make
natural
JSON
based
requests.
We
use
that
for
our
navigation
system.