►
Description
Houghton Mifflin Harcourt (HMH) has transitioned from a publisher best known for its textbook printing business into a global leader in education technology. But to do so, HMH had to evolve its infrastructure from monolithic to microservices-based architecture and give its development teams a reliable, scalable, and performant IaaS to build on. In this webinar, Robert Allen, Director of Engineering at HMH, will share how Linkerd helps them solve for service discovery, reliable failovers, telemetry, and real time provisioning in its cloud native environment, and enables its developers to make blue-green deployments for continuous application delivery.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
Join us for KubeCon + CloudNativeCon in San Diego November 18 - 21. Learn more at https://bit.ly/2XTN3ho. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
All
right
we're
gonna,
go
ahead
and
get
started.
I
want
to
remind
everyone
that
this
is
a
broadcast
and
I
it's
being
recorded,
so
you
can
refer
back
to
it
later.
Slides
will
be
posted
after
the
broadcast
as
part
of
the
recording
I
want
to
remind
people
that
if
you
do
ask
questions
through
the
Q&A,
we,
those
will
also
be
part
of
the
recording,
so
make
sure
to
ask
them
appropriately.
A
A
He
is
we've
left
through
15
to
20
minutes
for
questions
as
part
of
this
presentation.
So
we
are
happy
to
have
your
questions.
So
let
me
introduce
myself.
Those
of
you
who
are
kubernetes
contributors
have
seen
me
before.
There
were
multiple
hats:
I
am
the
kubernetes
community
manager
for
Red
Hat
Linux,
the
I,
where
I
work
on
our
kubernetes
contributions.
A
I
am
also
currently
the
release
lead
for
kubernetes
111,
which
goes
into
code
slush
today,
but
today,
I'm
actually
wearing
my
hat
as
CN
CF,
ambassador
and
I'm,
an
ambassador
because
I
started
the
I
portland
kubernetes
user
group.
Actually
cloud
native
user
group
plus
I
do
other
things
for
CN
CF
such
as
introducing
this
webinar.
It
is
my
tremendous
pleasure
today
to
introduce
Robert
Allen
senior
director
of
engineering
from
Houghton
Mifflin
Harcourt,
who
is
going
to
be
presenting
to
us
on
using
Lync
Rd
service
mesh
in
production.
A
What
have
you
heard
a
lot
about
service
meshes
lately,
liquidy
sto,
conduit,
others
are
interested
in
how
you
can
make
use
of
this
since
houghton
mifflin
is
already
I
deploying
this
or
has
deployed
this
we'll
find
out.
I
thought
we
would
all
be
really
intrigued
to
find
out
what
that
experience
has
been
like
and
what
it's
done
for
them.
So
welcome
Robert.
Thank
you.
B
B
B
B
B
Who
am
I
well
currently
I'm
the
senior
director
at
houghton,
mifflin
harcourt,
but
in
the
past
35
years
or
so,
I've
I
worked
as
a
auto
technician,
cabinet
maker
systems,
administrator,
Windows,
NT,
3.51
and
and
on
and
software
engineering
for
some
time,
I've
been
active
in
open
source
now
for
23
years,
going
all
the
way
back
to
contributing
to
the
PHP,
my
sequel
and
various
other
PHP
applications.
Since
that
was
my
my
first
language
some
23
years
ago,
I
lead
a
team
here
at
houghton,
mifflin
harcourt
called
the
bedrock
technical
services
group.
B
Our
primary
goal
is
to
make
the
bedrock
platform
safe
for
engineers,
not
engineers
safe,
for
the
bedrock
platform
that
bedrock
platform
has
over
the
years.
You
know
grown,
it's
been
here,
we
began
it.
I
began
it
when
I
started
working
here.
Almost
five
years
ago,
I
began
with
mesas
zookeeper
and
elasticsearch
collecting
the
logs
and,
as
time
went
on,
we
needed
the
orchestration
we
needed.
B
B
B
As
time
went
by
you
know,
I
I
met
William
and
Oliver
mason's
con
four
years
ago,
maybe
three
years
ago,
I'm
not
it's
been
so
long
and
before
they
actually
started
buoyant
and
we
talked
a
bit
about
what
we
wish
we
could
do
with.
You
know.
Finagle
and
ultimately
little
did
I
know
at
the
time
that
it's
exactly
what
linker
D
is
today
a
year
later,
one
of
the
guys
on
my
team
asked
it.
B
Have
you
seen
this
link
or
D,
and
so
looking
at
it
was
exactly
what
we
needed
as
our
services
scaled
out
over
time
and
the
amount
of
applications
that
were
running
in
the
stack.
It
became
more
and
more
difficult
for
us
to
actually
provide
rapid
delivery
of
infrastructure
for
the
350
engineers
and
the
approximately
a
thousand
jobs
running
in
her
cluster
today.
So
without
without
the
programmatic
characteristics
that
were
able
to
build
into
our
routing
with
linker
D,
we
would
not
be
able
to
provide
the
level
of
service
that
we
do.
B
B
Generally
Twitter
was
very
good,
with
Twitter
wrote,
finagle
and
Twitter
has
always
been
I
believe
very
good
with
their
their
delegation.
Tables
are
detached
that
they
use
where
you
can
write
five
lines
of
dtap
code
and
pretty
much
route,
everything
in
your
arsenal
and
do
it
between
a
local
environment
on
a
laptop
all
the
way,
through
your
various
stages
into
production
and
without
ever
changing
those
five
lines,
and
that's
really
the
that's
really
the
power
of
finagle
and
by
extension,
link
or
D.
B
Now,
as
you
drill
into
this,
you
know
some
of
these
d
tabs
are
very
short,
so
you
know
this.
Here's
a
six
line,
D
tab
that
you
know,
routes
service
to
the
admin,
port
and
mesas
for
a
you
know,
and
actually
three
different
well
potentially
to
infinity
when
it
goes
to
the
discovery
announcer,
it's
able
to
parse
that
out
into
an
own.
You
know
pattern.
B
Now.
This
is
a
much
more
complex,
B
tab.
These
actually
go
to
specific
services
to
prevent
any
kind
of
implied
or
implicit
destination
routing
and
which
sort
of
you
know,
as
you
expose
these
things
out
to
production
with,
like
your,
do,
you
have
to
think
about
some
of
the
security
characteristics
and
internally
you
want
to
be
able
to
utilize
the
l5
d
headers.
So
there's
the
the
the
context
deadlines
where
you
can.
B
B
So
every
router
in
linker
D
is
a
metric.
That
is
a
whole
lot
of
metrics,
and
since
we
have
a
thousand
jobs
running,
there
is
an
actual
measurement
for
each
one
of
those
jobs.
Some
of
them
are
not
busy
back
to
schools.
Coming
up
will
see
them
actually
picking
up
but
and
its
end
of
day
in
there,
and
some
of
them
are
busy
but
you're
able
to
get
all
of
these
metrics.
You
can
measure
your
latency,
your
request
volumes,
your
success
rates.
B
We
look
more
for
the
Delta
and
the
success
rate.
So
if
we
see
you
know,
spikes
in
in
error
rates
of
various
500
errors,
that's
when
we
started
learning
and
take
a
look.
There
was
a
bug
in
name
or
D,
which
is
what
provides
the
logic
for
the
routing
in
linker
D.
It
negotiates
the
request
was
zookeeper.
There
was
a
bug
that
we
ran
into
some
months
back
that
after
several
days
it
would
just
lock
up
and
stop.
You
know
resolving
requests
in
those
situations
we
would.
B
B
So
so
the
things
to
consider
we
have
so
we
have
for
the
last
four
years,
we've
used
Aurora,
Apache,
Aurora
and
we've
used
the
zookeeper
server
sets
for
our
discovery,
and
that
has
that's
worked
very
well,
but
we're
in
the
process
now
of
actually
moving
to
console
for
our
service
discovery,
mostly
because
tasks
that
are
running
in
a
maysa
agent
aren't
necessarily
hosts
aware
of.
What's
going
on
with
the
actual
service
health
check
service
registrations
in
console,
we
can
get
a
much
more
granular.
B
We
expect
to
have
that,
probably
in
the
next
three
months,
have
that
wrapped
up
and
and
switching
all
of
our
service
discovery
off
of
reliance
on
zookeeper
and
on
the
console.
It's
not
that
we're
not
hand
tied
zookeeper,
like
you
see
in
a
lot
of
shops,
max,
look
quite
the
opposite
with
it.
We've
had
very
good
success
with
zookeeper,
and
it's
worked
very
well
where
we
do
where
we
have
seen
trouble
is
that
there
seems
to
be
a
bit
of
a
learning
curve
in
general
and
with
350
engineers.
B
It's
not
an
easy
thing
to,
for
six
people
to
you
know,
run
a
cluster
and
provide
you
know,
outreach
continuing
education
on
you
know
working
with
zookeeper
proper
patterns.
It's
such
a,
whereas
with
console
we
found
that
to
be
a
much
easier
thing
for
people
to
pick
up
on
some
of
the
other
things
is
we
we
actually
enriched
the
requests
that
come
in
with
using
linker
day.we
plugins.
B
We
call
it
the
invigilator,
which
is
a
fancy
name
for
a
Proctor.
Basically,
what
it
does.
It
actually
looks
at
the
request
that
comes
in
it
validates
a
jot
token
for
service
requests.
It
validates
that
jock
token
and
then
provides
an
understanding
through
header
injection
of
you
know
what
the
what
claims
that
particular
request
store
has
and
it's
able
to
then
you
know
basically
authenticate
and
then
alter
the
request
downstream.
B
Based
on
those
details,
that
is,
that
is
a
work-in-progress
that
part
of
it
is
perfectly
functional
today.
The
things
that
we're
adding
to
it
more
and
more,
is
being
able
to
allow
a
customer,
a
student,
a
teacher,
a
school
district
cetera
to
choose
to
use
a
specific
version
of
the
API
or
a
UI
and
then
based
on
their
request
in
we.
We
look
at
the
the
authentication
information,
what
user
it
is,
and
then
we
alter
the
D
tab
to
go
to
specific
versions
of
an
application
being
a
beta
or
in
in
the
school.
B
B
They
would
rather
wait
nine
months
with
a
broke,
tap
location
or
a
bug
and
have
that
bug
fixed
and
change,
something
in
in
a
lot
of
cases,
and
so
it
makes
it
very
difficult
to
you
might
have
ten
customers
that
want
fixed
and
twenty
that,
don't
you
still
want
to
progress?
The
ten
you
need
a
you
need
a
really
good,
clean
way
of
having
you
know.
B
It
works
very
well,
it's
a
very
simple
process
of
writing
the
plug-in
and
once
you
get
started
and
truly
understand
the
level
of
things
that
you
can
do
with
that,
it
totally
opens
up
a
whole
new
world
with
Liberty,
and
it's
just
a
simple
jar,
basically
that
you
just
deploy
on
the
systems
and
it
you
know
loads
up
with
the
config.
It's
very
nice
quite
happy
with
that
other
net
I
mean
some
of
the
challenges
you
have
is
really
you
know,
especially
in
a
you
know,
company.
You
know
it's
been
around
so
long.
B
Adopting
a
new
technology
or
a
new
practice
has
to
be
done
through
influence,
and
it's
really
simple,
if
you
can
show
you
know
the
value
and
that's
really
what
we've
done
so
we
have.
We've
shown
the
value
in
being
able
to
using
linker
D.
We've
shown
the
value
of
being
able
to
deploy
applications
brand-new.
B
B
We
don't
have
to
make
changes
to
the
routing
or
Akamai
or
any
of
these
things
because
of
lincolni,
because
it
is
all
programmatic
and
that
there's
a
huge
benefit
to
that,
because
in
the
past
you
know,
infrastructure
would
take
six
weeks
or
six
months
to
get
a
server
and
that's
just
to
go
to
dev,
not
even
production
and
then
the
the
whole
process
of
actually
making
layer.
5
changes
at
you
know
something
like
Akamai
or
you
know
any
of
the
other.
B
You
know
gateway
services
that
are
out
there
that
can
take
months
or
weeks
and
it's
scary,
because
it's
very
easy
to
mess
those
up,
especially
when
you
have
a
paper
trail.
That's
two
weeks.
Long
and
it's
no
longer
in
the
engineers
control
there's
no
simple
way
of
backing
those
changes
out.
Now
we
control
everything
through
a
process.
You
know
of
code,
review,
continuous
delivery
and
we
no
longer
do
layer.
Five
changes
outside
of
the
control
of
the
engineers
and
that
alone
has
provided
far
more
value
than
all
the
other
things
combined.
B
I
think
it's
easy
to
forget
how
much
productivity
you
gain
from
making
a
simple
process
change
like
that,
because
it
just
becomes
part
of
the
day-to-day.
You
know
existence,
but
they're
they're
compounding
it's
like.
It's
the
opposite.
You
know
of
death.
You
know
it's
compound
interest,
it
pays
back
daily
and
it
allows
you
to
do
more
and
more
innovative
things
that
move
forward
yeah.
You
have
to
go
through
influence,
and
you
know
once
you
show
that
value,
it's
a
big
win.
On
the
financial
side.
B
B
C
B
A
B
That's
a
good
question,
especially
given
the
context
of
this
so
in
we
use
Mesa
and
Aurora
to
run
all
the
applications
in
our
infrastructure,
Omni
development,
and
what
that
means
is
that
we
we
run
about
a
thousand
jobs
and
it
grows.
It
grows
roughly
four
percent
every
month
and
of
those
thousand
jobs
they're
shared
on
resources
across
80.
You
know,
AWS
instances
the
port's
change,
they
just
get
random
ports
they're
on
a
random
host,
and
it
changes
constantly
so
anytime,
there's
a
partition
in
the
network
which
is
frequent
in
any
cloud
provider
or
any
network.
B
For
that
matter.
Those
services
get
moved
to
new
hosts
these
things
change,
so
it's
constantly
changing.
That
makes
it
very
difficult
to
actually
route
in
nowhere
to
route.
Your
your
request
to
those
services,
so
what
link
Rd
provides
to
provides
us
with
is
a
way
to
route
all
layer.
Five
requests,
which
is
you
know
you
above
the
network
layer
above
you,
know
the
actual
TCP,
but
really
it's
routing
it,
based
on
the
HTTP
requests
on
the
host
or
to
the
endpoints,
the
URL
of
the
endpoints,
and
what
it
does
is
it
interfaces
with
zookeeper.
B
In
our
case,
it
could
be
anything
to
beach
to
Bernie's
or
any
other
service
that
provides
discovery.
Information
on
your
services,
but
in
our
case
it's
zookeeper
using
the
server
steps
as
they're
called
in
that
realm,
which
tells
it
the
host
and
the
port
of
the
in
service,
and
there
may
be
a
pool
of
nine
of
those
instances
and
linker
D
will
find
the
least-used
and
it
will
route
that
request
to
it.
If,
for
some
reason,
it
fails
and
it's
a
retrial
error,
retrial
being
a
500,
you
know
5-xx
request
error
code.
A
B
B
A
A
B
B
We
weren't
originally
aware
of
this,
and
it
turns
out
that
it's
somewhere
around
50
and
I'm,
not
even
sure
it's
documented
anywhere.
We
discovered
it
quite
some
time
back,
I,
don't
know.
Maybe
four
years
ago
it
was
early
in
our
development,
we're
reusing
the
resources
so
much
on
these
servers
that
we
managed
to
get
AWS
to
expand
that
to
where
we're.
At
now.
B
It's
like
350
load,
balancers
connected
to
a
single,
auto
scaling
group,
but
they
aren't
going
to
give
us
anymore,
and
so
the
only
way
that
we
can
continue
getting
the
level
of
efficiency
of
resources
usage
is
to
stop
using
the
the
elby's
and
alb,
and
so
everything
else
has
really
just
been
a
win
from
there.
We've
gotten
a
lot
more
visibility
and
control
over
the
request
and
much
more
flexibility,
but
it
does
provide
a
little
bit
more
work.
But
it's
not
it's
not
terrible.
A
B
A
B
B
That
way
you
can
aggregate
or
or
drill
into
specifics,
on
each
of
the
routers
for
your
metrics.
But
the
way
it's
captured
right
now.
It
actually
captures
each
router
as
a
separate
measurement
and
that's
not
very
it's
not
very
good
clean
usage
and
it
creates
a
lot
of
sprawl.
But
that's
something
that
at
some
point,
I'll
try
to
get
around
to
proposing
a
change,
cool
and.
B
B
C
A
Thank
you
for
that
answer,
one
more
for
Eric
P
on
chat
and
again
I'm
going.
To
paraphrase
a
little,
so
you
build
all
the
structure
with
linker
D,
since
linker
D
was
released,
we've
gotten
a
couple
of
other
service
meshes
most
notably
histy.
Oh
and
conduit,
have
you
looked
at
those
other
service
mesh
tools
at
all
and
and
if
so,
what
were
your
thoughts.
B
Briefly,
looked
at
them,
there
hasn't
been
really
any
motivation
or
need
to
really
look
for
greener
grass,
and
this
is
it's
really,
you
know,
pragmatism,
linker
D
is
solving
our
problems.
It
has
an
easy
interface
for
us
to
work
with
and
frankly,
I've
got
two
thousand
other
things
that
actually
need
to
be
built
or
replace.
So
until
until
it's
not
solving
a
problem
or
it's
causing
a
problem,
I'm
not
I'm,
probably
not
even
going
to
be
looking
at
least
not
for
a
little
while
yeah.
My
experience
with
the
others
to
date
has
been
very
minimal.
C
B
A
B
Are
currently
not,
but
that
doesn't
mean
it
will
remain
that
way.
We
are
very
interested
in
in
giving
ourselves
in
a
position
where
we
can
give
or
back
anybody.
That's
ever
been
in
a
large
organization.
Probably
knows
that
I
mean
it's
not
easy
to
make.
That
shift
from.
You
know
the
closed-source
wall
garden
world
and
you
know
working
towards
being
a
much
more
open
organization.
There's
a
lot
of
steps
that
go
in
that
there's.
B
There's
the
legal
steps,
the
bureaucratic
steps,
but
then
there's
also
you
know
your
hygiene
steps
with
your
code
with
you
know
how
you,
how
you
expose
that
we
are
working
through
those
things,
but
we're
certainly
I
mean
there's
a
lot
of
areas
that
we
can
help
so
I'm
I'm
on
Twitter
and
github
you're.
More
than
welcome
to
you
know,
if
you
have
questions
or
anything
like
that,
let
me
know.
A
Okay
next
question:
another
one
from
a
deal
follow-up
to
a
deals
earlier
question.
It
seems
like
you're,
using
a
combination
of
tools
for
tracing
service
discovery
and
monitoring.
According
to
the
asker
link,
birdee
now
apparently
supplies
a
lot
of
that
functionality
integrated
with
linker
D.
Are
you
looking
at
consolidating
that
the
very
assertive
tool
sprawl
or
are
you
happy
with
the
current
set
up.
B
I'm
pretty
happy
with
my
current
set
and
the
reason
is
linker.
D
is
just
one
tool
in
you
know
a
lot
of
tools
that
I
have
and,
and
while
there
may
be
a
really
great
solution,
you
know
and
in
a
lot
of
cases
all
of
these
services
or
most
of
these
services
that
we
run
do
have
really
good.
You
know
one-off
solutions,
for
you
name
it,
but
what
I'm
more
interested
in
is
what
can
I
do
and
look
at
the
landscape,
because
ultimately,
I
I'm
not
locked
into
anything
that
we're
running.
You
know
something.
B
If
something
provided
the
you
know,
the
level
of
you
know,
control
and
second
level
scheduling
that
maysa
does
and
it
provides
some
benefit
or
value
that
you
know
far
outweighs
what
it
is
today.
I
then
I'll,
replace
it,
but
and
I'm
feel
that
way
about
anything
in
the
stack.
But
ultimately,
what
I
have
to
have
is
I
have
to
be
able
to
look
at
everything
as
a
single.
You
know
entity
and
be
able
to
understand
it
from
a
distance
I,
don't
want
to
drill
in
and
and
have
to
micromanage.
B
A
C
B
B
One
of
the
things
that
is
planned
to
be
built
soon
is
the
ability
to
the
you
know,
deploy
a
a
smaller
subset
of
a
production
system
with
different
settings,
JVM
settings
or
doling
settings
or
whatever
and
load
test
those,
but
not
really
load
test
them,
actually
run
them
in
production
measure
them.
How
they're
performing
and
tweak
those
settings
in
real
time
as
they're
running
with
you
know
replacing
our
instances.
B
A
B
We
have
a
taper
H,
a
proxy
provides
TLS
termination
based
on
the
hostname,
and
then
we
have
TLS
authentication
setup
between
all
the
linker
D
instances,
both
the
upstream
and
in
the
side,
cars
downstream
on
the
Mesa
stages.
All
communication
you
know
in
and
out
in
between
all
the
points
are
TLS
and
that's
all
managed
through
vault.
B
So
we
run
involved
of
all
issues
that
the
TLS
certificates
based
on
the
hosts
authentication
and
then
you
know
we
authenticate
those
requests
that
way,
really
that
in
the
main
reason
for
that
is
because
we
are
injecting,
you
know,
authentication
information
into
the
request
with
that
invigilator,
and
so
we
not
only
have
to
authenticate
the
request.
We
have
to
secure
the
request.
A
B
We
have.
We
do
believe
that
in
have
found
that
this
has
very
much
to
do
with
having
too
complex
of
a
routing
scheme
internal
routing
scheme,
where
it
was,
it
was
basically
filtering
down
through
too
many
routers
before
it
actually
got
to
the
service.
We've
improved
that
a
lot.
We
still
believe
that
we
can,
you
know,
really
bring
that
down,
but
on
average
we
see
requests.
B
Somewhere
around
total,
maybe
25
to
32
milliseconds
often
times
lower
than
that,
but
it
depends
on
you
know.
What's
going
on
at
the
time,
they've
been
acceptable
now
that
we
made
those
changes,
but
you
do
have
to
be
mindful
of
that
and
plugins
that
you're,
utilizing,
rewriters
etc.
All
of
those
things
can
very
much
impact
your
performance,
so
you
have
to
you
have
to
really
think
about
it,
and
this
is
also
a
big
part
of
why
we,
you
know
why
we
have
Zipkin
in
place
now
and
why
we're
capturing
all
that.
B
B
B
It's
definitely
something
you
have
to
watch
the
one
job
that
you
might
want
to
keep
in
mind,
since
there
is
Java
bass,
is
the
actual
refresh
time
on
on
host
name
resolution
will
cause
you
a
problem
if
you
expect
link
indeed
to
resolve
a
host
name
and
that
IP
address
may
change
it
won't.
Unless
you
go
in
and
actually
set
it
in
the
JVM
argument,
it
won't
actually
refresh
that
host
name
ever
again
by
default,
and
so
you
you
either
have
to
restart
the
service
or
gifts
be
without
until
you
set
that
JVM
setting.
B
Well,
I
would
assume
that,
first
of
all,
if
you're
looking
if
you're
looking
to
you,
know,
build
out
a
service
mesh,
you
should
have
already
spent
a
substantial
amount
of
time,
decomposing
your
services
and
understanding.
You
know
why
you
need
this
at
that
point.
It's
really
understanding.
You
know
what
fits
you
know.
Some
services
are
better
suited
than
others,
for
instance,
some
you
know.
B
He
put
me
on
the
spot.
I
think,
that's
the
you
know,
it's
really
understanding
you
know,
does
link
or
D
fit,
or
do
you
go
down?
You
know
Conda,
you
know.
Are
you
running
mazes?
Are
you
doing
static?
You
know
hostname
discovery.
Are
you
looking?
You
know
some
of
the
cases
it
works
really.
Well,
it
works
really
well
for
us,
but
we
had
you
know
if
we
were
using,
you
know,
say
kubernetes
or
something
we
might
not
be
using
it
I.
You
know,
I,
don't
know
it.
I,
don't
really
have
any
great
response.