►
From YouTube: Scheduling 68k COVID Tests with Linkerd - Dom DePasquale & Shawn Smith, The Pennsylvania State Univ.
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Scheduling 68k COVID Tests with Linkerd - Dom DePasquale & Shawn Smith, The Pennsylvania State University
In the summer of 2020, Penn State's software engineering team was tasked with building a system for scheduling the testing of students, faculty, and staff for their arrival back to campus in the fall. In this HIPAA-compliant testing and scheduling system, Linkerd was installed because of its reputation for security and observability. As the team came to learn in February 2021, the #1 reason to have Linkerd is to troubleshoot performance issues. The engineering team sent out over 68,000 invites to those returning to campus to log into the system and schedule COVID tests upon return for the spring semester. The large load on the system highlighted where bottlenecks and bugs really existed. In this presentation, members of that team will discuss how they utilized Linkerd to secure, monitor, and troubleshoot the microservice-based system.
A
So
this
is
science,
a
slightly
interesting
title
for
this
particular
presentation.
We
didn't
really
use
liquor
d
to
schedule
the
test,
but
linker
d
greatly
facilitated
our
ability
to
schedule
the
tests
six
to
eight
thousand
coveted
tests
in
a
very
short
period
of
time
and
we're
gonna
kind
of
talk
you
through
how
it
helped
us
troubleshoot
some
problems
and
get
us
over
some
humps.
A
Next,
please
so
a
little
bit
of
background.
So
last
march,
like
everybody
else,
we
were
all
kind
of
affected
by
what
happened
with
the
covet
outbreak
and
since
we
work
in
higher
education,
what
that
meant
for
us
is
that
we
had
to
find
a
way
to
send
all
of
our
students
home
very
very
quickly,
while
trying
to
keep
them
engaged
and
coming
up
with
a
plan
to
bring
them
back
safely
in
the
fall.
A
What
happened
was
we
reached
out?
We
had
a
bunch
of
vendors
that
we
were
dealing
with
for
doing
testing.
We
had
on-site
testing
and
we
had
no
way
to
tie
all
these
things
together.
So
we
quickly
took
a
built,
a
system
to
pull
all
the
pieces
together
to
go
all
the
way
from
testing
test
resulting
to
contact
tracing.
Unfortunately,
we're
built
on
top
of
a
micro
service
infrastructure
and
dom
has
done
a
really
great
job
of
terraforming,
a
lot
of
the
our
actual
backend
infrastructure
out.
A
A
So
what
we
were
doing
is
we
have
a
bunch
of
commonwealth
campuses
and
we
have
the
main
campus
and
we
were
sending
out
test
requests
invitations
for
tests
a
thousand
at
a
time
and
somebody
who
don
was
kind
enough
to
not
want
to
mention.
So
he
called
him
someone,
but
he
knows
who
he
is
chris.
I
wanted
to
see
just
how
quickly
we
could
kind
of
push
the
system
and
see
what
it
could
take
and
again.
This
is
something
that
we
didn't
really
plan
for
large
scaling.
A
A
So
why
linker
d
so
linker
d,
we
we
had
tried
other
service
meshes
previously
and
we
found
some
challenges
in
them.
In
the
configuration
we
couldn't.
We
had.
We
struggled
with
some
of
the
the
tools
that
we
wanted
and
then
I
was
at
kubecon
a
couple
years
back
and
I
went
to
a
presentation
on
linker
d
and
I
saw
how
easily
it
installed
how
smoothly
things
went,
and
I
immediately
texted
don
the
pr.
A
The
person
that
was
aforementioned
in
the
previous
slide
was
also
in
the
presentation,
also
texted
don
dom,
and
what
we
recognize
is
that
you
know
with
linker
d,
we
we
got
a
lot
of
capability
with
far
less
complexity.
Mutual
tls
is
great,
I
mean
who
doesn't
like
security.
Free
retries
is
great
because
we
get
tired
of
building
that
into
our
code,
but
the
real
bang
that
we
got
from
linker
d
is
the
observability.
So
observability
really
gave
us
the
opportunity
to
go
in
and
kind
of
visualize
and
see
things
at
a
new
level.
B
B
I
will
mention
quickly
that,
since
we
are
doing
both
running
two
clusters
on
my
laptop
and
the
load
test
tool
on
my
laptop,
there
will
be
some
resource
contention
and
there's
a
good
chance
that
some
of
the
performance
issues
we
see
aren't
necessarily
induced
by
latent
services.
It's
just
a
system,
resource
contention
problem.
B
But
the
way
the
east
and
west
clusters
are
laid
out
on
my
laptop
are
similar
to
what
our
environment
was
during
the
real
production
outage.
Well,
partial
lavish
east
is
representing
what
we
had
running
in
aws
and
west
is
representing
what
is
running
on-prem
at
penn
state
and
the
you
know
unhappy.
Stick
figure
here
with
had
a
browser
launched
from
the
invite
taking
you
to
the
scheduling
system.
B
The
scheduling
app
you
know
in
the
browser
was
calling
the
top
level
back
end
service
demo
service
x,
demo,
service
x,
depends
on
the
on
prem
on
these
three
demo.
Demo
servers
a
b
and
c
so
x
depends
on
a
b
and
c.
You
also
notice
that
a
and
b
depend
on
c
and
then
c
depends
on
two
simple
http
bins
and
demo
service
d.
B
Now
demo,
service
c
is
actually
our
rbok
service
and
which
is,
which
is
why
it's
so
dependent
everything
depends
on
it
from
demo
service
c.
It
depends
on
what
we're
simulating
here
is,
just
you
know,
three
random
little
services,
but
in
reality
it
would
have
been
authentication
and
authentication
and
authorization,
databases
or
services
that
are
out
of
our
control
out.
A
B
B
A
B
B
We
have
all
the
components
in
east:
we
have
the
components
and
a
quick
I'm
going
to
switch
tabs
one
more
time
and
quick
test
here,
just
make
sure
it's
still
running
yes.
So
when
I
call
0.3
I'm
calling
demo
service
x
and
that's
then
calling
making
calls
to
these
three
services
and
demo
services
calling
these
three
services.
B
A
B
So
we'll
see
here
in
in
our
deployments,
we
have
on
the
west
cluster
demo
service
abcd
and
in
our
east
cluster
demo
service
x,
already
we're
seeing
p95
latencies
of
28
seconds.
So
it's
going
bad
already,
and
this
is
what
it
was
like
once
we
sent
out
a
thousand
invites
and
then
all
of
a
sudden,
those
thousand
people
or
so
decided
to
click
on
the
link
to
start
signing
up
for
their
their
test,
scheduling
their
test
all
right.
B
B
These
two
dashboards,
so
here
we
have
demo
service
x
and
we
can
see
now
we
don't
have
a
very
high
request
per
second,
but
our
latency
is
just
terrible
over
here
the
success
rate
panel.
We
seem
to
be
okay
in
reality,
when
we
had
our
problems,
the
success
rate
was
not
100
the
whole
way
across
and
our
latency
was
terrible.
So
we
had
a
mix
of
you
know
the
best
of
both
worlds.
As
far
as
failure
was
going.
B
The
interesting
thing
is
the
reason
we
really
wanted
to
share.
What
we
went
through
here
is:
we
could
see
in
demo
service
x,
the
outbound
traffic
had
high
latency,
so
that
was
something
that
was
would
help
us
troubleshoot,
like
oh
okay.
So
we
have
this
terrible,
outbound
latency,
but
there's
nothing
down
here
in
this
dashboard
telling
us
that
we're
connected
to
anything,
that's
because
we
have
dependent.
B
A
B
So
we
this
is
what
gave
us
our
first
indication
that
it
must
be
happening
on-prem.
So
whatever
we're
talking
to
in
our
other
cluster
must
be
the
problem,
so
we
jumped
into
the
linker
d
just
like
here
we
had
a
separate
linker
d,
dashboard
and
separate
grafana
dashboards
to
dig
into
you
know.
We
started
poking
around
looking
at
all
the
different
services
that
our
service
x
was
dependent
on
and
we
could
see
like.
Oh,
these
are
all
failing
miserably.
B
B
B
B
B
B
B
So
in
while
this
was
happening,
while
we
were
trying
to
scale
out
components-
and
we
were
staring
at
these
graphs
trying
to
understand
what
was
happening-
all
that
kind
of
good
stuff-
we
had
one
of
our
or
maybe
a
few
of
our
other
teammates.
Looking
at
the
code
of
service
c
to
determine
if
there
was
any.
B
I
won't
go
into
the
details
about
why
it
was
the
check
was
there
and
why
it's
no
longer
important,
but
the
moral
of
the
story
is.
It
helped
us
understand
that
we
had
this
extra
code
in
demo,
service
c,
checking
d
for
no
good
reason
and
demo
service
d
ended
up
being
a
service
that
was
single
threaded
and
wasn't
meant
to
handle
this
type
of
load,
and
it
was
also
out
of
our
control,
so
our
load
test
is
currently
scaling
down.
B
A
B
B
B
B
Well,
I
tell
one
little
one
last
little
story
about
how
this
went
so,
the
way
this
happened
in
real
life,
it
was
there
weren't
breaks
like
they're
happening
with
my
load
testing
right.
This
was
just
constant
load
and
constant
users
and
a
team
of
us
frantically
trying
to
figure
out
what
was
happening,
and
you
know
lots
of
lots
of
stress
and
worry
and
all
that
kind
of
good
stuff.
We,
if
we
didn't
have
these
pictures
these,
that
dashboards,
that
linker
d
has
pre
pre-made
for
us
right.
This
is
all
out
of
the
can.
B
B
I'm
sure
we
would
have
figured
it
out
eventually
by
you
know,
looking
at
metrics
coming
out
of
ingress
logs
or
something
like
that,
but
because
we
had
linker
d
and
what
it
gives
us
out
of
the
box,
we
were
able
to
troubleshoot
fairly
quickly
where
the
bottleneck
was
because
of
the
latency
graph
and
the
outbound
traffic
latency
to
kind
of
tell
us
hey.
This
is
upstream
or
downstream,
depending
on
how
you
tell
your
stories
we'll
see
here.
B
It's
now
its
latency
is
super
low.
Now
because
it's
no
longer
dependent
on
that
problem
service,
and
this
is
the
same
type
of
experience
we
had
that
day.
We
got
rid
of
that
extra
check
and
all
of
a
sudden
everything
just
started
zooming
right
along
and
we
were
able
to
get
through
the
rest
of
our
testing
in
a
reasonable
amount
of
time
are
testing
our
invites
to
schedule.
The
testing.
B
This
load
test
is
almost
done.
Why
don't
we
just
wait
to
see
it
complete
there
we
go
so
this
is
demo
service
x,
our
top
level
service,
that
the
user's
browser
is
directly
connecting
to.
We
can
see
our
latency
now
is
much
much
lower
in
a
much
more
reasonable
range.
In
fact,
our
p99
is
two
seconds
which
is,
I
would
say
well
compared
to
60
seconds.
It's
super
good.