►
From YouTube: Lightning Talk: Honey, I Broke the Things: Debugging Gray Failures in Production! - Radha Kumari
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Lightning Talk: Honey, I Broke the Things: Debugging Gray Failures in Production! - Radha Kumari, Slack
Migrations are one of the most challenging tasks we do as infrastructure engineers.
These are sometimes long, tedious and come with many technical challenges of their own.
At Slack, we switched from HAProxy to Envoy Proxy for all ingress traffic. Overall, this migration was a success, and did not cause any downtime, but even so, we ran into several interesting edge cases that caused minor problems, such as failing a small percentage of requests, or increasing latency for requests, or sometimes an unhappy bot.
Troubleshooting these sorts of 'gray' failures can be difficult, so this talk will discuss some of those facepalm moments: how they were detected, steps taken to investigate them, and how they were solved.
Takeaways from this talk include a specific set of approaches for debugging such problems with Envoy Proxy and other web proxies that we learnt via these events along with some engineering practices that eases the stress during a large migration.
A
Hi
everybody,
my
name
is
Radha
Kumari
and
I'm,
a
Staff
engineer
on
the
Denmark
engineering
team.
At
smack,
we
look
after
inverse
load
balancing
TLS,
SSL,
DNS
CDN
service
networking,
among
many
other
things
in
2019,
our
team
decided
to
migrate,
Orla,
Ingress
load,
balancing
tier
from
proxy
to
onboard,
mainly
due
to
some
operation
overheads
with
Azure
proxy
and
some
great
features
that
onboard
proxy
offers.
This
talk
won't
go
into
more
details
about
why
we
decided
to
migrate
and
how
we
did
it.
We
wrote
a
detailed
blog
post
on
that.
A
Please
see
the
link
below
two
years
and
15
days
later,
this
migration
was
officially
marked
as
done
in
next
couple
of
minutes,
and
so
we're
going
to
talk
about
some
of
the
oops
moments.
During
these
migrations
steps
we
took
to
troubleshoot
and
mitigate
and
a
few
key
to
the
ways
towards
the
end.
So,
let's
get
started
during
our
web
load
balancing
tier
migration,
we
noticed
an
elevated
5xx
rate
from
our
web
API
endpoints.
This
was
detected
internally
and
we
started
digging.
A
We
try
was
among
the
first
things
we
looked
at
and
I
was
sure
we
had
retrace
configured
on
connect
failure
in
our
config.
With
that
Assurance
we
moved
on
and
tried
a
couple
of
things
to
get
that
5xx
error
date
done.
Unfortunately,
none
of
those
helped
One
Fine
Day,
one
of
the
team
member
noticed
that
there
was
a
typo
in
our
configuration
instead
of
retrying
on
connect,
hyphen
failure.
We
were
trying
on
connect
underscore
failure.
A
Now
the
question
is:
how
did
that
change
faster
testing
or
code
review
in
the
version
of
onboard
we
were
running
back?
Then
we
try
on
config
field,
it's
just
a
free
form
text
which
will
ignore
any
unknown
value
like
like
connect,
underscore
failure.
Every
time
we
change
anything
in
our
config.
We
do
a
validation
to
ensure
the
new
config
is
valid
before
we
replace
the
old
one
using
Envoy
validate
mode
since
Envoy
ignored
the
unknown
value
in
the
config,
as
the
config
was
valid.
This
change
missed
our
testing.
A
To
add
to
that,
we
had
the
same
typo
at
other
places,
and
tests
were
based
on
that
wrong
string,
which
is
another
reason
how
our
tests
failed
to
catch.
This
Envoy
wasn't
doing
any
validation
of
the
config
value.
This
Behavior
was
later
fixed
in
onward
version,
11
0.,
so
we
corrected
the
typo
in
our
configuration
to
fix
the
regression.
Yes,
we
also
fixes
our
tests
to
ensure
this
never
happens
again.
A
Next
up
this
was
during
our
internal
web
API
tier
migration.
We
had
just
finished
migrating
25
of
the
traffic
to
Envoy
in
AWS
region,
whereas
the
rest
25
were
still
going
to
achieve
proxy
in
the
same
region
after
12
hours
into
that
change.
Our
customer
experience
team
got
a
few
tickets
from
one
of
our
customers
that
they
are
getting
HTTP
response,
code,
404
or
some
API
endpoints,
but
only
for
25
of
the
request
they
were
sent.
A
This
will
soon
escalated
to
us
and
we
could
see
that
error
match
the
timing
of
the
rologs
we
did
upon
taking
when
we
found
that
the
request
was
appending
default,
SSL
Port
443
to
the
HTTP
host
header,
which
is
uncommon,
but
not
unsupported.
We
didn't
account
for
this
behavior.
During
the
migration
process.
Our
Edge
onboards
were
configured
to
match
only
on
the
hostname
and
not
on
the
offended
Port,
resulting
in
404,
as
onward
was
unable
to
find
an
outmatch
for
this
type
of
requests.
A
Now,
how
did
this
work
in
a
geography
answer
is
simple.
We
didn't
have
any
host
or
authority
based
routing
rules.
We
accepted
everything,
so
we
wrote
back
the
migration
to
ease
customer
Pane
and
later
on,
made
a
change
and
our
Edge
config
to
match
on
both
post
and
host
colon
Port.
Next
up,
this
was
by
far
the
most
difficult
issue
we
have
had.
Our
desktop
team
noticed
12
extra
latency
for
every
API
record
request
across
all
slack
lines.
A
This
issue
is
related
to
us
and
we
started
investigating.
We
tried
a
couple
of
things
in
terms
of
time
marks
and
such,
but
nothing
worked.
So
we
enabled
tracing
at
the
envoy
layer.
This
was
the
Turning
Point.
As
with
traces,
we
discovered
that
extra
time
was
taken
before
request.
Processing
starts
and
TLS
happens
before
that.
With
that
information,
we
started
looking
at
all
things
TLS
in
our
configuration
and
noticed
a
bug
in
our
PLS
session
ticket
case
container.
A
It
is
used
to
speed
up
TLS
handshakes,
which
improves
end
user
latency,
especially
for
the
customers
connecting
from
Asia
Pacific
region
in
case
of
slack
I'm,
not
going
to
talk
about
how
to
set
up
and
how
to
use
it
in
a
secure
way
as
it's
out
of
context.
So
going
back
to
the
issue
I
mentioned
a
minute
ago,
we
noticed
four
percent
external
latency
for
every
API
request
and
we
traced
it
down
to
a
bug
in
session
ticket
Keys
configuration
the
bug
goes
closely.
A
Sometimes
the
key
is
that
we
were
using
to
generate
these
session
tickets
were
not
synchronously
rotated
across
all
onwards.
This
means,
sometimes
the
request
would
come
with
a
key,
but
onward
doesn't
know
about
yet
and
therefore
required.
A
full
TLS
handshake,
ensuring
synchronous,
key
rotation
across
all
on
voice
fix
the
latency
issue.
A
With
that,
let's
look
at
some
of
the
things
we
learned
along
the
way.
Don't
underestimate
the
power
of
retries
and
timeouts.
We
have
mistakenly
missed
these
in
every
migration.
First
and
foremost,
aim
for
parity
parity
is
more
important
than
added
feature.
It
reduces
the
number
of
variables
you
roll
out
depends
on
keep
drawback
or
reward
plan
fast
and
simple.
A
Thank
you,
evaluate
risks
and
manage
expectations
accordingly,
in
any
big
migration,
there
is
always
the
risk
of
causing
outages
and
incidents,
and
those
are
always
disruptive,
especially
for
customers,
know
that
you
will
break
things
and
make
sure
this
is
communicated
to
the
business
during
initial
planning
basis,
business
or
organization
needs
to
be
aware
why
this
migration
is
happening
and
how
does
that
benefit
the
organization?
What
is
the
reward
plan,
and
how
long
does
the
report
take?
A
This
makes
rollouts
stress-free,
because
the
last
thing
you
want
is
having
to
justify
the
risks
or
the
migration
itself
in
the
middle
of
incident
or
similar
I
love
corn.
This
is
such
a
powerful
tool.
More
than
95
of
the
configuration
features
were
tested
using
card
ranging
from
routing
rules
to
TLS
and
much
more,
but
remember
it's
not
a
one
tool
to
test
all
the
things
with
that.
Thank
you
so
much
for
listening,
hope.
You
learned
something
from
our
theaters
and
we're
hiding
have
a
nice
day.