►
From YouTube: Keynote: Panic in San Francisco: The Critical Vulnerability That Wasn't - Shane Lawrence, Shopify
Description
Don't miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from 18 - 21 April, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Keynote: Panic in San Francisco: The Critical Vulnerability That Wasn't - Shane Lawrence, Staff Infrastructure Security Engineer, Shopify
In October, the OpenSSL team found a critical vulnerability in an open source library used by millions. They warned that they would disclose the bug and release patch a week later. Their early warning and quick resolution were commendable, but in the intervening days a flurry of speculation and concern set the blogosphere ablaze and Twitter atalking. On release day, some websites promising to report details of the vulnerability struggled to keep up with the traffic as herds of security specialists, developers, and sysadmins-turned-devops-turned-platform-engineers refreshed the page in anticipation. When details became available, many of us started to threat model the bug, evaluating how it might be used to harm our sytems. And most of us came to the same conclusion: it couldn't. The panic subsided, and the distraction arguably cost more than an exploit could have. In this talk, Shane will summarize the vulnerability and some of his team's efforts to prepare for and respond to it, then consider lessons learned from the experience. Attendees will hear suggestions for implementing strong security programs that allow rapid evaluation and response to supply chain threats so they can be prepared for the next vulnerability, whether it turns out to be a major risk or none at all.
A
A
Everybody
in
security
Circle
seems
to
be
talking
about
it,
but
we
don't
have
any
hard
facts
yet
and
in
my
mind,
I'm
trying
to
figure
out
what's
going
to
happen
next,
it's
pretty
uncommon
for
openssl
to
have
an
issue
like
this.
It's
an
example
of
why
we
encourage
you
not
to
roll
your
own
crypto
they're,
just
less
likely
to
make
mistakes
than
most
of
us
are
and
as
I'm
trying
to
figure
out.
What's
going
to
happen
next,
it's
bringing
me
back
to
the
last
time.
I
dealt
with
a
serious
openssl
vulnerability.
A
What's
going
to
happen
next
I
switched
to
another
tab.
Finally,
the
announcement
comes
I'm
thinking
to
myself
as
I
read
the
description
what's
vulnerable.
How
can
an
attacker
control
this?
What
could
even
fit
in
an
instruction
that
size
I'm
starting
to
realize
after
all
of
this
work?
And
all
of
this
worry
that
it's
a
dud?
A
Oh,
when
we
started
looking
at
this
we're
in
the
dark,
but
we
can
shed
some
light
on
it
now.
It
actually
started
way
back
in
September
of
2021,
when
a
new
major
version
of
openssl
came
out,
a
great
presenter
once
told
me
never
to
show
code
during
a
talk
so
I'm,
sorry
for
this,
it's
C,
that's
probably
a
capital
crime
just
bear
with
me.
So
this
is
a
bug
in
a
puny
code.
Decoder
now
punicode
is
a
way
of
taking
stuff.
A
That's
not
in
the
Latin
character
set
or
not
English
and
making
it
sort
of
english-ish.
It
made
it
more
familiar
and
easier
to
deal
with
for
the
people
who
developed
this
software,
it
decodes
text,
it
moves
it
to
a
new
place
in
memory
and
on
the
highlighted
line.
We
can
see
that
if
it's
written
more
than
the
maximum
it
was
supposed
to
then
stop
otherwise
keep
going.
A
So
more
than
a
year
passes.
It's
now
October
of
2022
and
the
vulnerability
is
first
discovered.
My
team
and
the
public
don't
know
anything
about
it
at
this
time,
but
our
omniscient
narrator
does
so
we
can
go.
Take
a
look
at
it.
This
is
taking
a
chunk
of
memory
and
in
this
case
one
chunk
too
much
four
byte
unsigned
integer
and
it's
moving
it
into
another
place.
Now
it's
supposed
to
fit
in
that
decoded
buffer,
but
it's
taking
up
just
a
little
bit
too
much
space.
This
is
called
a
buffer
overflow.
A
On
the
other
hand,
if
the
Overflow
is
used
by
something
else
later,
but
it
just
reads
that
memory-
it's
probably
just
going
to
cause
it
to
crash,
and
we
call
this
a
denial
of
service
or
dos
there's,
an
additional
failure
mode
for
dos
that
they
found
out
a
little
bit
after
this,
where
you
can
actually
fill
that
payload
with
a
whole
bunch
of
dots,
and
that
makes
it
way
more
likely
that
you
can
exploit
this,
but
well
you're.
Only
overflowing
it
with
dots,
so
there's
no
risk
of
rce.
A
You
only
have
a
higher
likelihood
of
Dos
and
again,
we
didn't
know
any
of
this
at
the
time.
The
very
first
that
anyone
heard
of
it
was
on
October
25th.
The
openssl
project
team
announces
a
new
update.
They
say
it's
a
security
update,
it's
the
highest
severity
level,
which
is
critical
and
right
from
the
start.
We
had
some
issues.
There
were
some
confounding
factors
that
made
this
harder
to
deal
with,
for
when
they
announced
it
without
telling
any
cve
numbers
or
a
name.
A
There
was
a
sort
of
unofficial
nickname,
but
people
were
reluctant
to
use
it,
and
so
everyone
that
I
was
coordinating
with
was
just
calling
it
The
open,
SSL
update,
which
made
it
difficult
to
communicate
because,
as
it
turns
out
on
the
exact
same
day,
they
were
releasing
another
open,
SSL
update
for
a
different
version,
but
it
was
totally
unrelated.
There
was
nothing
whatsoever
to
do
with
this
vulnerability.
A
We
got
our
first
notifications
from
Paid
vendors
and
this
activity
and
this
work
and
this
effort
that's
going
into
it,
keeps
increasing
and
increasing
and
increasing
until
we
finally
see
the
peak
anxiety
the
morning
after
Halloween.
At
this
time
there
were
conversations
on
social
media
I'm,
seeing
a
lot
about
it
on
Twitter
and
well.
Mastodon
was
mostly
silent,
as
was
the
custom
at
the
time,
but
people
were
panicked
elsewhere
and
I
think
this
comment
on
a
security
blog
actually
captured
the
Zeitgeist.
This
person
described
it
as
nasty
nasty
nasty.
A
Of
course,
this
is
before
we
have
any
information
about
what
it
is
or
what
it
does
so
November
1st
comes
around.
Finally,
we
get
the
details.
It's
downgraded
to
a
lower
severity
and
we're
getting
a
good
look
for
the
very
first
time
we
can
start
to
map
it
out
and
figure
out
the
defense.
Now
most
of
our
defense
actually
happens
to
the
left
or
to
the
right
of
this,
but
we
want
to
move
as
much
as
possible
to
the
left
for
next
time.
A
After
the
details
are
out
in
the
open,
we
can
take
a
look
at
what
it's
going
to
do.
We
can
map
it
out
and
see
how
this
works.
This
is
a
simplification,
but
it
gets
the
gist
of
it.
An
attacker
can
take
a
poison
certificate.
It
gets
decoded
it
overflows
if
it
puts
a
nasty
instruction
on
the
stack.
We
have
our
C
if
it
puts
a
little
chunk
or
a
whole
bunch
of
dots
somewhere
else
when
we
maybe
have
DOS.
A
So
we've
got
a
road
map
for
our
vulnerability,
but
we
still
need
to
know
what's
vulnerable
and
I
think
this
is
one
of
the
most
important
parts
for
us
at
Shopify
we
had
thousands
of
images,
hundreds
of
thousands
of
PODS
and
thousands
of
people,
and
we
have
to
find
out.
You
know
who
owns
these.
What's
in
them,
we
need
an
asset
inventory
and
we
need
to
know
what
those
assets
are
made
of.
So
has
anyone
heard
anything
about
s-bombs
lately
or
just
just
me?
A
So
s-bombs
aren't
new.
They
popped
up
eons
ago,
originally
I
think
mostly
for
tracking
licensing.
But
here
is
what
s-bombs
have
done
for
me
lately
before
the
vulnerability
was
even
described.
We
had
this
query
shared
by
my
colleague
that
was
able
to
tell
us
exactly
what
was
affected
and
exactly
where
it
was
running,
and
this
is,
as
you
can
see,
on
October
31st
before
we
really
know
the
details
of
the
vulnerability.
So
this
is
what
s-bombs
have
done
for
me.
A
Lately
we
saw
the
scary
Red
map
of
Badness,
and
now
that
we
know
what
we
have
running
this,
we
can
try
to
figure
out
our
mitigations
as
it
turns
out.
You
need
to
be
running
openssl
and
there
are
a
lot
of
other
Alternatives.
Now,
like
boring
SSL,
you
need
to
be
running
version
three
and
there
were
still
lower
major
versions
being
maintained
and
very
commonly
used,
and
so
those
weren't
affected
at
all.
A
You
need
to
convince
something
to
decode
it,
which,
as
it
turns
out
for
servers,
comes
after
chain
validation,
so
you'd
have
to
already
trust
a
poison
certificate
before
you
even
start
processing
this.
You
also
only
get
that
four
byte
overflow
into
used
memory
in
most
cases
which,
depending
on
how
the
compiler
lays
it
out,
it's
possible
that
something
bad
could
happen,
but
it's
very
unlikely.
A
So
I
I
saw
that
some
of
you
said
you
worked
on
this,
how
many
people
at
your
company
you
were
working
on
this
for
anyone
here.
Was
it
more
than
10
more
than
20.,
more
than
40.
yeah.
The
impact
of
this
misalignment
is
in
the
time
and
effort
spent
on
it.
Does
anyone
have
a
truly
unlimited
budget
or
unlimited
head
count?
A
Unlimited
time,
even
if
you're
not
constrained
by
your
policies,
there
are
intrinsic
limits,
you
can
only
interview
and
select
and
hire
an
onboard
so
fast
and
I
think
we
can
all
agree
that
the
fiscal
outlook
for
Tech
and
2023
doesn't
look
like
it
did
a
couple
of
years
ago.
More
companies
are
expecting
us
to
be
more
deliberate.
We
need
to
justify
our
expenditures
more
and
more
and
I.
Think
that's
what
matters
most
to
corporations
and
investors,
but
this
is
actually
what
I
care
more
about.
A
There's
a
more
significant
risk,
and
that
is
the
time
and
effort
spent
on
that
could
have
been
spent
on
automating
things
that
are
likely
to
be
mistaken
by
developers
and
leading
to
security
misconfigurations.
We
could
be
building
out
detection
for
a
more
severe
vulnerability
or
we
could
be
working
on
Outreach.
A
When
we
see
these
things,
we
need
to
use
the
available
info.
We
need
to
weigh
the
opportunity
costs
against
the
risk
of
compromise.
What's
the
business
impact?
What's
the
user
impact
in
our
case,
that's
millions
of
merchants
who
depend
on
us
for
their
livelihood
and,
of
course,
there's
user
privacy
to
consider.
So
this
is
why
we
had
dozens
of
people
working
on
this
when
it
was
first
announced,
and
this
is
why
I
would
urge
you
instead
of
just
saying
this-
is
a
waste
of
time
and
try
to
YOLO
it
next
time.
A
Try
to
prepare
ahead
of
time
so
that
the
effort
comes
before
rather
than
after,
and
just
to
be
very
clear.
I,
don't
hate
openssl
at
all,
I
think
they
actually
did
a
really
good
job
on
this.
They
announced
the
patch
very
quickly
after
it
was
reported
on
its
own
at
first,
it
seemed
like
it
could
cause
an
rce
and
they
responded
to
new
information
by
dropping
the
severity
to
high.
A
The
most
important
thing
if
we
want
to
reduce
the
effort
after
these
things
come
out,
is
to
have
an
upgrade
ritual
at
a
regular
Cadence
I
like
automation,
something
like
dependabot
is
great
here.
You
need
to
have
exception
lists,
because
there
are
going
to
be
things
that
you
aren't
able
to
upgrade
review
those
often
and
include
the
stakeholders
in
that
process,
so
that
they
are
continually
justifying
the
existence
of
this
out-of-date
software
track
the
time
since
the
last
update
for
all
of
your
assets.
A
A
I
really
like
old
software
at
home,
I
do
not
like
old
software
at
work.
In
a
previous
role,
I
was
Consulting
for
a
company
that
had
an
electron
microscope
and
it
only
had
drivers
for
Windows
NT
I
hated
that
machine,
but
there
was
no
way
I
was
going
to
convince
this
client
to
get
rid
of
it.
It
had
proprietary
drivers,
the
company
that
made
them
was
out
of
business
and
they're
not
going
to
throw
away
a
perfectly
good
electron
microscope,
because
Shane
told
them
to
you
might
have
a
different
example.
A
Maybe
your
boss
insists
on
reliving
their
Glory
Days
on
the
slopes
of
ski
free
or
playing
chips
challenge.
Maybe
they
insist
that
Windows
entertainment
package
just
isn't
the
same.
Unless
it's
running
on
a
486
with
Windows
3.
as
security
practitioners,
we
should
try
our
very
best
to
get
people
to
do
the
best
thing.
Sometimes
we
can
only
convince
them
to
do
something
like
hey,
maybe
get
that
machine
off
of
the
company
Network
track.
What
they're
doing
and
why
and
usually
I
would
encourage
you
to
do
detection
for
any
vulnerability
that
comes
out.
I
still
think.
A
That's
a
good
idea
and
my
go-to
is
usually
Falco
in
this
case.
Falco
is
great,
but
it's
only
going
to
read
the
first
80
bytes
of
a
buffer
by
default.
You
can
modify
the
snap
length
with
an
option
to
change
that,
but
it's
probably
not
worth
the
additional
overhead
in
this
case.
Instead,
let
Falco
look
for
other
suspicious
activity
from
as
yet
undiscovered
or
existing
vulnerabilities.
Just
not
this
one.
A
If
you've
got
some
sensitive
system
that
you
can't
upgrade,
and
you
really
need
to
mitigate
this
threat
or
you're
getting
hammered
with
a
DDOS
from
those
dots
we
talked
about
earlier
and
set
up
a
circata
IPS,
it
can
use
another
ebpf
detection
tool,
that's
geared
towards
Network
traffic
sniffing
or
just
use
fluent
D,
or
something
like
it
to
aggregate
logs
from
affected
systems
and
then
alert
on
that
poison.
Payload.
A
When
it's
detected,
we
saw
this
timeline
and
we
can
replace
the
dates
on
here
with
different
ones,
for
just
about
any
vulnerability
and
we're
going
to
over
and
over
and
over
again.
There
are
definitely
things
that
we'll
need
to
do
again
on
the
right
side
of
our
timeline.
But
if
we
consider
the
Inglorious
work
on
the
left
side,
it
is
much
more
impactful
in
addressing
these
vulnerabilities
and
it
has
the
greatest
impact
on
security.
A
So
if
we
want
to
be
prepared
for
the
next
threat,
whether
it's
nasty
nasty
nasty
as
one
commenter
said
or
another
dud,
we
can't
blame
maintainers
for
not
predicting
the
impact
on
our
environments.
Right
from
the
start,
we
can't
presume
to
be
some
sacred
group
that
just
issues
edicts
to
developers
telling
them
what
to
do,
and
we
certainly
can't
treat
every
new
risk
as
a
sensational
surprise.
What
we
can
do,
though,
is
we:
can
map
and
threat
model
our
environment?
A
We
can
build
Bridges
with
Builders
and
encourage
developers
to
follow
best
practices
by
working
with
them.
To
do
so,
we
can
start
preparing
right
now.
If
we
do
that,
we
can
uphold
our
responsibility
to
our
employers,
to
our
stakeholders,
our
customers
and
the
community.
We
can
protect
their
privacy,
we
can
earn
their
trust
and
we
can
be
responsible
with
their
resources,
and
maybe
just
maybe
we
can
even
preserve
our
conference
budgets.