►
Description
Establishing Container Trust at Scale [I] - Tim Mackey, Black Duck Software
Quantifying risks in a container image is a critical aspect of production deployments. With orchestration clusters supporting thousands of nodes, any risk assessment solution must work at production scale. Once a trusted image is deemed vulnerable, application risk increases, but which applications are impacted, and how far has trust been broken? Trust is established through best practices including the use of trusted image registries, static code analysis, fuzzing, strong perimeter defenses and deployment controls. Unfortunately, this trust model omits information flow.
Malicious actors succeed when applications are most vulnerable. When devising action plans in response to security disclosures, defenders must quickly assess both the impact and scope of the disclosure. This time to remediation requires accurate and actionable vulnerability assessments as applications are created, deployed and scaled. Enhancing security information flow accelerates risk mitigation at production scale.
About Tim Mackey
Tim Mackey is a technology evangelist for Black Duck Software specializing in the secure deployment of applications using virtualization, cloud and container technologies. Prior to joining Black Duck, Tim was most recently the community manager for XenServer and was part of the Citrix Open Source Business Office. Tim has held roles in mission critical engineering, performance monitoring, and large-scale data center operations. He has spoken globally on a variety of topics and at well-known events such as OSCON, LinuxCon, CloudOpen, Interop, CA World, Cloud Connect, USENIX LISA and the CloudStack Collaboration Conference. Mr. Mackey is an O'Reilly published author.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
The
presentation
part
of
this
session
could
conceivably
take
an
hour
and
a
half
I've
actually
done
it.
That
way
before
I,
don't
want
to
bore
you
with
an
hour
and
a
half
worth
of
me,
rambling
on.
If
all
goes
well,
I
will
have
a
demo
at
the
end,
to
show
how
some
of
this
actually
plays
out
in
the
real
world.
A
If
we
lack
time
to
be
able
to
pull
that
off,
then
come
down
either
to
the
Black
Duck
booth
or
assuming
that
I
get
it
all
built
up.
The
Red,
Hat
booth
and
I'll
be
happy
to
show
you
how
this
works
in
the
real
world
they're
in
the
middle
of
building
that
up
so
the
company
that
I
work
for
now
has
named
Black
Duck
software
and
we've
been
around
for
forever
14
15
years
and
we're
about
open
source
security,
and
that's
the
last
of
what
you're
going
to
hear
me.
Talk
about.
A
Black
Duck
you'll
see
the
logos
throughout,
but
this
is
a
project
that
I
started
in
on
about
this
time
last
year
to
figure
out
a
way
in
which
we
can
collaboratively
understand
what
trust
means
when
you
start
talking
about
containers
at
scale.
So
a
little
bit
about
me
right
now,
I
have
the
title
of
senior
technology
evangelist,
which
means
that
I
get
to
grind
code
and
figure
out
new
technologies
and
new
ways
of
solving
problems.
A
I've
been
open
source
for
ages
and
ages
and
ages,
and
most
recently,
I
was
a
community
manager
within
the
Zen
family
of
products
and
some
of
the
cool
things
that
I've
done
up
there
I'm
happy
to
talk
about
them
at
like,
but
they're,
not
part
of
this
talk
and
of
course
you
can
find
me
at
all
the
usual
social
places
since
I
never
know
exactly
what
the
skill
level
in
the
audience
is
like.
Technically,
this
is
an
intermediate
session,
but
that
is
your
own
subjective
interpretation
of
it.
A
How
many
people
here
are
new
to
container
orchestration
systems?
You've
not
really
played
around
with
them
for
very
much
longer
than
let's
say
the
last
couple
months.
Any
hands,
okay,
cool,
so
the
first
part
of
this
talk
is
going
to
be
more
of
a
grounding
as
to
what
the
problem
space
really
looks.
Like
second
part
of
the
talk
will
cover
some
of
the
general
use
cases
that
are
out
there
and
what
can
go
wrong
if
you
get
it
horribly,
horribly
wrong.
A
So
I'm
going
to
do
a
bit
of
a
review
of
what
the
container
build
process
looks
like
so
I'm
going
to
assume
that
the
organization
that
you're
working
in
has
some
form
of
get
repo
where
source
code
lives
and
that
there's
some
poor,
some
kind
of
approved,
binary
repository
that
could
be
from
Jay
frog.
That
could
be
from
a
docker
trusted
registry.
That
could
be
a
nexus.
That
could
be
something
that
you've
built
whatever
that
and
that
for
practical
purposes.
A
A
So
this
is
a
docker
file
which
I'm,
assuming
that
you've
all
seen
before
this
happens,
to
be
for
one
of
the
container
images
that
we
ship
as
part
of
our
off-site
solution
and
there's
the
base
image
from
which
it's
coming
from.
So
a
tag
is
sent
to
us
seven
out
of
the
center
project,
a
little
bit
of
software
hygiene
in
here
I'm
running
an
update
minimal
for
all
the
packages
that
are
in
there.
Then
I'm
copying
in
a
bunch
of
stuff,
that's
relevant
to
my
application
itself
and
I've
got
an
entry
point.
A
That's
going
to
launch
this!
Your
docker
files
are
going
to
look
very
similar
to
all
this.
Some
variation,
of
course
based
on
what's
going
on,
but
a
template
is
formed
and
if
I
look
at
the
docker
images
on
my
environment,
I
will
see
that
I
have
a
sent
OS
with
a
tag
of
sent
to
us.
Seven
I'll
have
this
hub.
Osc
scanner
thing
that
this
docker
file
was
referenced
to
and
I'll
have
an
image.
Id!
That's
in
there.
So
I
have
a
1,
9,
6
e.
A
That's
the
image
ID
for
sent
os
that
was
pulled
in
6
weeks
ago,
and
this
was
actually
on
my
development
machine
at
the
point
in
time
that
I
created
these
slides
a
couple
months
ago
and
16
hours
prior
I'd
built
this
hub.
Osc
scanner,
give
it
a
tag
of
4.2
and
and
ended
up
with
an
image
ID
of
three
nine
five
at
six
weeks
is
something
to
be
aware
of,
because
effectively
what's
happened
is
that
the
local
docker
engine
has
cached
this.
A
So
everything
that's
happened
in
the
outside
world
is
not
reflected
here
on
my
build
system
any
patches
any
anything,
so
something
have
in
the
back
of
your
mind.
If
I
look
at
the
docker
image
history
on
this
hub,
OSC
scanner
tagged
as
4.2.2
I
see.
Reading
from
the
top
down
I've
got
my
entry
point.
That's
16
hours
ago,
two
days
ago,
I
have
these
file
copies.
Ok,
that's
kind
of
interesting
that
lovely
yum
update
that
I
put
in
there
to
make
certain
I
have
the
software
hygiene.
A
Well,
that's
actually,
seven
days
old
and
then
six
weeks
ago,
I
see
the
pieces
coming
out
of
sent
to
us
seven.
So
that
gets
me
into
a
little
bit
of
a
weird
security
state,
because
I
believe
that
I'm
doing
the
right
thing
from
a
hygiene
perspective
by
having
this
yum
update,
but
it
actually
is
building
in
a
layer
that
is
a
little
bit
on
the
old
and
crusty
side.
A
Lord
only
knows
what's
happened
in
the
last
seven
days,
so
that
becomes
a
key
problem
when
we
start
looking
at
what
the
security
of
our
systems
are
and
where
the
trust
comes
from.
So
I
thought
when
I
was
building
this
image
when
I
first
started
out
that
I
was
actually
going
to
execute
that
yum
update
exactly
the
same
way
that
I
would
have
inside
of
say
of
yem,
but
that's
not
the
way
the
dockerfile
processed
it
and
it
increased
my
risk
and
decreased
my
trust.
A
That
now
gives
me
another
scenario,
so
I
have
a
cluster
of
one
note:
give
you
nice
and
simple
world's
a
happy
place.
I'm
gonna
pull
and
run
the
latest
web
app.
It's
going
to
be
give
me
this
kind
of
orange
e
color
I'm
gonna
scale
that
up
to
two
replicas
what's
happened
is
that
the
docker
engine
on
that
node?
Has
this
thing
cached
and
will
just
go
and
say,
give
me
another
one
of
these
latest
I.
Already
know
what
that
is:
I'm
gonna
pull
and
run
this
one
zero.
A
One
he's
got
kind
of
a
pinkish
tag,
I'm
gonna
scale
him
up,
I'm
gonna,
go
and
pull
by
that
pull
spec
of
sha-256
and
I
know
scale
him
up
and
then
I'm
going
to
in
my
registry
go
and
delete
that
1.0.1
and
I'm
gonna
scale
that
1.0.1
to
three
replicas.
This
works
because
the
cache
is
on
that
note.
It
doesn't
have
to
go
back
to
the
registry.
I
could
have
renamed
it
because
tags
themselves
are
human,
definable,
mutable
things.
A
So
if
I
wanted
to
go
and
push
something
else
and
re
tagged
it
as
one
zero,
one
I
can
change
the
thing
the
behavior
under
the
covers.
Now,
let's
add
a
second
note
in
and
then
now
let's
kill
that
first
note
now
the
orchestration
solution
has
survivability
on
the
images,
but
some
amount
of
time
has
passed
so
that
latest
is
whatever
got
pulled
down
is
latest
at
the
time
that
I
started
that
second
note
and
forced
survivability.
A
That's
not
quite
what
you
expect.
So
the
question
is:
how
do
I
know
exactly
what
I've
got
so
deployments
and
triggers
kind
of
play
into
this,
and
this
is
deployment
config,
and
the
objective
here
is
to
abstract
away
replication
controllers
and
pods
and
replica
sets
and
daemon
sets
and
so
forth,
such
that
you
end
up
with
consistent
behavior.
The
goal
is
to
define
a
set
of
behaviors
for
when
to
have
a
change
in
state.
A
So
if
I
go
and
I
say
that
I
want
to
have
when
an
image
changes
I'm
going
to
have
underneath
the
covers
a
mechanism
by
which
I
can
now
say
the
tag
on
this
a.k,
the
pull
spec
is
different
than
what
I
knew
it
was
at
the
time
and
I
will
pull
that
image
down.
Similarly,
I
can
have
a
config
change,
so
you
go
and
you
make
a
modification
to
a
config
map.
A
Or
what
have
you
as
my
build
mechanism,
and
it
just
happens
to
be
an
embedded
Jenkins
environment.
It
has
a
source
control
trigger
such
that
when
I
go
and
define
a
git
repository,
it
will
go
and
create
this
container
image.
It
will
go
and
move
it
to
it
into
an
internal
registry.
It'll
run
some
number
of
steps
of
pipeline
security
scans
that
I
defined
staging
tests
and
so
forth,
maybe
promoting
into
a
different
registry,
and
then,
what's
that
de
promotion
occurs
a
deployment
trigger
that
goes
and
says
gee
whiz,
but
I
want
four
of
these.
A
It's
pretty
neat
takes
a
little
bit
of
getting
your
head
around,
but
it
gives
a
mechanism
by
which
I
can
have
a
view
of
what
the
state
of
what
I
should
be
deploying
looks
like
now,
if
I'm
going
to
do
this
with
an
a/b
test.
Let's
say
that
I
have
two
nodes:
two
containers
that
are
running:
that's
on
the
right
hand,
side
and
there's
now
a
new
vulnerability
that
happens
to
be
associated
with
that
container
image.
A
That
is,
underneath
those
containers
well
I'll,
go
and
I
patch,
it,
let's
say
in
source
code,
put
a
new
dependency
in
place,
pull
the
patch
from
upstream.
What
have
you
have
that
source
code
trigger
go
and
automatically
create
the
patched
image?
Push
it
into
the
registry.
Have
that
deployment
trigger
kick
off,
define
that
I'm
doing
an
a/b
scenario.
Put
that
patch
thing
in
place.
Do
my
testing
and
replace
it,
and
now
I've
got
a
very
effective
patch
management
model,
but
that
doesn't
necessarily
gain
me.
A
Anything
with
respect
to
trust
and
so
I
have
to
take
a
little
bit
of
a
step
back,
and
you
can
tell,
by
the
blonde
in
my
hair
that
I've
been
around
for
a
little
bit
and
I've,
been
through
a
bunch
of
good
ideas.
I've
been
around
a
bunch
of
bad
ideas
and
for
practical
purposes
the
one
that
I
embrace
right
now
is
varying
names,
but
security,
driven
development
and
I'm
gonna
put
a
set
of
assertions
up
there.
A
Around
developers
aren't
probably
with
house
security
information
should
be
consumed
and
how
tests
should
be
created,
released
policies
around
security
are
in
place
and
trusted
components
are
all
in
there.
You've
got
security
testing
as
part
of
your
CI
loop.
Binary
artifacts
are
only
created
when
release
policies
are
met.
The
binaries
themselves
are
digitally
signed.
A
So
you
know
where
they
come
from:
container
images
are
built
from
trusted
images
and
they're
only
deployed
from
trusted
registries,
and
so
on,
so
on
so
on,
and
what
can
possibly
go
wrong
and
this
is
kind
of
the
bulk
of
the
time.
The
talk,
because
what
can
possibly
go
wrong
is
a
question
of
scale,
so
my
first
customer
for
this,
their
definition
of
scale
for
their
acceptance,
testing
forty
nodes,
changing
images
at
a
rate
of
about
a
hundred
a
day.
A
If
your
definition
of
scale
is
more
than
that,
please
raise
your
hand,
half
the
room
cool
now.
The
problem
is
that
everyone
has
a
slightly
different
definition
of
scale,
and
it
really
is
one
of
these.
You
know
when
you
get
there
and
it's
typically
one
of
these
scenarios,
where
you've
got
some
form
of
release
management
process
and
config
management
scenarios
in
place.
There's
some
government
regs
service
delivery
includes
terms
like
high
availability,
disaster
recovery,
multiple
data,
centers
service
level
agreements
and
so
forth.
A
That's
scale
proverbially,
that's
when
it
hit
the
fan,
and
so
I
love
this
prescient
article.
So
this
came
out
in
March
and
said
that
the
easiest
way
they
get
fired
in
2017
was
to
have
a
data
breach
and
little
did.
We
know
that
we're
going
to
have
a
rather
substantial
one
this
year,
but
before
that
one
was
disclosed.
We
had
a
lovely
report
out
of
IBM
in
the
Ponte
Motta
Institute.
That
puts
some
stakes
in
the
sand.
A
The
average
cost
of
the
data
breach
being
over
seven
million,
the
lost
business
associated
with
it
being
four
million
and
importantly
that
206
days,
that's
over
half
a
year
for
people
to
recognize
that
they
had
the
breach
and
contain
it.
That's
the
industry
average,
that's
not
so
good,
and
then
along
comes
Equifax
and
well.
A
We
know
a
little
bit
of
the
timeline
and
I'm
actually
going
to
dive
into
that
a
little
bit,
but
it
can
get
slightly
worse
because
well,
they
had
a
couple
days
later:
36
percent
decrease
in
their
stock
price
ouch
and
bad
things
happened.
So
the
idea
is
that
we
should
always
be
questioning
everything.
There
are
new
data
regulations
on
the
horizon.
If
you've
not
heard
about
something
called
gdpr,
it's
an
e
regulation.
That
starts
in
May,
we're
in
the
US
there's
a
certain
level
of
well,
that's
a
European
problem.
A
We
don't
need
to
deal
with
it.
No,
we
all
need
to
deal
with
it
at
some
level,
we're
touching
a
European
person
and
the
penalty
is
four
percent
of
our
organization's
revenue
or
20
million
euro.
Whichever
is
greater,
there's
already
been
determinations
that
some
organizations,
because
they
touch
data,
could
be
implicated
in
all
of
this.
If
you're
not
already
aware
of
it
make
certain
that
your
organization's
are
planning
around
it
so
question
everything.
So
where
does
your
base
image
actually
come
from?
Who
owns
it?
Who
updates
it?
What
is
the
health
of
that
image?
A
Why
should
I
be
trusting
that
if
I'm
building
it
my
build
environment
should
I
be
trusting
my
build
servers?
Is
there
any
way
that
a
foreign
container
can
start
in
my
orchestration
platform?
Who
has
the
rights
to
modify
these
things?
What
happens
if
that
base
images
registry
goes
away
and
I
need
to
patch
it?
What
happens
if
an
update
mirror
goes
down
when
a
security
disclosure?
What
happens
to
my
patches?
What
happens
to
my
system?
A
A
A
They
expect
that
the
procurement
manager
has
some
level
of
relationship
with
the
vendor
that,
when
something
goes
end-of-life
that
they're
going
to
be
notified
that
gee
whiz,
but
here's
an
upgrade,
it's
going
to
cost
you
a
boatload
of
money,
but
what
the
hey
or
here's
a
set
of
patches-
and
we
know
that
you're
kind
of
in
your
sunset
of
this
version.
But
why
not
define
security
processes
all
part
of
how
the
proprietary
software
world
works?
A
Open-Source
is
very
different,
open
source,
as
most
of
you
know,
if
you're
not
engaged
with
the
community,
you
know
nothing.
The
community
could
decide
that
the
version
that
you're
using
was
the
worst
possible
implementation
of
that
functionality
on
the
planet
and
they
collectively
went
off
to
the
beach
got
themselves.
A
A
bunch
of
margaritas
had
a
great
old
time,
came
back
and
said:
yeah
we're
not
gonna,
make
those
same
mistakes
again
and
created
version
2,
and
if
you
didn't
know
that
that
happened,
and
you
want
to
update
version
1
you're
effectively
only
a
fork
security
disclosures,
don't
look
the
same
way
throughout
this
talk.
I'm
gonna
have
a
bunch
of
things
that
I
highlight
and
have
for
the
people
in
the
back
of
the
room.
I'm
gonna
actually
read
out
what
I've
got
highlighted.
A
This
is
a
security
patch
release
note
from
MediaWiki
year
or
so
ago,
and
this
is
pretty
typical
for
how
these
things
play
out.
So
the
first
bit
of
yellow
says
this
is
a
maintenance
fix.
Very
special
pages
resulted
in
fatal
errors,
that's
their
security
disclosure,
which
is
incredibly
helpful
if
your
MediaWiki
admin.
They
also
put
a
note
about
end-of-life.
That
said,
hey
please
note
that
one
twenty
four
six
marks
the
end
of
support
for
the
124th
series
of
releases.
Technically
this
ended
a
few
weeks
ago
with
the
release
of
one
twenty
one.
Twenty
six
zero.
A
However
one
twenty
four
five
had
issues
along
with
other
versions,
so
it's
not
fair
to
fix
them.
That's
a
really
cool
thing,
that's
kind
of
how
open-source
works,
we're
all
trying
to
make
certain
that
the
world
is
a
good
place
for
our
consumers
and
do
the
right
thing
and
you'll
note
that
this
is
a
one
26
25,
24
and
23
release.
A
That's
not
how
most
organizations
expect
to
consume
security
notes.
They
don't
expect
end-of-life
notices
to
be
embedded
in
them.
They
expect
some
level
of
awareness,
so
the
onus
is
on
us
as
the
consumers
to
kind
of
do
the
right
thing
now,
I
mentioned
Equifax
and
one
of
the
cool
things
that
I
like
to
do
is
go
and
take
a
given
voluntary
and
decompose
it
to
say
who
knew
what?
When
and
what
was
the
real
problem.
A
So
the
whole
Equifax
scenario,
if
I
go
back
to
August
of
2012,
that's
when
the
code
that
ultimately
became
this
bug
was
introduced,
oops
and
that's
what
it
looked
like.
Those
are
the
lines
and
specifically
there's
a
bunch
of
checks
up
front
in
this
billed
error,
message
routine
and
there's
a
return
of
a
localized
piece
of
text.
A
A
Then
we
have
the
disclosure
and
the
patches
available,
and
those
of
you
who
are
familiar
with
git
formatting
will
note
that
there
is
one
line
changed
that
was
the
patch
and
that
one
line
was
that
localized
find
text
could
actually
return
null.
So
we
needed
to
test
for
that
and
make
certain
that
we're
returning
the
right
things.
A
If
you
go
back
and
look
through
all
the
history
that
protection
code,
that
was
originally
there
kind
of
evolved
over
time,
and
so
it
was
it
actually
protection
coding
longer
as
a
scenario
where
code
that
works
can
actually
become
problematic.
When
you
end
up
with
a
review
that
becomes
looks
good
to
me,
because
unless
you've
got
the
tests
in
place,
you
don't
know
that
what
was
working
is
in
fact
changed
to
become
what's
broken
know.
A
So
that's
the
six
of
March
the
disclosure
is
published,
as
in
media,
is
aware
of
this
and
a
week
after
that,
the
national
vulnerability
database,
where
more
people
get
their
security
information
than
hacker
news
and
that's
a
whole
separate
talk,
happens
to
have
information
on
this
up
until
that
point
in
time.
The
national
vulnerability
database
for
CVE
2017
5638
was
a
placeholder,
no
information
whatsoever.
Unless
you
were
actively
engaged
with
the
community,
you
knew
that
there
was
a
something,
but
you
didn't
know
what
that
something
was
so
at
this
point,
Trust
has
been
broken.
A
A
That's
something
you
never
want
your
employer
to
be
doing
just
so.
You
know,
and
I'm
gonna
again
read
the
highlighted
elements
on
March
9th
equifax
disseminated
the
US
cert
notification,
aka
the
notification
from
the
nvd
internally
by
email,
requesting
that
applicable
personnel
responsible
for
Apache
struts
installations
upgrade
their
software.
We
now
know
that
the
vulnerable
version
of
Apache
struts
with
an
echo
fax,
was
not
identified
or
patched.
A
A
Unfortunately,
however,
the
scans
did
not
identify
the
Apache
struts
vulnerability
prepared
testimony
October
3rd,
so
we've
got
a
bit
of
a
timeline
here
now:
May
13th,
some
hacks
were
truly
successful
and
it
took
them
to
the
29th
of
July
to
figure
that
all
out
for
those
of
you
want
to
take
a
picture
of
this.
This
is
the
point
where
it's
a
good
time
take
a
picture.
We
had
almost
1700
days
from
the
time
that
the
code
was
introduced
to
the
time
the
patch
was
available
and
various
morphs
of
that
code.
A
A
Kind
of
a
bit
of
a
problem,
so
how
can
we
actually
do
a
little
bit
better?
We
need
to
be
establishing
trust.
We
need
to
be
doing
this
at
scale
because
problems
are
going
to
creep
in
problems
going
to
be
moving
through
it.
So
we
have
to
identify
that
security
is
a
layered
process
and
it
starts
with
understanding
the
role
that
security
tools
play
in
defining
trust.
A
So,
if
I
have
the
realm
of
all
possible
vulnerabilities,
the
first
scenario
is
I:
have
some
static
analysis
tools,
I
have
some
fuzzing
tools,
I'm
doing
some
injection
analysis
and
for
practical
purposes.
This
is
all
going
to
focus
on
code
that
your
organization
creates
you're,
going
to
have
one
hell
of
a
time
going
to
your
CTO
VP
of
Engineering
VP
of
ops
and
saying
I
want
to
go
and
put
this
really
expensive,
scan
tool
on
the
Linux
kernel,
OpenSSL
Tomcat.
What
have
you
he's
gonna
say,
but
that's
somebody
else's
problem
and
he's
right,
she's
right!
A
That's
looking
at
your
code,
not
upstream
vulnerability,
analysis
or
dependency
analysis
or
software
composition
analysis,
depending
on
the
term
that
a
various
analysts
will
use.
That's
looking
at
what
is
in
fact
in
your
environment,
so
3,000
disclosures
in
2015
found
by
researchers,
I
got
4,000
plus
and
2016.
As
of
the
1st
of
November
over
13,000
disclosures
this
year,
that's
about
35
a
day
in
your
organization.
A
How
would
you
go
and
determine
whether
or
not
your
infrastructure
today
is
vulnerable
to
a
given
CDE
you'd
have
tooling
for
that,
but
unless
you're
continually
updating
that
tooling
and
running
it
on
a
continuous
scan
basis,
looking
back
at
the
history
of
time,
you
might
miss
a
few
things
and
that's
where
things
get
really
problematic.
So
I've
got
a
couple
of
other
decompositions.
How
many
people
heard
about
the
Mariah?
This
was
the
620
get
bit
per
second
attack
on
Krebs
own
security
last
year.
Oh,
not
too
many
people.
A
Well,
this
was
an
attack
that
came
through
a
bunch
of
internet
of
things,
devices
doorbells
nanny,
cams,
thermostats,
fridges,
internet-enabled
toothbrushes,
Barbie
playing
play,
houses
that
all
kinds
of
ways
through
it.
It
was
a
vulnerability
that
was
originally
disclosed
in
2004
and
if
you
read
the
security
disclosure
for
it,
it
does
not
describe
anything
like
what
the
world
looks
like
today.
In
fact,
if
you
look
at
it,
it
talks
about
OpenSSH
allowed
TCP
forwarding.
A
A
Let
that
one
sink
in
for
a
little
bit.
This
would
only
be
exploitable
if
you
were
connected
to
a
public
network
and
used
a
well-known
default
password
admin
admin
work
for
you
to
admin
password.
That
was
what
was
used
for
a
lot
of
these
devices.
Now
the
cool
thing
in
all
this
is
that,
if
you
read
through
it,
you
don't
necessarily
apply
it
to
what
the
world
looks
like
today,
because
the
descriptions
don't
match
up
and
that's
kind
of
true
for
a
lot
of
these
older
things.
A
The
same
vulnerability
that
befell
Equifax
befell
the
Canadian
Revenue
Agency
Canada's
version
of
the
IRS
difference
in
behaviors.
They
detected
it
pretty
quickly.
They
detected
that
it
was
in
there
a
file
system.
They
would
on
the
evening
news
and
said
we
have
turned
off
our
a
file
system
in
the
middle
of
tax
season,
because
we
have
found
a
vulnerability
that
could
disclose
personal
information
for
every
single
citizen
that
your
identity
could
be
compromised
and
we're
going
to
fix
it,
we'll
let
you
know
when
we've
turned
it
back
on
and
if
we
need
to.
A
We
can
extend
tax
season
because
well,
that's
kind
of
our
job
and
difference
in
behavior
between
the
two.
How
do
we
have
responsible
disclosure
and
then
the
last
scenario
is
why
please
God?
Are
there
still
two
hundred
thousand
publicly
accessible
websites
that
are
vulnerable
to
heartbleed
in
2017?
Should
we
not
have
learned
something
by
now
the
moral
this
story
isn't
so
much
about
Hartley?
It's
about
the
long
tail
of
systems
that
we
manage.
We
all
have
legacy
infrastructure.
It's
all
part
of
our
trust
scenario.
Bad
things
can
happen.
A
We
need
to
be
continually
understanding
what
it
is
that
we
own
and
reassessing
that
trust,
and
for
me,
that's
focusing
on
factors
that
truly
impact
risk.
I've
talked
about
vulnerable
open-source
components.
Forks
forks
of
Forks
code
ends
up
in
your
environment
from
a
multitude
of
sources.
There
are
something
like
1,200
Forks
of
OpenSSL,
which
version
are
you
running
and
I
got
five
minutes
left,
so
this
is
going
to
be
fun
to
blast
through
a
few
more
slides
impact,
a
point
in
time
decisions
I
mentioned
about
going
to
the
beach
and
having
a
great
old
time.
A
But
what
if
that
beach
experience
was
version?
1.2
of
a
given
component
happened
to
be
no
longer
going
to
be
developed
anymore,
and
so
there's
this
massive
change
set
in
your
future
to
go
and
consume
version.
Two.
How
would
you
know
what
kind
of
impact
is
in
that
scenario?
Is
there
going
to
be
a
security
patch
for
something
that
you've
done?
What
does
the
commit
velocity?
Look
like?
A
We
don't
patch
containers,
we
re
spin
them,
but
we
still
have
to
look
at
how
patches
work
so
again,
looking
at
the
same
equifax
patch,
if
I
move
one
version
further
along,
it
fixes
the
same
of
the
vulnerability
that
befell
Equifax,
but
it
introduces
new
issues.
So
is
the
patch
version
worse
or
not
subjective,
and
there
were
three
other
scenarios
where
the
patch
version
actually
is
worse
than
where
you
are
at
least
in
terms
of
quantity.
A
I
mentioned
the
product
that
we've
been
working
on
I'm,
going
to
put
this
into
an
an
open
shift
context.
So
for
those
of
you
who
don't
know
open
shift
is
a
fairly
opinionated
version
of
Nettie's
supply
is
a
bunch
of
things
that
you
would
have
to
make
the
decision
as
to
how
to
compose
your
environment.
One
of
the
things
that
they
add
in
is
an
integrated
registry
and
a
mechanism
known
as
an
image
stream
and
for
practical
purposes.
A
An
image
stream
is
nothing
more
than
a
mapping
between
that
registry
representation
and
a
bunch
of
tags
for
where
the
image
comes
from
just
think
of
it.
That
way.
Of
course,
you
can
have
external
registries
that
are
managed
by
this,
and
some
of
the
representations
will
end
up
inside
the
internal
registry
with
image
stream
mappings.
A
What
we
did
was
we
put
a
piece
of
glue
in
the
middle
and
we
monitor
for
every
single
scenario
under
which
an
image
can
be
utilized
within
a
cluster
new
image
being
created
through
a
build
mechanism
imported
into
the
system
utilized
within
a
pod
as
a
container
or
an
it
container.
It
really
doesn't
matter
when
it
comes
into
existence,
we're
going
to
figure
out
whether
we
need
to
scan
it
will
scan.
A
Are
we
impacted
by
this
and
get
that
answer
that
quickly?
That's
the
goal,
and
so,
if
fundamentally,
is
a
case
of
sources
of
truth
and
identifying
where
things
are
maintaining
a
security
source
of
truth,
so
that
you
know
where
things
are
going
in
we're
not
the
only
source
of
truth
and
what
I'm
asserting
is
that
there
are
other
tools
like
those
static
analysis
tools,
those
fuzzing
tools
that
need
to
become
sources
of
truth.
That
might
be
some
of
what
we
see
from
graffia
so
ver
time.
A
Today,
still
very
much
more
of
a
spec
than
anything
else,
layer,
the
container
security
pieces
in
place
in
an
open
shift
environment
you've
got
a
very
lockdown
host
running
a
minimal
Linux.
You've
got
something
called
open
s
cap
that
can
go
and
assess
the
policy
and
security
state
of
that
host
with
us.
You
can
take
a
look
at
every
single
container
image
and
what
the
open
source
room
risk
is
associated
with
it
and
bring
that
all
back
so
that
it's
automatic
there's
no
human
interaction
required,
because
everything
requires
end-to-end
visibility.
A
Otherwise
things
can
fall
through
the
gaps.
During
his
testimony,
echo
fat,
former
CEO
of
Equifax
said
this
guy
or
this
team
was
responsible.
Well,
maybe
they
just
regressed
and
rolled
back,
something
that
hadn't
been
patched
because
the
patch
broke
something
this
happens.
You
want
to
be
able
to
know,
so
you
can
assess
that
risk.
So
I
probably
have
time
for
just
one
question
with
one
minute
left
and
I'm
apologized
for
not
getting
to
the
demo,
but
like
I
said
I
can
do
this
about
90
minutes.
Any
questions.