►
Description
Don't miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from 18 - 21 April, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
How to Secure Your Supply Chain at Scale - Hemil Kadakia & Yonghe Zhao, Yahoo
In this session we will present a high-level system that protects against attacks — like unauthorized access, exploiting known vulnerabilities, injecting malicious software — by integrating open source tools such as Grafeas, Sigstore, Screwdriver, Kyverno & Anchore. In short, providing a unified solution for securing various aspects of the software supply chain. As one of the top ten visited websites on the Internet, Yahoo's massive scale across hybrid cloud and mobile platforms makes the security of our brands paramount — especially in today's evolving software supply chain landscape. This talk will deep dive into our primary use cases of source code scanning, security misconfiguration detection, vulnerability management, and protecting K8s deployments using dynamic policies. Attendees will leave with a framework for successfully managing the same tools Yahoo uses to simplify the developer experience.
A
B
A
A
A
So
what
is
the
problem
and
what
are
the
various
attacks
that
are
possible
in
the
software
supply
chain?
There
are
quite
a
lot
of
places.
An
attacker
can
Target
your
infrastructure,
starting
from
compromising
your
Source
Code
system
example
being
the
php's
git
server
when
they
were
attacked
or
compromising
your
build
system.
Resentish
example
being
the
solar
winds
attack,
or
they
could
also
trick
users
to
use
bad
artifacts,
which
is
known
as
typo
squatting
and
that's
been
happening,
often
in
package
managers.
A
So
why
is
software
supply
chain
security
important
to
us?
These
are
some
of
the
recent
attacks
and
headlines
related
to
software
supply
chain
security
at
the
bottom.
You
can
see
the
most
recent
attack
on
Pi
torch,
where
a
nightly
version
of
pytorch,
which
was
released
during
the
holiday
season,
most
likely
had
a
rogue
package
which
was
able
to
siphon
off
some
sensitive
data
from
user
system.
This
attack
is
known
as
dependency
confusion
and
that's
been
affecting
a
lot
of
the
development
environments
and
package
managers
right
at
the
top.
A
So
here's
an
obligatory
side
showing
you
some
scary
figures.
Industry
data
suggests
anywhere
between
85
to
97
percent
of
Enterprise
code
base
is
using
open
source.
This
means
that
most
of
your
application
consists
of
code
that
you
didn't
write.
The
vulnerabilities
in
third
party
or
open
source
dependencies
can
pose
a
significant
security
risk.
A
Supply
chain
attacks
are
on
the
rise
and,
according
to
the
recent
report
by
sonar
type,
the
supply
chain
attacks
have
gone
up
by
742
percent
year
over
year.
Over
the
last
three
years,
three
out
of
five
companies
have
been
targeted
due
to
the
supply
chain
attacks
based
on
angkor's
report
as
well.
So
what
is
the
industry
doing
in
order
to
secure
software
supply
chain?
B
Thanks
Hano
Yahoo
indeed
has
a
large
software
span
system.
We
are
supporting
yahoo.com,
which
is
amongst
the
top
10
visited
website
in
the
world.
We
have
a
lot
of
services
and
products
serving
high
low
traffics
internally.
There
are
about
60
000,
build
jobs
running
and
about
5
000
images
published
to
the
registry
on
a
daily
basis
and
at
the
same
time
there
are
more
than
700
kubernetes
clusters
running
more
than
100
000
ports,
and
that
is
still
now
the
whole
picture
of
Jaco's
software
suspension.
B
There
are
still
a
lot
of
other
kinds
of
artifacts
not
being
deployed
to
the
kubernetes
Clusters
like
to
the
on-prem
virtual
machines,
so
Yahoo
suffers
a
bench
and
scale
is
large
not
only
in
the
direct
quantity
but
also
in
terms
of
tuning
choices
within
Yahoo.
We
have
different
teams
with
different
requirements
and
needs,
and
each
team
may
have
their
own
path
to
deliver
their
software.
B
The
variety
of
tools
makes
securing
software
suspension
of
Yahoo
challenging,
so
we
must
simplify
the
problem
at
hand
when
we
first
started
our
journey.
We
decided
to
Pilot
the
security
measures
and
produce
real
value
in
a
certain
software
suspension.
Has
we
choose
the
cloud
native
path,
that
is
a
GitHub
Enterprise,
a
source
called
measurement
tool
screwdriver
as
a
build
pipeline,
internal
site
registry
as
artifact
store
and
finally
kubernetes
as
a
deployment
environment?
B
B
Okay.
So,
as
we
all
know,
software's
pension
security
has
a
lot
of
aspects,
even
if
we
have
only
one
path,
there
are
still
a
lot
of
security
controls
and
the
best
practice
to
follow,
but
luckily
we
do
not
need
to
start
from
scratch,
because
we
have
already
had
some
fundamental
security
controls
in
place
like
static
code
scanning
Ripple,
Branch
production
to
progress,
reverse,
and
so
on.
B
So
after
evaluating
the
existing
security
controls
and
open
source
standards,
we
realized
there
were
three
major
gaps.
Firstly,
even
if
we
have
static
code
scanning
to
check
the
proprietary
code,
we
are
not
able
to
detect
the
vulnerabilities
or
open
source
dependencies,
so
we
decided
to
introduce
software
conversation
analysis
essay
to
fill
this
Gap.
B
In
addition,
we
are
not
able
to
check
the
software
and
block
the
deployment
after
the
code
is
released
from
the
Repository.
So
to
fill
this
Gap,
we
decided
to
add
two
checkpoints
in
our
software
development
lifecycle,
while
in
the
build
stage
another
in
the
deployment
stage,
we
will
work
you
through
for
this
Recaps.
What
did
we
do
in
the
following?
Slides?
B
First
software
conversation
analysis
also
known
as
essay
as
Hannah
said
earlier.
Most
of
most
of
our
application
consists
of
code.
We
didn't
write.
Traditional
code
scanning
will
just
look
at
the
flow
and
logic
of
your
proprietary
code
and
reports.
If
there
is
any
potential
vulnerabilities,
it
will
not
look
how
the
open
source
dependencies.
B
They
can
identify
the
women's
Source
dependencies
package
names
and
address
check
against
the
vulnerability
database
and
finally,
report.
If
they
find
any
issue,
they
could
also
probably
raise
per
requests
to
bump
up.
The
version
of
your
open
source
dependencies,
which
will
in
most
cases,
can
feel
fix
the
potential
vulnerabilities.
B
In
practice,
we
use
saved
and
drive
to
generate
s-bomb
and
vulnerability
data
to
do
the
assessment.
We
also
provide
an
option
to
block
a
specific
pipelines
if
we
found
any
severe
vulnerability.
So
in
the
next
time,
if
another
log4j
crisis
occurs,
we
can
take
actions
and
prevent
those
vulnerable
images
from
being
published
to
our
registry.
B
B
So
there
are,
there
are
a
lot
of
use
cases.
For
example,
if
an
image
is
not
signed
or
it
is
from
an
untrusted
registry,
or
it
still
contains
any
unsolved
vulnerabilities,
we
can
block
it
or
at
least
inform
the
inform
Engineers
for
those
violations
to
achieve
the
deployment
time
verification
we
utilize
a
dynamic
animation
control
in
kubernetes.
B
It
allows
us
to
implement
HTTP
workbook
that
can
receive
admission,
requests
check
against
predefined
policies
and
finally
decide
whether
or
not
to
develop,
deploy
a
resource
and
in
practice
we
use
both
Yahoo
proprietary,
webhook
and
Kevin
to
achieve
our
goal.
So
let's
talk
about
some
of
the
tracks,
video
during
the
deployment
time.
B
First,
one
is
ensuring
prominence
of
images
exists.
Provenance
is
a
bunch
of
records
that
tell
you
various
this
image
comes
from.
It
can
include
information
like
who
commits
the
code,
which
Ripple
and
branch
is
this
image
originally
from
build
information,
and
so
on.
We
collect
our
prominence.
Data
from
GitHub,
screwdriver
and
registry
basically
covers
all
three
major
steps
or
software
suspension
and
those
data
are
stored
in
the
you
know,
provenance
database
and
this
provenance
store
is
based
on
Graphics,
which
is
an
open
source
project
that
standardized
provenance
format.
C
B
B
B
B
B
B
C
B
B
B
B
B
B
B
B
A
Thanks
younger
so
just
a
brief
summary
on
what
tools
we
have
been
using
and
the
timeline
on
how
we
reach
there.
We
started
our
journey
by
around
mid-2020
when
we
started
our
journey,
not
a
lot
of
Open
Source
tools
and
solutions
were
available.
So
we
built
a
lot
of
In-House
tooling
and
also
used
graphius
as
the
provenance
store
by
by
early
mid
2021.
A
We
were
able
to
collect
most
of
the
source
provenance
and
around
10
percent
of
the
built
Providence
were
through
our
build
pipelines
later
2021,
we
open
source,
graphius
RDS,
which
adds
supports
to
postgres
database
with
graphius
early
2022
six
store
had
emerged
as
a
really
promising
solution
for
both
signing
container
images
and
also
a
testing
software.
So
we
started
exploring
that
internally
for
some
of
our
build
pipelines
and
while
doing
that,
we
were
also
working
on
adding
deployment
checks
which
jung-ho
showed
earlier.
A
So
we
use
both
kivarno
and
Yahoo's
proprietary
policy
checks
later
that
year
and
beginning
this
year,
we've
Incorporated
some
of
the
cosine
functionality
to
our
existing
build
pipelines
and
are
also
working
on
guac
for
visualizing.
The
s-bomb
data,
which
is
generated
by
sift
and
gripe,
so
what
did
we
learn
from
our
journey
so
far?
A
A
If
the
tool
being
built
can
do
its
job
behind
the
scenes
or
if
it's
it
can
be
integrated
with
the
tools
itself,
it
reduces
the
onboarding
and
learning
efforts
by
developers
and
by
doing
that,
we
reduce
the
friction
between
the
security
team
and
the
development
team,
because
they
don't
need
to
update
their
projects
on
a
regular
basis,
and
this
can
be
done
behind
the
scenes.
This
also
helps
us
in
reducing
the
time
we
go
to
market
or
we
go
to
actually
the
deployment
time
an
example
to
Showcase.
A
This
lesson
is
when
we
started
collecting
the
provenance.
As
mentioned
earlier,
we
we
use
graphia,
so
we
integrated
Source
metadata
collection
as
part
of
GitHub
web
books
and
for
getting
the
build
metadata.
We
integrated
it
with
the
built
a
down
step.
So
any
build
which
runs
in
Yahoo
was
able
to
send
the
provenance
to
the
graphia
store
that
way
it
gave
us
around
70
to
80
percent
of
provenance.
The
remaining
20
is
because
they
don't
use
the
standard
tooling
and
the
same
when
we
tried
to
do
for
admission
web
hook.
A
So
that
brings
me
to
my
next
Point
that
is,
and
one
of
the
most
important
lessons
we've
learned
is
around
adoption
and
Adoption
of
tools
and
services
like
the
admission
web
hook
across
the
company.
If,
even
if
there
is
a
small
portion
of
the
company
which
is
not
onboarded
to
using
these
tools,
we
may
still
be
susceptible
to
software
supply
chain
attacks.
So
we
need
to
ensure
that
everyone
in
the
T
company
uses
or
gets
onboarded
to
the
admission
webhook
and
example.
A
For
this
being,
we
made
the
webhook
as
default
for
all
eks
deployments,
but
there
were
some
other
teams
which
were
using
non-standard
tools
or
different
deployments
so
for
in
order
for
them
to
be
onboarded,
we
had
to
either
create
a
Helm
chart
so
that
they
can
install
it
manually
or
we
needed
to
get
behind
them
to
make
sure
they
can
actually
adopt
it.
So
it's
still
a
challenge
and
we
are
still
working
on
increasing
the
adoption
within
the
company.
A
A
We
we
should
engage
with
the
open
source
community
and
also
try
to
make
meaningful
contributions
that
increases
the
expertise
in
that
and
also
gives
back
to
the
community,
and
the
last
one
is
around
continuous
feedback.
So
performing
continuous
testing
on
your
solution
and
getting
periodic
feedback
from
stakeholders
is
important.
Ensuring
the
requirements
and
use
cases
are
covered
is
important
to
the
success
of
the
project.
A
C
A
Sure
so
oppa
is
a
great
tool
and
it's
quite
powerful.
So
when
we
started
our
journey,
we
we
started
working
with
oppa,
but
as
we
proceeded,
we
we
realized
that
oppa
is
so
powerful
and
the
needs.
What
we
have
is
not
as
granular,
which
oppa
offers
and
also
it
as
learning
a
new
language
since
oppa
needs
to
have
some
expertise
for
learning
the
language.
A
B
Yeah
for
permanence
check,
as
I
show
in
the
in
the
demo,
so
there
are
basically
three
Nursery,
maybe
about
several
checks,
so
we
can
check
the
Source
repo
information
and
build
pipeline
information
and
kind
of
what
what
what
registry
is
it
from
and
actually
for
the
performance
but
not
from
products,
but
during
the
there's.
Other
kinds
of
the
data
storage
in
the
graphics
database
is
a
vulnerability
data,
so
we
can
just
fetch
one
ability
data
along
with
the
prominence
data.
It
is
no
problem,
but
we
can
check
with
problems.
B
A
So
I
can
take
that
so
internally
we
do
mirroring
of
most
of
the
open
source
packages
and
repos.
So
we
eventually
so
obviously
it
doesn't
go
through
our
build
pipelines
because
those
are
open
source,
so
we
won't
have
provenance
for
those
builds,
but
for
artifacts
and
making
sure
those
are
internally
signed.
That's
what
we
will
make
sure
we
do
it
as
part
of
provenance
check.
A
Yeah,
that's
a
good
question,
so
it's
always
a
challenge
to
get
a
lot
of
folks
working
on
things
like
this,
because
it's
unless
you
are
attacked,
there's
not
a
lot
of
value.
You
can
prove
to
the
execs,
so
I
would
say
around
four
to
five
people
working
on
this
and
this
not
a
full
time
or
that's
not
the
only
project
they
work
on.
So
it
varies
and
I
would
say
around
60
percent
is
worked
on
this
project
over
the
past
three
years
or
so.
A
B
I
want
to
add
on
that,
because
we
have
already
finished
most
part,
mostly
the
coding
part
or
the
infrastructure
building
part.
Now
we
are
more
focused
on
how
to
increase
an
adoption
number,
how
to
collect
those
numbers
from
our
system
and
make
a
list
of
them
that
inform
the
engineers
to
improve
the
the
their
system.
A
A
For
the
wrong
repository
right
yeah,
so
when
a
CI
CD
job
tries
to
push
an
artifact
which
it
generates
to
artifactory
or
oci
registry,
we
we
need
to
make
sure
or
we
need
to
explicitly
ask
the
artifactory
to
only
allow
artifacts
generated
by
pipeline
X
instead
of
pipeline
y
being
sending
it
to
it.
So
if,
let's
say
there
is
a
node
package
which
has
authorization
to
be
created
by
pipeline,
X
artifactory
would
have
that
information.
B
A
B
Has
an
authorization
like
we
internally,
we
use
a
product
called
Essence,
it
is
also
versus
a
product
is
kind
of
our
back
our
back
system.
This
is
also
a
CSF
product,
so
you
can
find
in
this
in
the
in
the
website
and
every
time
we
send
the
prominence
to
the
graphics
database.
We
actually
send
a
kind
of
a
mission
required
to
the
essence
and
if
you
can
guess
like
a
permission,
they
can
just
put
the
performance
into
database.
Otherwise
the
gear
cannot
so
yeah.
That's
what
we
did.
B
A
Yeah
I
think
those
decisions
were
made
way
before
we
actually
did
it
and
that's
been
happening
for
a
long
time,
but
we
are
now
now
with
cosine
and
six
store
being
widely
adopted.
We
are
looking
at
exploring
that
solution
of
having
either
short-lived
keys
or
ephemeral
keys
by
going
the
keyless
route
to
use
those
instead,
yeah.
B
A
No
yeah
I,
don't
think
it's
a
requirement
and
I
may
be
wrong
again,
but
we
are
working
with
the
right
folks
within
Yahoo
to
make
sure
that
we
change,
because
it's
been
you
always
been
around
for
so
long.
It's
and
the
open
source
tools
have
been
coming
in
recently
as
well.
So
it's
a
change
of
say
learning
curve
as
well.
So
that's
what
we're
working
on
sorry
good.
A
So
we
we've
just
started
collecting
those
and
doing
the
vulnerability
checks.
The
plan
is
to
tie
the
s-bombs
with
the
freshness
policy
so
that,
as
as
we
ensure
that
an
image
or
an
artifact
cannot
be
deployed
for
more
than
six
months,
which
is
a
stale
image
having
an
s-bomb
which
is
older
than
six
months.
Also
may
not
add
value,
because
we
won't
have
any
artifacts
which
are
running
for
old,
which
are
older
than
six
months
So.
A
B
A
D
D
A
A
B
B
A
So
we
currently
integrate
our
the
SD
command
with
screwdriver
provides
with
all
the
standard
templates
so
before
it
gets
published
to
artifactory
We've
integrated
that
SD
command
to
all
all
of
the
existing
pipelines.
So
it
gets
scanned
and
determines
whether
to
proceed
to
publish
it
or
not
so
yeah
we
use,
as
mentioned
gripeshift,
to
get
that
information
and
some
propriety
code
to
figure
out
whether
to
push
or
not.