►
From YouTube: Learning from Supply Chain Failures and Best Practices in Other Industries - Demian Ginther
Description
Don't miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from 18 - 21 April, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Learning from Supply Chain Failures and Best Practices in Other Industries - Demian Ginther, Superorbital, LLC
Supply chains are critical in many industries, but are only gaining attention as vitally important in the software industry in the past couple years. What can we learn from established supply chain best practices, and from the biggest failures in various industry supply chains? How can we apply that to our own work in securing our own critical infrastructure? In this talk we will discuss the evolution of supply chain processes in the physical world. We’ll explore what parts of physical supply chains apply to our work, how they have been implemented in those paradigms, what sorts of failures can and have occurred, and how we can utilize the lessons learned in our own software supply chain pipelines.
A
A
little
bit
about
me,
I'm
Damian,
Ginther
I
am
a
distributed
systems
engineer
at
Super,
orbital
I'm,
going
to
talk
today
about
some
supply
chain
failures
and
practices
in
the
non-software
world
that
we
can
learn
from
and
apply
to
our
own
work
and
software.
A
little
bit
about
me.
I've
worked
in
sysadmin
for
over
20
years,
moving
into
what
what's
now
called
devops.
A
During
that
time,
I've
worn
a
lot
of
hats.
Security,
though,
has
always
been
at
the
core
of
what
I
do.
I
put
some
of
my
employers
up
here,
but
I've
done.
Network
Administration
incident
response,
penetration
testing,
sysadmin
physical
security
for
a
data
center
I've
worked
for
governmental
agencies.
So
if
anybody
wants
to
talk
about
any
of
that,
please
feel
free
to
say
hello.
A
So
what
is
a
supply
chain?
This
is
kind
of
a
silly
question
here
at
this
conference,
but
you've
probably
heard
about
Supply
chains
and
other
Industries
like
food,
food
and
goods
and
commerce,
medical
and
chemical
entertainment
has
a
supply
chain,
energy
utilities.
They
all
share
a
lot
of
common
Concepts,
but
the
overall
definition
of
a
supply
chain
is
here:
it's
a
network
of
entities,
individuals,
groups,
resources
and
Technology
involved
in
the
creation
and
sale
of
a
product
or
put
differently
the
entire
system
of
producing
and
delivering
a
product
or
service.
A
So
in
the
past
few
years,
we've
seen
a
lot
of
growth
in
the
complexity
of
software
and
a
huge
growth
in
open
source
software
and
we've
seen
a
lot
of
threats
emerge
in
in
many
many
parts
of
the
supply
chain.
For
that
Focus
had
been
a
lot
on
endpoint
protection,
phishing
other
email
issues
like
that
and
in
around
2020
it
started
to
become
painfully
clear
that
the
supply
chain
for
the
software
everyone
was
relying
on
was
really
becoming
a
big
threat.
A
So,
of
course,
there
had
been
some
supply
chain
issues
before
2020,
but
in
in
2020
there
was
all
the
solar
winds,
Orion
attack,
and
it
was
an
infection
of
a
system
that
built
commercial
software.
There
was
an
estimated
18
000
government
entities
and
Fortune
500
Fortune
500
companies,
nine
federal
agencies
and
more
than
a
hundred
other
companies.
A
A
And
keynote
crashed
on
apologize
for
something
I,
don't
know
that
was
weird,
so
there
was
a
self-hosted
git
server
that
they
were
using
someone
forged
some
malicious
commits,
along
with
a
forge
developer
signatures,
and
there
was
an
rce
added
to
the
PHP
code.
The
remediation
for
this
included,
moving
to
2fa,
moving
to
hosted
GitHub
turns
out.
Maintaining
your
own
infrastructure
is
difficult,
but
it
also
impacted
The
credibility
and
the
perception
of
PHP
in
in
a
lasting
way.
A
In
my
opinion,
in
mid-2021
the
build
system
for
home
brew,
if
anybody's
ever
used
that
to
install
software
in
their
Mac,
there
were
vulnerabilities
in
the
build
system
which
allowed
untrusted
actors
to
inject
arbitrary
code.
Using
a
pull
request
to
the
repository,
it'll
basically
allow
anybody
to
insert
any
code
they
wanted
into
anything
in
the
Brew
ecosystem.
A
So,
obviously
you
know
configuring
your
pipelines
and
secure
configuration
of
those
pipelines
is
important
as
well
as
trusting
your
developers
and
then
you
know,
we've
heard
a
lot
about
log4j.
In
this
conference
there
was
a
vulnerability
discovered
in
the
release
code,
which
allowed
unsophisticated
attackers
to
fully
control
servers.
A
There
was
a
huge
cross-section
of
devices
and
software
that
were
affected
and
a
lot
of
people
didn't
actually
know
that
they
had
log4j
in
their
software,
because
it
was
largely
a
dependency
added
by
other
software
packages
and
the
initial
incidents
that
caused
these
failures,
maybe
were
not
preventable.
I
think
determined.
Attacker
can
and
will
eventually
find
a
hole
somewhere,
but
the
follow-on
impacts
and
the
overall
exposure
of
information
and
the
cost
to
remediate
them
I
think
could
have
been
reduced
by
better
supply
chain
security.
A
Where
did
these
raw
ingredients
come
from
transparency?
Who
made
the
product
what
what's
in
the
product?
What
are
their
qualifications,
safety
and
quality
in
the
in
the
non-software
supply
chain?
These
are
things
like
storage
of
the
product,
freshness,
proper
packaging,
but
certainly
safety
and
quality
are
relevant
Concepts
to
apply
Logistics.
In
this
case,
we
can
think
about
logistics
for
software
as
our
pipelines
and
resource
utilization
of
those
pipelines,
tracking
distribution.
A
Where
is
the
software
deployed?
What
versions
deployed
do
I
know
what
dependencies
that
version
has
can
I
track
that
piece
of
software
down
quickly
if
I
need
to
after
sales
tracking?
So
where
is
the
product
in
use?
Who
are
my
customers
here
are
some
things
that
don't
so
much
apply:
production,
processing,
packaging
storage,
physical
distribution,
these
kind
of
things
most
of
this
is
for
wholesale
and
retail
sale
of
physical
goods,
and
there
are
some
parallels
that
you
could
draw,
but
mostly
aren't
applicable
to
what
I'm
trying
to
discuss
here.
A
So
where
did
the
honey
that
you
ate
yesterday
originate
and
what
plants
did
it
come
from
and
why
might
that
be
important
for
allergy
mitigation
and
for
the
taste
of
the
honey
and
that
kind
of
thing
as
Scott
Lee
Scott
Gottlieb
points
out
in
this
quote?
The
public
and
economic
costs
of
non-transparent
Supply
chains
are
considerable.
A
So
if
we
apply
this
concept
of
transparency
to
our
software
supply
chain,
if
you
think
about
the
last
time,
you
used
a
node
or
a
python
package
in
your
project,
do
you
always
go
and
check
out
their
code
and
then
follow
the
rabbit
hole
to
look
at
all
the
code
that
they
included
and
what
their
dependencies
are?
You
know
we
all
know
this
joke
about
the
node
modules
being
heavier
than
an
actual
black
hole.
How
long
did
your
last
npm
install
take,
and
did
you
recognize
all
those
packages?
A
How
do
you
determine
where
you
have
the
products
installed
and
how
do
you
detect
if
there's
a
new
vulnerability
in
those
products,
and
how
do
we
make
sure
we're
not
this
XK
CD
cartoon,
where
we
build
this
fantastic
new
product
but
somewhere
in
the
dependency
chain?
There's
a
project,
that's
under
supported,
vulnerable
or
problematic.
A
Let's
talk
about
safety
and
quality
in
many
other
Industries
safety
and
quality
is
about
freshness
of
the
product,
whether
it
was
stored
properly
and
how
outside
factors
might
have
affected
it,
rotten
food,
stale,
food,
those
kind
of
things,
degradation
to
exposure
to
light
or
moisture,
particularly
that's
a
problem
with
chemical
and
medical
supply
chains,
but
definitely
applies
to
others
or
in
the
case
of
the
entertainment
supply
chain,
the
safety
of
the
workers,
the
safety
of
the
truck
drivers
in
in
logistics
and
transportation.
Much
of
that
doesn't
apply
to
software.
A
Luckily,
software
doesn't
go
bad,
it
doesn't
have
an
expiration
data.
You
could
argue
that
it
probably
should,
but
these
Concepts
do
apply
to
software.
But
most
of
this
is
the
safety
and
quality
of
software
comes
down
to
the
previously
covered
traceability
and
the
ability
of
you
as
the
developer
to
make
sure
that
you
are
auditing
and
reviewing
what
you
put
in
your
code
and
that
you
are
not
including
toxic
ingredients
in
your
code.
B
C
A
We
aren't
driving
trucks
or
ships
here,
but
there
is
a
pipeline
that
is
analogous
to
those
physical
world
Concepts.
So
here
are
some
common
tools,
both
of
them
both
open
source
and
paid,
and
you
can
use
these
for
those
analogous
processes.
It's
not
a
full
list,
but
these
are
tools.
I've
used
and
thought
were
good
quality,
but
whatever
tools
you
choose
to
use,
you
must
make
sure
to
incorporate
best
practices
using
security
at
every
step
scanning.
During
builds.
Storing
your
codes
securely
making
sure
that
your
repositories
are
secure.
Contributors
are
vetted.
A
Contributors
are
aware
of
security,
regularly
scan
your
old
builds
and
s-bombs
everybody's
talking
about
s-bombs
for
new
vulnerabilities.
This
is
something
that
was
mentioned
in
the
keynote
there's
a
gap,
I
think
in
the
the
way
that
we
handle
s-bombs
after
they're
created.
We
don't
want
them
to
just
sit
there
and
gather
dust.
A
New
vulnerabilities
are
discovered
all
the
time,
so
it's
possible
that
old
s-bombs
will
show
you
new
vulnerabilities
scan
your
software
before
you
deploy
it.
Use
an
admission
controller
to
deny
entrusted
containers
ensure
that
you
have
a
way
to
know
what's
running
and
where
and
keep
a
record
of
what
you
deployed
and
what
version
it
is,
so
you
can
find
it
in
a
hurry
in
case
you
need
to
so.
Let's
talk
about
failure.
A
That
was
in
the
title
of
the
talk,
so
I
love
failure,
I
think
it's
the
best
way
to
learn
in
the
moment.
It
can
be
pretty
awful.
The
dire
consequences
can
happen,
but
you
see
across
a
lot
of
other
Industries
like
Toyota
uses
the
concept
of
Kaizen
or
continuous
Improvement.
In
order
to
have
continuous
Improvement,
you
must
be
open
to
failure
and
you
must
identify
your
mistakes
and
work
toward
developing
Solutions
in
many
ways.
This
is
also
what
drives
agile
software
development.
You
iterate,
you
determine
what
didn't
work.
A
A
A
The
source
of
this
outbreak
was
identified
and
the
produce
was
recalled
and
on
the
surface
it
looks
like
a
success,
but
economic
analysis
shows
that
the
impact
of
the
outbreak
could
have
been
lessened.
Had
there
been
more
transparency
and
traceability
for
the
affected
produce.
If
there
was
had
been
a
system
in
place
that
allowed
consumers
to
know
where
their
food
was
grown,
it
could
have
potentially
reduced
the
time
to
determine
the
source
of
the
lettuce,
determine
change
the
scope
of
the
recall
and
would
have
affected
the
subsequent
drop
in
in
lettuce
sales.
A
Paper
had
to
be
tracked
through
Farm
Processing,
packaging
distribution
sales
and
since
then
Walmart
has
implemented
a
blockchain
based
tracking
system,
which
seems
like
Overkill
to
me,
but
I
I
didn't
design
it.
It's
reduced
the
time
to
Mere
seconds
actually
to
determine
that
entire
chain,
which
I
think
is
excellent.
A
Another
case
here,
the
peanut
Corporation
of
America
between
2006
and
2009.
They
were
leasing,
peanuts,
contaminated
with
salmonella
into
the
food
supply,
and
there
were
over
700
infections.
Nine
people
died
and
there
was
a
big
cover-up
by
the
people
who
ran
the
company
and
the
eventual
Fallout
of
this
was
a
federal
prosecution
of
officials
on
a
76
count
indictment.
They
got
successful
convictions.
There
were
fines,
huge
huge
amount
of
fines.
A
Companies
only
know
their
direct
supplier
where
they
got
the
peanuts
and
their
direct
customer.
So
some
companies
purchase
the
contaminated
peanuts
from
a
third
party
without
knowing
that
the
original
source
of
the
peanuts
was
the
peanut
Corporation
of
America.
So,
even
after
the
recall,
this
meant
they
continued
to
sell
the
products
because
they
didn't
have
sufficient
visibility
into
the
entire
supply
chain
and
we,
as
software
developers,
we
don't
have
any
such
excuse.
We
we
have
the
technology
to.
If
we
were
selling
peanuts.
A
We
could
know
the
individual
customers
who
bought
the
peanuts
and
the
peanut
farmer
could
know
the
name
of
the
person
who
purchased
each
batch
of
peanuts
and
the
customer
can
know
who
farmed
the
peanuts
and
each
party
can
know
where
all
those
peanuts
went.
You
know
from
from
start
to
finish
it's
possible
to
do
that.
I,
don't
think
we're
necessarily
quite
there,
but
I
think
it's
certainly
certainly
a
possibility
with
the
technology
that
we
have.
So
we
can
take
from
this
that
it's
not
enough
to
know
what
your
dependencies
are.
A
A
What
does
the
software
or
container
or
the
helm
chart
contain
what
pieces
of
software
are
used
in
the
creation
of
your
software,
and
you
want
you
want
to
use
a
tool
that
creates
a
multi-level
s-bomb,
because
it's
not
enough
to
just
know
your
own
dependencies
you've
got
to
know
the
dependencies
of
those
also
those
dependencies
as
well
and,
very
importantly,
in
order
to
lighten
the
burden
of
creating,
storing
and
signing
those
s-bombs.
You
must
use
an
automated
process.
A
You
cannot
do
this
by
hand,
it's
it's
too
onerous
to
do
that
and
two
prone
to
mistakes,
not
auditable,
and
you
should
sign
your
s-bombs.
You
should
attest
that
they
are
what
they
say.
They
are
because
of
the
theory
of
Integrity.
You
want
to
make
sure
that
your
s-bomb
is
is
maintains
its
integrity
and
that
it
has
not
been
modified
since
it
was
created,
there's
a
lot
of
question
about
how
we
can
verify
that
it's
correct
as
well
and
I
haven't
seen
a
good
solution
for
that,
yet,
but
I
hope
hope
one
crops
up.
A
A
A
It
could
have
mitigated
some
of
the
losses
experienced
and
then
also
poor,
inventory
and
tracking
of
goods
on
the
ship,
because
it's
just
a
bunch
of
giant
containers
on
the
ship
and
and
there's
poor
counting
in
a
lot
of
those
systems
meant
that
suppliers
and
customers
didn't
really
know
if
their
goods
were
on
that
ship
at
all
and
they
were
in
limbo.
Regarding
whether
their
goods
were
affected,
when
they
might
arrive,
so
caused
a
lot
of
churn
in
the
in
the
supply
chain,
so
there
was
a
ripple
effect
of
that.
A
It
was
a
six
day
bottleneck
that
was
stuck
there
for
six
days
until
they
finally
dug
it
out
and
it
created
months
of
effects
and
a
huge
financial
effect.
Six
like
60
billion
dollars
is
that's
a
lot,
so
you
know
what
we
can
take
from
this
is
that
it's
important
to
have
infrastructure
and
resources
available
to
support
a
plan
B
and
maybe
a
plan
C
and
A
Plan
B
Plan
D.
If,
if
you
know,
if
you're
not
creating
a
plan,
that's
tenable
in
in
the
first
place.
A
In
order
to
ensure
that
your
supply
chain
can
be
resilient
to
disruptions
and
unforeseen
events.
You
should
dedicate
some
thought
to
redundancy
if
your
budget
allows
use
multiple
regions
in
your
cloud
provider,
if
you're
on
premise
use
multiple
data,
centers
lacking
that
please
use
redundant
power
and
cooling.
A
I
worked
for
a
data
center
that
had
cooling
failures,
often
at
one
in
the
morning
and
I
had
to
get
up
and
drive
to
the
office
and
flick.
A
switch.
Don't
be
me
if
your
supply
chain
pipeline
breaks
down,
have
a
plan
in
place
for
how
you're
going
to
track
and
trace
your
software.
A
human
process
for
that
kubernetes
can
assist
with
some
redundancy
and
some
self-healing,
and
you
want
to
make
sure
that
you
have
monitoring
and
alerting
in
place
for
resource
utilization
errors.
A
Visibility
into
your
applications
logs
are
great
and
they
can
assist
in
troubleshooting
and
maintaining
integrity
and
traceability
of
your
code
pipeline.
But
they're
not
enough
logs
are
just
indicators
of
what
already
happened.
So
in
order
to
avoid
problems
before
they
happen,
you
need
to
start
developing
a
system
which
verifies
everything.
So
we're
starting
to
talk
about
a
zero
trust
architecture
here
and
if
you
take
the
time
to
actually
develop
a
plan
B,
you
must
practice
that
plan
b
or
plan
C
and
D.
A
If
your
human
processes
are
too
slow,
maybe
you
can't
get
around
the
Cape
of
Africa
fast
enough
to
prevent
the
goods
from
spoiling.
Your
plan
may
not
be
helpful
if,
in
the
event
of
an
emergency
or
unforeseen
event
like
that,
and
if
you
want
to
develop
a
really
resilient
systems,
I'd
recommend
looking
into
chaos
engineering
so
that
your
engineers
can
start
to
know
what
happens
in
each
part
of
your
supply
chain.
When
things
start
to
fail.
A
It
was
determined
that,
because
of
the
implementation
of
a
manual
processes
for
identifying
transfusion
materials
and
patients,
a
nurse
or
attack
would
go
in
and
they
would
look
at
the
the
patient's
bracelet
and
then
they
would
look
at
the
the
blood
transfusion
materials.
There
was
a
48
error
rate.
This
is
terrible
and
people
can
die.
You
know
you
give
them
the
wrong
blood,
they
will
pass
away.
So
this
was
just
due
to
human
error
manual
mistakes,
so
they
implemented
an
electronic
system
where
they
had
a
scanner.
A
They
could
scan
the
bracelet
scan
the
materials
and
it
led
to
a
massive
increase
in
traceability
and
error,
error
reduction,
which
obviously
led
to
better
outcomes
for
the
patients.
Not
not
many
people
died
and
after
2005
they
discovered
that
traceability
was
over
99.
So
a
very
small
error
rate.
How
did
this
happen?
Automation
is
what
allowed
this
to
happen,
always
automate
your
systems.
Manual
systems
are
impossible
to
audit,
maintain
or
properly
rely
on,
and
in
your
automation.
A
A
Verification
is
the
Cornerstone
of
a
good
security
program
and
the
basis
for
a
zero
trust.
Architecture.
I
won't
get
into
the
complete
zero
trust
architecture.
Conversation
here,
because
I
only
have
six
minutes
left,
but
some
good
things
to
think
about
are
using
strong
authentication
in
your
processes,
using
least
access
policies
and
make
sure
that
you're
validating
your
application.
Behavior,
don't
just
assume
that
your
application
is
going
to
behave
the
way
that
it
says
on
the
tin
and
use
this
same
validation
and
verification
procedures
for
all
components
of
your
system,
including
your
supply
chain.
A
You
cannot
have
a
security
system
which
relies
on
on
human
beings
to
manually
review,
scan
results,
detect
vulnerabilities.
You
also
can
have
a
system
which
relies
entirely
on
computer
automation.
Yet
anyway,
maybe
chat,
GPT
can
start
analyzing
and
fixing
our
vulnerabilities
someday.
Someone
has
to
look
at
logs.
Someone
has
to
look
at
vulnerability,
scan
output
and
your
team
has
to
take
action
to
remediate
and
preferably
you'll
have
another
team
using
separation
of
Duties.
A
That
will
verify
that
the
remediation
has
occurred
and
audit
audit
you,
but
in
the
meantime
you
can
automate
some
of
the
easy
stuff
so
things
like
version
bumps
and
dependencies
with
dependabot
or
other
tools
like
that
use
one
of
those
tools
to
analyze
your
dependencies
alert
you
that
there's
updates
have
it
make
a
PR.
If
you
can
automate
tests,
use
tdd
to
build
your
software
so
that
you
can
know
if
a
version
bump
breaks
your
software.
A
That
will
reduce
the
need
for
your
team
to
manually
test
when
versions
change
if
they
can
just
go
into
GitHub
and
see
that
well,
this
passed
all
the
tests,
even
though
it
upgraded
a
package
you
can.
You
can
merge
that
use
automated
scans
in
your
pipelines
make
sure
that
if
a
vulnerability
is
discovered,
it
will
prevent
emerge.
A
If
you're
using
containers
or
kubernetes,
it's
pretty
easy
to
implement
an
admission
controller
which
will
ensure
validation
and
verification
of
all
these
things
that
we're
talking
about
here,
so
I'm
not
going
to
read
through
this
slide.
We've
talked
about
these
things,
but
that's
that's
my
talk
and
here's
some
credits
and
any
are
there
any
questions.
A
A
B
C
A
Think
I
think
somebody
mentioned
yeah
I,
think
somebody
mentioned
in
the
keynote.
Yes
in
the
Keynotes
yesterday
about
using
stars
as
a
as
kind
of
a
dark
pattern.
It's
I
mean
it's
difficult
to
know.
I
personally
I
look
and
see
when
the
last
commits
were-
and
you
know
the
you
know,
the
rate
at
which
it's
been
developed,
but
that's
some
annual
process
too
so
go
ahead.
A
Blockchain
has
a
very,
in
my
opinion,
narrow
application
set
and
I
think
that
a
lot
of
things
that
people
want
to
use
blockchain
for
could
be
much
better
managed
with
you
know
traditional
database
type
systems,
I,
don't
I,
don't
know
why
it's
not
more.
You
know,
I,
don't
know
why
people
haven't
more
done
more
experimentation
with
it,
but
I
do
know
that
it
is
a
huge
thing
for
like
physical
Supply
chains.