►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Deep Dive: TUF - Justin Cappos, NYU & Trishank Kuppusamy, Datadog
When VCS systems, build pipelines, or signing servers are compromised, attackers get to distribute malicious versions to millions of unsuspecting users. We present how Datadog used TUF and in-toto to develop, to the best of our knowledge, the industry’s first end-to-end verified pipeline that automatically builds integrations for the Datadog agent. That is, even if this pipeline is compromised, users should not be able to install malware. We will show a demonstration of our pipeline in production being used to protect users of the Datadog agent, and describe how you can use TUF + in-toto secure your own pipeline.
To learn more: https://sched.co/GrdC
A
Sorry,
sorry
and
I'm
a
security
engineer,
data
dog
so
before
joining
data
dog
I
worked
on
my
PhD
at
NYU
Tandon,
with
Justin
Kappas,
who
is
my
PhD
advisor
and
where
I
led
the
research
and
development
of
tough.
At
the
same
time,
docker
implemented
docker
content,
Russ
and
notary,
which
used
stuff
to
secure
the
distribution
of
container
images.
A
This
led
to
tough
becoming
a
member
project
of
the
CNC
F,
which
were
very
proud
of,
and
so
a
pleasure
today
to
take
you
into
a
deep
dive
into
tough
and
show
you
how
we
can
use
the
secure
software
updates
and
various
demanding
settings.
So,
let's,
let's
go
straight
into
it.
First,
let's
remind
ourselves
that
a
problem
we're
trying
to
solve
experts
agree
that
software
updates
are
the
most
important
thing
you
can
do
to
remain
secure
online.
A
This
is
because
updates
fix
security
vulnerabilities
that
can
be
remotely
exploited
by
attackers
alerts
crucial
to
update
software
to
remain
secure
online.
An
important
part
of
the
picture
is
often
neglected,
and
that
is
a
compromise
of
the
repository
used
to
serve
software
updates.
When
this
happens,
the
impact
is
huge.
Attackers
get
to
compromise
millions
of
devices
all
at
once.
The
popularity
of
repositories
is
precisely
what
makes
them
attractive
targets
to
attackers,
especially
nation-state
attackers,
who
have
the
resources
and
motive
to
carry
out
such
attacks.
A
So
we
believe
it
is
prudent
to
assume
that
it's
a
question
of
when
not
if
a
repository
compromise
is
going
to
happen
now
a
goal
is
not
to
prevent
a
compromise.
Rather,
even
if
attackers
have
compromised
the
repository
and
have
access
to
signing
keys
kept
on
the
repository,
the
damage
that
they
can
do
must
be
limited
to
the
greatest
possible
extent.
A
We
can
use
the
update
framework
or
tough
for
short,
as
the
foundation
for
building
compromised
resilience
offer
Abbas
trees
before
I
talk
about
tough.
Let
me
quickly
dispatch
some
background.
A
repository
is
a
server
that
contains
both
packages
and
metadata.
A
package
is
the
smallest
unit
of
update,
which
could
be
either
a
software
library
or
an
application.
A
Having
dispatched
at
background
out
of
the
way,
we're
now
ready
to
see
how
tough
uses
a
few
simple,
yet
powerful
design
principles
to
ensure
that
repositories
can
severely
limit
the
impact
of
a
key
compromise
as
well
as
recover
from
them
first
different
types
of
metadata
assigned
by
different
roles
using
different
keys,
so
that
the
impact
of
the
key
compromised
is
minimized
and
does
not
necessarily
affect
the
security
of
the
whole
system.
The
four
top-level
roles
controlled
by
administrators.
Let's
take
a
look
at
them
one
by
one.
A
The
snapshot
role
which
builds
on
top
of
that
indicates
which
packages
have
been
released
together
at
the
same
time,
but
different
developers
under
Abbas
tree
building.
On
top
of
that,
the
time
stem
role
provides
a
quick
summary
of
whether
any
metadata
or
package
has
been
updated
in
the
repository
at
all.
A
Finally,
the
root
role
acts
as
the
root
of
trust
for
the
whole
system
and
distributes
the
public
keys
for
all
four
top-level
roles,
including
itself,
putting
it
all
together
when
you
use
the
rolls
with
different
keys
to
sign
different
types
of
metadata.
The
impact
of
key
compromised
does
not
necessarily
affect
the
security.
The
whole
system
or,
as
grandma
would
say,
don't
put
all
of
our
eggs
in
one
basket.
It's
good
to
listen
to
Grandma.
A
The
second
design
principle
is
that
a
minimum
number
of
keys
may
be
required
to
sign
off
a
metadata
file,
so
that
attackers
must
compromise
this
trash
whole
number
of
keys
in
order
to
install
malware.
So,
for
example,
you
could
require
a
tree
out
of
four
different
keys,
any
three
three
of
them
to
require
to
sign
them
at
a
data
file
before
you
trust
it
or,
as
grandma
would
say,
when
you
have
to
launch
nuclear
missiles,
make
sure
you
use
different
keys.
Do
you
know
what
they
do
on
these
actual
systems?
A
The
third
principle
is
that,
since
keys
can
be
compromised,
you
need
ways
to
revoke
keys
the
first
ways
to
revoke
keys
explicitly
by
publishing
new
keys
to
replace
old
ones.
Alternatively,
what
we
do
in
tough
is
by
default,
set
expiration
timestamps
on
all
the
metadata
files
which
acts
as
a
natural
incentive
for
you
to
replace
and
rotate
keys
anyway.
A
Last
but
not
least,
one
of
the
most
overlooked
design
principles
that
tough
allows
for
using
a
diversity
of
cryptographic,
hashing
and
signing
algorithms,
so
that
the
compromise
of
any
one
of
them
is
not
sufficient
to
cause
total
failure.
You
could
even
do
interesting
things
like
try,
post
quantum
cryptographic
schemes
at
the
same
time
now,
let
me
hand
it
off
to
Justin,
who
also
is
the
consensus
builder
for
the
tough
project
to
talk
about
how
it
has
and
does
evolve
all.
B
Right,
thank
you
very
much
tree
Xiang.
So
I'm
too
excited
to
just
stand
here
behind
the
podium,
so
I'm
gonna
bounce
around
a
little
bit
as
I
as
I
talk
through
this.
So
one
of
the
things
that
I'm
really
excited
about
in
in
working
on
the
tough
project
is
the
community.
So
really
what
what
we
have
is.
We
have
quite
a
stable
spec
and
you
know
for
doing
updates
in
a
secure
way,
but
this
back
gets
tweaked
and
updated
as
different
people
use
tough
in
all
sorts
of
cool
different
domains.
B
So
what
I'm
gonna
do
now
is
I'm
going
to
walk
you
through
four
of
the
different
things
that
have
made
it
into
tough
and
talk
to
you
a
little
bit
and
just
kind
of
give
you
a
little
glimpse
about
what
our
communities
like
and
what
it's
like
to
be.
You
know
to
contribute
in
this
space,
so
the
first
thing
I'm
going
to
talk
about
is
a
tap
that
we're
currently
in
the
process
of
finalizing
tap
8.
B
It
deals
with
key
rotation
and
salt
revocation,
and
so
let's
say
you
have
a
scenario
here
where
you
have
someone
Jane
Jane
can
either
be
a
project
or
gene
can
be
an
individual
developer.
In
this
case,
we'll
just
have
Jane
be
an
individual
developer
and
Jane
happens
to
be
a
really
popular
individual
developer.
So
lots
of
people
are
like
hey,
you
know,
I
like
Jane's
stuff,
so
I'm
gonna
use,
Jane's,
stuff
and
I'm
gonna
refer
to
Jane's
Jane's
packages.
B
So
if
you
know
a
B
and
C
want
to
install
things,
they're
all
gonna
say
hey,
you
know:
Jane
has
this
library
that
we
like
to
use.
For
instance,
Jane
has
a
key
okay
and
now,
let's
say
that
the
Jane
goes
and
buys
a
new
laptop
or
something
and
get
two
new
gets.
A
new
key
gets
a
new.
You
know
token
for
this
new
device.
B
Now
Jane
has
a
bit
of
a
problem
because
Jane
has
to
go
and
tell
a
and
B
and
C
all
to
update
their
information,
referring
to
Jane,
to
say,
hey
use
this
new
key
okay
and
that's
a
pain
in
the
butt
right.
Also,
let's
say
that
Jane,
you
know
some
evil
hacker
gets
in
and
steals
Jane's
old,
a
key
and
now
Jane
has
to
really
quickly
get
a
and
B
and
C
to
revoke
it,
because
all
those
users
are
potentially
at
risk.
Only.
B
It
turns
out
that
B
is
maybe
out
on
vacation
or
something
like
that.
So
this
is
not
only
you
know,
kind
of
a
usability
problem
for
people,
but
it
also
can
be
a
security
problem
all
right.
So
the
solution
is
something
called
self
rotation
or
revocation,
and
this
is
the
ability
for
Jane
to
go
and
when
Jane
wants
to
go
and
have
a
new
key
or
something
she
can
sign
something
that
says:
hey.
You
used
to
trust
this
one
key
for
me
now.
You
should
trust
these.
B
Like
you
know,
an
accurate
set
of
Jane's
keys
in
there
is
able
to
go
and
say:
don't
trust,
Jane
anymore.
So
it's
a
way
to
go
and
if
your
keys
get
out
there,
somebody
steals
them
and
then
tries
to
do
a
rotate
or
something
you
can
say.
No,
no,
no!
No.
We
need
to
self
revoke.
You
shouldn't
trust
things
in
this.
B
In
this
domain-
and
this
is
work
that
I've
been
doing
along
with
Hannes
from
the
O
camel
community,
who
really
came
with
this
use
case
as
part
of
their
integration
and
has
been
led
in
many
ways
by
my
student,
marina,
Moore
who's.
Put
a
lot
of
effort
into
this.
So
another
cool
thing
that
we
have
that
we
actually
integrated
quite
a
while
ago
is
a.
A
B
Called
tap
five,
this
deals
with
splitting
repository
location
across
URLs,
so
I'll
talk
kind
of
quickly
about
this.
So
sometimes
you
want
to
refer
to
multiple
different
repositories
and
you
might
have
a
repository
where
you
only
want
to
get
some
software
from
it
like.
Maybe
it
hosts
a
whole
bunch
of
different
things,
but
you
really
want
to
say
this
one
package:
you
should
get
it
from
here,
but
you
shouldn't
really
get
other
stuff
from
there.
B
B
What
can
happen
if,
let's
say
you
have
a
service
that
you're
hosting
in
China
and
the
Chinese
government
comes
and
like
seizes
your
server
or
you
know
it's
trying
to
compel
you
to
do
different
things.
How
you
know
there
are
still
ways
to
use
this
to
make
your
system
be
effectively
secure.
So
you
know
this
was
a
really
fun
process,
working
with
folks
at
core,
OS
and
and
docker
and
elsewhere.
B
In
order
to
to
add
this
improvement,
so
the
the
third
tap
I'm
going
to
talk
about
deals
with
multi
repository
consensus,
and
this
is
a
scenario
where
you
have
this
sort
of
problem.
So
let's
say
that
we're
going
to
kind
of
flip
the
model
a
little
bit
here,
so
the
place
where
this
came
up
for
this
is
actually
in
the
context
of
automobiles
in
automobiles,
at
a
first
glance,
at
least
to
me
as
a
naive
person
four
years
ago.
It's
like
hey,
it's
not
that
different.
B
B
You
know
almost
never
have
their
own
network
connection
out.
They
usually
go
out
through
something
like
a
telematics
unit,
which
is
a
big,
complicated,
very
likely
to
be
compromised
piece
of
hardware
on
on
your
network,
and
so,
if
you,
if
you
just
kind
of
flip
your
mindset
for
a
moment
and
imagine
that
your
repository
should
be
telling
you
what
to
install
rather
than
the
devices
making
any
decisions
about
that,
then
we're
gonna
see
that
there's
going
to
be
a
problem
here
for
a
moment.
B
So
let's
say
that
on
this
repository,
you
keep
a
key
that
you're
going
to
use
to
sign
things
online.
You
keep
this
key
on
the
repo.
The
repo
is
gonna
know
then
tell
you
install.packages,
x,
y&z,
with
these
hashes.
The
problem
is:
is
that
when
the
attacker
goes
and
breaks
in
here,
then
the
attacker
has
the
ability
to
tell
you
to
install
whatever
horrible,
like
crypto,
minor
or
key
logger
or
whatever
bad
software.
They
want,
because
you
have
this
one
repo
and
you
have
this
key
for
it.
B
So
you
say:
okay
well,
I've
learned
something
about
you
know
from
treaty
Onix
discussion
before
that
offline
keys
are
better
if
we
can
get
away
from
them
with
them
from
usability
standpoint,
and
so
maybe
I'll
just
put
my
key
in
a
safety
deposit
box,
and
then
you
know
or
a
threshold
of
keys
and
not
have
to
worry
about
it.
But
the
problem
is:
is
that
you
can't
be
responsive,
then,
when
a
car
comes
to
you
and
says
hey
what
packages?
You
know
what
am
I
supposed
to
install
how's
this
supposed
to
work?
B
You
have
to
sort
of
immediately
be
able
to
customize
and
tell
it
you
need
to
have
these
different
pieces
of
software
installed,
there's
a
surprising
amount
of
customization
that
goes
in
and
and
they
will
install
different
packages
on
different
cars
with
different
service
models
depending
on
different
use
cases.
You
know
the
government
vehicles
will
get
different
software
for
their
same
ECU's
and
things
like
that
as
others.
So
it's
very
customized,
even
some
of
the
big
rental
car
companies
and
things
have
their
own
way
of
doing
things.
So
it's
it's
it's
difficult.
B
So
the
idea
behind
tap
four
is
to
use
actually
two
repositories.
So
with
two
repositories,
you
can
have
an
image
repository
that
says
these
are
valid
software
pieces
of
software.
That
could
be
installed,
and
it
can
say
things
like
this
is
a
valid
piece
of
software
that
can
be
installed
on
this
model
of
brake
controller
right
and
this
is
signed
with
an
offline
key
and
then,
in
addition
to
this
there's
another
repository
called
the
directory
repository
that
says,
and
you
as
a
vehicle
should
install
these
things
and
it
has
to
match
both
of
these.
C
B
I
got
a
demo
to
show
you,
and
then
it's
like
check
this
out.
This
is
running,
obtain
your
tough
variant
and
I'm
like
cool.
Why
don't
you
tell
everybody
not
an
owner?
Now
we
don't
have
approval
to
say
this
through,
like
the
18
levels
of
Management.
It's
really
strange,
very
odd
domain,
but
we've
had
obtained
be
adopted
by
one
of
the
big
three
US
automakers
by
major
automakers
in
Asia.
We
have
a
major
automaker
in
Europe
that
is
inking
the
deal
now.
B
It's
all
completely
open
source,
free,
everything's,
available
heavily
audited
by
security
experts,
both
an
automotive
and
traditional
security
standpoint.
If
anyone
else
would
like
to
look
at
it,
we
welcome
you
to
do
it,
we're
currently
about
halfway
through
the
I
Triple
E
isto
standardization
process.
B
B
Is
you
know
the
way?
It
is
an
effective
way
to
distribute
your
software,
but
it
really
only
solves
part
of
the
problem.
So
you
know
you
have
your
version
control
system,
your
build
system,
and
you
know
your
CI
CD
system,
and
eventually
you
get
to
a
point
where
you
want
to
distribute
your
software
tough
steps
in
there
and
does.
B
But
it's
you
know.
It
makes
sure
that
that
bad
things
don't
happen
in
this
last
step.
But
what
about
all
the
other
steps?
What
about
what
happens
so
the
Antonio
project
actually
takes
cryptographically
signed,
secured
metadata
from
other
parts
of
this
process
and
then
goes
and
is
able
to
integrate
this
so
that
a
user
that
goes
to
install
your
software
knows
that
there
wasn't
tampering
done
so
the
same
way
that
if
you
go
and
you
buy
a
bottle
of
aspirin,
how
do
you
know
that
nothing
bad
is
in
that
aspirin?
B
Well,
the
FDA
approved
the
drug
first
of
all,
the
factory
that
made
it
has
certain
health
and
safety
standards
that
had
to
go
into
doing
it.
There's
a
tamper,
resilient
seal
on
the
bottle.
There's
lot
numbers,
there's
all
this
stuff
in
there
that
goes
into
protecting
that
supply
chain.
For
that
aspirin
that
you
put
into
your
body
right.
B
Wouldn't
it
be
nice
if
we
had
the
same
kind
of
protection
for
the
software
that
we
trust
our
lives
to
right?
So
that's!
What
in
Toth
is
all
about,
and
it
integrates
really
well
with
all
sorts
of
existing
pieces
of
software
and
signing
in
things
like
this.
Actually,
in
the
process
of
doing
this
work,
we
found
some
flaws
in
the
way
that
get
signing
worked
and
get
actually
changed
to
use
our
method
of
get
signing.
B
So
there
are
admission
controllers,
not
I,
guess
I
should
say:
there's
a
you
know
a
kubernetes
and
mission
controller
for
for
toff,
which
probably
comes
as
there's
no
big
surprise
to
anyone
in
the
room,
but
there's
also
a
mission
controller
for
in
toto
that's
being
used
in
production
by
some
very
large
sites.
You
can
other
folks
that
have
gone
in
built
and
deployed.
This
include
control,
plane,
it's
being
used
by
the
reproducible,
builds
community
to
do
reproducers
for
Debian.
B
If
you'd
like
to
sign
up
in
run
a
reproducer
and
and
be
someone
else
generating
in
toto
metadata,
so
people
can
be
sure
that
information
wasn't
stolen.
That
would
be
great.
We
have
an
official
Jenkins
plug-in
and
have
had
a
lot
of
people
work
very
hard
on
this
and
with
that
I'll
hand
over
to
tree
trunk,
to
tell
you
more
about
it,
data
dog
and
the
way
in
which
data
dog
took
tuff
and
in
toto
and
fuse
them
together
in
a
cool
and
interesting
way.
So
tree
shock
takes
away
preciate
that.
A
So
I
wanted
to
get
Michael
Bay
to
produce
some
fireworks
and
explosions
and
stuff
like
that,
but
I
was
told
we
ran
out
of
budget
special
effects,
so
I'll
do
my
best.
So
let
me
talk
about
how
we
took
tuff
further
at
data,
so
data
dog
is
a
SAS
platform
that
analyzes
infrastructure,
metrics
your
application,
performance
and
logs,
so
that
your
develops
teams
can
quickly
detect
troubleshoot
and
prevent
issues
occurring.
The
system
the
data
dog
agent
is
software
that
runs
on
your
hosts.
A
Ide,
collects
events
and
metrics
from
your
host
and
sends
them
to
data
dog
where
you
can
analyze
your
monitoring
and
performance
data
and
finally,
the
agent
integration
stem
cells
are
plug-ins
that
collect
metrics
and
services
running
an
infrastructure,
so
they're
more
than
100
of
them
to
come.
Install
out-of-the-box
whenever
you
install
our
agent
for
now
on
when
I
say,
integrations
I
mean
agent
integrations,
which
are
to
be
distinguished
from
say.
A
Our
cloud
integrations
okay,
so
the
problem
is
as
follows:
the
agent
releases
software
on
a
six-week
cycle
and
the
latest
versions
of
integrations
are
always
bundled
with
them.
Then,
however,
we
also
wanted
to
publish
new
integrations
independently
of
the
age
and
release
cycle.
This
is
so
the
customers
can
beta
test
new
features
and
bug
fixes
immediately
before
we
release
them
as
part
the
next
stable
agent.
A
A
Now
what
can
go
wrong
with
this,
when
no
one
has
compromised
infrastructure?
Nothing,
but
if
your
developer
key
is
being
compromised,
then
the
CI
CD
system
will
automatically
build
and
release
more
or
when
your
were
shin
control
system
has
been
compromised
or
when
your
CI
CD
system
has
been
compromised
or
when
your
container
image
registry
used
to
build
packages
has
been
compromised
or
when
both
your
key
and
file
servers
have
been
compromised.
I
think
you
get
the
idea.
A
Unfortunately,
this
is
considered
a
state-of-the-art
in
the
industry.
The
conventional
wisdom
has
been
that
there
is
a
necessary
trade-off
between
security
and
availability.
Now
we
cannot
expect
developers
to
build
and
sign
packages
whenever
there's
change
to
the
source
code.
This
is
because
source
code
can
change
all
the
time
and
it
is
unreasonable
to
expect
developers
to
do
this
expensive
job,
because
that
is
what
CIC
these
systems
are
good
for,
and
it
generally
do
do
this
better.
A
A
A
Each
step
may
be,
for
example,
a
developer
writing
source
code
or
a
container
packaging
that
source
code
into
a
zip
file.
Every
step
must
produce
a
signed
link
or
attestation,
but
the
input
that
had
received
and
the
output
that
produced
when
end-users
put
together
this
series
of
links
or
sine
data
stations,
they're
able
to
inspect
whether
a
package
was
produced.
According
to
your
prescribed
series
of
steps,
the
data
doc
agent
integration
supply
chain
looks
as
follows.
In
the
first
step,
one
of
our
developer
signs
Python
source
code
using
a
Yubikey.
A
We
use
UV
keys
to
prevent
exfiltration
of
signing
keys
and
unauthorized
signatures.
Every
signature
requires
the
entry
of
a
secret
pen
and
or
a
manual
touch
operation,
there's
two
levels
of
authentication
there
in
the
second
step:
oops
technical,
glitch,
Oh,
a
beautiful
presentation,
okay,
very
good,
all
right
anyway,
back
to
order.
A
On
the
end
user
side,
the
data
dog
agent
would
transparently
download
this
metadata
and
wheels
and
ensure
that
the
source
code
inside
the
wheel
contains
exactly
the
source
code
that
one
of
our
developers
produced
now
putting
it
all.
Together,
we
use
stuff
to
bootstrap
how
every
part
the
entire
system
is
to
be
trusted.
We
use
offline
keys
with
tough
to
securely
distribute
the
route
of
trust
for
the
whole
system.
A
A
The
key
point,
though,
is
that
this
offline,
bootstrapping
of
trust
and
protecting
developer,
signing
keys
with
you
be
keys,
is
what
gives,
in
total,
meaningful
security
guarantees
in
this
case,
in
short,
using
both
toughen
in
Toto
allows
you
to
build
tamper,
evident
CI
CD
pipelines,
despite
running
an
untrusted
hardware,
on
the
cloud
okay.
So
what
does
this
tamper-evident
pipeline
bias?
A
A
Furthermore,
we
require
the
entry
of
a
secret
pin
and
a
manual
touch
operation
to
authorize
signatures,
which
I
will
show
you
shortly
does
we
believe
we
have
done
the
best?
We
can
to
significantly
minimize
the
risk
of
a
key
compromise
without
hampering
developer
usability
and
the
future,
we
may
require,
say
a
trash
hole
of
two
developer
signing
keys
to
be
able
to
trust
the
integration
at
all.
A
Now
what
the
tamper-evident
pipeline
shines
in
is
in
how
it
protects
us
again:
man-in-the-middle
attacks
against
our
infrastructure
itself,
so,
for
example,
attackers
a
compromised,
our
github
repository
cannot
cause
malicious
integrations
to
be
released.
This
is
because
stuff
and
in
total,
would
block
this
malicious
integrations
from
being
installed
by
the
agent,
as
the
signatures
would
not
match
what
we
expect
from
our
developer.
Signing
keys.
A
Likewise,
when
our
container
image
registry
has
been
compromised
or
even
when
both
our
key
and
file
servers
have
been
compromised
now,
all
of
the
security
has
got
to
be
simply
unusable
right.
That
is
the
reputation
of
security.
After
all,
so
let's
put
this
to
the
test,
let
me
quickly
show
you
what
our
developers
and
then
uses
experience
okay,
so
this
is
one
of
our
developers
from
their
point
of
view
how
the
source
code
will
be
released.
So
what
happens?
Is
that
they
would
check
out
a
branch.
A
And
edit
source
code
and
so
on,
and
when
they
are
ready
to
release
their
source
code
to
build
new
wheels,
what
they
would
do
is
use
an
internal
cool
tool,
called
data
doc
check
staff,
which
basically
does
things
like
bump
version
numbers
and
so
on,
and
we
simply
use
this
to
call
in
todo
to
call
Yubikey
which
in
turn
would
sign
metadata.
So
let
me
show
you
how
painful
and
owner
is
Dennis.
A
Oh,
it's
telling
me
to
touch
my
she's
precisely
what
I'm
gonna
do
so,
even
if
this
mall
were
sitting
in
my
machine
trying
to
trying
to
get
me
to
sign
something
I
would
know.
So,
let's
see
what
that
actually
did
that
checked
in
a
signed
file
to
their
to
their
github
repository
and
what
it
looks
like
it's
like
this.
A
It's
a
sign
files
you
can
see
and
it's
got
the
hashes
to
follow
Python
source
code.
What
happens
after
here
when
you
push
code
to
github
our
pipeline
actually
does
a
lot
of
the
complexity
right
takes.
All
of
this
metadata
produces
new
tough
metadata
makes
new
wheels
and
releases
them.
So
a
few
minutes
later,
when
users
one
install,
we
put
them
to
an
equally
painful
procedure.
So
let's
see
how
painful
it
is.
I.
A
All
right,
so
this
is
a
new
feature
right.
So
what
happens
when
the
install?
So
you
first
download
a
whole
lot
of
bootstrapping
of
metadata
of
trust.
The
long
story
short
is
that
tough
takes
care
of
darling
is
supply
chain
for
you,
which
sign
using
offline
keys.
It
downloads
the
public
keys,
used
to
verify
the
supply
chain.
It
downloads
link,
metadata
produced
by
containers
in
between,
and
importantly,
the
developer
signed
metadata,
which
is
here,
I
believe
so
anyway.
A
At
the
end,
it
also
tells
you
that
in
total
verified
that
they
will
contain
exactly
the
source
code
that
our
developers
produce.
So
you
want
to
see
what
it
looks
like
the
nitty-gritty
details,
what's
actually
happening
behind
the
scenes
if
you're
interested
it
we're
both
output.
Is
that
the
key
interesting
part
is
this:
we
literally
unzip
the
wheel
you're,
simply
as
if
all
we
transparently
do
it
for
users
and
check
that
the
Python
source
code
inside
is
precisely
what
was
signed
with
my
yubikey
earlier.
A
So,
to
conclude,
sorry
so
to
the
best
of
our
knowledge,
this
is
the
first
tamper-evident
CIC
ICD
pipeline
in
the
industry,
by
integrating
both
tough
and
in
toto.
We
believe
that
we
have
set
a
new
standard
for
the
secure
delivery
of
software
dated
our
customers
can
enjoy
beta
testing
this
with
agent
6.8,
which
was
released
just
earlier
today,
so
get
it
while
still
hot,
so
to
conclude,
tough
serves
the
crucial
foundation
for
building
compromised
resilient
software,
parse
trees
and
various
demanding
settings,
including
automatically
and
securely
delivering
software
in
modern
DevOps
environments.
A
So,
as
we
speak,
as
Justin
alluded
to
earlier
obtained
is
going
through
a
standardized
standardization
process
for
Regals
in
the
North
American,
auto
industry,
at
the
very
least.
So
let
me
quickly
acknowledge
the
gracious
help
of
many
people
who
helped
to
make
tough
what
it
is
today.
In
particular,
our
recent
deployment
at
data
dog
has
helped
us
the
battle
test
boat
toughen
in
toto
and
makes
them
even
better.
D
B
A
B
A
B
So
yeah
right,
but
but
there
there
are
problems,
even
even
in
that
regard
like
because
in
practice
I
don't
know,
I
mean
maybe
some
companies
just
let
anybody
push
to
master,
to
go
and
build
stuff
out
in
production,
with
like
no
code
review
or
whatever
else.
In
some
cases,
organizations
actually
want
to
look
at
code
and
actually
want
to
go
through
a
process
and
being
able
to
take
whatever
process
you
use.
Non-Judgmentally,
so
in
toto
is
not
judgmental.
Tough
is
not
judgmental.
D
B
Well,
the
procedure
is:
is
that
whoever
develops
the
tool
that
you're
gonna
do
install
with?
Does
a
one-time
integration?
And
you
don't
even
know
it's
there?
It's
the
same
thing
toggling
like
like.
We
in
fact
didn't
find
out
that
some
groups
were
using
tough
until
after
they
had
you,
know,
gone
and
either
been
protected
from
an
attack
or
typo'd
something
and
then
started
to
have
tough,
related
error
messages
because
they
type
out
in
an
expiration
time
on
a
key
it
caused
it
to
expire
artificially.
So
it's
totally
transparent
to
users.
A
If
I
could
add
another
point
to
that,
the
other
point
is:
how
do
you
distribute
a
developer
publication
that
needs
to
be
a
secure
process
by
itself
and
that's
a
famous
problem
with
GPG
key
distribution?
You
don't
know
how
you're
getting
the
right
keys
whether
the
key
has
been
revoked.
That's
what
tough.
B
Basically,
how
do
you
handle
the
fact
that
you
start
with
source
code
and
then
something
you
know
some
compilation
stuff
happens,
and
then
you
have
a
binary
right.
Okay,
so
there's
a
bunch
of
there's
a
bunch
of
things
with
this.
So
the
the
easiest
and
most
pervasive
answer
based
on
what
we've
already
done
is
to
just
say:
reproducible
builds.
Basically,
the
you
know
we
have
worked
with.
The
reproducible
builds
community
for
a
couple
years
now,
quite
a
while
been
very
active
with
them
and
they're
reproducers
are
actually.
You
know.
B
There
are
reproducers
at
multiple
sites
that
are
generating
in
toto
metadata
and
doing
this,
and
this
gives
you
assurance
that
if
somebody
broke
into
one
of
those
reproducers
and
started
to
monkey
around
with
it
that
it
would
be
detected,
you
have
to
break
into
all
of
them.
So
that's
sort
of
the
the
short
and
easy
answer
in
Santiago
who's
made
it
in
go
ahead.
Why
don't
you
jump
in
and
answer
this.
E
A
E
Actually,
while
we
were
talking
the
reproducible
build
some,
it
just
ended
and
probably
announced
announcements
in
there
is
that
there's
an
app
transport
for
Debian.
That,
basically
does
this
for
you
as
Trish
shank
says
it
needs
to
be
transparent
for
people.
They
just
need
to
do
apt,
install
app,
get
install,
get
a
debian
package
and
well
know
if
it
was
actually
tampered
with
or
not.
There's
also
something
to
be
said
about
name
spacing
in
the
in
the
case
of
data
dog.
E
Well,
it's
it's
very
clear
that
you
that
developers
can
actually
change
anything
in
the
repository,
but
you
can
also
do
name
spacing
at
the
level
of
Pio
files
right.
The
team,
that's
in
charge
of
changing
the
localization
scripts
or
writing
Unicode
Association
scripts
shouldn't
be
able
to
change
a
make
file
and
the
team
that's
in
charge
of
updating
your
messin,
build
scripts
shouldn't,
be
changing
your
your
source
code
and
so
on
and
so
forth.
E
E
E
That's
true:
I
were
speaking
with
a
couple
of
FinTech
people
and
they
actually
take
the
whole
source
code
and
they
compile
it
themselves.
That's
a
that's
a
trade
of
that
you're
you're
willing
to
make
in
terms
of
security
actually
most
of
these
vulnerability
scanners.
They
just
learn
hashes
of
known
malicious
files
and
if
your
compilation
is
not
bit
by
bit,
reproducible.
Well
good
luck
with
that,
because
your
file
is
different.
This
time.
B
Let
me
add
one
other
thing
to
this,
which
is
that
bit
by
bit
reproducibility,
you
think
in
a
complex
environment
is
nice,
but
this
isn't
like
core
to
what
it
means
to
be
in
toto.
If
you
have
an
environment
that
you
think
is
like,
like
you
just
want
to
know,
did
it
actually
build
on
our
freaky
build
server?
B
We
can
we
can
validate
that
even
better
if
you
have
something
like
some
sort
of
TPM
set
up
or
something
that
you
trust,
that
that
make
sure
that
process
more
trustworthy,
we
can
validate
all
that
within
the
framework.
So
this
is
part
of
the
kind
of
non-judgmental.
If
you
go,
the
extra
mile
for
reproducible
builds,
we
take
advantage
of
it.
If
you
can't
for
your
environment,
we
take
advantage
of
what
you
got.
D
B
That's
by
your
build
server,
having
an
HSN
but
EPM,
whatever
else
like
that,
and
that
will
get
you
into
you
know,
possibly
in
the
news,
because
someone
broke
into
your
built
server
as
they've
done
with
fedora
and
a
couple
of
other.
You
know
historical
situations
like
it,
but
it's
better
than
somebody
just
being
able
to
build
somebody
else
somewhere
else
and
and
have
stuff
to
be
trusted.
So
you
know.
E
A
It's
interesting:
you
should
mention
that,
because
there's
a
proposal
called
sub
resource
integrity.
If
I
remember
it
correctly,
the
basic
idea
is
you:
would
you
would
use
TLS
to
distribute
the
hashes
of
all
the
JavaScript
files
that
you're
using
and
I
think
you
could
do
even
more?
So
if
you
see
the
end,
tries
to
substitute
and
give
you
something
else,
you
would
know
it's
a
small
lighter
version
of
that,
but
not
not
that
not
the
whole
shebang.
A
A
A
B
Okay,
so
the
question
is:
is
that
when
you
have
a
user
go
and
get
information
from
a
repository?
Is
there
a
better
solution
than
trust
on
first
use,
and
the
the
answer
is:
is
that
yes,
but
it
depends
on
your
domain
so
often
like
this
happens,
one
of
two
ways:
either.
It's
trust
on
first
use
the
first
time
you
fire
up
your
client
or
your
client
itself
comes
shipped
with
some
root
keys
or
something
like
that
in
it.
B
A
D
E
E
E
I
think
a
nice
side
effect
of
using
in
toto
is
that
it
introduces
process.
So
it's
for
things
like
this
I
I
know.
The
the
specific
example
like
a
month
ago
was,
like
everyone
was
screwed
to
the
sky,
and
it
was
pretty
much
like
hey
this
very
nice
developer.
Data
tool
didn't
have
any
interesting,
interesting
it
for
like
a
while
and
some
guys
some
random
guy
was
like,
can
I
have
it
and
well
all
of
the
people
like
them.
E
You
see
and
they
were
like
well
you're
handing
like
crucial
code
to
like
some
random
got
it
some
random
guy
in
theory,
within
total,
you
can't
have
this
processes
also,
well-defined
and
well.
You
could
also
have
the
fact
that,
well,
you
trust
this
guy,
but
you
don't
trust
anyone
else
and
then,
as
a
consequence,
you
can
say:
hey
I'm,
downloading
the
the
layout
for
this
piece
of
piece
of
software
and
suddenly
the
guy
who's
supposed
to
like
tag
this
release.
It's
not
doing
it
what's
happening
right.
B
Because
in
toto
makes
you
make
all
this
information
visible,
there
are
people
that
are
very
excited
and
looking
at
all
this
metadata
and
trying
to
create
things
like
scores
about
how
concerned,
you
should
be
that
there's
something
wrong
with
this
project.
So,
all
of
a
sudden,
if
that
project
goes
from
a
green
smiley
face
to
you,
know
like
a
orange
concern,
frown
then
maybe
before
you
deploy
that
in
your
infrastructure,
you'll
look
a
little
more
closely
at
that
and
try
to
suss
out
what
might
be
happening.