►
From YouTube: Rootless, Reproducible, and Hermetic: Secure Container Build Showdown - Andrew Martin, Control Plane
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Rootless, Reproducible, and Hermetic: Secure Container Build Showdown - Andrew Martin, Control Plane
Rootless container image builds (as distinct from rootless runtimes) have crept ever closer with orca-build, BuildKit, and img proving the concept. And they are desperately needed: a build pipeline with an exposed Docker socket can be used by an attacker to escalate privilege - and is probably a backdoor into most Kubernetes-based CI build farms. With a slew of new rootless tooling emerging including Red Hat’s buildah, Google’s Kaniko, and Uber’s Makisu, will we see build systems that can securely build untrusted Dockerfiles? How are traditional build and packaging requirements like reproducibility or hermetic isolation being approached? In this talk we: - Compare the strengths and weaknesses of modern container image build tools - Explore the safety of untrusted image builds - Live demo attacking container build pipelines - Chart the history and future of container image build tooling
https://sched.co/MPYp
A
Is
falling
and
welcome
to
Brutus
with
useful
and
somatic
very
sad
state.
My
consent,
I
is
not
make
it
today,
but
Jesus
with
us
and
spirits
and
let's
go
hi
Manny
I,
don't
like
I'm
a
lover
of
all
things,
grateful
you
could
say:
I'm
a
bill
fanatic
and
an
applicant
of
continuous
everything,
my
family,
which
is
senior
ski
instructor
engineering
practices
with
folks
on
executing
and
terrorized
bills,
and
how
to
attack
them.
A
A
A
A
Any
privilege
to
this
building
modes
of
cloud
service
markets
and
hopefully
preventing
against
licious
fighting
fires
building
without
exactly
or
second
axis,
for
class
privilege
escalation
attacks.
Of
course,
everything
in
Linux
is
bad
and
discretionary
access
control
to
those
files
is
specified
by
users
so
running
inside
inna
as
cruise.
But
after
you,
snakes,
mace
means
that
any
other
databases
that
odd
samples
move
that
user
closer
and
closer
the
groups
on
post
some
built
tools
will
create
a
username
space
to
get
the
crowds,
probably
others
great
lengths
to
avoid
doing
so.
A
From
tail
piece
right
to
so
many
touch
points
to
mission,
so
many
places
in
username
space
with
user
IDs
into
gas
different
user
IDs
is
not
a
fitness
I
switched.
We
are
transportation.
Shipping
give
the
impression
of
a
different
user,
take
words
in
the
user
database,
UID
0,
but
obvious
straight
use
by
any
user,
and
the
posters
had
100
rupees
users
only.
A
And
my
personal
objections
of
white
light
syndrome,
the
ship
and
that's
upset
because
languished
for
the
lips
curl
mailing
list.
For
many
years
it
has
been
reflected
the
political
helling
city
picked
up.
This
patch
set
run
relentless.
This
will
allow
dynamically
piping
user
IDs
similar
to
use
to
make
spaces,
but
it
is
filesystem
Evelyn
a
different
approach.
Different
compromises
is
super
exciting.
So
that's
one
one
rootlessness
now
about
reproducibility.
A
We
want
reproducibility
to
stop
that
happy
real
life
prior
acts
like
x-ray,
ghost
and
attack
the
Xcode
idea
that
we've
been
in
combat
deployments
malware
in
able
to
access
and
troll
the
title
of
facts
distinguish
Jesus
at
22,
conduction
Paris,
also
developing
compilers
half
the
virus,
that's
enabling
code.
This
may
sound
familiar
and
the
links
for
this
plus
tax
problems.
A
A
Sex,
like
Debian,
have
been
moving
towards
fully
reproducible
builds
for
all
of
their
packages.
This
creates
an
independently
verifiable
path
from
spidery
to
sort
sorry
from
source.
The
binary
code
there's
a
lot
of
work
going
on
at
the
distribution
level,
and
we
want
to
be
able
to
extend
that
work
into
container
images.
A
A
This
is
fundamentally
a
question
of
trust
from
our
build
environments
through
our
tools
and
the
output
software,
the
packages
and
the
libraries
that
are
consumed
in
the
build
and
runtime
of
the
system,
and
this
trust
goes
all
the
way
back
to
the
identity
of
the
user,
committing
changes
to
the
lines
of
code
in
your
source,
repo.
But
what
happens
if,
like
Ken,
we
do
not
trust
that
our
build
tools
and
supply
chain.
Well,
we
can
rebuild
the
same
package
in
multiple
locations.
A
If
our
builds
are
deterministic
and
reproducible,
we
can
hash
and
sign
the
outputs
and
compare
them
to
other
builds.
Any
non
matching
builds
are
then
subjected
to
scrutiny.
Attacking
this
system
requires
either
a
shared
supply
chain
attack
or
an
individual
assault
on
each
and
every
node,
in
the
build
system
raising
the
expense
of
the
attack
quite
significantly.
A
This
requires
GPG
signatures
for
each
stage
of
a
build
we
can
run
duplicate,
distributed,
builds
verify
all
of
our
output
signatures
and
check
that
the
code
is
built
correctly
and
to
give
us
some
certainty
that
our
build
environments
have
not
been
compromised
or
that
they
are
all
equally
compromised.
So
to
achieve
this
property
of
reproducibility
for
OCI
images,
we
require
local
or
pin
dependencies,
no
non-deterministic
Network
rules
or
any
non
determinism
in
general
and,
of
course,
an
identical
product.
A
Every
time
no
time-based
outputs,
identical
output
ordering
ultimately
bit
for
bit
similarity
so
noble
and
tamper
proof
outputs.
Of
course
the
OCI
image
is
addressed
by
sha-256
of
its
contents.
We
can
easily
see
when
any
of
these
conditions
are
unmet
in
an
individual
OCI
build.
However,
non-deterministic
actions
like
network
call
can't
be
identified.
A
Excuse
me
can't
be
identified
in
docker,
run,
commands
and
so
require
a
user
to
add
things
like
hash
validation
for
downloads
that
will
fail,
build
on
incorrect
values
and
essentially
trick
the
docker
image
cache
into
being
reproducible.
Speaking
of
docker
images,
a
brief
diversion
onto
OC
IB
to
the
version
1
spec
is
a
tad
ropey.
In
places
it
was
derived
from
the
state
of
the
tooling
written
in
go
lang
as
of
the
version
it
was
released
in.
That
includes
some
bugs
from
that
version
of
the
language
OC.
A
A
There
are
links
at
the
end
of
the
talk
and
on
to
the
final
property
hermit
ISM
related
to
the
practice
of
hermitage
seclusion,
isolation
and
independence.
In
container
terms.
Our
builds
should
not
impact
each
other.
They
should
not
leak
state
or
indeed
be
knowable
by
another
build,
and
we
shouldn't
rely
on
anything
outside
of
build
context.
Hermit
ism
also
makes
running
sorry
building
images
in
multi-talented
environments
practical.
This
means
we
can
share,
build
farms
across
projects
with
similar
trust
boundaries
and
reduce
the
cost
of
running
many
build
workers
and
generally
reducing
CI
cycle
time.
A
Okay.
So
how
do
we
attack
a
build
if
dockerfile
is
untrusted?
Malicious
commands
in
the
run
directive
can
attack
the
host?
They
can
attack
things
on
the
host,
none
loopback
network
ports
services.
They
can
enumerate
other
Network
entities,
the
cloud
provider
build
infrastructure
routes
to
other
services,
metadata
API
s
or
despite
a
trusted
docker
file.
A
malicious
or
compromised
image
specified
in
the
Fromm
directive
has
access
to
any
build
secrets
that
are
added
or
pulled
in
mounted
into
that
image.
At
runtime,
a
built,
I'm,
sorry
or
a
malicious
image
has
a
nefarious
on
build
directive.
A
Happily,
they
are
less
used
these
days,
but
nevertheless
it's
arbitrary
code
execution
in
a
descendent
image
or
with
docker
and
docker.
We
can
get
to
the
hosts
as
we're
running
in
a
privileged
container
which,
as
we
know,
turns
off,
set
calm
up,
Armour
capabilities
or
namespaces
and
leaves
us
running
as
close
to
the
host
as
we
could
possibly
be,
and,
of
course,
we
can
get
out
of
a
container
in
all
the
traditional
ways
by
attacking
the
kernel
so
to
protect
our
builds.
What
do
we
do?
We
can
prevent
network
or
Internet
bound
egress?
A
Why
is
this
needed
anyway?
It's
better
to
pull
our
dependencies
pre
build
or
from
a
local
repository,
so
we
know
that
they
won't
change
and
we're
de-risked
from
a
remote
outage
affecting
our
deployments.
We
can
isolate
ourselves
from
the
hosts
kernel.
Vms
are
not
really
the
essence
of
containerization,
but
nevertheless
this
is
how
our
cloud
providers
run.
Our
beer
up
run
our
container
images.
Anyway,
we
can
run
run
commands
with
directives
inside
the
dock
file
as
a
non
root
user.
A
In
the
builds
filesystem,
we
don't
want
access
to
the
docker
socket
or
to
change
global
states.
In
a
container
and
finally,
we
can
run
build
processes
as
a
non
root
user
or
in
a
user
name
space.
This
prevents
leakage
of
UID
privileges
from
the
hosts.
We
should
be,
as
always
in
as
many
namespaces
as
possible.
So
do
you
want
to
get
burned.
This
is
where
you
can
get
burned.
I
will
just
leave
these
here
for
posterity.
These
slides
will
go
up
on
Twitter
shortly
afterwards
and,
of
course,
on
the
shed
web
page
too.
A
Okay,
docker
build
version.
Two
is
called
build
kit
and
amongst
a
host
of
useful
features,
including
secrets
mounting.
It
can
run
rootless
thanks
to
a
key
hero
Sudha,
who
is
speaking
right
now.
In
fact,
I
recommend
that
you
watch
his
talk
when
they're
when
they're
published
this
protects
the
system
from
a
potential
class
of
bugs
in
build
kits
or
container
D,
or
run
see
that
rootlessness
uses
rootless
kits,
which
is
a
Cahiers
project.
A
build
of
Linux
native
fake
root,
emulator
and
everything,
including
build
kit,
can
run
inside
a
user
name
space.
A
So
not
the
nicest
story,
but
the
compromises
here
are
we're
protecting
ourselves
from
all
these
other
attacks
instead,
and
this
is
a
trick
that,
as
I
say,
many
other
of
these
build
tools
require
you're,
probably
using
it
already.
It
is
integrated
with
daca
from
version
1806,
and
it
can
run
as
a
standalone.
Daemon
rootless,
build
kit
can
be
executed
in
kubernetes
courtesy
of
proc
mounts
in
the
security
context,
pod
security
policy.
This
allows
paths
in
proc
to
not
be
masked,
which
built
kit
uses
to
spin
up
nested
containers.
A
What
that
means
is
some
things
in
containers
are
still
shared
with
the
hosts,
but
we
have
a
virtual
overlay,
a
file
system
perception
overlay
almost,
which
means
that
when
we
access
certain
parts
in
proc,
our
container
runtime
gives
us
fake
values
so
that
we're
isolated
from
directly
meddling
with
the
hoax
kernel.
Essentially
unmasking
proc
gives
us
some
namespace
busting
superpowers.
So
again,
there
are
compromises
in
some
of
these
mechanisms.
A
Everything,
of
course
it's
balanced.
Based
on
your
threat
model.
It
is
also
likely
to
derive
an
entitlement
mode
inspired
by
Moby
entitlements,
which
will
allow
fine-grain
permissions
around
run,
commands
in
the
dock
file,
which
is
super
nice,
and
finally,
it
has
an
optimized
parallel
step
runner
that
is
unique
for
OCI
builders.
As
far
as
I
am
aware
on
to
just
frases
image
or
image.
This
is
a
derivative
of
Bill
kit
that
reimplemented
kit
interfaces
to
be
unprivileged.
A
It
still
requires
set
UID
binaries,
as
does
build
kits,
and
a
lot
of
work
is
just
done
here
to
run
out
because,
of
course,
apt
is
installing
things
that
is
a
root
level
operation.
In
this
case
it
allows
sets
of
subordinate
user
and
group
IDs.
These
are
all
quite
nicely
documented
in
the
repos.
If
you
want
to
poke
around
there
and
as
you
might
expect,
it
goes
all
out
with
namespaces
and
set
comp
and
as
it's
using
a
user
name
space,
it
is
fully
and
privileged
on
the
host.
A
Nice
Alexi
has
an
honorable
mention,
as
it
is
the
old,
faithful
of
container
runtimes
powering
the
first
version
of
docker
and
still
used
extensively
across
the
opuntia
States.
Despite
using
a
different
image
formats,
it
actually
supports
OCI
containers
via
dedicated
scripts.
If
you
use
LXE
with
this
script,
you
can
run
OCI
containers
fully
rootless,
not
proliferated
much
but
damn
cool.
You
muchy
is
one
of
the
original
OCI
manipulation
tools
and
takes
a
different
approach
to
everything.
A
So
far,
it's
built
by
a
run
C
and
kernel
maintainer
called
alexis
RI
and
uses
some
funky
tricks
to
simulate
root.
Fulness,
in
fact,
he's
the
back
T
to
run
C
maintainer
means
rootless
features
required
to
get
this
project.
Work
have
been
ported
back
upstream
into
run,
C
and
again.
This
is
a
sort
of
three
or
four
year
labor
of
love
from
a
small
pool
of
developers,
for
which
we
must
be
eternally
grateful.
You
much
use
rootless
and
it
doesn't
require
any
special
file
systems
to
be
rootless.
A
It
currently
doesn't
use
any
kernel,
namespacing
solve
EFS
based,
and
it
should
be
noted
that
a
lot
of
tooling
in
this
space
operates
by
recursively
toning,
a
directory.
This
is
quite
a
lot
of
overhead
and
you
machi
uses
an
extended
file.
Attributes
get
around
this
particular
overhead.
It
is
component
of
a
wider
built
tool
and
may
be
extended
shortly
to
include
its
own
builder
watch
that
space
Hanako
is
probably
one
that
you
have
all
heard
about.
A
A
It
does,
however,
use
the
root
user
inside
the
container
is
required,
so
it
has
permission
to
unpack
an
image
and
extract
that
Tarble
into
the
file
system.
Then
also
the
ability
to
run
docker
file
run
directives
as
the
root
user.
This
opens
it
up
to
some
of
the
issues
mentioned
previously.
I,
don't
know
where
the
Calico
was
vulnerable
to
the
recent
run.
Cc
ve
I'd
love
to
talk
to
you.
A
If
anyone
here
works
on
Conoco-
and
there
is
an
open
issue
to
do
with
hardening
against
malicious
from
containers
from
images
which
is
a
little
niche,
but
it's
a
good
sign
when
people
are
reporting
the
high
crowning
fruit
fruit,
because
the
load
dangling
items
have
already
been
dealt
with
so
really
nice
solution
and
great
to
see
the
innovation
in
this
space
builder
is
red-hats
answer
to
build.
It
can
run
in
various
modes
with
the
rootless
mode
being
preferable
for
our
use
case.
A
Slurp
for
net
NS
is
an
interesting
addition.
This
is
more
work
from
a
lexer,
the
run
c
maintainer
and
Akihiro
the
build
kit
maintainer
or
one
of
them.
So
networking
engineer
sorry,
networking
operations
are
privileged
and
in
order
to
bypass
the
requirement
for
routes
to
interact
with
these
long-established
networking
capabilities,
slurp
creates
a
dedicated,
tap
interface
network
namespaces,
it's
a
lot
faster
than
alternatives
and
is
seeing
general
adoption
in
the
land
of
ruthlessness.
A
Makisu
is
ubers
approach
to
these
requirements.
It's
similar
to
Kanak
Oh,
in
that
it
runs
inside
an
unprivileged
container
somewhere,
but
it
goes
to
great
lengths
to
ensure
cache
ability.
It
runs
Redis
for
distributed
caching,
so
it
is
designed
to
operate
a
huge
scale
and
it
provides
local
caching
options,
including
exploration,
and
the
hash
bank
emits
annotation,
so
uber
have
extended
the
dockerfile
syntax
somewhat
to
give
them
massively
distributed
and
parallelized
caching,
which
again
is
a
nice
approach.
A
Another
Google
tool
in
here
has
Kanaka
was,
if
any
of
you
have
tried
to
use
basel,
it
is
academically
perfect
from
a
user
experience.
It
is
far
more
difficult
to
interact
with
it
has
much
hermit
ism,
but
a
great
cost
run
commands
do
not
exist,
and
there
is
much
hoopla
consternation
wailing
gnashing
of
teeth
online
about
this
basel
fulfills.
A
And
hosted
tool
now
again
deeply
isolated,
very
hermetic,
and
here
is
an
option,
but
probably
not
a
usable
build
target
for
organizations
with
the
kind
of
security
requirements
that
really
care
about
secure
container
builds
for
everybody
else.
Cloud
builds
great.
There
is
even
a
Jenkins
plugin
for
those
of
us
that
can't
escape
his
looming
specter
and,
of
course,
it's
reliant
on
the
cloud
provide
a
security
model
which
is
not
always
fully
disclosed.
A
Everybody
can
achieve
rootlessness
today,
thanks
to
work
shifts
over
the
last
couple
of
years,
especially
in
run
see,
but
implementation
specific
caveats
abound.
No
build
tools
are
fully
unreproducible
and
fallible.
Why
design
output
is
a
function
of
run
behavior
and,
as
such
only
bars
will
really
achieves
this
without
diligent
use
of
the
run
directive.
We
can
still
put
non-deterministic
behavior
into
those
directives
which
mean
the
reproducible
element
of
these
builds
is
compromised.
A
A
So,
which
I
owe
ya
sorry
here
we
are
yep,
so
this
means
that
not
only
can
you
benchmark
these
tools,
but
you
can
also
run
them
relatively
easily
in
parallel.
So
what
do
we
think
the
gift
started?
It's
slightly
the
wrong
time
and
trusted
bills
should
be
great,
but
they
are
not
realistically
quite
ready.
There
are
attacks
against
everything,
that's
not
contain
a
builder
and
even
contain
a
builder.
We
are
trusting
someone
else's
security
model,
which
is
great,
but
of
course
it
doesn't.
A
Work
for
on-prem
builds
as
I
say,
and
in
the
wild
the
kind
of
isolation
we're
talking
about
for
this
kind
of
isolation.
People
are
using
virtual
machines
around
fully
untrusted
builds
by
fully
untrusted
I
mean
we
are
pulling
data
from
the
internet.
We
are
allowing
users
to
supply
executable
code
that
we
run
inside
our
estates.
Currently,
VMs
are
used
as
the
ultimate
security
boundary
around
that,
which
is
fine.
No
project
has
fully
pursued
the
untrusted
dream,
but
I
expect
us
to
get
closer
to
utopia
in
2019.
A
What
else
will
happen
in
2019?
Well,
rootless
Runcie
is
already
here:
user
Nettie's
is
an
unprivileged
kubernetes
in
a
user
name
space
again
from
a
key
hero
shift,
FS
may
well
land
soon,
courtesy
of
the
canonical
team.
The
container
versus
hypervisor
battles
continue
in
data
centers
throughout
the
lands.
Building
containers
in
hypervisors
seems
a
sensible
compromise
at
this
point
when
we
don't
trust
them
and
heresy.
Yoona
kernels
are
still
here,
although
they
don't
provide
the
most
obvious
container,
build
context
at
this
time.
A
I
suspect
it's
going
to
be
a
good
year.
Happily,
I
have
some
extra
bits
to
add
to
the
torque.
We
have
just
open
sourced
cube,
Seck
version
2,
which
is
static,
analysis
and
security
risk
scoring
for
container
for
human
axis
resource
manifests.
There
is
a
hosted
version.
There's
an
open-source
version.
There's
a
Cooper
NetID
admission
controller.
There
is
a
keep
control
plug-in.
Please
kick
the
tires.
Tell
us
what
you
think
it
passes
the
same
acceptance
test
Suites
as
version
one,
but
it
has
become
more
stringent.
It
outputs
something
like
this.
A
So
it's
just
blobs
of
JSON
in
in
an
array
giving
you
essentially
a
per
document
analysis
and
also
control
plane
are
releasing
a
number
of
dev
SEC,
ops,
flavored
tools.
These
are
essentially
pipeline.
Scripts
and
they've
said
cops
is
a
service.
If
that's
something
that
interests
you
please
swing,
pass
the
booth
give
me
a
grilling
and
with
that
here
are
all
the
links
and
thank
you
for
your
attention.
B
A
Mechanic
Oh
project
is
probably
the
the
best
adopted
and
the
most
proliferate
ERDs.
It
is
designed
as
being
kubernetes
native
and
they've
patched.
These
cuba,
Nettie's
proc,
mask
features
into
security
contacts,
so
you
can
explicitly
set
up
on
security
policy
that
allows
you
to
run
calico
that
that
is.
That
is
a
really
nice
solution
that
there
are
these.
You
do
still
have
roots
nested
down
inside
there,
so
everything
comes
as
a
compromise,
but
I
really
like
Hanako.
The
the
new
docker
builder
stuff
is
also
berry.
A
A
So
so
Knicks
essentially
prevents
you
from
getting
into
the
pendency
hell
by
being
fully
deterministic
I.
Guess
probably
the
way
to
describe
it.
I
do
not
know
enough
about
it,
but
what
I've
asked
people
then
I
believe
you
have
an
entirely
separate
operating
system
chain
down
to
down
to
your
sort
of
kernel,
build
so
I,
don't
know
at
what
point
it
would
integrate
with
all
this
tooling.
Maybe
you
could
tell
me
yeah.