►
From YouTube: Linux Distribution Build Tools for Custom Container Images - Nisha Kumar & Joshua Lock, VMware
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Linux Distribution Build Tools for Custom Container Images - Nisha Kumar & Joshua Lock, VMware
A typical container image builder takes a base OS from somewhere, runs scripts to add and modify all the things needed for an app to run, then deploys to a registry. This leads to large images which developers try to trim down by using multistage builds, removing files and squashing filesystem layers. Building container images in this way makes it difficult if not impossible to ascertain the license and security implications of using these images. How do we generate app specific build and runtime images without having to maintain our own base OS images and build machinery? Fortunately, this is a problem that has been solved in the Linux distribution world for some time. This talk will outline some popular tools and compare them against the requirements for a declarative and reproducible container OS builder which is not reliant on any external infrastructure.
https://sched.co/Uaas
A
So
hello,
everybody
we're
here
to
talk
about
some
of
our
experiences
in
investigation
into
using
Linux
distribution
tools
for
building
container
images.
I've
been
talking
about
using
Linux
distribution
tools
for
about
a
decade
and
never
seen
this
many
people.
So
thanks
for
turning
up
my
name
is
Joshua
I'm,
a
software
engineer
in
VMware's,
open
source
program
office
and
I
work
on
sort
of
ecosystem
security
issues
for
the
most
part
and.
B
I'm
Nisha
I
am
also
an
open
source
engineer
at
the
open
source
program
office
at
VMware.
I
am
the
tech
lead
for
containers
and
the
open
source
Technology
Center,
which
is
the
technical
arm
of
the
open
source
program
office.
It's
nice
to
see
so
many
people
interested
in
this
topic,
I
usually
talk
about
compliance
and
I
use.
That
means
that
I
usually
speak
to
only
10
people
or
less.
B
So,
when
talking
with
Joshua
about
this
problem,
we
found
that
actually,
it
overlaps
with
a
lot
of
security
concerns
as
well.
So
we
kind
of
boil
down
our
concerns
for
looking
at
Linux
distro
distro
tools
into
these
three
concerns,
so,
like
I,
explained
compliances
finding
the
software
Bill
of
Materials,
finding
the
licenses
that
govern
the
distribution
of
that
container
and
then
delivering
corresponding
sources
for
those
components
that
are
installed.
Security
involves
finding
what
CVS
in
the
container
imagined
that
and
also
what
oh
we're
in
the
supply
chain.
B
A
So
before
we
get
too
far
into
the
meat
of
this
talk
just
need
a
level
set
and
make
it
clear
that
when
we
talk
about,
containers
were
effectively
referring
to,
you
know,
container
images
and
we
think
of
them
as
somewhat
analogous
to
a
packaging
format.
They're
not
Richard,
like
a
lot.
The
packaging
formats,
those
of
us
that
come
from
a
dish
world
are
used
to
they're,
fairly
loose
and
forgiving,
and
they
don't
contain
a
bunch
of
the
rich
metadata
that
some
of
us
expect
they're,
probably
most
similar
to
a
VM
appliance.
B
A
If
we
think
of
them
as
a
packaging
format,
there's
a
few
desirable
properties
that
we
would
like
to
preserve
on
packages,
we
really
want
a
package
to
be
repeatable.
If
we've
got
the
metadata
for
a
package
and
we
try
and
rebuild
it,
we
expect
to
be
able
to
produce
an
equivalent
thing
and
we're
on
the
package
to
be
identifiable.
We
need
to
understand,
what's
in
it,
be
able
to
reason
about
the
license,
compliance
implications
of
its
contents
and
the
security
vulnerabilities
that
may
or
may
not
exist,
and
we
really
want
the
contents
to
be
recent.
B
Okay,
so
those
are
the
properties
that
we
would
like,
as
Joshua
mentioned,
I
would
like
to
talk
about
how
current
practices
of
building
containers
break
these
desirable
properties
and
I'm
gonna
use
the
example
of
a
docker
file,
although
all
the
other
container
build
tools,
kind
of
follow
the
same
principles
more
or
less
so,
for
example,
if
you
were
to
have
a
run
line
in
your
docker
file,
this
can
use
any
kind
of
script
that
you
want.
You
can
pass
any
kind
of
command
that
you
want,
and
these
are
usually
non-deterministic
once
you
have
that
installed.
B
It's
really
hard
to
repeat
this.
The
second
one
is
using
things
like
app
get
update
or
apt-get
upgrade.
What
happens
here
is
that
you
are
updating
software
on
the
fly
at
a
given
time
during
build,
and
so
the
property
that
happens.
The
problem
that
happens
with
this
is
that
if
you
were
to
use
the
same
docker
file
and
build
the
container
image
six
months
later,
you
have
no
guarantee
that
you
are
installing
the
same
software
in
your
container
image.
The
next
one
is
using,
add
or
copy.
This
is
adding.
B
There
is
no
understanding
of
what
these
files
are
once
the
container
is
created
and
shipped,
and
it
is
very
difficult
to
reason
about
what
those
files
are,
because
they
came
from
an
external
pipeline
and
then
the
last
one
I
want
to
point
out
is
using
from
in
the
docker
file,
so
docker
hubs
tags
are
not
immutable,
they
are
rolling
tags
and
they
very
much
get
updated
based
on
whether
the
supplier
wants
to
update
them
or
not.
So
even
if
you
were
to
pull
from
latest
that
not
that
actually
may
not
be
latest.
B
B
Okay,
so
here's
the
deal
with
distro
tools,
they
are
historically
been
really
difficult
to
understand
and
use.
I
do
want
to
point
out
here
about
the
the
two
key
notes
that
you
heard
today.
One
was
about
making
kubernetes
easy
to
use
and
the
other
one
was
about
threat
modeling.
These
are
two
diametrically
opposite
things
that
you
are
expecting
your
tools
to
do,
and
this
is
very
hard
to
achieve.
So,
on
the
one
hand,
you
want
these
to
be
very
developer
friendly
tools
such
that
they
will
not
feel
you
know
the
it.
B
It
doesn't
feel
hard
to
use
them,
but
on
the
other
hand,
you
want
this
to
these
tools
to
produce
the
production
grade
software,
so
software
of
high
quality.
So
there
are
several
things
to
balance
out.
One
is
ease
of
use,
so
the
reason
why
docker
is
so
popular
is
because
it
is
really
easy
for
any
app
developer
to
become
an
operating
system
vendor.
So
this
is
this
helps
adoption,
but
on
the
other
hand,
you
want
your
tools
to
give
you
all
that
comprehensive
information.
B
In
order
for
you
to
make
engineering
decisions
about
whether
this
artifact
is
okay
to
ship,
the
next
one
is
the
ecosystem
that
this
tool
exists.
Exists
in
darker
is
popular
because
there
are
so
many
available
components
that
an
app
developer,
an
app
developer,
can
put
together
and
create
the
app
for
deployment.
On
the
other
hand,
it
is
very
much
difficult
to
reason
about
the
trusted
nosov
these
components
and
the
last
one
is
that
we
want
the
tool
to
be
very
fast
to
use,
so
developers
can
iterate
over
their
software
as
fast
as
possible.
B
A
So
at
this
point,
nature
and
I
have
been
talking
for
a
while,
and
we
had
sort
of
the
seed
of
an
idea.
Can
we
use
Linux
distribution
tools
to
create
container
images?
You
know,
nm
are
identifiable
repeatable
and
recent
or
with
more
recent
contents,
and
so
we
wanted
to
serve
our
actually
experimenting
with
the
tools
trying
to
figure
out
whether
that
was
possible
and
how
far
we
could
get.
A
And
that's
what
we're
going
to
talk
about
next
and
the
first
thing
we
had
to
do
was
choose
some
tools
to
experiment
with,
and
the
space
of
available
distribution
tools
is
quite
large
and
we
needed
to
constrain
it
for
the
you
know
purposes
of
this
investigation.
So
we
did
a
sort
of
finger
in
the
decision-making
process
based
on
gut
feeling
and
experience
and
picked
these
four
tools
and
we
start
out
with
Debian,
because
it's
this
extremely
reliable
long-lived
project,
with
a
massive
amount
of
software
available
already
packaged
in
this
system.
A
And
then
we
looked
at
some
embedded
tools,
which
is
the
sort
of
the
world
I
come
from,
so
we'd
look
to
build
root,
which
is
an
embedded
image.
Creators
kind
of
designed
for
building
Linux
based
firmware
and
doesn't
care
about.
You
know
updating
the
software.
It
just
assumes
that
you're
going
to
replace
the
image
when
you
need
to
create
an
update,
which
is
somewhat
similar
to
a
container
system,
whereas
Yocto
project
is
the
other
end
of
the
spectrum,
is
a
distribution
builder.
A
So
it
has
a
lot
more
affordances
for
building
multiple
images
with
a
shared
core
and
kind
of
like
products
queues,
but
it
has
different
trade-offs
because
of
that
and
then
finally,
we
want
to
look
at
the
system
called
geeks,
which
is
based
on
some
functional
concepts,
functional
distribution,
content,
concepts
which
are
interested
in
repeatability
and
immutable
operating
systems,
which
shall
seem
highly
relevant
in
a
container
space.
The
disadvantage
for
many
is
that
it
uses
schemas
it's
common
expression,
language
and
so
parentheses
are
intimidating
to
a
lot
of
us,
but
it
had
some.
B
So
what
we
wanted
to
do
was
try
to
build
different
types
of
container
images
using
these
disre
tools,
and
we
want
to
evaluate
them
based
on
these
four
things,
ease
of
use
that
it
should
be
like
fairly
straightforward
to
use
these
tools.
The
output
image
size
should
be
lesser
than
what
is
already
available
on
docker
hub
the
amount
of
engineering
effort
it
takes
to
actually
make
the
container
image
and
the
quality
and
the
presence
of
software
bill
of
materials
that
helped
us
reason
about
things
like
compliance
and
security
vulnerabilities.
B
So
we
start
off
with
our
first
experiment,
which
is
build
a
base
OS
for
developing
golang
applications.
The
first
tool
we
looked
at
was
de
bas
de
bas
is
actually
really
easy
to
use.
It's
just
a
wrapper
around
Deb
bootstrap
that
was
written
in
gold.
It
is
only
supported
for
Debian
and
Debian
based
two
operating
systems
like
a
bun
too.
B
So
if
you
have
a
new
bundle,
if
you're
working
with
an
imprint
of
operating
system,
it's
pretty
straightforward
to
use
the
image
size
that
God
created
over
here
was
actually
half
of
the
image
size
that
was
available
on
around
400
megabit
megabytes.
The
engineering
effort
in
there
was
it
was
fairly
straightforward
to
build
an
image
using
this
dabbas
uses
something
called
uses:
a
yamo
file
that
you
feed
into
the
system.
It's
pretty
straightforward
to
make
this
yamo
file.
B
It
looks
kind
of
like
a
docker
file
after
if
you
squint
enough,
and
so
it
this
should
be
like
fairly
straightforward
for
an
app
developer,
who
is
used
to
developing
with
docker
files.
To
pick
up
the
downfall
of
this
I
mean
the
the
downside
of
this
is
that
the
software
bill
of
materials
is
not
there
and
there
is
no
seamless
way
of
getting
it.
I.
B
B
Okay,
so
the
second
one
we
tried
was
build.
Route
build
route
was
not
that
straightforward
to
use.
There's
a
lot
of
documentation.
I
will
not
comment
on
the
helpfulness
of
this
documentation,
but
it
took
a
long
time
to
go
through
it
to
figure
out.
You
know
what
exactly
you
can
do
with
the
help
of
some
tutorials
that
went
missing
all
of
a
sudden,
so
we
I
could
not
build
an
image
for
golang
development
per
se,
because
this
is
not
supported
in
build
route.
B
A
It
doesn't
have
so
much
of
a
learning
curve
as
a
learning
cliff
that
you
can
gradually
inch
up,
and
so
this
is
of
course,
hence
the
sad
face
and
one
of
the
nice
things,
though,
is
that
the
image
sizes
you
get
are
pretty
good
to
get
to
that
place.
So
you
have
to
put
a
lot
of
engineering
effort,
and
you
know
the
reason
that
your
project
is
quite
difficult
to
use
these
captions
and
a
lot
of
concepts
about
building
operating
systems
in
order
to
user.
A
So
you
have
to
play
in
that
legwork
and
then
write
a
lot
of
configuration
files
on
top
of
it
to
generate
an
image,
but
it
does
generate
a
really
nice
software
bill
of
materials.
You
get
a
full
list
of
all
of
the
contents
and
the
licenses
and
with
a
minor
configuration
tweak,
you
can
get
all
of
the
license
files
and
the
sources
for
your
image
and
have
your
full
compliance
footprint.
So
it
gets
a
very
happy
face
for
that
and
then
I
started
looking
at
geeks,
which
is.
A
Has
an
unfortunate
loan
sort
of
usability
spectrum
because
it's
got
this
really
nice
unified
command
line
tool
to
drive
it
and
but
as
soon
as
you
have
to
step
outside
of
the
command
line
tool,
all
you
have
to
write.
You
know
several
lines
of
scheme
with
a
bunch
of
boilerplate
that
it's
not
necessarily
relevant
to
the
content
use
case,
because
that's
not
something
that
all
has
paid
too
much
attention
to.
It
generates
really
large
images.
B
Okay,
so
we
actually
managed
to
get
to
where
the
status
quo
is
still
stands.
You
can
still
make
a
base
OS
that
you
can
develop
your
app
on
top
of,
and
that
fits
with
the
way
that
we
do
our
current
development,
which
is
choose
a
base,
OS
install
your
apps
dependencies
and
install
the
app,
and
so
it
works
pretty
well
in
that
case,
but
we
wanted
to
push
the
envelope
a
little
bit
and
see
if
we
can
build
an
actual
ideal
container,
which
is
just
the
app
and
all
of
its
dependencies
just
its
dependencies.
A
We
can
get
too
far
into
this,
so
we
have
to
talk
a
little
bit
about
packaging
software.
So
in
the
distribution
space
you
create
a
package
you
describe
effectively
what's
in
the
package
and
how
to
build
and
install
it
as
part
of
a
manifest
file
effectively,
and
it
has
a
paired
archiving
format
that
tells
you
how
to
tells
the
package
manager.
A
Sorry,
it
has
an
archive
format,
that's
basically
a
tar
file
and
then
the
package
manager
can
read
the
metadata,
understand
the
tree
of
dependencies
for
an
application
and
install
what
is
required
into
the
operating
system
at
runtime
and
the
way
I've
described.
This
probably
gives
away.
You
know
our
next
idea
is:
can
we
leverage
the
package
manager
to
install
only
what's
required
into
an
image?
B
Okay,
so
our
second
experiment
is
to
make
a
container
with
enough
to
run
the
app
so
the
app
and
all
of
its
dependencies,
and
what
we
found
is
that
when
we
get
to
the
situation,
the
steep
learning
curve
becomes
steeper,
because
each
of
these
districts
are
different
ecosystem
and
they
have
different
ways
to
express
your
metadata
different
ways
to
express
how
you
want
to
build
and
different
infrastructure
to
package.
Your
application
and
the
the
hard
part
about
this
is
that
they
actually
don't
have
much
support
for
applications
built
for
in
the
cloud
native
space.
B
B
There
are
some
tools
that
help
you
automate
this,
but
again
the
support
is
quite
minimal.
So,
for
example,
if
you
are
working
with
Devas,
there
are
tools
to
help
you
automate
Debian
izing
your
applications.
Based
on
what
language
you
write.
Your
applications
in
the
trouble
here
is
that
there
is
no
straightforward
way
of
installing
a
locally
build
Debian
package
in
your
container
image
using
de
bas,
and
the
second
thing
is
that
you
have
to
Debian
eyes
all
of
your
dependencies,
and
this
becomes
pretty
harrowing.
B
If
you
are
developing
in
nodejs,
for
example,
so
build
root
has
documentation
that
covers
the
infrastructure
available
to
package
your
applications
based
on
what
language
and
ecosystem
you
use
again
steep
learning
curve
and
very
difficult
to
reason
whether
this
is
actually
supported
or
not.
You
can
you
the
only
thing
you
can
do
here.
B
This
trial
and
error
see
which
one
works
in
which
one
breaks
the
Yocto
project
has
a
thing
called
recipe
tool
that
you
can
use
to
package
your
applications
based
on
what
language
you're
writing
in
the
thing
is
that
it
doesn't
support
all
of
the
languages
that
we
mentioned
before
that
cloud
native
developers
are
used
to
and
go.
X
has
a
tool
called
gooks
import
once
again,
no
support
for
cloud
native
languages.
A
So
we
wanted
to
take
a
step
back
and
figure
out
if
we
could
actually
achieve
the
end
goal
without
having
to
go
through
all
the
rigorous
work
of
packaging,
a
bunch
of
cloud
native
software,
and
it
turns
out
as
the
slide
indicates,
that
we
can
get
a
bit
further
with
most
of
the
tools
and
most
of
the
way
with
another
one
of
the
tools.
So
the
debian
build
group
and
actually
project
tools.
A
We
can
start
off
with
a
minimum
based
operating
system,
install
our
packaged
application
and
all
this
dependencies
and
get
a
smaller
container
image
with
less
bloats
and
a
smaller
attack
surface.
But
Geeks
has
this
really
nifty
command-line
option
called
geeks
pack
and
what
that
does
is
it
takes
an
application,
gathers
all
of
its
dependencies
and
packs
it
into
some.
A
Distribution
space
where
the
tool
started,
you
can
do
something
like
generate
a
tar
ball
with
relocatable
binaries,
so
you
can
run
Postgres
on
any
Linux
system
with
a
Lib
C,
regardless
of
whether
it's
got
you
know,
Postgres
and
all
of
this
dependence
is
installed
and
regardless
of
whether
your
roots
are
anything
like
that
and
they
actually
do
have
an
output
format
for
generating
generating
ativ
one
compatible
container
images.
So
I
was
able
to
generate
a
Postgres
container
image.
A
There
was
a
few
hundred
megabytes
so
probably
a
little
bit
on
the
large
side,
but
this
is
all
the
manual
blow
and
and
localization
and
things
that
we
discussed
earlier
and
but
I
do
just
have
in
this
container
image
gross
all
of
its
runtime
dependencies
and
nothing
else.
So
I'm,
not
shipping,
most
of
an
operating
system
just
to
run
my
database
server,
which
is
kind
of
the
point
of
containers.
A
B
So
I
suppose
the
big
takeaway
from
our
talk
is
that
we're
partway
there,
but
we're
not
all
the
way
there.
The
tools
that
we
have
talked
about
come
from
the
perspective
of
distro,
and
usually
that
comes
from
the
perspective
of
enabling
this
distro
and
a
certain
on
different
types
of
hardware.
And
what
needs
to
happen
is
there
needs
to
be
some
kind
of
switch
where
we
have.
We
understand
that,
there's
a
homogeneous
hardware
that
we
are
enabling,
or
that's
already
enabled
from
us,
and
to
actually
work
from
the
app
developer
side.
B
There
are
actually
projects
that
are
working
on
this,
so
fedora
is
trying
to
reduce
the
size
of
their
images
for
container
use,
so
is
Debian,
and
there
are
compelling
reasons
to
help
these
folks
and
you
can
help
them
because
they're
all
open-source
projects
you
can
submit
tweaks
to
the
documentation.
So
they
become
more
easy
for
an
app
developer
to
pick
up
and
use
the
you
can
submit
tests
in
pull
requests.
B
So
they
work
for
the
container
use
case
and
you
can
submit
use
cases
from
the
app
developer
side,
because
these
use
cases
are
not
necessarily
visible
from
the
distro
side,
because
they're
just
looking
at
hardware,
okay,
so
actionable
learnings
things.
You
can
do
right
now
in
your
organization,
introspect
your
containers
to
see
what
is
installed
in
them.
There
are
several
tools
that
help
you
do.
B
This
dive
helps
you
figure
out
the
layout
of
your
container
image,
such
as
all
the
layers
that
are
in
it
and
what
files
there
might
hold
container
diff
is
a
tool
that
helps
you
figure
out.
The
difference
between
in
the
files
for
each
of
your
container
bills
and
turn
is
the
tool
that
I
mentioned
before.
That's
the
tomb
that
I
maintain
you
can
use
it
to
get
software
Bill
of
Materials
and
sources
for
known
operating
systems
like
Debian,
okay,
and
you
can
get
halfway
there
with
some
of
the
tools
that
we've
talked
about.
B
Deb
Oz,
as
I've
said,
is
really
straightforward
to
use
and
easy
to
pick
up
for
any
like
naive,
developer
naive
as
far
as
operating
systems
are
concerned,
so
you
can
use
Deb
us
to
make
the
base
OS
and
the
development
tools
that
you
need,
or
the
base
OS
and
the
runtime
dependencies
that
you
need
and
use
that
base
OS.
When
you're
deploying
your
software,
because
at
least
you
know
the
software
Bill
of
Materials
for
your
base
OS.
So
you
have.
B
You
have
certainty
of
what
exactly
you're
shipping
goo
exhale
geeks
has
a
lot
of
potential
in
creating
container
images
that
are
just
just
enough
with
the
app
and
the
dependencies.
But
you
need
to
write
a
lot
of
glue
code
in
order
for
it
to
get
there.
It's
kind
of
like
it's
almost
there,
but
it
needs
this
last
mile.
So,
as
Joshua
said,
the
the
container
has
a
bunch
of
a
bunch
of
manuals
and
software
Bill
of
Materials
and
metadata
that
you
don't
necessarily
need
in
the
container
image.
B
But
it
is
there,
so
you
can,
when
you're
pushing
your
container
image,
you
can
push
all
of
those
artifacts
along
it,
along
with
it
as
well,
so
that
information
is
not
lost.
You
can
also,
upstream
all
these
changes
to
go
extinct.
So
the
user
experiences
more
seamless,
okay-
and
here
are
some
resources
for
you
to
read.
If
you
are
so
inclined,
there's
the
de
bas
github
repo
repo
there's
debian
packaging
manual,
there's
the
build
route
manual
and
the
geeks
manual
and
the
Yocto
project
mega
manual,
which
I
did
not
look
at.
B
A
C
B
So
the
the
image
that
we
were
working
on,
which
was
just
the
base,
OS
plus
the
golang
image,
it's
actually
quite
smaller
than
the
golang
image
that
you
get
from
docker
hub.
So
if
you
were
using
multistage
pills
and
you
were
using
from
golang
that
image
you
get
is
much
larger,
like
the
image
that
I
created
was
half
the
size.
B
The
other
thing
about
it
is
that
when
you're
using
docker
multi-stage
builds,
your
build
stage
disappears,
so
that
breaks
your
build
repeatability,
but
in
if
you
were
using
something
that
you
know
for
sure
what
is
installed
in
it,
then
that
is
not
something
that
you
care
about,
because
you
already
know
what's
in
it.
So
in
that
does
that
answer
your
question?
Okay,.
A
Yes,
will
you
come
say
that
we
wanted
to
try
and
look
at
what
was
available
out
of
the
box,
and
so
that's
what
we
focused
on
for
this
talk.
But
yes,
we've
made
we've
done
that
kind
of
trickery
in
the
past.
We
just
want
to
see
now.
It
feels
like
a
lot
of
work
for
compared
to
doc
files
and
that's
really
the
baseline.
We
had
one
in
something
that
was
closer
to
dockerfile
usability.
A
Yeah,
so
what's
the
workflow
for
most
of
these
tools-
and
it
varies
a
lot
by
tool
so
most
of
them,
you
run
them
on
the
host
operating
system.
You
use,
you
know
super
user
privileges,
you
feed
it
some
configuration
files
and
it
generates-
and
you
know
an
artifact
and
the
only
one
Yocto
project
gets
special
extra
mention
there,
because
it
doesn't
require
any
super
user
privileges.
A
B
You
can
use
harbor
for
that
or
you
can
push
it
to
docker
hub
on
your
repo,
so
kind
of
the
does
that
does
that
make
sense
yeah,
so
some
of
them
do
that
some
of
them
don't
and
that's
part
of
the
work
that
we
need
to
do
so.
For
example,
gigs
makes
a
container
image
format
like
with
the
manifest
and
one
layer.
The
thing
is
that
that
one
layer
contains
everything,
Geeks
has
something
that
it's
called
a
store.