►
From YouTube: Building Container Images In Kubernetes: It’s Been a Journey! - Laurent Bernaille & Eric Mountain
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from April 17-21, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Building Container Images In Kubernetes: It’s Been a Journey! - Laurent Bernaille & Eric Mountain, Datadog
Speakers: Laurent Bernaille, Eric Mountain
Almost all of Datadog now runs on Kubernetes, but for a long time we needed dedicated nodes running Docker to build container images. We have recently migrated container image builds to Kubernetes and it's been an interesting journey!
The main challenge to build container images inside Kubernetes is to achieve it without additional privileges. We will explain why we chose buildkit in rootless mode, the architecture we ended up using, as well as the challenges we faced.
Building container images in rootless mode worked flawlessly for over 90% of our images, but for the remaining 10% we encountered complex and interesting issues. We will dive into these problems and explain in detail how rootless builds work and why they sometimes behave differently. We will also explain how we addressed these issues together with the community.
A
Hello,
everyone,
my
name,
is
Lauren
bernai
and
I'm
here
with
Eric
and
I
mean
we're
very
happy
to
be
here
today.
It's
our
first
coupon
in
person
since
2019
in
North
America.
So
that's
that's
great
to
to
be
here
with
you
So
today
we're
going
to
to
discuss
our
migration
of
the
way
we
build
container
images
to
kubernetes
and
as
you
you
will
see,
it
was
a
bit
eventful
and
it
got
pretty
interesting.
A
So
before
we
start,
we
both
work
for
datadoc,
so
we
have
a
few
numbers
and
of
about
datadag
on
the
left
hand,
side
of
the
slide.
What
matters
is
with
our
an
obserability
company
and-
and
we
do
a
lot
of
things
in
the
obserability
space,
but
today
we're
not
going
to
talk
about
dialogue,
the
product,
we're
going
to
talk
about
data
dogs
infrastructure,
because
we
both
work
on
the
dialogue
infrastructure
teams.
A
That
was
working
fine
until
sometime
when
the
company
had
grown
a
bit
and
we
were
having
many
more
Engineers,
many
more
builds
every
day
and
Docker
machine
was
starting
to
hit
limits.
We're
hitting
API
rate
limits
with
the
AWS
API,
and
it
was
starting
to
be
tricky
to
to
to
scale
with
the
skill
of
the
company,
and
so
we
migrated
our
workers
to
kubernetes
right.
This
was
easy
because
at
that
time
we
had
enough
of
knowledge
of
kubernetes
a
datadog
to
be
able
to
do
that.
A
The
next
step
in
this
journey
is
some
customers
were
starting
to
ask
us
for
armed
binaries
right,
because
we
provide
the
data
log
agent,
for
instance,
that
people
use
for
monitoring
and
people
were
starting
to
use
Arm
CPUs.
So
we
need
to
provide
them
with
arm
binaries,
and
so
what
we
did
is
because
we're
running
on
kubernetes
we
set
up
kubernetes
nodes
with
arm
CPUs
and
we're
running
Builds
on
on
this.
On
this
node
to
get
native
builds.
A
Well,
as
you
can
imagine,
the
next
step
will,
of
course,
to
be
able
to
do
that.
We
need
to
have
Docker
images
that
supported
architecture,
and
so
we
wanted
to
have
multi-arch
images
to
do
this.
We
transition
our
Runners
from
running
simple
Docker,
builds
to
running
Docker,
build
X,
which
allows
you
to
do
multi-art
build
by
relying
on
emulation
with
qmu.
A
So
this
was
to
be
honest.
This
was
magic.
It
just
worked
right.
We
were
able
to
provide
to
get
images
which
were
both
working
on
x86
and
arm
by
using
emulation.
However,
as
you
can
imagine,
for
some
builds,
we
moved
from
10
minutes
on
Native
x86
to
more
than
an
hour
on
arm
emulated
right,
so
it
wasn't
ideal,
and-
and
today
the
talk
is
going
to
focus
on
this
part
like
how
we
build
images
and
how
we
transition
to
kubernetes.
A
So,
as
I
was
hinting
I
mean,
this
system
was
starting
to
show
its
limit
the
way
we're
building
Docker
images,
it's
what
the
last
workload
running
outside
of
kubernetes,
which
was
starting
to
be
a
pain
for
the
team
managing
it,
because
everything
else
was
managed
with
a
single
team
by
a
single
team,
doing
everything
on
kubernetes
and
but
this
team
had
to
manage
dedicated
instances
outside
of
kubernetes
to
for
these
workloads
it.
It
also
requires
investing
in
this
Legacy
platform
right
because
we
wanted
to
do
military
art
builds
and
Native
Ambience.
A
A
So,
if
you
look
at
how
people
do
this,
because
many
people
have
tried
and
tried
to
do
it,
you
have
multiple
ways.
The
first
one
is
to
use
Docker
in
Docker,
so
what
you
do
is
similar
to
what
I
was
describing
before,
which
is
you
create
a
kubernetes
pod
in
which
you
mount
the
docker
socket
and
from
there
you
do
local
builds.
A
A
A
The
second
one
Standard
Builders
actually
worked
pretty
well
and
something
we
we
we
try
them
and
they
work,
but
they
were
a
bit
more
complex
to
use
because
you
had.
If
you
want
to
do
multi-arch
images,
it
means
you
have
to
distribute
jobs.
You
need
to
run
one
job
on
x86
nodes,
one
job
on
arm
nodes
and
one
job
to
assemble
the
multi-arch
image
and
push
it
to
the
registry,
so
it
works,
but
it's
more
complex,
buildkit
D,
although
the
build
demon
was
very
attractive
to
us,
because
well,
the
ux
is
great
right.
A
A
So
what
would
this
look
like,
like?
Let's
get
back
to
our
built-in
for
running
in
kubernetes,
so
well,
what
we
wanted
to
achieve
is
simply
this
right
when,
when
a
job
is
to
build
a
new
image,
what
what
we
want
to
do
is
one
of
the
worker
is
going
to
run
the
build
it's
going
to
do
a
Docker,
build
X,
build
command
and
contact
the
build
kit,
the
demon
and
do
the
build.
A
What
is
nice
is
because
it's
remote
built
and
build
kit
support,
build
ex
supports
it.
We
can
actually
have
demons
running
on
x86
and
demons
running
on
arm
nodes
and
build
X
is
able
to
build
to
to
use
the
remote
Builders
to
build
the
native
image
on
the
next
86
node
and
a
native
image
on
an
arm,
node,
assemble
them
and
create
a
multi-arch
image.
A
So
what
do
we
mean
by
root
lesbians?
Well,
it's
built
where
the
main
demon
inside
kubernetes
is
running
as
non-root
right
as
the
normal
user.
However,
as
I
was
saying
before,
we
also
need
to
be
rude
to
run
some
commands,
and
this
is
where
user
namespace
come
come
into
play
right.
A
user
namespace
is
a
way
to
simulate
a
root
user,
but
you're
not
root
in
on
the
host.
A
You
just
root
in
this
limited
namespace
with
limited
capability
and
user
namespace
are
very
powerful
and
they
interact
with
many
low-level
candle
implementations
like
capability
mounts
or
security
modules.
If
you're
curious
about
how
this
works,
there's
this
great
presentation
from
Arc
Hero
at
kubecon
in
2019,
where
he
explained
exactly
how
this
works.
A
So
to
give
you
a
very
quick
overview.
This
is
how
it
looks
like
you
have
a
built
it
worker.
The
pid1
is
running
as
uid1000
so
and
then
privileged
because
you
so
that's
completely
fine,
exactly
what
we
want
and
then
everything
else
is
running
on
a
username
space
and
the
uid
is
zero,
but
I
put
a
star
because
it's
not
really
zero.
It's
zero
inside
this
namespace,
but
it's
not
true
on
the
host.
A
A
very
quick
example:
it's
very
easy
to
to
simulate.
If
you
want
to
try
it
on
the
Linux
machine,
this
command
will
create
a
username
space
and
in
there
you
can
see
where
root
in
the
namespace,
but
we
can't
touch
a
file
that
requires
being
a
root
on
the
host.
So
that's
why
the
touch
Etsy
X
fails
because
we
just
root
in
this
time
space
we're
not
root
on
the
host.
We
can't
do
something
that
requires
root
permission
on
the
host,
and
if
we
touch
a
file
we
can't
we
can
modify
you.
A
A
So
this
is
the
the
the
main
intro
now
we're
going
to
Deputy
to
interesting
and
fun
things
to
be
to
be
fair
when
we
started
working
with
buildkit,
everything
mostly
works
right.
More
than
80
of
the
builds
just
worked
out
out
of
the
box,
but
today
we're
going
to
focus
on
the
20
that
were
interesting
because
otherwise,
with
the
fun
right,
so
we're
going
to
present
to
you
like
three
different
issues
and
we're
going
to
go
into
increasing
complexity.
So
the
first
one
is
well
actually
simple
enough.
A
A
Well,
when
we
run
this
inside
our
rootless
building
environment.
Here
is
what
we
got.
So
that
was,
of
course,
very
surprising,
because
this
is
like
the
most
basic
Docker
file
you
can
use
and
something
that's
interesting
is:
we've
got
an
operation,
not
permitted
message,
but
it
says
something
about
mounts,
so
something
might
be
happening
with
the
file
system.
A
So
that's
pretty
surprising,
let's
look
into
it.
So
what
we
wanted
to
understand
is
what
was
happening.
So
what
we
did
is
we
s
traced?
The
build
kit,
D
Daemon
and
we
started
the
build
and
we
looked
for
permission,
errors
right
and
this
one
is
actually
pretty
interesting
right.
We
have
this
command
that
is
trying
to
set
an
extended
attribute,
and
it's
actually,
this
SC
Linux
attribute,
and
it's
failing
and
well
I
made
the
joke
about.
It's
always
DNS,
because
I
tend
to
talk
about
networking
issues
and
to
be
fair.
A
It's
often
DNS,
but
it's
also
quite
often
SC
Linux
and
the
only
thing
I
need
I
know
how
to
do
with
SC.
Linux
is
how
to
disable
it
I'm.
Sorry,
foreign,
so
I
mean
we
had
a
very
good
inside
a
very
good
idea.
What
might
be
happening
and,
and
so
what
we
did
is
we
downloaded
the
layers
of
the
image
extracted
the
turbo
and
look
at
the
content
of
the
turbo
and,
as
we
suspected
there's
actually
SC
Linux
Libor
labels
on
the
files
right
and
it
turns
out.
A
If
your
root
on
a
host
user
namespace,
you
can
modify
the
security
context
of
a
file.
This
is
what
we
do
at
the
top
of
the
slide.
However,
if
you're
in
a
username
space,
which
is
the
second
part
of
the
slide
and
we're
entering
the
username
space
used
by
Bill
kit
D,
we
can't
modify
the
SC
Linux
attribute
of
a
file
right
and
the
kernel
disallow
it.
You
can't
do
it
if
you're
in
a
username
space,
which
makes
sense.
A
So
this
is
the
issue
that
we
open
an
issue
Upstream
to
be
fair,
there's
no
magic,
nothing!
We
can
really
do,
but
it's
pretty
easy
to
mitigate
right
either
you
remove
the
SC
Linux
attributes
by
pulling
and
pushing
the
file
or
you
use
an
image
without
any
SC
Linux
label
which
to
be
fair,
is
something
I
would
recommend
you
do,
and
we
were
looking
at
that,
because
this
one
was
an
upstream
image
and
the
one
just
the
release
just
after
when
we're
using
actually
didn't
have
the
labels
anymore.
A
So
the
docker
file
is
a
bit
more
complex,
but
once
again
no
rocket
science
right.
We
just
downloading
a
depth
file
and
we're
trying
to
install
it
and,
of
course,
as
before,
this
works
perfectly
fine.
If
you
do
Docker
build,
it
works
perfectly
fine,
if
you
use
build
kit
in
root
mode.
However,
when
you
use
buildkit
in
rootless
mode-
and
you
do
exactly
this
for
this
specific
app
the
build
times
out
so
because
we
are
very
scientific,
we
retry
it
right
and
this
time
it
failed
again,
but
it
felt
in
a
very
different
way.
A
It
fell
with
an
error.
Saying
well
address
already
news.
That's
very
weird!
So
at
that
point,
where
we're
like,
let's
try
and
understand,
what's
happened
what's
happening,
so
we
tried
again
well.
This
site
was
consistent
at
least
we're
seeing
the
same
error
address
already
news,
so
it
was
very
confusing
to
us.
So
what
we
did
is
said:
well,
let's
try
from
scratch.
Let's
delete
the
build
keypad
and
try
again
well
this
time
the
build
times
out.
Okay,
we're
back
to
what
we
had
at
the
beginning.
Well,
let's
try
again
this
time.
A
A
Okay,
that's
what
so,
maybe
package
installation
is
starting
a
demon
right.
It
happens.
Sometimes,
let's
do
the
second
build
when
we
do
the
second
build
this
time.
Netstat
is
showing
that
the
pot
is
bound
and
package
installation
fails
with
address
already
news,
so
we're
getting
somewhere
right.
It
looks
like
package,
installation
is
starting
a
demon
and
it
looks
like
the
demon
is
still
running
when
we
do
the
second
build,
which
makes
little
sense,
because
it's
a
completely
separate,
build.
A
So
can
we
reproduce
this
by
using
a
different
method?
So
we
use
this
very
simple
reproducer,
so
you
have
the
docker
file
on
the
top
left
of
the
slide
and
the
script
on
the
top
right.
So
it's
pretty
simple.
We
just
start
from
Ubuntu.
We
had
the
script,
we
run
the
script
and
we
call
the
script
is
done
and,
as
you
can
see
here,
everything
works
fine
except
we
never
get
to
the
last
line
of
the
docker
file
because
the
build
hangs.
So
exactly
what
we're
saying
before.
A
So
it's
a
good
thing:
we've
reproduced
what
seems
to
happen
is
well.
If
we
looked
into
the
Container,
we
actually
see
sleep
still
running,
which
is
what
we
suspected
before
right.
Remember.
Some
map
was
still
running
and
that's
why
the
port
was
bound,
so
it
seemed
that
the
process
is
leaked
or
and
never
stopped.
A
So,
let's,
let's
look
exactly
at
what's
happening
under
the
hood,
so
this
is
the
anatomy
of
the
build
kit
worker.
So
we
have
the
bilkit
demon.
When
we
start
the
build,
build,
decks
is
going
to
do
an
exec
in
the
Pod.
It's
going
to
start
the
build
steps.
So
here
it's
going
to
run
bash
bash
is
going
to
run
slip
in
the
background
and
what's
important
here
and
we're
going
to
come
back
to
this
later
is
there's
no
process
sandbox.
We
can
see
all
the
processes
in
the
container.
A
A
So
at
that
point
we're
like
well,
we
got
curious
right.
What
has
what
happens
if
we
actually
kill
sleep
inside
this
hanging,
build
build
pod?
Well,
we
were
able
to
kill
it,
but
it
was
never
garbage
collected
because
it's
and
it
became
a
zombie
which
was
also
kind
of
interesting
to
us.
So
at
this
moment
we
took
a
step
back.
How
does
it
work
usually
so
build
step
use
usually
run
in
a
process
sandbox
and
when
the
step
finishes,
all
the
process
inside
the
sandbox
are
killed.
A
However,
in
rootless
mode
we
then
build
kit
with
this
flag.
That
is
very
clear.
Like
this
flag
says:
well,
you
don't
get
a
process
sandbox
and
because
we
don't
have
a
Sandbox,
we
can't
keep
track
of
all
the
processes
starting
during
the
build
and
we
can't
clean
them
and
we
can
really
clean
them
up.
So
this
got
us
two,
but
why
do
we
need
this
flag
and
the
reason
we
need
this
flag
is
because
of
the
way
process
work
in
containers.
A
If
you
want
to
create
a
new
process,
which
is
what
we
would
need
to
do
to
create
a
Sandbox
for
our
build
step,
we
actually
can't
because
there's
a
Kernel
check,
that's
called
Mount
to
revealing,
which
is
verifying.
If
you,
the
process,
you
have
access
to
is
partially
masked
or
not,
and
if
it
is
partially
masked
it
won't.
Let
you
create
a
new
proc
FS
and
that's
why
we
actually
need
our
worker
to
not
try
and
and
create
sandbox
for
the
processes
and
that's
why
all
the
processes
are
seen
inside
the
build
kit
process.
A
Namespace
in
in
conclusion,
I
mean
there's
no
real
solution
to
date.
We
we
chatted
about
it
with
maintenance
in
this
issue
extensively
and
there's
no
real
way
to
do
it.
There
are
potentially
multiple
mitigations.
The
one
we
use
we're
using
for
now
is
we're
making
sure
that
our
Docker
files
are
not
certain
demons
in
the
background
or
if
they
do,
we
explicitly
stop
them.
So
that's
easy
enough.
A
Something
that
could
be
interesting
in
the
future
is
kubernetes
expose
a
notion
that
is
called
proc
man
type,
where
you
can
tell
that
a
specific
container
will
have
a
process
that
is
not
masked
so
fully
accessible,
which
means,
if
you
remember
that
mount
to
revealing
will
be
okay
and
we
will
be
able
to
create
a
new
proc
FS
for
our
build
step
inside
the
sandbox
inside
the
container.
Sorry,
so
that's
extremely
promising,
but
it's
the
feature
has
been
in
the
alpha
in
kubernetes
since
1.12.
A
B
Okay
yeah,
so
our
last
talk
was
at
kubecon
was
ghost
in
the
runtime.
We
like
ghosts
so
this
time
ghosts
in
the
file
system,
so
we're
just
going
to
build
a
go
program.
So
here
it's
the
local
volume,
provisioner
fairly
straightforward.
We
clone
the
repository
check
out
a
specific
tag
and
then
run
go
build
what
could
possibly
go
wrong?
B
Well,
we
get
a
compilation,
error,
consistent,
read,
redeclared.
It's
clearly
declared
in
two
files
here,
read.go
and
consistent
read.go,
but
the
strange
thing
is:
when
you
build
this
yourself
on
your
laptop
or
in
a
Docker,
build
not
in
rootless,
it
works
perfectly
fine,
so
what's
happening,
let's
have
a
look
at
the
directory
contents.
Well,
indeed,
in
this
vendor
directory,
we
have
both
files,
consistent,
read
and
read.
B
Because
in
the
master
Branch
you
just
have
read.go
and
at
the
tag
we
have
just
consistent
read.gov,
so
we
clearly
shouldn't
have
these
two
files
at
the
same
time,
so
we
check
at
each
layer
the
git
clone
shows
that
we
have
read.go.
That's
expected
the
checkout.
The
tag
shows
that
we
have
only
consistent
read.go,
so
that's
fine
too,
but
we've
already
seen
that,
in
the
end
that
the
next
step
we
have
both
files
for
some
reason
so
something's
wrong
with
the
file
system.
B
B
So
the
idea
is,
you
have
a
set
of
directories
that
are
the
base
and
you
want
to
expose
a
mount
point
where
changes
can
be
made,
but
without
affecting
the
the
original
files,
and
so
what
overlayfs
does
is
it
has
an
intermediate
or
what
we
call
the
upper
layer
which
records
the
changes
that
have
occurred
on
the
file
system.
That's
exposed
to
to
processes
through
the
mount
point,
and
so,
for
instance,
a
file.
That's
deleted
will
be
marked
in
the
change
layer,
the
upper
layer
as
a
tombstone
file.
B
B
So
that's
all
very
good.
Let's
try
and
reproduce
the
steps
of
the
of
build
kit
here.
So
first
we
unshare
the
username
space.
That's
what
buildkit
is
doing.
That's
because
we're
running
in
rootless.
We
create
some
directories
to
to
provision
our
overlay
file
system,
so
we
have
layer,
one
which
is
the
lower
layer.
B
Where
we're
going
to
do
the
git
clone,
then
we
have
a
the
layer
2,
which
will
be
our
Mount
points
that
we
will
Expose
and
we
have
the
layer
2
diff,
which
is
going
to
be
the
changes
that
occur
through
the
overlay
file
system
and
so
for
in
particular,
when
we
actually
do
the
git
checkout.
After
mounting
the
file
system,
the
L2
diff
directory
is
going
to
record
the
changes
that
are
made.
B
So
that's
all
very
good.
We
look
at
what
we
end
up
with
in
each
of
the
directories
and
we
see
so
layer,
one,
the
checkout.
We
have
read.go
fine,
Layer
Two
difference.
We
have
consistent
read.go,
that's
expected,
that's
the
new
file
and
in
the
mount
Point
that's
exposed.
We
see
only
consistent,
read.go,
that's
consistent
with
what
we've
seen
before.
But
it's
we
haven't
reproduced
the
problem
at
this
point.
B
So
now,
let's
pile
on
the
step
three
of
our
build
where
we
list
the
directory
contents,
because
it's
the
equivalent
of
when
we
build
and
we
get
compiler
error.
So
we
unpount
the
previous
overlay
file
system.
We
provisioned
some
new
directories
for
the
the
difference
layer
for
for
our
step,
three,
the
layer
3
directory,
which
will
be
the
mount
point
that
we
expose
and
we
Mount
this.
B
So
one
thing
to
note
here
is
that
in
the
lower
directories
we
actually
have
two
directories,
because
we
have
first
the
changes
that
were
made
by
the
get
checkout
and
then
we
have
the
base,
which
is
the
git
clone,
and
we
list
the
the
files
and
we've
reproduced
the
problem.
We
we
see
both
files
so
clearly,
there's
something
wrong
here.
B
B
So
where
is
it?
Well,
maybe
we've
missed
something,
and
yes,
we
have
so
in
overlayfs,
there's
actually
an
optimization
for
directories
which
are
where
all
this
contents
have
been
deleted
and
but
the
directory
still
exists.
You
have
an
opaque
flag.
Basically,
it
avoids
overlay
FS
from
having
to
recurse
and
subdirect
in
under
layer
directories.
If
all
the
contents
have
been
suppressed,
it's
an
optimization,
but
how
does
it
work
so.
B
B
Now,
if
we
rerun
the
same
command
but
in
the
host
username
space,
the
initial
username
space,
we
do
see
the
extended
attribute,
so
it
is
being
set.
But
that
also
is
a
bit
strange
because
trusted
the
trusted
namespace
of
extended
attributes
is
actually
subject
to
permissions.
You
can
only
set
a
trusted
extended
attributes
if
you
are
capsys
admin.
You
have
the
system
administrator
capability
in
the
host
username
space,
which
is
not
the
case
in
our
overlayfs
sequence
here
in
rootless,
so
we
shouldn't
be
able
to
set
that
extended
attribute,
and
yet
we
have
so.
B
B
So
we
still
don't
quite
understand
certain
things.
We
do
understand
that
we're
having
the
attribute
set,
but
we
can't
read
it.
So
that's
why,
in
step
three,
we
see
both
files
because
we
can't
read
the
extended
attribute.
So
the
opacity
is
not
being
honored
fine,
but
in
step
two,
why
don't?
We
have
the
same
problem,
because
we
can't
read
that
extended
attribute
either,
and
it
turns
out
that
well,
if
we
think
about
it,
fast
systems
do
a
lot
of
caching.
B
So
again
we
reproduce
our
case.
We
reach
the
point.
This
is
step
two.
We
reach
the
point
where
we
list
the
files
and
we
see
consistent
read.go.
Only
so
opacity
somehow
is
being
respect
is
being
honored,
even
though
we
don't
expect
it
to
be.
We
now
drop
the
caches,
and
lo
and
behold,
the
two
files
are
there.
B
So,
in
short,
the
kernel
we're
using
added
a
patch
to
make
user
overlay
FS
work
inside
username
spaces,
but
it
was
a
partial
change.
It
only
changed
the
set
and
the
remove
of
the
extended
attribute
and
thanks
to
caching.
Well,
sometimes
the
opacity
is
honored
and
sometimes
it
isn't,
which
leads
to
some
rather
interesting
behaviors.
B
The
other
nice
thing
is
the
build
kit.
Actually,
the
overlay
implementation
adds
user
accept
support
when
it's
available,
and
so
all
we
had
to
do
was
wait
for
kernel
511
to
be
available
for
our
distribution
for
Ubuntu
and
then
simply
roll
it
out
to
our
nodes
for
new
our
nodes,
and
from
that
point
it
just
worked.
B
So
our
messages
are
that
bill
kit,
you
know,
is
really
good.
We
we've
had
a
very
good
experience
with
it.
It
gives
us
remote
builds.
It
gives
us
multi-op
images
really
easily
rootless
is
or
was
a
little
bit
bleeding
edge,
but
you
know
with
the
changes
in
kernel:
511
overlayfs
in
user
namespaces
has
become
really
very
usable.
Well,
it
works.
Fine
I
think
really
the
only
problem.
We
still
have
slightly
that
you
know
that
could
affect
us.
Is
the
process,
sandboxing
and
so
I.