►
From YouTube: Fast Image Pulls Using IPFS And Opportunistic Caching - Christian Weichel & Manuel de Brito Fontes
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from April 17-21, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Fast Image Pulls Using IPFS And Opportunistic Caching - Christian Weichel & Manuel de Brito Fontes, Gitpod
Speakers: Christian Weichel, Manuel de Brito Fontes
Image pull times pose a considerable challenge when optimising for fast container starts within Kubernetes, due to potentially large images or network topology, bandwidth and egress cost constraints. Container runtimes offer layer-based node-local caches which help improve pull-times when there’s high layer-reuse, but find their limits when clusters need to scale quickly or there’s little control over the images which are used. We present the results of our efforts to bring down pull times down, which brought about considerable pull time improvement. Our goal was to optimise performance and networking cost, without imposing limits on the container images themselves. We went through several iterations which combined eStargz, nerdctl-registry with an in-cluster IPFS deployment. Using an opportunistic pull-through caching mechanism, we were able to considerably bring image pull times down without imposing extra burden on users (i.e. folks deploying the pods). We have been operating this setup in production on gitpod.io for over six months. In this session we will provide insight into our learnings, backed by the real-world data and observations we have gathered.
A
Thank
you
all
for
sticking
around
last
day
conference
appreciate
you
all
showing
up
so
we're
going
to
talk
about
fast
image
polls
using
ipfs
and
opportunistic
caching,
and
my
name
is
Chris.
I
work
at
gitpod
and
with
me
on
this
stage
should
be
Alejandro,
who
is
actually
in
a
tab
in
a
Google
meet
in
this
video
browser
right
now,
so
we'll
pull
them
in
later
for
the
Q
a
session.
A
So
we
both
work
at
gitpod
and
what
we
do
is
we
turn
the
hours
and
days
that
it
takes
to
set
up
a
deaf
environment
into
seconds
and
the
reason
I'm
bringing
that
up
is
part
of
this
deaf
environment
is
a
Docker
image
that
you
can
choose
or
that
we
can
build
for
you
and
so
just
to
show
a
quick,
slightly
sped
up.
Demo
I
was
just
on
GitHub
clicked
the
button
and
then
it's
preparing
the
workspace
pulling
the
image
right,
and
this
is
the
face.
A
A
So
that's
what
gitpod
does-
and
this
is
the
context
in
which
we're
talking.
So
we
pull
a
lot
of
different
images
so
over
the
any
sort
of
period
of
seven
days,
we'll
pull
more
than
ten
thousand
distinct
images
measured
by
their
manifest,
not
necessarily
so
they
might
have
some
layer
overlap,
but
it's
ten
thousand
distinct
images
and
we
cache
more
than
a
terabyte
of
of
image
layers,
and
these
images
vary
greatly
in
size
anywhere
between
200
megabytes
to
several
tens
of
gigabytes.
A
A
So,
just
to
tell
you
a
bit
about
where
we're
coming
from
from
a
Time
perspective
and
where
we've
been
heading
so
about
12
months
ago,
when
we
embarked
on
a
let's
get
this
faster
Journey
P95
workspace
startup
times
was
more
than
10
minutes
and
our
Prometheus
histogram,
just
you
know,
stopped
at
10
minutes
within
like
beyond
that
it
doesn't
matter
you're
going
to
assume
the
system
is
broken.
Anyways,
so
more
than
10
minutes,
and
today,
we've
brought
this
down
to
120..
A
That
is
still
a
very
long
time
to
be
waiting
in
front
of
a
computer
and
until
you
come
work,
so
we'll
keep
on
working
bringing
that
down.
But
this
is
by
far
the
largest
reduction
that
we've
been
able
to
achieve,
and
a
good
part
of
this
is
because
of
the
caching
mechanisms
that
we'll
discuss
in
this.
In
this
talk,
the
p50
startup
time
more
than
halved
as
well.
So
we're
coming
from
about
24
seconds
down
to
about
10
seconds
today,.
A
So
what
have
we
tried?
Well,
first
of
all,
we
tried
nothing
the
Baseline,
so
gipport
workspaces
are
essentially
kubernetes
pods
and
then
some
and
the
Baseline
was
to
just
use
the
kubernetes
mechanisms
that
are
out
there.
So
we
would
rely
on
the
layer,
reuse,
essentially
that
would
that
would
happen
within
the
nodes
and
that
is
sort
of
the
12
month
ago.
Situation
that
you
just
saw
me
refer
to,
then
we
try
to
pre-pool
images.
So
what
we
would
do
is
we
would
put
a
Daemon
set
in
place.
That
would
pull
well-known
images.
A
A
We
basically
looked
at
what
was
used
last
week
and
assumed
that
that
was
going
to
be
used
next
week
and
we
would
pre-pull
that
the
problem
with
this
approach
is
that,
especially
during
rush
hour,
so
in
the
morning
when
a
lot
of
people
fire
up
their
deaf
environments
will
have
scale
up
of
nodes
too,
and
then
the
demon
set
will
come
in.
There
are
going
to
be
a
lot
of
image,
pulls
on
that
note
happening
already.
A
So
not
only
is
the
demon
set
ineffective
if
it
even
exuberates
the
problem,
because
it's
going
to
do
pulls
on
top
of
the
pulse
that
uses
actually
are
waiting
for
The
Next
Step
was
to
pre-bake
that
into
VM
images.
So
we
run
on
on
k3s
on
gcp
with
VM
images,
so
we
just
pre-baked
those
into
the
VM
image
so
that
once
the
VM
came
up,
it
wouldn't
need
to
pull
those
commonly
used
images
that
also
worked.
A
It
worked
considerably
better
than
pre-pulling,
specifically
because
pulling
the
VM
image
is
faster
than
pulling
a
lot
of
individual
tiny
top
comparatively
tiny
top
files.
The
downside
of
that
is
that
we
would
need.
We
would
churn
those
VM
images
very
often
like.
In
order
for
this
to
be
effective,
we
would
need
to
produce
new
VM
images.
Very
very
often
they
would
loot
like
this
would
lose
its
Effectiveness
in
about
three
days
and
so
that
just
wasn't
a
viable
path
forward.
Also,
it
helped
a
lot
on
the
p50.
A
It
did
not
help
a
lot
on
the
P95
right.
The
P95
is
caused
by
people
bringing
images
that
are
not
commonly
used
but
large,
and
this
would
specifically
help
with
images
that
are
commonly
used,
because
we
just
take
like
the
10
or
15
most
commonly
used
images,
and
we
pre-baked
those
so
we'll
help
a
lot
with
p50,
not
with
P95,
and
all
of
this
then
led
to
looking
into
other
means
of
speeding
up
the
the
image
pull
times,
and
so
we're
not
alone
in
this
endeavor.
A
It
is
sort
of
the
way
we've
we've
been
looking
at.
It
and
one
is
sort
of
distributed
file
systems
where
you
try
and
distribute
the
the
layers.
Essentially
right,
that's
what's
what's
expensive
at
the
end
across
the
different
nodes,
so
you
don't
have
to
pull
them
from
a
registry
that
is
potentially
far
away
and
slow
to
reach.
A
A
It
doesn't
just
have
to
be
vs
code,
it
can
be
jetbrains
products,
for
example,
and
so
we
require
a
bunch
of
layers
to
sit
on
top
of
what
a
user
configured
and
it's
the
IDE
layers
and
then
there's
also
some
gitpod
specific
tooling
that
we
require
within
the
same
file
system
of
your
of
your
workspace,
and
we
could
just
build
out
those
combinations
right.
So
whenever
we
update
our
own
images
or
whenever
there
are
ID
images,
we
could
just
multiply
that
out
with
the
images
that
users
have
configured.
A
This
would
lead
to
a
to
an
explosion,
quite
literally
of
images
that
we
need
to
take
care
of.
It
would
also
not
provide
a
very
good
user
experience,
because
whenever
we
update
one
of
our
images
and
we'll
deploy
anywhere
between
1
to
15
to
20
times
a
day,
you
know
you'd
have
to
rebuild
this
every
single
time.
A
So
instead,
a
registry
facade
is
what
workspace
pods
actually
pull
from.
So
the
image
that
is
used
will
not
be
directly
pulled
from
some
remote
registry,
but
instead
it
will
point
to
an
instance
of
registry
facade,
usually
running
on
the
same
node
and
then
registry
facade.
Will
dynamically
assemble
the
Manifest
to
be
pulled.
A
A
So
I'm
broadly
going
to
assume
you
all
know
what
ipfs
is
I
by
no
means
an
expert
on
this,
so
I
will
not
claim
to
be
broadly
speaking.
Ipfs
is
a
peer-to-peer
based
distributed
file
system
and
for
the
context
of
this
talk,
that's
good
enough.
A
description.
There
is
much
much
more
that
goes
into
it,
and
I
want
to
be
aware
of
that
of
that
depth
without
needing
to
go
into
it.
A
So
what
we
did,
then,
is
we
embarked
on
the
Journey
of
trying
to
speed
up
image
poles
and
what
we're
going
to
show
here
is
what
that
was
specifically
done
around
April
April
to
May
earlier
this
year.
So
some
of
the
statements
about
other
projects
that
we'll
make
might
no
longer
be
true
and
I'd
if
any
of
the
maintainers
of
those
projects
are
in
the
room.
I'd
very
much
welcome
them
to
speak
up
in
in
the
Q
a
session
and
bring
corrections
to
to
what
I'm
saying
so.
A
The
very
very
first
thing
that
we
did
is
we
looked
into
nerd
control
or
not
cuddle
ipfs
registry,
and
what
node
cuttle
ipfs
registry
does.
Is.
It
is
a
registry
very
much
like
our
own
registry
facade
that
you
can
run,
for
example,
on
the
same
node
and
that
you
can
pull
from
and
the
way
you
pull
from
it?
Is
you
address
the
image
using
an
ipfs
Content
ID
using
an
ipfsc
ID?
So
instead
of
doing
I,
don't
know
GCR
or
something
something
you
would
do
a
local
host,
50
50?
A
A
This
got
us
sort
of
got
our
feet
wet
with
ipfs.
It
was
not
the
solution
to
our
problem.
First
off,
you
need
to
know
the
the
CID,
the
content
ID
and
the
way
you
would
usually
do
that
is.
You
would
use
nerd
control
ipfs
push
and
you
would
push
into
that
registry,
and
that
would
give
you
the
CID
that
you
could
then
later
use
to
reference
and
at
the
time
there
was
no
content
distribution
across
nodes,
because
the
example
did
not
incorporate
ipfs
cluster,
and
this
has
since
been
fixed.
A
So
we
started
introducing
this
with
registry
facade
and
the
first
attempt
was
to
use
stargaze
the
Stargate
snapshotter,
and
so
the
Stargate
snapshotter
has
a
feature
where,
if
the
oci
descriptor
of
a
blob
contains
an
ipfs
URL,
it
will
pull
from
that
right.
Something
that
say
your
your
regular
overlay
snapshot.
I
could
not
do,
and
so
the
idea
was
that
we
would
use
registry
facade
and
if
a
request
comes
in
we'll
get
the
Shah
of
The
Blob
out
of
redis
to
translate
that
into
the
content
ID
of
of
ipfs.
A
And
if
that
blob
was
not
present,
then
we
would
have
registry
facade.
Take
that
blob
put
it
into
ipfs
stream
it
as
it
was
being
downloaded
by
the
consumer
and
then
set
that
Shah
hash
to
the
CID
to
the
resulting
content.
Id-
and
this
is
the
opportunistic
caching
part
right-
we
will
basically
start
caching
a
blob
the
moment
it
is
used.
If
we
haven't
cached
it
before
with
that
translated
CID,
then
we
could
make
the
corresponding
entry
in
the
oci
descriptor
and
have
the
Stargate
snapshotter
pull
that
from
within
ipfs
at
the
time.
A
So
we
would,
we
would
have
needed
to
stargazify
all
images
in
order
to
use
this
all
right,
and
that
is
something
that
we
could
have
done
and
it's
quite
interesting
to
look
at
the
lazy
pulling
aspect
of
that.
But
it
is
not
the
path
that
we
wanted
to
go
down.
Also,
it
would
have
required
cons
a
considerable
amount
of
excess
compute
just
to
to
do
that,
given
the
amount
of
images
that
we
need
to
handle.
A
A
A
We
have
registry
for
start
running
and
we
have
ipfs
node
running
all
right
so
that
all
of
this
is
already
node
local,
ideally
and
ipfs
distributes
the
individual
blob
content
to
those
workspace
nodes
where
that
content
will
be
needed
so
that
we
don't
have
to
pull
across
nodes
a
lot.
That
was
the
idea,
and
this
actually
really
helped
improve
image
pull
times.
A
However,
the
intersone
traffic
that
that
caused
was
very
expensive,
so
we
would
span
anywhere
from
like
up
to
50
to
100
nodes
on
a
given
day
from
close
to
five
up
to
100
and
then
back
down
again.
So
there
would
be
a
lot
of
content
redistribution
happening
within
this
ipfs
cluster
and
there's
no
awareness
at
this
point
of
availability
zones.
So
we
would
be
paying
inter-region
traffic
for
this
sort
of
distribution
of
of
content.
That
ipfs
was
doing
on
top
of
that.
A
The
ipfs
nodes
lift
on
the
workspace
nodes
right
where
the
actual
workloads
happen,
and
so
they
scale
up
and
down
considerably
throughout
the
day
and
as
they
scale
down
we're
going
to
lose
content
that
we
previously
had
cached.
And
so
our
cache
Miss
rates
would
go
up,
as
our
cluster
would
scale
down
and
the
next
day,
we'd
essentially
start
from
zero,
because
we
would
have
lost
most
of
what
was
cached
the
day
before,
because
those
nodes
had
scaled
down.
A
So
the
fourth
iteration-
and
this
is
what
we
operate
today-
makes
this
whole
thing
region
aware.
So,
instead
of
running
the
ipfs
node
and
cluster
proxy
on
the
same
nodes
as
the
work,
the
workloads,
we
run
them
in
a
separate
setup.
And
at
this
point
we
we
operate
in
in
three
availability
zones.
So
we
quite
literally
have
three
ipfs
nodes
running
which
farscale
currently
works.
A
A
This
is
is
cumulative,
so
this
is
not
just
one
node
and
you'll
notice,
the
the
yellow
line
in
the
top
graph,
if
you
can
I,
took
that
out
and
put
that
in
the
lower
graph.
So
this
is
the
the
exact
same
graphs.
It's
just
one
doesn't
have
the
green
line,
and
what
you'll
notice
is
that
step
change?
This
order
of
magnitude
change
and
download
speeds.
A
The
other
thing
to
look
at
is
to
download
request
rate
I.E.
What
are
we
actually
pulling
from?
You
know?
Is
registry
facade
proxying
to
an
upstream
registry,
or
is
it
polling
using
its
own
ipfs
cache,
and
what
you
can
see
here
is
for
one
sort
of
the
this
peaking
behavior
and
that
corresponds
to
the
to
the
scale
and
to
the
load
that
the
system
sees
throughout
the
day
and
then
again
the
yellow
line
is
proxying
to
an
upstream
registry
and
the
green
line
is
pulling
from
ipfs
and
I
zoomed
in
here.
A
So.
In
conclusion,
we
gained
considerable
startup
time
improvements,
specifically
through
the
reduction
of
bandwidth
requirements
towards
Upstream
Registries
and
by
bringing
that
data
much
closer
to
where
it's
needed
in
a
very
opportunistic
fashion.
So
we
do
not
actively
pre-warm
caches
and
we
just
rely
on
what
users
pull
and
we
will
add
that
to
the
cache,
assuming
that
it
will
be
needed,
it
will
be
needed
again.
There
is
a
time-based
eviction
mechanism
that
I
have
not
spoken
about.
A
A
A
So
I
think
the
the
main
benefit
you'll
see
the
main
benefit,
not
So
Much
from
the
amount
of
data
really,
but
rather
the
variance
of
data
I
want
to
say
so.
If
you
keep
pulling
the
same
image
over
and
over
again,
the
layer
reuse
that
happens
on
the
Node
will
likely
be
sufficient.
If
it
is
a
large
variety
of
images
that
you
need
to
pull
over,
which
you
have
no
control,
then
something
like
this
will
be
useful.
The
moment
you
can
reasonably
predict
what
it
is.
C
Hi
great
talk
by
the
way.
Thank
you.
So
one
of
the
aspects
you
mentioned
that
this
helped
with
was
reducing
Network
bandwidths
in
a
way
or
reducing
load
on
the
actual
registry,
because
you're
mostly
pulling
from
ipfs,
did
it
have
a
latency
impact
for
cases
where
you
missed
the
cash,
for
example,
because
then
you
essentially
have
two
hops.
A
A
Much
okay,
the
cost
of
a
cash
Miss
is
essentially
pulling
from
the
Upstream
registry.
It
is
very
cheap
because
we
will
detect
the
cache
Miss
like
we
assume
that,
what's
in
redis
and
what's
in
ipfs
is
on
parity,
and
so
the
check,
if
something
is
in
cash,
is
essentially
a
redis
get,
which
is
a
very,
very
fast
operation.
D
A
E
A
E
A
D
A
So
let
me
rephrase
the
question
just
to
make
sure
I
understood
correctly.
The
question
is
essentially
is
what
we
just
showed
open
source
right.
So
it
is
the
all
the
code
of
what
I
just
presented
can
be
found
in
in
our
repo
github.com
IO.
So
pod
and
they're
in
the
registry
facade
component
you'll
you'll
find
the
code
that
I
spoke
of
hivl
license.
H
A
Yeah,
that's
a
great
question,
so
we
also
pull
from
like
the
Upstream
registry
that
we
would
pull
in
a
cache.
Miss
case
is
also
one
that
we
control.
It's
essentially
Google
artifact
registry,
and
we
have
still
found
that
by
caching,
this
very
close
to
the
node.
We
do
see
considerable
speed
improvements,
so
I
would
vager-
and
this
is
you-
know-
speculation
at
this
point
that
the
most
the
most
benefit
we
see
here
is
from
the
locality
rather
than
the
replication
itself
right.
A
I
Yeah,
so
the
idea
of
caching
helps
when
the
same
images
are
pulled
multiple
times
on
a
given
note
for
newly
provisioned
notes.
Have
you
look
into
using
network
SI
storage
to
you
know,
host
all
the
images
on
the
volume
and
I
know
it
doesn't
work
with
the
NFS
like
overlayfs,
doesn't
work
with
NFS,
but
have
you
export
options
like
that.
A
The
main
reason
is
because
we
did
not
want
to
dive
too
much
into
how
the
how
the
nodes
themselves
operate
again,
we're
coming
back
to
get
part
deployment
models
and
sort
of
the
degree
of
control
that
we
have
over
the
infrastructure
that
we
run
on.
So
we
did
not
want
to
sort
of
assume,
for
example,
that
we
can
meddle
with
content
of
these
content
store
or
something
to
that
tune.
J
A
So
the
there
is
a
policy
on
what
registry
facade
can
pull,
what
it's
allowed
to
pull.
In
the
end,
the
authentication
like
registry
facade
itself
is
just
authenticated
against
a
single
external
registry
and
that's
where
it
will
always
pull
from.
So
it's
not
sort
of
authentication
transparent.
If
that
makes
sense,
which
is
really
an
artifact
of
how
registry
facade
is
used
within
the
context
of
gitpod
I'm
certain
that
this
context
could
be
expanded
further,
so
I
do
remember
when
Docker
introduced
cost
on
on
manifest
pools.
A
You
know
we
saw
a
bunch
of
pull
through
proxies
popping
up,
notably
from
Alex
Ellis,
for
example,
who
wrote
one
and
he
basically
built
a
transparent
authentication
pass-through
which
you
would
need.
We
just
opted
not
to
to
build
this
out
of
the
architectural
setup,
but
conceptually
that
would
well
be
possible.
K
Yeah
pretty
cool
thanks
for
introducing
this
I.
Do
you
benefit
from
the
ipfs
peer-to-peer
distribution
like
if
one
of
your
nodes
contacts
the
ipfs
in
its
AZ,
and
it's
not
there?
Does
it
pull
from
the
other
nodes
if
it's
in
another
AC
or
does
it
go
to
the
Upstream
like?
Is
that
automatic,
or
did
you
have
to
write
a
lot
of
stuff
for
it.
A
That
is
a
great
question
which
I
actually
cannot
answer.
I
would
like
to
defer
that
to
Alejandro
I'll,
repeat
it
for
Alejandro.
The
question
is
what
if
there
is
a
cache
Miss
within
an
availability
Zone,
will
that
node
talk
to
like
that
ipfs
node
talk
to
another
ipfs
node,
or
is
it
just
a
cache
Miss?
In
the
end.
L
L
A
Yeah,
so
to
rephrase
the
question
to
make
sure
I,
understood
correctly,
is
what,
if
an
availability
Zone
goes
down.
What's
the
impact
of
of
something
like
that
and
its
likelihood,
so
the
impact
of
it
would
be
a
loss
of
caching
in
that
particular
region,
which
would
be
a
degradation
of
service,
and
we
would
treat
it
as
such.
At
this
point
we
have
not
seen
that
so
the
notes
are
they
cycle
right,
they
cycle
regularly
and
so
we'll
just
have
it
reinitialized
with
the
other
three
nodes.