►
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Registries After Dark - Daniel Mangum, Upbound & Jon Johnson, Google
You know the registry as your most boring friend. You push and pull images, and it just works, but have you ever taken the time to really get to know it? What is the registry really like behind that unassuming OCI specification? What does it do when it’s not just distributing your images? Maybe it gets a little crazy on the weekend - maybe it has hidden talents you don’t know about. What would happen if this thankless hero went rogue? In this talk we will demonstrate unconventional registry implementations, including those that serve self-modifying and dynamically generated images. Along the way we’ll also take a look at how clients can utilize registries in interesting and unexpected ways, e.g. as a content-addressable key-value store or a general-purpose directed acyclic graph database. Attendees will walk away with a better understanding of what guarantees OCI images and registries provide, as well as how they can exploit the registry’s flexibility to benefit their own use cases.
A
Hey
folks,
my
name
is
dan
mangum
and
I
work
at
a
company
called
upbound
and
I'm
joined
today
by
john
johnson
from
google.
First
of
all,
we
want
to
thank
you
for
taking
time
out
of
your
kubecon
experience,
whether
attending
virtually
or
in
person
to
attend
our
talk
today.
We
hope
that
wherever
you
are,
you
are
safe
and
well.
Today
we
are
going
to
spend
some
time
talking
about
what
you
have
likely
heard
referred
to
as
container
registries.
B
B
I
think
that's
a
little
sad.
I
also
think
it's
not
true.
So
let's
look
at
how
the
registry
sees
the
world
and
what
happens
when
you
interact
with
it.
So
first,
let's
go
over
all
the
data
structures
that
a
registry
has
to
deal
with
it's
going
to
be
more
or
less
oci
101,
so
the
fundamental
primitive
in
oci
is
what's
called
a
content.
Descriptor.
You
can
see
here
it's
a
very
simple
structure
with
three
fields:
there's
a
media
type,
the
digest
and
the
size.
B
So
for
a
given
piece
of
content,
a
descriptor
tells
us
how
to
interpret
those
bytes,
how
many
bytes
we
should
expect
to
read
when
we're
fetching
it
and
the
exact
hash
of
the
content
of
those
bytes.
So
we
know
exactly
what
we
get
when
we
go
to
fetch
it,
and
this
allows
us
to
fetch
content
over
the
internet
safely,
and
this
is
the
most
crucial
bit
of
oci
and
what
I
think
is
most
interesting.
B
You
might
notice
that
those
three
properties
map
pretty
directly
onto
http
headers,
so
it's
very
straightforward
to
serve
this
content
via
http
and,
as
you
might
have
guessed,
the
registry
api
speaks
over
http
http,
so
the
media
type
maps
to
this
content
type
size
maps
to
a
content
length
and
the
digest
maps
to
this
oci
kind
of
specific
thing.
Docker
content
digest.
B
If
we
drew
some
lines
between
these
objects,
we'd
have
this
wonderful
graph
and
you
can
see
anywhere.
There's
an
arrow
is
kind
of
a
content
descriptor
from
one
object
that
references
another,
and
you
can
see
here
that,
while
some
objects
are
manifest,
other
objects
are
blobs
that
the
registry
doesn't
really
know
that
much
about
right.
We
just
have
arrows
coming
into
those
and
then
coming
out.
B
There
are
a
couple
other
interesting
features
that
the
registry
knows
about
that
might
as
well
mention
so
for
legal
reasons.
You
might
not
want
clients
uploading
certain
pieces
of
content.
As
an
example
say,
microsoft
doesn't
want
you
distributing
the
windows
based
layers
to
work
around
this.
The
registry
has
to
understand
this
concept
of
foreign
layers
that
need
to
be
downloaded
from
somewhere
else
say:
microsoft.com,
not
the
registry
you're
pulling
from
another
interesting
feature
is
similar
to
kubernetes
manifests
oci.
B
Has
this
concept
of
an
annotation
map
where
you
can
just
stuff
arbitrary
strings,
so
we're
not
really
just
serving
tar
balls
right.
We
have
one
this
nice
content,
addressable
object,
storage
thing
for
blobs.
We
also
have
this
interesting
merkle,
dag
api
for
the
manifest
so
and
beyond
that,
like
there's
this.
A
B
Api,
where
you
can
store
arbitrary
things
that
have
type
safety
features
and
are
memory
safe,
because
you
know
how
large
they
should
be.
They're
tamper
proof
thanks
to
their
digest
and
they're
human
readable,
because
they're
json,
thanks
to
kubernetes,
are
also
ubiquitous
and
that
you
always
have
access
to
a
registry
more
or
less,
because
it's
cloud
native
and
most
registries
support
some
kind
of
garbage
collection,
but
you
can
also,
you
know,
put
interesting
arbitrary
metadata
on
things.
B
A
When
we
talk
about
sending
data
between
computers,
we
often
make
distinctions
between
different
types
of
data.
For
instance,
you
likely
don't
feel
like
pulling
a
container
image
is
the
same
operation
as
watching
a
video
on
youtube
and
in
some
respects
it's
not.
However,
all
data
transmission
ultimately
boils
down
to
an
order
of
operations
and
bits
on
a
wire.
As
john
mentioned,
the
distribution
spec
broadly
defines
two
categories
of
data
blobs
and
manifests.
A
Let's
start
out
by
talking
about
the
former
in
the
common
case,
blobs
often
represent
gzip
tarballs,
containing
a
file
system
layer
or
in
simpler
terms,
a
compressed
directory
of
files.
However,
this
is
certainly
not
a
requirement
as
while
the
oci
image
spec
defines
a
set
of
acceptable
media
types,
the
distribution
spec
has
no
such
requirement
learning.
All
of
this
was
a
surprise
to
me,
because
I
had
only
ever
seen
the
registry
in
professional
settings.
Little
did.
A
I
know
that
the
registry
was
actually
very
open-minded
so
different
than
the
image
spec
registries
are
frequently
referred
to
as
a
form
of
content.
Addressable
storage.
What
does
that
mean?
If
you
have
ever
used
a
blob
storage
solution?
Let's
say
an
s3
bucket:
you
know
that
you
can
download
content
from
the
bucket
using
a
url.
This
url
frequently
indicates
the
location
of
the
content,
but
may
not
inform
about
the
actual
bits
in
the
content.
A
This
means
that
when
you
request
content
from
a
given
location,
the
onus
is
on
you
to
check
and
make
sure
that
you
are
given.
What
you
asked
for
this
is
frequently
referred
to
as
location
addressed.
Content
address,
on
the
other
hand,
means
that
we
can
ask
for
exactly
what
we
want
by
submitting
a
digest
of
the
content
rather
than
a
location.
A
Now
there
is
nothing
that
says
that
a
registry
may
not
return
some
malicious
content
even
in
a
content
address
system,
so
clients
should
still
check
that
the
digest
of
the
content
they
receive
matches
what
they
ask
for.
So
you
could
say
the
registry
is
pretty
well
connected.
It's
rare
that
I
ever
ask
it
to
find
some
content
for
me,
and
it's
unable
to
do
so.
A
Another
thing
you
may
not
know
is
that
the
registry
is
a
great
listener.
It
knows
that
folks
communicate
in
different
ways,
so
it
offers
a
few
different
mechanisms
to
submit
content
when
pushing
a
blob.
A
client
may
be
able
to
upload
all
of
the
content
in
a
single
request,
or
it
may
require
doing
so.
Incrementally
the
distribution
spec
accommodates
both
of
these
scenarios
by
supporting
multiple
upload
workflows.
A
Let's
take
a
brief
look
at
the
different
strategies
and
the
advantages
and
drawbacks
to
each
the
most
common
method
for
uploading,
a
blob
is
chunked.
Typically,
this
doesn't
actually
mean
that
the
blob
is
being
uploaded
in
multiple
chunks,
for
instance,
we'll
see
in
a
moment
that
docker
will
upload
via
the
chunked
method,
but
then
send
the
full
blob
in
the
patch
step.
A
The
advantage
to
this
method-
and
one
reason
it
is
most
commonly
used,
is
that
it
supports
resumable
pushes
this
means
that
if
a
connectivity
is
lost
during
upload,
a
client
can
resume
pushing
content
without
re-uploading.
What
is
already
there.
Another
advantage
is
that
if
the
content
is
stored,
uncompressed
the
client
can
compress
and
calculate
the
digest
of
the
compressed
content
in
a
streaming
fashion
without
having
to
actually
write
the
compressed
content
to
disk.
First,
the
distribution
spec
also
supports
uploading,
a
blob
monolithically
via
two
different
strategies.
A
A
The
last
strategy
is
a
single
post
and
it
is
specified
as
optional
and
in
practice
is
hardly
ever
used.
Why?
Well,
an
important
thing
to
note
in
the
previous
two
workflows
is
the
step
of
obtaining
a
session
id.
This
is
returned
in
the
form
of
a
location
url,
but
that
location
does
not
necessarily
have
to
point
back
to
the
registry.
This
means
that
a
registry
may
offload
the
actual
upload
to
a
different
service,
something
that
would
not
be
possible
in
the
single
post
scenario.
A
So
now
that
we've
seen
the
multiple
different
methods
for
uploading,
blobs
and
manifest
the
registry,
let's
actually
look
at
how
a
real
client
goes
about
uploading,
a
multi-arch
image
in
this
case.
So
here
we're
at
iaker.o
cubecon,
which
is
the
namespace
that
we're
going
to
push
this
image
to
and
we're
using
crane
copy
here,
which
is
going
to
take
the
crossplane
crossplane
v
1.4
image
from
docker
hub
and
copy
it
over
into
our
yacker
registry.
Here
in
the
ui
here,
you're
going
to
see
the
different
operations
that
are
taking
place
as
it
goes
through
this.
A
A
In
just
a
moment
there
you'll
see
and
see
that
it
points
to
each
of
these,
and
if
we
also
look
at
the
different
operations
that
took
place
starting
off
the
beginning,
you'll
see
that
we
first
checked
to
find
the
existence
of
the
various
blobs
we
were
going
to
upload
before
then
starting
to
do
our
uploads.
We
are
patch
requests
and
eventually
committing
an
upload
for
each
with
a
put
request.
A
A
A
A
A
The
manifests
for
these
resources
are
bundled
into
an
oci
image
for
easy
distribution
by
building
a
stream
of
yaml.
Writing
it
to
a
file
named
package.yaml,
then
packaging
it
up
in
a
gzip
tarble.
The
result
is
an
artifact
that
can
be
pushed
to
any
registry,
even
ones
that
are
on
the
more
picky
side.
A
Strategies
like
the
one
crossplan
employs
are
useful,
but
there
is
nothing
that
says
a
given
image
is
a
crossplain
package
outside
of
the
way
that
we
parse
it.
This
presents
some
pros
and
cons,
and
some
of
the
negatives
of
the
approach
are
attempting
to
be
solved
by
the
oci
artifacts
initiative,
which
aims
to
take
the
flexibility
of
the
distribution
spec
and
provide
a
standardized
method
for
defining
new
artifact
types.
A
A
A
drawback
to
this
approach
is
that
the
explosion
of
media
types
and
the
fragmentation
of
tooling
to
handle
oci
artifacts
is
really
exacerbated
when
designing
the
crossplane
packages.
One
of
the
things
we
wanted
to
ensure
was
that,
while
we
do
offer
dedicated,
tooling
users
could
interact
with
crossland
packages
using
nothing
more
than
their
existing
registry
client
of
choice
as
more
and
more
artifact
and
media
types
are
introduced.
We
start
to
lose
out
on
the
original
benefit
of
the
registry
and
related
tooling,
that
it's
ubiquitous.
A
So
now
we've
seen
two
ways
that
registries
can
be
used
to
distribute
data.
That
is
not
meant
to
be
run
as
a
container,
but
we
really
haven't
pushed
it
to
the
limit.
Yet,
if
we
look
at
the
two
data
types
in
the
distribution,
spec
manifests
and
blobs.
It's
clear
that
manifests
are
the
limiting
factor
here.
After
all,
a
blob
is
just
content
and
content
is
just
bits
in
the
previous
two
examples:
we
either
shoehorned
our
content
into
an
accepted
media
type
or
defined
a
new
one.
A
A
A
So
the
next
thing
I'm
going
to
do
after
I've
gotten
my
location,
is
make
a
patch
request,
with
the
content
of
my
upload
and
you'll,
see
here
that
in
our
ui
we
actually
see
that
upload,
that's
present,
but
it
hasn't
yet
tied
to
been
tied
to
a
blob.
Yet
so
the
next
thing
we
need
to
do
is
actually
commit
this
upload
and
say
that
you
know
this
is
the
the
full
amount
of
content
that's
present
in
this
blob
and
also
provide
the
digest
that
should
exist
for
the
blob.
A
The
last
thing
I
want
to
do
is
go
ahead
and
pull
this
blob,
and
that
can
easily
be
done
with
a
get
request
as
well,
and
I'm
just
going
to
give
the
output
of
this
bob
into
a
different
jpeg.
Pull.Jpg
and
you'll
see
that
we
we
represented
that
poll
here
in
the
in
the
ui,
and
if
we
go
ahead
and
open
up
our
port.jpg,
we'll
see
that
we
got
the
exact
same
image
back.
B
B
B
B
However,
if
you
deploy
things
by
tag
like
most
people,
your
trusted
computing
base
just
got
a
lot
bigger.
Can
you
trust
your
registry
domain?
Can
you
trust
that
no
one's
management
in
your
client
or
manipulating
what
mirrors
you
use
or
have
stolen
credentials
for
your
registry
or
even
have
bypassed
your
registry
credentials.
B
B
B
Why
am
I
so
passionate
about
this?
The
answer
is
typos
and
if
you
know
me,
you'll
think
that's
funny,
but
when
I
first
started
at
google,
I
noticed
a
pattern.
A
lot
of
our
customer
support
requests,
including
from
internal
teams,
ended
in
the
exact
same
way.
Eventually,
someone
noticed
there's
a
typo
and
they
close
the
book.
B
This
one
in
particular,
happened
a
lot.
You
notice,
we
run
the
google
container
registry,
not
the
google
registry
container,
you
might
be
thinking
who
cares
that
pool
failed
and
they
eventually
noticed
the
typo.
Well,
what?
If
the
pool
didn't
fail
in
the
case
where
you're
trying
to
pull
by
a
digest,
your
client
would
check
the
response
it
gets
from
the
server
against
the
digest
you
requested,
and
it
would
immediately
fail
if
they
didn't
match.
B
But
if
you're
trying
to
pull
an
image
by
tag,
grc.io
could
respond
with
any
valid
image,
and
your
client
wouldn't
complain.
When
I
first
realized
this.
I
was
kind
of
terrified.
I
immediately
tried
to
register
grc.io
but
found
that
somebody
already
owns
it.
What
would
happen
if
that
person
was
a
malicious?
I
eventually
convinced
myself
that
this
wasn't
a
huge
problem.
I
just
don't.
A
B
B
A
B
B
So
I
highly
recommend
just
watching
artem's
talk
or
reading
the
blog
post,
but
I'm
going
to
give
a
very
hand-wavy
explanation
of
bit
squatting.
So
computers
are
imperfect,
sometimes
a
bit
will
go
rogue
and
just
flip
from
a
0
to
a
1
or
vice
versa.
Many
things
can
cause
this,
for
example,
just
a
lot
of
heat
or
faulty
manufacturing
process,
or
even
cosmic
rays
from
outer
space.
B
B
A
B
Ioka.Io
this
was
an
enticing
purchase
because
it's
both
a
single
bit
flip
and
a
single
character
away
from
docker.io
on
a
qwerty
keyboard.
It
would
take
a
little
too
long
and
probably
be
unsafe
to
actually
flip
a
bit
via
radiation.
So
I'm
going
to
cheat
for
a
demo
and
just
pull
from
yakker.io
directly.
B
B
B
B
So
what
I've
done
for
the
demo
is
set
up
a
registry
that
just
proxies
your
requests
back
to
docker
hub,
but
before
serving
it
to
you
mutates
the
image
to
inject
a
malicious
payload
in
this
case
that
malicious
payload,
just
logs
a
message
and
immediately
runs
your
intended
image.
But
you
could
certainly
imagine
doing
something
much
more
nefarious,
like
bitcoin
mining,
a
reverse
shell,
proxying,
all
the
traffic
to
your
pod.
B
B
B
So
it's
hard
to
really
say
concretely
how
likely
that's
to
happen,
but
based
on
some
numbers,
docker
have
put
out
earlier
this
year.
I
think
that
maybe
possibly
this
is
something
you
should
worry
about
at
300
billion
pools
we're
getting
close
to
just
very
large
numbers,
so
this
stuff
does
happen
in
practice.
A
A
But
what
if
we
threw
caution
to
the
wind
and
really
took
advantage
of
the
flexibility
of
the
distribution
spec?
Well,
we
could
build
a
chat
service
and
in
fact
we
did
since
we
aren't
able
to
be
in
person
with
folks
at
kubecon
this
year.
We
thought
we
could
at
least
do
a
live
demo
as
we
round
out
this
talk
with
one
of
the
most
extreme
examples
of
pushing
the
registry
to
the
limit
feel
free
to
join
us
at
eocor.io
chat
to
talk
about
how
this
presentation
went
and
ask
any
questions
you
might
have.
A
So
how
does
this
work
it's
built
on?
What
john
has
advised
us
to
never
do
pull
by
tag
since
the
content
of
a
tag
is
dynamic,
we
can
essentially
build
our
manifest
just
in
time.
Each
layer
of
our
chat
image
contains
a
log
of
messages
in
between
polls.
New
messages
are
appended
to
an
in-progress
upload.
The
next
time
someone
pulls
the
chat
image.
We
commit
that
upload
as
a
blob
then
serve
a
manifest
that
points
to
it
and
all
the
other
blobs.
A
On
the
client
side,
when
the
image
is
run,
all
previous
chat
messages
are
sent
to
standard
out.
The
binary
connects
to
a
websocket
service
on
yakker.io
and
the
user
is
shown
any
new
messages
that
are
sent
and
is
able
to
send
their
own.
The
websocket
service,
not
only
brokers,
communication
between
users,
running
the
chat
image,
but
also
uploads
new
messages
to
the
in-progress
upload
in
the
chat
repository
and
the
registry.