►
From YouTube: From `docker push` to Bytes on Disk: Inside Distribution - Wayne Warren & Adam Wolfe Gordon
Description
Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
From `docker push` to Bytes on Disk: Inside Distribution - Wayne Warren & Adam Wolfe Gordon, DigitalOcean
If you use containers, at some point you've probably done a `docker pull` or a `docker push`. But, have you ever thought about how those operations work? How does a container image travel to persistent storage in the cloud? What does it look like when it gets there? We hadn't thought much about these questions until we started building DigitalOcean Container Registry (DOCR) on top of the CNCF Distribution codebase in 2019. Working on DOCR required us to learn a lot of the answers and we're excited to share them. In this talk we'll pull back the curtain on how Distribution works. From your registry client, to the OCI Distribution API, to the CNCF Distribution codebase, to bytes on disk, we'll explain exactly how a container image makes it from your computer to the cloud, what it looks like when it gets there, and what happens when you ask for it back. We'll also touch on less-standardized topics such as authentication and the evolving garbage collection implementation in Distribution.
A
B
B
A
Before
we
get
into
a
bunch
of
details,
let's
answer
a
really
basic
question:
what
is
a
container
registry
to
answer
that?
Let's
start
with
something
most
of
us
have
probably
seen
or
done
before,
and
that's
a
docker
push,
so
you
docker
push
some
ascii
arrows
fly
across
the
screen
and
your
container
image
is
now
somewhere
in
the
cloud
later.
Your
co-worker
or
some
random,
stranger
in
the
internet
or
a
deployment
system,
can
pull
that
image
and
run
it
somewhere
else.
So
this
is
the
power
of
containers.
A
As
we
all
know,
you
can
share
the
image
and
it
should
just
run
on
somebody
else's
computer.
The
thing
that
stores
those
images
in
the
cloud-
that's
a
container
registry,
so
we
know
what
a
container
registry
does
for
us,
but
what's
underneath
it's
a
content,
addressable
data
store,
and
what
does
that
mean?
A
It's
an
object
store,
so
it
you
can
put
data
there
and
it's
content
addressable.
That
means
that,
for
each
object
in
that
store,
instead
of
being
identified
by
some
arbitrary
key
or
path
name,
it's
identified
by
a
digest
of
its
contents
in
container
in
a
container
registry.
We
have
a
few
different
kinds
of
objects.
First,
we
have
blobs.
These
are
the
the
basic
object
that
that
are
stored
in
a
container
registry,
and
then
we
have
manifests.
A
A
So
tags
here
are
drawn
as
a
different
shape
because
they're
special
they
have
human,
readable
names
and
they're
mutable.
I
mean
not
a
different
shape,
but
a
different
color.
So
since
manifest
and
blobs
are
identified
by
their
contents,
they're
immutable
change
the
contents
and
you
change
the
identifiers
well
tags.
On
the
other
hand,
you
can
update
later
so
there
are
many
different
registry
implementations,
but
the
one
we'll
talk
about
today
is
a
cncf
project
called
distribution.
A
This
was
previously
the
open
source
registry
implementation
released
by
docker.
They
donated
to
the
cncf
in
2020,
and
it
is
the
reference
implementation
for
other
container
registries,
but
it's
also
used
by
many
people
in
production,
which
includes
us
at
digitalocean.
Our
hosted
container
registry
platform
is
built
on
top
of
this
code
base.
B
So
we've
heard
a
little
bit
about
what
a
container
industry
is
what
it
looks
like
inside.
One
sort
of
mystery
that
we
haven't
talked
about
is
those
ascii
arrows
that
fly
across
your
screen.
When
you
do
a
push,
how
does
your
container
image
actually
get
to
the
registry
and
that
data
get
into
the
cloud?
B
And
the
answer
to
that
is
the
oci
distribution
spec.
So
this
is
the
http
api.
That's
been
standardized
that
registry
clients
use
to
talk
to
registry
servers.
I,
like
I
said
it's
a
standardized
api.
It
reached
v1
last
year,
so
now
it's
really
official
and
it's
vendor
neutral.
It
was
originally
written
by
docker
to
go
with
their
registry
implementation,
but
it's
been
adopted
and
updated
by
the
open
container
initiative
or
oci,
which
is
a
community,
so
it
is
vendor
neutral.
There
are
lots
of
different
implementations
on
both
the
client
and
the
server
side.
A
So,
starting
with
what
a
push
and
a
pull
look
like
from
an
api
perspective
in
the
distribution
spec
and
then
we'll
dive
into
the
distribution
code
base
to
see
how
it's
structured
and
how
it
implements
image
pushes
on
the
server
side
and
then
finally,
we'll
wrap
up
with
a
garbage
collection,
which
is
basically
the
process
of
cleaning
up
or
freeing
up
disk
space.
After
deleting
images.
B
So,
let's
start
by
looking
at
what
happens
in
the
api
in
this
distribution
spec
when
you
push
a
docker
image
or
a
container
image
to
a
registry.
Remember
this
spec
is
standardized.
It
is
vendor
neutral.
So
everything
we're
going
to
talk
about
in
this
section
is
applicable
to
any
implementation
of
it.
We're
going
to
use
docker,
as
our
sort
of
example
commands,
but
that
is
not
important.
You
could
be
using
some
other
tool,
so
remember
this
diagram
from
earlier.
B
This
shows,
what's
in
the
container
registry,
this
is
all
the
objects
that
need
to
make
it
to
the
registry.
When
we
do
a
push-
and
we
talked
earlier
about
how
the
manifest
is
sort
of
pointers
to
these
layers,
it
references
the
layers
that
means
that
as
a
client,
we
need
to
push
the
layers
first
and
then
the
manifest,
because
the
spec
says
that
a
registry
is
allowed
to
reject
an
object
that
has
references
to
non-existent
objects.
B
You'll
notice
that
I
didn't
talk
about
the
the
tag,
that's
because
the
tag
actually
gets
pushed
as
part
of
pushing
the
manifest
it's
sort
of
an
additional
piece
of
metadata.
That's
attached
to
the
manifest
it's
not
really
a
separate
object
in
the
registry,
even
though
we
kind
of
think
of
it
that
way.
Conceptually.
B
The
other
thing
to
point
out
before
we
talk
about
the
the
mechanics
of
the
api
is
that
different
container
images
can
share
layers.
So
in
this
diagram
we
have
this
layer
called
d655
and
so
on,
and
it
is
shared
between
two
different
images:
one,
that's
already
in
the
registry,
the
one
that's
on
the
bottom,
that's
full
saturation,
and
this
one
that
we
want
to
push
to
the
registry
on
top.
B
But
we
don't
have
to
do
that
as
an
optimization.
We
can
figure
out
that
the
registry
already
has
that
layer
and
then
we
can
skip
pushing
it
and
most
real
implementations
on
the
client
side
are
going
to
do
this
so
jumping
into
the
spec
and
what
the
requests
and
responses
look
like.
The
first
thing
we're
going
to
do
is
check
whether
the
layers
that
we
need
that
we
want
to
push
are
already
in
the
registry
and
to
do
that,
we
use
this
head
endpoint.
B
We
identify
the
blob
using
its
digest,
which
is
a
hash
of
its
contents,
and
the
registry
is
going
to
return
a
200
response
code.
If
it
already
has
that
blob,
if
it
doesn't
have
the
blob,
it's
going
to
return
a
404
and
then
we
know
that
we
need
to
push
it
once
we've
figured
out
which
blobs
we
need
to
push
we're
going
to
actually
push
them,
and
this
is
a
little
more
complex,
because
the
distribution
spec
gives
us
three
different
ways
to
do
it.
These
three
different
methods
all
do
effectively
the
same
thing.
B
They
do
three
different
steps,
but
they
do
it
in
different
combinations
of
requests.
So
the
three
steps
that
we
need
to
do
are
first
initiate
and
upload
the
initiating
the
upload
lets.
The
registry
know
that
we're
about
to
push
some
data
to
it
and
it
will
return
us
a
session
id
that
we
can
then
use
in
subsequent
requests
to
refer
to
that
upload.
B
B
Finally,
we're
going
to
finalize
the
upload
when
we
finalize
the
upload,
we
provide
the
digest
that
we've
calculated
of
our
layer,
blob
and
the
registry
then
can
calculate
the
the
checksum
itself
make
sure
that
it
got
the
right
data
that
what
we
said
we
were
going
to
send.
It
is
actually
what
we
sent
it.
This
finalization
step
is
important
for
another
reason
other
than
integrity.
It's
also
important
to
make
sure
that
the
registry
isn't
going
to
make
in
progress
uploads
available
to
download
for
clients.
B
So
we
we
don't
tell
the
registry
the
digest
until
the
end.
That
way,
it's
not
is
not
even
able
to
make
that
upload
available
to
a
client
to
download.
Until
we've
pushed
all
the
data-
and
it
knows
that
it's
there
so
each
of
these
different
methods-
these
three
ways
of
pushing
like
I
said,
does
those
those
three
steps
in
different
combinations.
B
Clients
can
make
really
any
assumptions
they
like
about
that,
and
the
assumption
that
docker
makes
is
that
it
can
transmit
that
entire
blob
in
a
single
patch
request.
So
if
you've
got
a
10
gigabyte
file
system
layer,
you're
going
to
get
a
10
gigabyte
patch
request
a
10
gigabyte
body,
that's
important
to
know
if
you're
implementing
a
registry
or
hosting
a
registry
like
we
are
because
there's
all
kinds
of
timeouts
that
you
might
assume
are
reasonable.
That
turn
out
not
to
be
reasonable.
When
you
have
a
client
making
a
gigantic
patch
request,.
B
One
other
option
that
the
spec
allows
is
called
blob
mounting.
Everything
in
a
registry
is
scoped
to
a
repository
which
is
kind
of
a
namespace.
That's
what
you
might
think
of
as
the
image
name
when
you're
thinking
about
your
container
images
as
a
client.
If
you
know
that
you
share
a
layer
with
an
image,
that's
in
another
repository.
B
B
Maybe
it
got
deleted,
but
if
it
does
allow
it-
and
it
does
already
have
the
layer
you
and
you
include
these
extra
parameters
on
the
post
request
when
you
initialize
the
upload
it'll
return
a
two,
I'm
just
making
sure
I
have
the
right
code,
it
will
return
a
201
response
to
your
request,
and
that
means
that
you
don't
have
to
go
through
the
rest
of
the
sequence
of
pushing
your
layer.
That
layer
is
already
there.
It'll,
just
reuse
it
from
that
other
image.
B
B
Once
the
registry
has
all
of
our
layers,
we
can
push
the
manifest,
and
this
is
relatively
simple.
There's
only
one
way
to
do
it.
The
little
variation
that
there
is
is
that
there's
this
parameter
called
reference
in
the
url
and
that
can
be
either
a
digest
or
a
tag.
I
say
that
if
it
quacks
like
a
digest,
it's
a
digest
and
the
way
that
it
digests
quacks
is
it
has
a
colon
in
the
middle
separating
the
algorithm
like
sha-256
from
the
checksum,
are
the
hash,
which
is
a
big
long,
hex
string.
B
B
If
you
push
using
a
tag
that
does
implicitly
create
the
digest
reference
in
the
registry
as
well,
because
remember,
everything
in
the
registry
is
identified
by
the
hash
of
its
contents
by
this
digest.
So
the
tag
just
if
you
push
by
a
tag,
you
now
have
two
ways
to
refer
to
your
manifest
instead
of
just
the
digest,
but
you
can
push
either
way
so
spec
wise.
Here's.
What
the
image
push
looks
like
end
to
end.
First,
we're
going
to
check
whether
each
of
the
layers
that
we
want
to
push
exists
in
the
registry.
B
Using
that
head
end
point
that
we
saw
then
we're
going
to
push
each
of
the
layers
that
we
need
to
push.
I
in
this
example
using
the
the
three
part
post
patch
put
chunked
upload
and
then
finally,
we're
going
to
put
the
manifest
using
either
that
tag
or
the
digest
like
we
just
saw
in
the
previous
slide,
and
that's
it
at
the
api
layer
of
abstraction.
B
So
as
a
reminder
back
to
this
diagram
again,
these
are
the
objects
in
the
registry
as
a
client.
This
is
the
things
that
we
need
to
pull
and
if,
if
we
want
to
pull
those
layers,
we
need
to
know
which
ones
they
are
so,
as
you
might
expect,
this
is
kind
of
the
opposite
of
the
push.
We
need
to
pull
the
manifest
and
then
we
can
pull
the
layers
once
we
have
the
manifest
and
know
which
layers
they
are
pulling
a
manifest
looks
a
lot
like
pushing
a
manifest.
B
You
remember
the
duck
you
can
use
either
the
tag
or
the
digest,
and
the
only
difference
here
is
that
it's
a
get
endpoint
instead
of
a
put
in
point
pulling
blobs,
is
much
simpler
than
pushing
blobs.
It
is
a
single
endpoint.
It's
the
same
as
that
head
endpoint
that
we
use
to
check
for
existence,
except
now
it's
a
get
and
you'll
get
back
the
the
data.
That's
in
the
layer.
That's
that's
really
pretty
simple!
B
So
spec
wise
again,
here's
the
the
end
points
that
are
used
when
you
pull
an
image.
We've
included
the
two
head
endpoints
in
there
that
you
can
use
to
check
for
existence.
You
don't
have
to
do
that
when
you're
pulling
an
image,
but
you
might
want
to
it's
just
as
an
optimization
to
make
sure
it
has
everything
that
you
need,
but
really
you're
just
going
to
use.
A
A
A
So,
in
an
ideal
world
you
might
expect
oci
abstractions
to
be
a
relatively
straightforward
mapping
between
types
and
their
operations
in
the
real
world
and
especially
in
an
open
source
project
built
over
a
long
time
by
many
contributors
with
different
goals
and
ideas
about
how
code
should
be
structured
and
different
use
cases
for
that
code.
Things
tend
to
get
a
little
bit
messy,
so
I
don't
want
to
belabor
the
details
too
much
here
or
to
disparage
the
code
as
by
representing
it
as
overly
complex.
A
So
the
last
interface
I'd
like
to
visit
is
the
storage
driver.
This
interface
is
what
enables
oci
abstractions
and
the
container
registry
more
generally,
to
target
a
variety
of
back-end
storage
systems
from
in-memory,
which
can
be
useful
for
local
testing
to
production-ready
object,
storage
systems
like
an
s3
api,
so
there's
the
aws
s3
api
and
various
open
source
implementations
as
well
as
digitalocean
spaces,
which
is
an
s3
api
as
well
so
for
a
registry
operator
who
wanted
to
make
distribution
work
with
a
new
backend.
A
This
is
the
interface
that
they
would
have
to
implement
to
do
so,
and
then
they
could
run
a
registry
on
top
of
whatever
their
back
end
is
all
right.
So
up
to
this
point,
we've
talked
about
the
broad
strokes
of
http
interactions
involved
in
image,
pushes
and
pulls,
as
well
as
a
high
level
overview
of
the
distribution
internals
next
we'd
like
to
make
good
on
the
promise
of
the
talk
and
discuss
how
these
elements
fit
together
from
docker
push
to
bytes
on
disk.
A
More
specifically,
we'll
take
a
look
at
the
http
patch
function
for
the
blobs
uploads
endpoint.
Here
you
can
see
the
http
endpoint,
where
n
is
the
name
of
the
image
within
the
registry.
Also,
we've
referred
to
it
as
the
repository,
because
that's
the
name
of
you
know
an
image
in
the
spec,
and
then
s
is
the
session
uuid
shared
across
all
chunk
upload
patch
requests.
A
So
what
exactly
do
we
mean
by
from
docker
push
to
bytes
on
disk
in
the
context
of
the
blob
upload
patch
request?
What
we
want
to
do
is
highlight
all
the
interface
methods
and
objects
involved
in
the
seq
and
the
sequence
in
which
they're
called
to
store
a
chunk
layer
in
the
configured
backend.
So,
in
our
case,
the
configured
backend
is
an
s3
api.
A
During
the
authentication
phase,
we
obtain
credentials
from
the
patch
request,
header
validate
those
credentials
in
a
way
that
depends
on
the
method
in
use.
So
I've
listed
a
few
potential
authentication
methods,
basic
auth,
bearer
token
or
jwt
and
oauth
after
we
authenticate
we
can
move
on
to
resuming
the
session.
A
A
I
realized,
by
the
way
that
this
is
maybe
going
into
a
little
bit
more
detail
than
you
might
want
out
of
a
101
session,
but
this
the
slides
are
available
online
if
you're
interested
to
take
a
closer
look
anyway.
The
the
next
phase
is
the
data
upload
phase,
where
we
stream
the
incoming
patch
request
body
using
the
s3
blob
writer
we
just
received
when
resuming
the
session
and
practically
speaking,
this
is
implemented
using
an
io
copy.
A
That
reads
from
the
patch
request
body
and
writes
to
the
s3
blob
writer
as
bytes
are
received
from
the
client
connection.
The
I
o
copy
will
call
right
on
the
s3
blob
writer
repeatedly
as
bytes
are
streamed
in
from
the
patch
request
body.
So
we
don't
buffer
anything
in
memory
here.
Well,
we
don't
buffer
the
entire
request
in
memory.
I
should
say
we
do,
however,
buffer
a
little
bit
so
internally
for
our
s3
back
end.
A
A
The
create
multi-part
upload
is
only
ever
invoked
on
the
first
patch
request
and
then
the
complete
multi-part
upload
doesn't
happen
during
any
patch
request.
Instead,
it
happens
during
the
final
put
in
the
chunked
upload
sequence
of
post
patch
put
so
yeah
this
more
or
less
wraps
up
our
somewhat
hand.
A
Now
that
we've
covered
how
the
bytes
get
onto
the
disk
kind
of,
let's
consider
how
they
get
off
the
disk
for
distribution.
This
happens
through
a
process
known
as
garbage
collection
in
programming
languages.
Garbage
collection
is
a
memory
recovery
feature
where
the
runtime
detects,
not
all
programming
languages,
some
programming
languages,
where
the
runtime
detects
unused
blocks
of
allocated
memory
and
deallocates
them
for
to
free
them
for
subsequent
use.
Similarly,
in
distribution,
garbage
collection
is
a
disk
usage
recovery
feature
to
understand
what
garbage
collection
is
in
the
context
of
distribution.
A
Let's
revisit
the
container
registry
diagram
from
earlier,
where
we
had
just
pushed
a
new
image
and
updated
the
latest
tag
to
point
at
it.
The
old
image
represented
here
by
the
top
manifest
is
now
in
an
untagged
state.
While
this
manifest
could
be
deleted
directly
through
the
api,
untagged
manifests.
Can
be
garbage
collected,
that
is,
they
can
be
detected
as
untagged
and
automatically
deleted
once
a
manifest
is
deleted
either
manually
or
by
garbage
collection.
A
Such
blobs
are
now
eligible
for
either
manual
deletion
or
garbage
collection
similar
to
untagged
manifests
after
garbage
collection
is
run.
Disk
disk
space
is
freed
for
use
by
subsequent
image.
Pushes
so
garbage
collection,
then,
is
really
a
convenience
feature
that
removes
the
need
for
the
user
to
manually
delete
things
that
could
be
deleted.
A
So
one
resulting
risk,
then,
is
that
a
delete
that
happens
by
one
user
during
another
user's
right
could
lead
to
data
loss
for
the
writing
user,
because
the
writing
user
may
see
the
blob
as
existing
before
they
begin
to
push
their
image
and
decline
to
push
it
or
before
they
begin
to
push
their
layer
and
decline
to
push
it
assuming.
It
will
be
there
at
the
same
time
that
another
user
is
deleting
that
layer.
B
That's
that's
it
for
today.
That's
our
talk
about
container
registries.
Hopefully
you
learned
something
from
this
or
or
learned
some
things
you
already
knew.
Maybe
if
you
already
knew
about
registries,
I
think
we
do
have
a
little
bit
of
time
for
questions
before
we
get
there.
I
did
want
to
say:
we've
got
stickers
they're
on
the
the
table
there.
If
you
want
some
stickers
and
and
we're
also
happy
to
chat
after
and
I
have
more
stickers
in
my
backpack
if
we
run
out
so.
B
C
B
You
can
run
you
can
repeat
as
well.
Oh
yes,
sorry.
The
question
was
whether
you
can
run
distribution
inside
a
kubernetes
cluster
to
give
you
some
locality
of
where
your
images
are.
The
answer
is
yes,
that's
a
really
common
use
case
for
it
you
can
run
distribution
pretty
much
anywhere
and
and
like
we
alluded
to
there's
lots
of
different
storage
backends,
so
you
could
store
images
on
local
disk.
You
could
store
them
in
local
object,
storage
like
in
your
own
cloud
or
in
your
own
data
center.
B
You
can
also
use
s3
or
s3
compatible
storage,
so
you
can
absolutely
do
that.
Distribution
does
also
have
a
mirroring
mode.
That's
explicitly
meant
for
that.
We
don't
use
that
for
our
registry,
we're
using
it
in
sort
of
the
other
mode
where
it's
the
source
of
truth,
but
that
is
an
option.
That's
there
as
well.
D
A
That,
well,
that
would
differ.
I
mean
that
depends
on
how
how
your
product
is
implemented.
So
at
digitalocean
we
have
a
container
registry
product
and
we
use
a
bearer
token,
a
jwt
for
authentication
and
they
have
an
expiry
set
on
them.
So
when
we,
when
a
user
schedules,
garbage
collection
which
happens
through
the
digitalocean
api,
we
mark
the
registry
as
read-only
modes
and
we
no
longer
issue
right,
capable
jwts
and
then
the
experience
on
our
jwts,
I
think,
is
that
15
or
16
minutes.
So
garbage
collection
is
then
scheduled
to
begin
once.
B
In
the
generic
distribution,
I
think
it
actually
goes
into
a
different
mode
where
it
just
doesn't
accept
any
right
requests
but
yeah,
since
we
are
serving
multiple
users
and
we
want
them
to
each
be
able
to
garbage
collect
independently.
We
have
to
yeah
do
some
things
with
with
tokens
to
make
that
work.
E
B
Sure
so
the
question
was,
with
this
read
only
mode
for
garbage
collection.
What's
the
user
experience
for
that,
the
answer
is
that
that
you
get
effectively
an
authentication
failure,
at
least
in
our
implementation
of
it.
I
think
in
the
generic
implementation
you
get
either
an
authentication
failure
or
some
other
kind
of
http
error
back
if
you
try
and
do
a
write
operation.
B
Sure
so
the
question
was
given
that
the
client
doesn't
provide
the
digest
until
the
end
of
the
the
layer
push.
How
do
you
handle
simultaneous
uploads
of
the
the
same
layer?
Potentially,
the
answer
is,
it
would
just
get
uploaded
twice
and,
and
the
last
one
will
win
they're
going
to
get
uploaded
into
the
same
at
the
end
of
that
upload,
they're
going
to
be
moved
into
the
same
location
in
the
backend
storage,
so
that
the
second
one
will
will
end
up
overriding
the
first
one,
but
they
have
the
same
content.
B
B
A
I'm
not,
I
don't
know
if
I
can
answer
that
question
specifically,
but
one
thing
we
didn't
cover
here
is
what
a
manifest
is,
which
is
a
it's.
It
is
itself
a
json
blob,
so
it
gets
stored
actually
as
a
blob
in
the
registry
and
it-
and
it
contains
all
of
the
information
about
your
image
and
like
not
just
the
layers
but
different
annotations,
and
you
know
the
time
it
was
created,
largely
dependent
on
the
runtime
that
built
the
image.
So
different
runtimes
may
have
different
extra
metadata.
A
B
Yeah,
so
the
question
is:
when
you
push
an
image
and
you're
using
the
digest,
is
it
the
the
image
digest
or
the
image
id?
So
I
think
the
image
id
is
specific
to
your
container
runtime.
So
your
your
sort
of
local
container
client,
what
gets
used
for
the
upload
is
that
the
digest
of
the
manifest.
So
it
is
the
the
hash
of
the
manifest
contents,
so
like
wayne,
said
that
the
manifest
is
a
json
file.
Basically,
so
you
take
a
hash
of
that
json
file
and
that's
the
image
digest.
A
Yeah,
so
the
question
is
what
digest
algorithms
are
supported
for
the
the
hash
digests,
and
there
are
two
officially
supported
by
the
spec
that
shaw
256
in
sha
512.
in
theory,
somebody
could
propose
another
one,
like
shaw
384.
I
think
I
think
shot.
384
is
mentioned
in
the
in
the
docs.
If
you,
if
you
read
the
spec
itself,
but
it's
not
like
officially
meant
to
be
supported
and
that
that
actually
gets
into
a
tricky
area
of
because
you
have
to
verify
the
digest
at
when
you
receive
the
blob.
A
A
If,
at
the
end
of
a
chunked
upload
the
digest,
really
shocked
512,
then
distribution
will
go
and
download
the
entire
blob
from
the
back
end
and
recalculate
it
at
whatever
algorithm
was
specified,
but
you
could
also,
in
theory,
calculate
all
supported
algorithms
in
in,
like
you
know,
concurrently
as
you're
taking
the
the
chunk
upload.
Does
that
answer
your
question?
I
kind
of
got
off
on
a
tangent
there.
Okay.