►
From YouTube: P2P Docker Image Distribution in Hybrid Cloud Environment with Kraken - Yiran Wang & Cody Gibb, Uber
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
P2P Docker Image Distribution in Hybrid Cloud Environment with Kraken - Yiran Wang & Cody Gibb, Uber
Docker image is a foundational building block of container based infrastructure, but distributing high volume of docker images in a multi-zone, hybrid cloud system has been a scaling problem for many. In this talk, we will walk through different approaches we investigated for distributing docker images and introduce Uber’s own solution, Kraken, a P2P docker registry loosely based on BitTorrent protocol, and capable of distributing terabytes of images within seconds. This talk will cover: - Image distribution solutions and their characteristics - Whether you need a p2p solution - Architecture, performance and security features of Kraken - How to deploy Kraken in a Kubernetes cluster
https://sched.co/MPcz
A
Okay
cool,
so
let's
get
started
so
this
talk
is
about
crackin,
a
p2p
talk
originally
developed
at
uber
and
how
we
got
there.
So
my
name
is
Iran
and
this
is
a
Cody.
We
have
nano
team
member
could
Evelyn,
but
she
couldn't
make
it
here
because
of
visa
issues.
So
this
is
a
current
cracking
team
at
Auburn.
Of
course,
there
are
other
contributors
who
contributed
to
this
project
and
we
wish
there
are
more
future
contributors.
So
the
agenda
today
is
so
first
we'll
spend
a
couple
minutes,
explain
how
our
image
and
the
dr.
A
Terry
works,
because
I'm
not
sure
if
everyone
here
are
very
familiar
with
their
internals,
then
we'll
talk
about
a
few
common
ways
you
can
use
to
speed
up
talk
her
poo.
If
that's
bothering
you
retinol,
then
we'll
talk
about
p2p
solutions
we
exported
before
Kraken,
and
then
we
talked
about
how
we
are.
How
were
you
working
to
solve
the
all
the
territory
related
issues
and
we'll
have
a
few
minutes
in
the
end
for
questions
so
to
begin
with,
let's
let
me
explain
how
docker
image
works,
so
the
cream
is
actually
a
very
simple
packaging
format.
A
It's
mostly
composed
of
fuel
tar
files,
so
each
docker
image
layer
is
a
very
regular
tar
as
compressed
there's.
One
special
layer
called
the
image
config
that
contains
like
a
murmur
manoeuvrable
use.
Our
entry
point,
those
metadata
so
that
we
may
trans
or
contains
image
manifest,
is
another
regular
tree,
sound
file
that
contains
the
list
of
all
the
layers.
The
last
thing
is
a
docker
image
tag,
so
we
had
to
talk
her
pool
open
to
latest
that
want
to
latest
string
is
a
tag
so
the
tag
to
manifest
mapping
study
in
docker
registry.
A
So
again
so
very
simple
formats.
Basically
apart
or
tars,
you
can
create
a
limit
using
like
shell
script.
Personwho.
Neither
can
alert
you
any
tenderizer.
Toc
registry
is
also
a
very
simple
piece
of
technology
used
to
distribute.
Docker
images
expose
expose
some
very
common-sense,
REST
API
s,
so
I
can
use
them
to
cat
attack
cat
manifest
and
cat
blob.
That's
it.
A
You
can
easily
implement
one
in
like
a
couple
weeks,
so
knowing
what
his
docker
image
and
was
at
a
registry
and
talk
about
how
her
pool
works
against
very
simple,
so
at
the
beginning
of
docker
pool,
we
talked
her,
will
resolve
attack
to
manifest
they
put
down
the
manifest,
take
a
look
at
the
list
of
image
layer,
shots
and
then
put
down
each
image
layer
separately.
In
the
end,
it
decompress
all
the
layers.
A
So,
given
that
talk,
abou
is
basically
a
bunch
of
HTTP
requests
that
put
down
if
they're
individual
is
very
simple
to
do
a
speed
up
dr.
Poole.
So
so
the
first
thing
you
should
do
is
actually
try
to
optimize
your
docker
image
to
make
it
a
more
homogeneous,
but
I
mean
usually
looking
to
have
a
common
image.
For.
If
you
have
a
lot
of
images
comment,
you
should
have
a
common
base
image
and
a
looper.
We
also
have
a
docker
file
template.
It's
almost
there
out
developers,
don't
write
their
own
docker
file.
A
So
by
doing
this,
all
the
docker
file
look
very
similar.
They
have
a
share
of
your
common
common
layers.
We
also
to
multistage
peeled.
So
the
end
result
imagery
is
very
small
in
the
end.
So
recently
there
also
a
lot
of
build
suicide
supported,
distributed
air
cash
for
in-app
amake
tsuzuku
that
over
can
come
from
Google
built
kit
from
docker.
There's
a
talk
going
on
right
about
all
these
tools
so
yeah
they.
We
also
make
the
teeth
between
images
smaller.
A
So
if
you
look
into
those
so
after
you
have
done
optimizing,
your
docker
images,
if
used
to
have
issue
with
docker
pools-
and
you
should
start
looking
at
how
to
optimize
your
hereditary.
So
the
first
the
simplest
thing
to
do-
is
actually
to
deploy
a
layer
of
engine
X
cache
in
front
of
territory.
So
very
idea
of
opacity,
workloads
uber
use
that
for
a
couple
years,
it's
very
scalable,
very
simple,
and
this
works
better.
A
If
you
apply
a
connection
limit,
but
solo
person,
he
using
the
criteria
and
the
Internet's
cache
in
2015,
but
at
late
2016
in
Linux,
is
no
longer
enough.
That's
because
of
boobers
workload.
So
at
over,
we
have
a
lot
of
our
images.
They
are
all
fairly
big
either
to
size
is
about
wengie.
But
these
days
we
see
a
lot
of
10g
images
at
over.
A
We
also
schedule
batch
processing
jobs
on
top
of
our
compute
class,
sir,
those
jobs
they
they
permit
is
crazy,
much
more
than
status
jobs
so
concurrent,
so
they
could
concurrently
a
pool
tens
of
thousands
of
images
every
machine
so
because
of
this
large
talker
pool.
You
can't
really
use
things
like
set
for
HDFS,
because
those
those
oceans
active
than
for
each
machine
for
different
file
you've
had
gone
for
the
same
talking
image
on
all
the
machines.
Actually,
it
doesn't
scale
very
well.
Another
scenario
that
has
been
bothered
us
bothering
us
is
host
maintenance.
A
So
imagine
you
have
a
machine
cluster,
pure
ethically,
have
to
up
with
colonel
up
with
OSF,
take
down
hundreds
of
them
offline
at
a
time,
and
whenever
that
happens,
all
containers
running
all
those
machine
need
to
be
rescheduled.
So
if
you,
if
you
have
like
plant
a
container
vending
machine
and
you
take
down
control
machines,
and
then
you
are
rescheduling
to
southern
containers
at
the
same
time,
so
as
a
$2,000
pool,
all
most
of
them
are
actually
different
images.
A
So
this
is
where
nginx
actually
cannot
handle,
because
if
you
are
putting
a
lot
of
different
images
but
cache
the
standard
works,
the
cache
get
it
past.
Eight
recent
years,
uber
also
started
building
a
lot
of
smaller
data
centers.
So
there's
a
that
create
a
new
problem.
That
is,
how
do
you
create
hobby
replica
images
across
multiple
zones
uber
also
on
top
of
so
purpose,
have
one
prompted
the
center
also
use
TCP,
and
otherwise
there
is
no
common
storage
solution
available
across
all
three
of
them.
These
are
all
issues
we
need
to
solve.
A
So,
given
the
issues
we
started
looking
into
a
p2p
solutions,
so
our
design
design
consideration
is
first,
it
has
to
be
optimized
for
its
entire
internal
usage,
which
is
a
little
different
from
for
the
PPP
in
in
a
while
in
open
Internet.
We
also
hope
the
solutions
who
have
low
maintenance,
because
I
have
a
very
small
team.
We
have
three
people
and
we
work
half
that
half
time
on
this
and
how
to
maintain
like
thousands
of
hosts.
We
don't
have
time
to
you,
know
debug.
A
If
you
every
day
there
shouldn't
be
any
single
point
of
failure.
If
any
any,
if
any
machine
your
new
solution
fails,
we
shouldn't
have
to
work
hard
when
the
fix
time
machine
we
should
be
able
to.
You
know,
take
a
few
days
and
look
at
it
like
that's
a
week
or
something
also,
we
had
to
handle
two
kinds
of
workload
as
I
mentioned
before.
So
first
is
tens
of
thousands
of
same
image
for
batch
processing
jobs
and
the
suddens
of
unique
images
in
the
case
of
hosts
maintenance.
A
Another
thing
is
we
also
care
about
do
completion
time.
So,
if
you
think
about
how
you
were
loading
up
within
the
cluster,
you
care
about
the
last
one
last
motion
I.
Actually
they
finished
auger
pool
you
don't
want
a
CUDA
p50.
You
want
a
coupie
99
100,
so
this
is
also
where
solutions
like
BitTorrent
actually
doesn't
work,
because
Pierce
onesies
didn't
for
multi
mised
for
like
opening
tonight,
it's
more
defensive.
It
had
to
punish
bad
behaviors
optimize
for
the
for
the
people
who
download
the
fast
and
that's
not
what
we
need.
A
So
the
first
thing
we
thought
about
is
way
to
some
kind
of
layered
structure
by
layer.
Thrush
I
mean
it's
great.
Actually
we
employ
deploy
a
simple
solution
would
be
you
deploy
more
layers
of
in
genetics
but,
as
we
talked
about
before
and
genetics
under
work
where
well,
when
you
have
a
lot
of
unique
images
because
there's
nothing
to
cache
a
slightly
fancier.
Variation
of
this
is
you're
somehow
constructed
differently
dynamically
for
each
each
file.
Each
image,
but
after
some
co-creation
realized
tree,
is
actually
not
a
structure
or
not
actually
not
optimal.
A
For
this,
because,
realistically,
virtually
should
be
relatively
fat.
Think
of
Melaka
Petri
and
as
we
get
fatter,
your
your
upload
speed
of
the
parents
actually
get
care
to
get
lower.
So
it's
not
ideal
for
our
big
files.
It's
also
very
hard
to
maintain
circuitry
topology,
because
if
you
think
about
energy
every
point,
every
node
is
a
single
point
of
failure,
so
every
failure
need
to
be
handled
is
a
little
hard
to
implement.
But
that
being
said
there
are,
there.
Are
companies
actually
implemented.
Such
solutions,
for
example,
led
from
Facebook,
but
it's
a
very
complex
project.
A
I
took
them
mad
for
years
right
Ober.
We
want
a
solution
simpler
than
that.
So
another
thing
we
consider
the
in
the
beginning
is
a
central
over
overseer
model.
So
the
Halep
idea
is
you
divide
each.
It
fell
into
smaller
data
chunks
for
in
about
4
megabytes
or
slightly
bigger,
and
you
have
where
you
have
a
central
component
that
make
all
the
scheduling
decisions.
You
said
that
we
should
know
the
transfer
has
transmitted
it
needs
to
wear,
so
it
makes
sense
it
could
at
the
universal
optimal
solution
in
theory.
A
But
if
you
do
some
compilations,
actually
you
have
to
go.
The
central
oversee
I
have
to
handle
a
very
high
amount
of
QPS.
We
are
not
sure
we
are
able
to
implement
such
overseer.
So
we
give
up
this
approach.
It's
also
not
clear
what
happens
if
you
or
overseer
fail,
but
again
there
are
other
companies
who
did
this.
That
is
turning
falaya
from
Alibaba.
They
have
this
central
like
super
node
that
decide
everything
we
actually
talked
to
them
recently
they
are
there.
Our
company
is
indeed
the
kewpie
is
on
the
super
node.
A
So
after
some
exploration,
we
found
a
ideal
model
that
is
a
random
graph,
so
so
in
this,
in
this
model
you
still
need
a
central
component,
but
the
central
component
responsibility
is
not
much.
It
just
make
a
connection
decision
decide
who
connected
boom.
The
actual
data
transfer
case
is
still
made
by
individual
node
in
the
video
notes,
so
such
graph
is
called
so
if
you
also
enforce,
if
you
know
that
you
have
a
fixed
number
of
IPs,
then
you
get
a
random
regular
graph.
A
So
subsequent
factories
have
very
very
nice
properties,
so
it
has
very
good
connectivity.
There
are
like
metal
pass
between
I
need
to
are
between
those
and
the
distance
between
any
any
nodes
actually
fairly
small.
This
is
now
acting
a
new
idea
which
isn't
they
found.
There
is
a
paper
form
to
some
stuff
called
jellyfish,
describing
such
a
network
topology,
it's
very
very
great.
So
we
did
some
simulation.
A
So
if
you
drop
one
single
cedar,
a
single
upload
or
in
this
network
instantly,
every
other
node
is
able
to
download
the
data
at
80
percent
of
max
data
speed.
So
also
our
simulations
actually
hoped
for.
We
assume
the
relative
latency,
if
it
did,
a
proper
simulation
number
should
be
approaching
90%.
So,
as
you
can
easily
tell,
such
a
model
is
also
very
resilient
to
failures,
it's
very
hard
to
create
a
natural
partition
in
sexual
connection
growth.
So
this
this
is
what
we
went.
A
B
Can
you
hear
me
it's
this
cool?
Yes,
I'm
Cody
I'm,
one
of
the
developers
on
Kraken,
so
I'm
gonna
talk
a
bit
about
how
we
used
cracking
to
solve
these
peer-to-peer
problems.
So
first
some
vocabulary
so
we're
on
the
same
page.
A
torrent
is
what
we
call
a
file
that
has
been
broken
into
multiple
pieces.
Typically,
four
megabytes
each
pieces
are
transferred
independently.
A
peer
is
a
participant
in
the
torrent
network.
B
Connected
peers
transfer
pieces
between
each
other
and
appear
with
a
hundred
percent
of
pieces
is
called
a
seeder
I'll
also
be
throwing
around
the
word
blob
around
a
blob
is
just
a
file.
So
so
this
is
the
Kraken
architecture.
We
have
three
components,
an
agent
that
runs
on
every
host.
This
is
the
peer.
The
agent
implements
a
docker
registry
interface,
so
you
execute
dock
or
poles
against
your
local
agent
and
that
pulls
the
image.
Through
the
peer-to-peer
network.
We
have
an
origin.
The
origin
is
a
dedicated
seater.
B
They
see
data
from
some
pluggable
storage,
back-end
such
as
s3
HDFS,
even
another
darker
registry.
The
origins
form
a
self-healing
hash
string,
so
you
can
add
more
origins
to
distribute
load.
If
an
origin
dies,
no
big
deal,
they
seal
the
gap.
This
is
in
line
with,
like
our
low
maintenance
philosophy.
If
an
Origin
dies,
we
don't
want
to
get
woken
up
to
fix
it.
B
You
know
we
can
look
at
it
a
week
later,
even
and
then
tracker
tracker
maintains
a
global
view
of
who
the
peers
are,
who
the
cedars
are
who's,
trying
to
download
what
data
and
it
does
all
of
this
in
memory
again.
Trackers
also
form
a
self-healing
hash
strings,
so
you
can
add
more
trackers
to
distribute
load
if
a
tracker
dies
no
big
deal,
they
fill
the
gap.
B
So
some
key
features
of
this
architecture
cracking
is
just
like
cashing
and
distribution
layer.
We
don't
want
to
be
in
the
business
of
owning
data,
of
doing
backups,
of
maintaining
a
database
again,
there's
just
three
of
us.
We
don't
have
time
or
resources
to
do
that
so
because
our
origin
serves
data
from
another
source
like
s3,
we
can.
We
can
suffer
from
total
data
loss
and
recover
automatically,
so
you
could
wipe
the
disk
of
every
single
cracking
component,
restart
everything
and
no
one
would
even
notice
you
just
start
reseeding.
B
The
data
back
into
the
network
we
take
a
minimal
dependency
set.
Your
darker
registry
is
usually
the
at
the
base
level
base
layer
of
your
your
infrastructure.
It
is
essentially
what
deploys
all
other
services
so
we're.
The
only
dependency
we
can
take
is
DMS,
which
is
actually
optional
and
then
finally,
constant
adjustable
blobs,
so
old,
blobs
and
cracking
are
constant
ingestible.
What
this
means
is
the
blob
identifier
is
a
hash
of
the
blog
content.
By
definition,
this
means
that
blobs
are
immutable.
B
B
So
how
does
peer
discovery
work
if
I'm,
an
agent
and
I
want
to
download
some
torrent,
I
contact,
my
tracker
and
I
say:
hey
I'm.
This
peer
I
want
to
download
this
torrent,
and
the
tracker
says:
ok
here
are
50
random
peers,
sorted
by
preference
to
go,
try
and
connect
to
so
the
most
preferred
are
other
completed
agents
seeding.
The
torrent
next
are
the
origins.
B
Ok,
the
designated
seeders
and
then
finally,
the
lowest
priority
are
other
importers
agents,
since
they
might
not
have
all
the
data,
if
any
so
once
I
have
this
list
of
50
I,
just
iterate
down
the
list
until
I
have
10
connections
and
remember
a
random
graph.
You
have
a
connection
limit
peers,
reject
connections
once
they
are
at
that
limit,
so
I
might
have
all
down
this
list
until
I
get
10
peers
that
want
to
have
open
connection
slots.
B
B
B
So
this
actually
isn't
the
whole
picture
to
make
darker
poles
work.
You
actually
need
to
do
one
step
initially
to
resolve
the
darker
tag.
You
know
Alpine
3.6,
into
the
image
Shaw,
so
old
data
interact
in
this
content-addressable,
which
means
that
we
have
to
map
this
human
readable
tag
into
a
sha-256
digest
so
build
index
is
a
super
rudimentary
piece
of
technology.
B
It
has
a
pluggable
storage
back-end
like
our
origin.
It
doesn't
own
any
data.
We
just
put
tags
somewhere
else,
has
no
consistency.
Guarantees
turns
out
having
a
consistent
key
value
stores.
You
know
kind
of
a
hard
problem
to
solve,
but
it
also
has
one
more
neat
feature
for
powering
image:
replication
between
clusters.
So
uber
we
have
you
know
several
on-prem
data,
centers
cloud
zones
and
cracking
clusters
in
each
zone.
B
B
Security
is
important
for
us.
We
support
mutual
TLS
between
all
our
central
components.
The
peer-to-peer
traffic
doesn't
go
through
TLS,
but
we're
not
too
worried
about
data
integrity
because
all
the
blobs
are
content.
Addressable
they're
immutable
if
I'm
a
peer
and
I
download
some
blob,
I
recalculate
the
hash
of
the
blob
and
make
sure
it's
what
I
expected.
B
So,
let's
look
at
some
results,
so
we
ran
this
pretty
simple
test.
We
took
a
three
gigabyte
image
and
we
concurrently
downloaded
it
onto
2600
hosts.
This
is
what
a
typical
batch
processing
job
looks
like.
It
looks
like
a
d'hubert
and
on
each
agent
we
enforced
a
0.3
gigabyte
per
second
speed
limit,
so
this
would
make
the
theoretical
max
download
speed
10
seconds.
The
results
were
that
the
p50
was
10
seconds,
p99
18
seconds
and
the
max
was
32
seconds.
B
So
essentially,
if
you
had
a
batch
processing
job
on
2,600
hosts
with
the
3
gigabyte
image,
you
could
launch
that
job
within
30
seconds,
which
definitely
met
our
requirements
when
we
were
trying
to
solve
this
scaling
issue.
So
what
does
this
actually
look
like
in
production
in
our
busiest
zone?
Cracking
distributes
close
to
2
million
blobs
a
day.
Most
of
these
are
medium-sized.
Around
a
gigabyte
and
the
daily
peak
is
usually
around
20,000
1
gig
blobs,
distributed
within
30
seconds.
So
with
our
old
registry
and
Gen
X
setup.
This
would
have
instantly
cause
an
outage.
B
Basically,
the
batch
jobs
would
start.
Oh
geez,
that
should
all
just
start,
timing
out
and
deployments
would
start
failing
and
we
would
start
getting.
You
know,
complaints
from
users
now
this
you
know
these
Peaks
happen
every
day
and
nobody
even
notices,
so
some
optimizations
we
made
along
the
way
so
in
the
random
regular
graph.
Like
you
have
a
connection
limit,
we
found
that
keeping
that
connection
limit
pretty
low
is
actually
pretty
useful.
It
lowers
the
overhead
of
the
entire
network,
I
think
in
production
we
use
10
as
a
connection
limit.
B
Another
optimization
is
to
aggressively
disconnect
unneeded
connections.
So
when
you
have
a
connection
limit
and
you're
going
to
reject
connections,
if
your
your
connection
limit
is
full
I
want
to
do,
I
want
to
disconnect
connections.
I,
don't
need
to
make
room
for
other
peers
that
want
to
download
data
from
me.
So
idle
connections.
Connections
between
cedars
are
all
aggressively
disconnected,
and
this
is
the
effect
of
rebalancing.
The
network
and
always
making
data
available
to
new
newcomers
pipelining
is
a
technique
where
you
maintain
a
request:
queue
of
size
n
for
each
connection.
B
Basically,
instead
of
sending
you
know,
I
request
a
piece:
I
wait
for
a
response:
I
request,
a
piece:
I
wait
for
a
response:
I'll
just
do
request,
request,
request
and
then,
as
the
responses
come
in,
I,
send
more
requests.
This
results
in
less
idle
time
between
peers.
You
know
I
always
have
something
to
chew
on
if
I'm,
if
I'm
getting
lots
of
requests
and
then
finally
in
game
mode,
this
is
something
we
borrowed
from
BitTorrent,
basically
for
the
last
few
pieces
of
a
download
just
to
request
them
from
all
your
connected
neighbors.
B
Initially
we
had
this
idea
that
we
could
prefer
peers
on
the
same
rack,
to
reduce
network
overhead
in
the
rack
switches.
The
problem
here
is
that,
once
you
start
making
more
intelligent
connection
decisions,
you
end
up
forming
disjoint
graphs,
so
you
we
would
see
an
entire
rack
of
peers
format,
interconnected
web
between
each
other
and
basically
star
for
data,
and
it
turns
out
you
don't
even
need
to
do
this.
B
If
your
network
is
over
provisioned,
which
ours
are
so
we
threw
this
out
pretty
quickly
and
then
this
next
one
reject
incoming
connections
based
on
number
of
mutual
connections.
So,
if
I'm,
a
peer
and
I'm
connected
to
another
peer
and
a
third
peer
comes
in
and
connects
to
both
of
us,
that
would
make
a
mutual
connection
right,
and
so
one
of
those
connections
actually
isn't
needed.
We
could
transitively
send
the
data
between
the
the
middle
peer,
so
we
were
a
little
worried
about
getting
tight
clumps
of
interconnected
peers.
B
We
call
this
like
graph
density
problems,
but
in
reality
we
haven't
seen
any
issues
with
this.
We
just
use
strict
randomization
and
things
work
out
pretty
well,
so
some
takeaways
this
initially
we
set
out
to
solve
this
darker
registry
scaling
problem,
but
the
solution
we
ended
up
designing
was
agnostic
to
two
darker
images.
It
just
works
on
content-addressable
blobs,
which
means
you
can
integrate
with
any
storage
system.
So
we
started
with
darker
images,
but
now
we
use
Kraken
an
uber
to
power,
config
distribution,
machine
learning,
data,
large
s3
files.
B
Basically
we
have
this
peer-to-peer
network
deployed
in
our
data
center
and
we
can
use
it
any
way
we
want,
and
the
pluggable
storage
backends
really
makes
this
possible.
So
next
PDP
solutions
work
with
data
centers.
There
was
some
concern
that
deploying
a
PDP
solution
would
overload
the
network
slow
things
down
for
production
services,
but
we
found
this
to
not
be
the
case
and
it
works
pretty
well.
B
Randomization
works.
We
started
with
random
policies
because
they're
easy
to
implement,
but
it
turns
out
they're
pretty
well.
They
work
pretty
well
and
they're
easy
to
reason
about
trying
to
be
too
intelligent
about
how
you
make
connections
what
pieces.
Your
requests
can
sort
of
make
things
more
difficult
for
you
and
then.
B
Finally,
this
goes
without
saying:
get
something
working
before
Optimas
optimization,
it's
very
difficult
to
predict
how
PDP
works
without
experimenting
yeah,
so
future
plan
right
now
cracking
mostly
addresses
medium
sized
blob
around
a
gigabyte,
but
we
have
some
use
cases
for
a
massive
number
of
tiny
files
that
we
want
to
distribute.
We
want
to
improve
observability,
so
you
saw
that
simulation
that
we
have.
B
We
want
tighter
integration
with
other
registry
registry
implementations,
so
this
means
adding
more
docker
registry
backends,
such
as
GCR
or
artifactory,
and
then,
of
course,
tighter
integration
with
kubernetes
modeling
images
as
resources
to
speed
up
deployment
times
further.
So
this
is
our
rolling
github
everything's
open
sourced
we're
looking
for
more
contributors.
If
you
have
a
storage
back-end
that
you
want
to
add
we're
happy
to
take
a
looking
at
it,
yeah
I
think
that's
it.
Okay,.
C
That's
good
cuz,
I
kind
of
blew
up
my
voice
yesterday,
so
you
mentioned
beginning
the
talk
that
one
of
the
things
you
wanted
to
get
out
of.
This
was
not
just
multiple
parallel
poles
of
a
single
image,
but
also
the
ability
to
deal
better
with
many
poles
of
unique
images.
Did
you
see
sort
of
similar
gains
on
that
metric
to
what
you
saw
and
you
put
in
your
presentation
about
parallel-
pulls
against
the
same
image.
A
D
A
Because
if
you
check
out
our
github,
we
have
a
how
much
heart
so
I
can't
deploy
this
with
one
command,
but
normally
you
probably
want
to
configure
the
config
file
little
bit,
it's
just
so
it
can
connect
to
your
hostel
registry.
Normally
what
happens
with
doctorates
reviews
you.
You
have
some
custom
configurations.
A
E
Okay
and
I
was
just
wondering
if
you
looked
at
I'm,
actually
pushing
blobs
on
two
nodes
before
30
years.
So
as
far
as
I
can
tell
at
the
minute,
you
do
like
a
docker
pull
and
that
wall
starts
pulling
the
the
blobs
from
the
network
onto
the
note.
But
if
you
actually
looked
at
pushing
them
before
that
happens,.
A
A
Story
here
so
Cooper
started
doing
using
docker
in
2015,
so
at
that
time,
uber
to
basically
develop
today's
own
operation
framework.
That's
actually
similar
to
cronikeys,
except
the
upgrade
on
that
framework
is
much
faster,
Lancome
radius.
We
upgraded
like,
for
example,
5
500
container.
That
would
take
5
minutes.
A
But
if
you,
if
you
do
a
back
up
word
on
communities
that
could
take
half
an
hour
to
an
hour
because
on
communities
when
you
do
a
rolling
upgrade,
every
batch
have
to
you
know,
download
the
imagery
again
and
if
not,
each
batch
take
a
few
minutes
at
least
so
end
up
with
very
long
time.
So
our
plan
for
this
is
we.
We
are
wrecking
our
own
communities
scheduler.
A
Their
support
in
place
upgrade
so
once
you
have
once
you
have
such
a
scheduler,
then
you
know
which
machine
will
be
hosting
the
next
version
of
the
images.
So
we
can
preheat
all
those
Tocker
demon-like,
all
those
machines.
So,
ideally
you
can.
You
can
do
a
very
fast
running
up
great
because
you
preheat
all
the
machines
with
images
and
all
you
do
when
you
know
going
out
with,
is
just
a
start.
A
container
and
to
how
spec
so
yeah,
but
that
takes
quite
a
lot
of
work
for
every
plan
to
get
that
down.
F
Hey
two
quick
questions:
the
first
one:
how
do
you
think
about
retention
and
deletion
of
images?
Under
what
circumstances
do
you
delete?
Have
you
run
experiments
that
tell
you
which
strategies
work
better?
Second
question:
your
network
presumably
also
does
real
work
and
not
just
you
know,
distribution
of
images.
How
do
you
balance
your
network
band
with
between
the
tasks
of
actually
seeding
and
loading
and
Ozma,
and
distributing
these
images
with
the
real
work
that
your
clusters
are
supposed
to
do
so.
A
The
first
question
is
actually
a
very
tricky,
so
there
are
two
kinds
of
image
talking
talking
about
years,
although
the
first
line
is
about
deleting
from
dr.
demon,
a
sacraments
body
beating
from
Kraken,
so
photogra
demon
at
our
policy
is
if
the
image
is
not
used
for
one
hour,
it
will
get
eveything
for
crack
and
we
just
use
you
know
everywhere-
wheels
about
I
think
our
default
tdrs
six
hour.
So
if
a
layer
is
not
seeded,
this
not
is
not
not
used
anywhere
than
this
DVD
in
six
hours,
and
that
worked
out
pretty
pretty
well.
A
Repeat,
second
also,
so
in
Kraken
we
implement
our
own
battling
logic
and
that
worked
pretty
well.
We
try
to
make
work
and
consume
less
than
30
percent
of
our
planned
waste,
and
that
has
been
has
been
doing
pretty
well,
because
all
the
rest
of
network
command
are
not
consuming
as
much
bandwidth
as
cracking
itself.
So
together
we
add
up
less
than
hundred
percent.
G
A
Better,
so
is
our
simulation.
There
was
a
magic
number,
that's
between
5
to
10.
So
if
you
random,
regular
graph,
if
the
degree
number
by
design
Evernote
is
between
5
to
10,
you
actually
get
the
best
performance.
That's
that's
why
we
picked
50,
actually,
I,
don't
think
there
is
work
with
the
reason
just
not
work.
That's
production!
A
A
Today's
actually
works
pretty
well
in
production,
actually
I
think
our
worst
case
scenario
can
still
download
at
about
60%
like
Mike.
Whatever
about
her
speed
limit,
you
specify
that
node
will
be
able
to
tunnel
at
6%
of
the
60%
of
the
speed
limit.
That's
that's
okay,
but
want
to
push
to
put
that
to
90%
I.
Think
currently
is
our
implementation
is
not
ideal.
So
there
are
some
small
small
tweaks.
We
can
make
to
push
that
enough.
G
B
So
I
think
it
can
handle
way
more
than
we
then
we
use
our
clusters
are
around
8,000
notes,
but
when
we
just
like
hammer
it
for
benchmarking,
theoretically,
you
get
a
handle.
You
know
tens
of
thousands
more
than
that
so
everything's
in
memory
and
you're
not
storing
very
much
data
about
each
appear,
literally
just
IP
port
and
a
single
bit
as
to
whether
the
peer
has
completed
the
torrent
or
not,
and
so
yeah
QPS
isn't
really
a
bottleneck
for
us.
Okay,.
A
A
Project
uber
has
a
lot
emotion
about
when
Gbps
network
card,
and
so
so
it
worked.
Also,
if
you
look
at
this
graph,
so
we
played
treat
each
zone
as
a
different
cluster.
So,
within
those
eyes
the
network,
some
more
reliable,
then
there's
no
problem
between
yeah.
If
there
are
problem,
maybe
replication
take
a
little.
That's
that's.
Okay,.
I
Hello,
you
mentioned
the
generic
nature
of
Kraken.
Do
you
see
this
applying
in
some
sort
of
CDN
capacity,
for
example,
some
edge
deployment
of
kubernetes
nodes?
Perhaps
something
like
a
digital
billboard
deployment
across
the
nation
for
delivering?
Just
you
know,
JPEGs,
to
show
on
the
digital
billboard
is.