►
From YouTube: Dragonfly Intro: Plugin Framework and New Distribution Strategy - Haibing Zhou & Ben, Ye,
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Dragonfly Intro: Plugin Framework and New Distribution Strategy - Haibing Zhou, eBay & Ben, Ye, N/A: Individual
In cloud native world, image distribution is never an easy problem when the number and size of container images scale up. It has to be fast and resource efficient and to be cloud native. This session shows how Dragonfly solves this problem, and how it can increase the speed of image distribution while keeping the operation effort as less as possible. Meanwhile, this session shares the latest update on Dragonfly project where plugin framework is coming into the picture, and how this can benefit dragonfly users, as an example, this sessions shares how this plugin framework gives a chance for new decentralized distribution strategy.
https://sched.co/UajR
A
A
So
today's
agenda
has
four
parts.
The
first
part
is
Jennifer
overview.
I
will
tell
you
about
what
is
dragonfly
and
Jennifer
is
history
in
the
second
part,
I
will
talk
about
Jung
flies
architecture
and
how
to
jam
fire
components.
Work
together
in
the
third
part,
I
will
introduce
our
design
for
the
decentralized
the
mode
of
dragonfly.
In
the
last
part,
I
will
tell
you
something
about
our
future
work
as
a
roadmap.
Okay,
let's
get
into
the
first
part.
A
A
A
So
you
can
see
that
there
is
a
network
bottleneck
in
your
image
registry,
because
the
load
is
very
high
in
that
image
registry,
so
the
whole
download
process
will
be
very
slow.
So
how
can
we
solve
this
problem?
What
about
using
a
p2p
based
distribution
solution?
That
is
what
Jennifer
does.
So,
if
one
pier
already
download
download
the
image
file,
then
any
other
peers
can
just
directly
download
from
that
cure.
A
So
Jenna
fly
is
aimed
to
solve
the
bottleneck
of
file
and
the
image
distribution
in
large-scale
data
centers
to
prove
its
performance.
We
did
a
test
between
two
image
distribution
solutions,
as
first
is
just
using
docker
pool
it's
a
native
way.
The
second
way
is
using
Django
flag.
So,
as
you
can
see
from
this
diagram,
as
the
concurrency
becomes
much
larger,
the
load
on
the
center
image
registry
becomes
much
higher
in
the
native
way,
so
the
average
download
time
increases
rapidly.
A
Okay,
next
lat,
let
me
introduce
about
the
dragonflies
history,
so
Jeong
fly
was
born
in
Alibaba
in
2015.
It
was
initially
a
file
distribution
system
because,
as
a
time
container
technologies
not
widely
adopted
in
China
two
years
later,
John
fly
was
open
source
and
it
became
one
of
the
fundamental
infrastructures
in
Alibaba
and
at
a
time
it
can
distribute
3.4
petabytes
data.
Amongst
one
year
later,
general
I
joined
the
CN
CF
and
it
became
one
of
the
sandbox
project
and
more
than
20
companies
are
using
dragonfly
in
in
their
production
environment.
A
A
A
Now
let
me
talk
about
our
integration
with
other
CN
CF
project.
As
for
container
engine
translight
can
natively
support,
support
container
d.
Its
configuration
is
very
simple
and
I
will
show
you
in
the
next
few
slides.
As
for
harbor.
Harbor
is
a
very
famous
image
registry
and
the
jennifer
supports
preheat
functionalities,
so
the
harbor
users
can
trick
a
can,
send
HTTP
request
to
join
a
fly
and
the
jungle
flight
will
do
some
image
of
Perficient
for
later
image.
Distribution.
Matrix
is
another
very
important
aspect.
A
A
So
this
all
about
the
first
part,
the
next,
let's
get
into
dragonfly
architecture
so
for
file
distribution.
Jeong
fly
mainly
needs
two
components.
The
first
is
John
fly
server
because
super
node
and
another
role
is
dragonfly
client,
which
is,
if
get
so.
How
do
they
work
together
does
see
when
the
client
starts.
It
will
register
itself
as
a
peer
to
super
node
server.
So
when
one
peer
wants
to
download
some
file,
it
will
send
a
create
task
request
to
the
super
node
server.
A
So
what
is
a
task
you
can
consider
it
as
I
want
to
download
this
file.
So
this
is
a
task.
So
after
the
supernal
server
receives
this
request,
it
will
generate
task
ID
and
send
the
ID
back
to
the
client.
So
next
step
give
gap.
Client
will
can
use
this
ID
to
track
the
pieces.
Information
of
that
file
because
super
node
is
the
central
server.
It
maintains
the
whole
status
of
this
cluster,
so
it
such
as
peers
information,
and
it
also
knows
whether
the
request
a
file
exists
in
the
cluster
or
not.
A
So
after
the
scheduling
the
client
can
download
that
file
from
another
peer.
So
after
has
this
process
finalized,
the
client
will
send
an
update
to
the
server.
So
this
step
is
just
say:
I
have
already
download
this
file
so
later,
when
other
peers
want
to
download
the
same
file,
they
can
directly
download
from
me
okay.
A
Additionally,
there
is
kind
of
situations
that
the
requested
file
is
not
present
in
the
current
cluster,
so
the
super
node
will
act
as
a
CDN
nodes
to
download
from
the
sauce
the
end
well
directly
Donald
from
the
supernote
server.
So
that's
basically
how
drunk
I
work
in
the
file
distribution.
Next,
let's
get
much
deeper
in
each
component.
A
A
A
A
A
Actually,
our
solution
is
quite
simple.
We
just
start
a
proxy
in
each
host
and
we
configure
the
registry
mirrors
to
that
proxy
so
that
when
your
container
engine
won't
receive
these
download
requests,
it
will
send
the
tunnel
image
requests
to
your
configure.
The
registry
mirror
so
that
drug
fly
will
know
what
image
you
want
to
download.
So
these
are
the
two
examples
of
configuring,
docker
and
continuity.
A
A
So
the
whole
step
is
that
user
use
a
talker
command
line
to
pull
some
kind
of
images
and
the
finalized
image
request
will
reach
to
the
DfE
mo
then
diff
da
mo
will
folk
a
Gift
Guide
process
and
then
Gift
Guide
process
will
try
to
connect
to
the
center
server,
which
is
super
node.
It
will
register
itself
and
download
from
this
p2p
network
for
the
next
few
steps.
This
is
quite
similar
to
file
distribution,
so
I
will
skip
this
part.
A
Next,
let's
talk
about
our
today's
main
topic:
the
decentralize,
the
mode
of
John's
life.
So
someone
might
ask
so
what
does
it
mean?
So
it's
John,
four
I
already
decentralized,
but
actually
it
is
not
fully
decentralized,
because
the
give
get
clients
part
is
decentralized
decentralized,
but
we
still
need
to
maintain
the
super
node
server.
A
So
some
of
our
users
give
give
give
us
some
feedback
that
it
is
not
easy
to
maintain
the
super
node
server.
It
is
quite
heavy
and
the
currently
it's
cons
go
easily
because
we
don't
have
a
perfect,
perfect
edge
resolution
for
super
nodes
and
also,
as
I
have
mentioned
earlier,
super
node
will
sometimes
work
as
a
CDN
node
to
fetch
the
files
from
source.
So
it
has
a
high,
relatively
high
demand
it's
hardware,
such
as
disk,
I/o
and
network
bandwidth.
A
Can
we
just
find
a
way
to
run
soup
dragonfly
without
super
node?
Just
like
this
diagram,
can
we
just
use
this
gift
demo
running
in
each
node?
So
how
do
we
manage
to
do
this?
Actually,
in
order
to
solve
this
problem,
we
should
first
know
what
super
node
to
us.
Actually,
it's
just
a
server
which
keeps
cluster
information
in
its
memory,
such
as
peers,
information
file,
pieces
information,
tasks,
information
and
then
it
will
do
some
scheduling
based
on
these
information.
A
A
Actually,
we
found
a
great
tool
called
surf.
Surf,
was
open
source
by
hash
cop
and
is
it
is
what
I
use
in
decentralized
applications.
It
uses
gossip
protocol
to
communicate
with
each
peers,
and
it
also
has
many
great
functionalities,
such
as
membership
management,
failure,
detection
and
recovery.
So
when
one
peer
fails
and
then
other
peers
can
detect
the
failure
quickly
and
try
to
recover
that
failed
peer
as
for
custom
event,
propagation,
that's
what
we
need,
because
we
want
to
integrate
with
Jennifer
Iwase
surf.
A
A
So
what
other
things
should
we
do?
The
first
thing
is
defining
a
tracker
interface,
because
in
the
traditional
architecture
we
only
have
two
main
rules.
The
first
is
super
node
server.
Another
is
thief,
gate
client,
so
these
two
components
talk
only
we
as
a
super
node
API.
So
in
the
decentralized
mode
we
don't
have
super
node
anymore
so,
but
we
still
want
to
keep
the
compatibility
how
to
achieve
this.
A
Pure
discovery
is
another
very
important
thing
in
in
the
p2p
network,
because
one
peer
wants
to
join
the
concert.
It
should
first
know
the
address
of
it
of
the
other
peers
in
the
cluster.
So
how
do
we
want
to
implement
this
feature
because
we
want
dragonfly
to
adapt
to
many
different
scenarios?
So
we
want
this
feature
to
be
pluggable.
A
So
here
we
list
the
three
three
kind
of
ways
to
do
peer
discovery.
The
first
is
using
criminales
api
kubernetes
has
many
resources
and
the
api.
So
when
the
peers
starts,
it
can
fetch
this
community's
information
right,
API
such
as
notes
or
endpoints,
so
through
this
API
they
can
connect
to
the
peers
in
the
after.
A
A
The
next
way
is
using
surf
agent
serve,
is
not
only
library
for
doing
gossip,
but
it
is
also
a
certain
area
server,
so
you
can
run
it.
So
in
this
situation,
we
can
initially
start
surf
agent
in
our
cluster
to
work
as
a
dummy
node.
So
when
you
want
to
start
other
peers
in
your
cluster,
we
can
just
directly
connect
to
that
dummy
node
and
join
these
p2p
cluster.
A
So
the
next
thing
is
about
our
custom
events
for
Django
fight.
So
what
type
of
events
should
we
define
here?
We
define
three
type
of
events.
The
first
is
about
five
events,
because
trying
to
fly
itself
is
a
file
and
image
distribution
system,
so
we
should
propagate
the
file
pieces.
Information,
among
other
appears,
so
here
we
define
three
type
of
our
events.
First
is
download
initialization
event.
This
is
used
to
notify
other
peers
that
I
start
downloading
some
file
and
then
later
mix
they
can
download
from
this
pier.
A
The
second
type
of
our
event
is
download
progress
event
because
in
Java
we
split
the
file
downloading
process
into
several
pieces.
So
this
is
used
to
that.
Other
peers
know
that
how
many
pieces
have
you
already
downloaded
and
whether
the
whole
download
process
is
is
finished
or
not.
So
when
the
process
is
finished,
they
can't
download
from
up
here.
The
last
event
is
file
remove
event
so
because
the
disk
has
maximum
capacity,
so
sometimes
you
need
to
do
like
garbage
collection
or
clean
up
your
disk
space.
A
So
when
you
delete
these
files,
you
need
to
notify
other
peers
as
well,
so
that
they
can't
download
this
file
from
up
here
and
another
two
type
of
event.
The
first
is
lens
event,
because
when
you
download
a
file,
the
peer
needs
to
calculate
the
file
ends.
So
when
file
lens
is
calculated
by
one
peer,
it
can
propagate
this
event
to
other
peers,
so
that
other
peers
don't
need
to
calculate
these
file
ends
again.
A
Another
type
of
event
is
recovery
event,
because
in
a
distributed
system,
peers
may
have
a
possibility
to
be
a
crashed
so
when
it's
crushed
and
recovered
and
needed
to
synchronize
with
other
peers
to
gather
latest
caster
information,
so
we
enter
restarts.
It
can
send
this
recover
event
to
other
peers,
then
other
peers
can
send
back
the
recover.
The
latest
cost
her
information
to
that
pure.
A
So
this
is
the
final
design
for
our
decentralized
architecture.
Actually,
this
is
quite
similar
to
the
previous
diagram,
but
in
this
architecture
we
don't
have
super
node
anymore.
We
just
only
have
each
host
running
DFP
MO.
So
when
the
FDA
Minh
starts,
it
will
try
to
connect
to
the
in
each
node
or
some
other
peer
discovery
mechanism.
It
will
send
a
registered
request
and
then
you
need
no
will
send
back
the
peers
already
in
that
cluster.
A
So
after
the
first
two
steps,
the
peer
will
join
into
this
gossip
cluster.
Then
they
can.
It
can
synchronize
with
other
peers,
such
as
fire
information
and
the
peers
information.
So
in
the
first
step
where
it
wants
to
download
some
kind
of
files,
it
can
do
the
scheduling
itself
because
it
has
enough
information.
So
it
can
know
whether
the
requested
file
is
present
in
other
peers
or
not,
if
not,
it
can
directly
download
from
the
source
or
the
registry
or
the
registry.
So
if
the
files
is
present,
it
can
just
directly
download
from
other
peers.
A
A
Besides,
we
also
want
to
support
more
computing
architectures
like
arm
so
as
for
ecosystem,
we
want
to
integrate
with
open
tracing
and
it's
better
to
make
dragonfly
more
communities
up
kubernetes
native,
so
we
want
to
support
operator
as
well.
So
if
you
are
interested
in
our
future
work,
you
can
check
out
about
the
roadmap
in
our
github.
A
B
A
D
You
guys
approached
any
of
the
major
cloud
providers.
Google
AWS
see
if
they
can
support
it
on
GAE
was
like
I,
don't
think
I
get
the
chance
to.
You
know,
set
up
the
registry
mirror
on
the
nodes
right
like
this
is
like
pretty
awesome,
but
I
don't
think
I
can
use
it
in
an
environment
where,
where
I
don't
get
to
manage
all
the
config.
E
A
Yeah,
because
in
this
slide,
I
only
shows
a
way
to
configure
the
register
mirrors,
but
in
Jung
fly
we
also
supports
the
proxy
process
by
using
the
environment
input
variable,
you
can
check
out
our
document.
We
can
configure
the
HTTP
proxy
or
HTTP
proxy
in
the
darker
in
docker
or
container
disk
configuration
file
yeah,
it
works
yeah.
D
B
Probably
related
to
the
first
questions:
how
about
authorization
so
normally,
however,
hosts
a
bunch
of
images
right
and
user
a
can
pull
image
a
but
user.
B
cannot
pull
image,
be
right
so
and
in
kubernetes,
plain
user
normally
set
da
couple
secrets.
So
does
the
secret
information
can
be
handled
dragonfly
properly.
A
Actually,
currently,
trans
I
don't
have
the
support
for
community
resources
secrets,
as
for
integrating
with
harbor.
Actually
Java
already
supports
these
preheat
functionality,
but
in
harbor
side
they
are
trying
to
provide
an
interface
for
other
different
kind
of
PGP
solutions
such
as
kraken,
and
maybe
you
have
already
heard
about
it
so
for
this
integration
in
harbor
UI,
there
will
be
a
button.
A
E
But
actually
I
don't
think
dragonfly
needs
to
know
those
secrets.
So
when
you
put
image
it's
actually
from
the
docker
container
deposit,
just
the
HTTP
header,
when
you
pass
it
was
proxy
proxy,
just
translated
to
send
help
to
the
British
by
itself,
just
no
interception
or
I
guess
the
probe
just
just
works.
E
E
E
As
as
ben
jonson
is
during
our
design
phase,
it's
not
into
year
I
at
least
know
in
today's
version.
It's
during
our
design
phase,
but
the
most
collaboration
work
is
to
you
know:
I
proposed
this
topic,
I'm
proposed
at
the
center
as
the
design
doc
and
that
accurate
as
the
experiment
in
urban
environments
we
are
using
surf.
We
are
using
the
distribution
without
a
center
as
a
node,
so
we
are
walking
together
the
more
moving
or
internal
implementation
to
public,
so
hopefully
I
guess
should
be
so
stay
tuned.
D
E
The
question
is
how
about
the
performance
of
surf?
We
didn't
do
the
test
MP
at
surf
level,
but
we
do
the
MP
at
a
higher
level.
We
really
test
the
performance
when
you
like
I'll
test
case
shows
when
you
have
2000
notes.
All
of
them
want
to
put
the
1
point:
5
gigabytes
image,
which
normally
some
Ubuntu
image,
1.5
I,
think
the
result
is
good.
Almost
5
minutes,
90
percentile
can
download
the
image
successfully,
but
the
surf
performance
appear
from
the
official
document.
It
says.
So.
E
G
A
So
so
for
this
question,
actually
we
have
a
integrated
check,
because
so
I
will
talk
about
this
integration
track
because
we
do
the
integration
based
on
the
hash
of
the
of
the
file
of
the
file
URL
and
some
other
things,
because
we
can
configure
the
so
md5
during
the
during
the
start
of
each
client
so
in
this
p2p
network.
So
this
is
a
file
from
one
URL.
A
E
D
F
E
So
I
think
the
first
question
is:
what
does
really
protecting
scale?
Was
the
numbers
right
so,
first
of
all,
the
decentralized
mode
is
not
in
public.
So
what
I'm
going
to
share
is
a
specific
numbers,
so
we
are
running
like
two
thousand
notes,
with
five
or
ten
proxy
nodes.
That's
that's
a
little
cheap
proxy,
but
the
most
use
case
is
for
big
images
like
because
we
are
running
a
simulation
assume.
It
means
you
get
you're,
not
running
the
simplest.
You
may
get
like
one
point:
five
gigabytes
or
2.5
gigabytes
or
some
maybe.