►
From YouTube: Intro: Dragonfly - Jin Zhang & Zuozheng Hu, Alibaba
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Intro: Dragonfly - Jin Zhang & Zuozheng Hu, Alibaba
As container technology becomes more and more widespread in the industry, how to distribute images efficiently and safely is a new challenge for engineers. Project Dragonfly is an open source intelligent P2P based image and file distribution system. Its goal is to tackle all distribution problems in cloud native scenarios. Currently Dragonfly focuses on being: • Simple: well-defined user-facing API (HTTP), non-invasive to all container engines • Efficient: CDN support, P2P based file distribution to save enterprise bandwidth • Intelligent: host level speed limit, intelligent flow control due to host dection • Secure: block transmission encrytion, HTTPS connection support In this presentation, we will focus on the distribution of container images through Dragonfly. We will review and provide solutions to challenges faced by organizations, including mass distribution, secure transmission, bandwidth cost. Real world use cases will be discussed in the session.
https://sched.co/Nrqe
B
B
A
B
A
B
A
B
A
A
Therefore,
the
software
package
to
be
distributed
has
increased
and
due
to
the
bottlenecks
in
Puli
images,
our
distribution
efficiency
is
very
low.
So,
in
order
to
address
this
bottleneck,
we've
used
a
p2p
to
increase
the
distribution,
efficiency
of
images
and
software
package.
With
more
and
more
scenarios,
we
have
been
proving
and
adding
new
features,
for
example,
dynamic,
compression,
etc,
and
we
are
trying
to
build
a
dragonfly
into
a
comprehensive,
a
file
and
image
distribution
system
for
all
scenarios.
This
is
important
milestones
of
dragonfly.
A
Was
born
in
June
2015,
so
the
first
version
was
released
within
the
group
in
June
2015,
mainly
supporting
p2p
file,
distribution
and
CDN
in
September
2015.
That
is
the
three
months
later
we
were
able
to
support
image
of
distribution.
We
developed
a
local
proxy,
so
we've
downloaded
images
and
put
on
local
client
in
November.
2016
dragonfly
cover
the
whole
Alibaba.
B
A
With
1.5
billion
downloads
every
month
we
finally
became
part
of
the
infrastructure
of
Alibaba
in
November
2017.
That
is
one
year
later
in
order
to
address
more
problems
and
further
improve
distribution,
efficiency
and
lower
bandwidth,
cost
we've
developed
automatic
free
city
memory
file
systems
and
through
dynamic
compression,
we
have
lowered
the
bandwidth
cost,
especially
cross
region
and
cross
country.
Bandwidth
cost
and
we've
also
became
open
source
in
November
2017
and
in
November
2017.
We
were
the
first
to
support
cinco's
day
sales
campaign
of
the
Alibaba.
A
B
A
A
A
It
can
address
various
kinds
of
file
and
image
distribution
problems
under
all
scenarios
and
in
November
2018
dragonfly
joined
CN
CF
sandbox
project.
At
the
same
time,
we
integrated
with
a
hopper
I,
don't
know
whether
you
know
about
hubber.
It
provides
image
registration
services.
We
are
now
integrated
with
a
harbor.
B
A
B
B
A
B
A
Green
one
is
native
and
orange
is
dragonfly.
With
the
increase
in
downloads,
you
can
see
that
the
average
cost
of
native
has
grown
exponentially,
whereas
if
we
use
a
dragonfly,
the
average
cost
remains
the
flat,
in
spite
of
the
increase
in
number
of
downloads
and
with
the
expansion
of
a
scale,
dragonflies
advantages
became
more
and
more
obvious
when
reaching
certain
scale
using
native
will
probably
lead
to
explosion
of
registry.
That
means
the
image
danos
would
have
fail,
whereas
dragonfly
could
address
all
the
problems
faced
by
native.
B
B
B
A
B
A
B
A
B
A
A
B
A
A
A
B
A
A
A
B
B
For
example,
there
are
a
lot,
but
of
course,
every
time
the
upper
block
is
limited.
For
example,
we
have
two
blocks
and
now
the
client
can
only
download
one
block.
So
do
we
choose
one
from
the?
Actually,
we
use
deep
back
to
pack
back
to
back.
What
do
you
mean?
So
that
means
that
when
the
client
downloads
the
different
blocks
now
we
know
it's
block,
8
and
9
and
the
client
can
only
just
one.
Then
we
would
choose
9,
because
9
is
the
closest
to
10.
B
B
B
Then,
among
the
target
clients,
we
need
to
do
another
screening,
and
now
we
can
use
affinity.
We
can
use
affinity
to
define
which
clients
is
has
a
better
affinity
to
my
claim,
and
also
we
have
the
work
limiter
to
choose
the
lowest
load
client.
For
example.
Now
we
have
to
target
client
a
and
B
for
a
a
is
providing
three
blocks
out
and
two
is
providing
in
the
two.
So
then
would
just
be
because
that
the
load
there
is
the
lowest
and
then
the
speed
to
respond
to
the
requesting
client
is
faster.
B
B
We
have
the
multi
thread
continuing
so
that
we
can
synchronize
the
image
we
have
the
breakpoint
in
resume,
so
we
can
generate
the
seed
and
then
for
catcher,
that
is
the
city
in
cash.
So
it
includes
the
the
cash
testing
and
also
the
cash
dynamic,
GC,
and
then
compressor.
That
is
the
dynamic
compressor
and
we
will
dynamically
decide
whether
this
vote
or
this
file
should
be
compressed
and
also
the
compression
strategy
is
block-level.
B
That
means
for
the
whole
file.
We
want
to
compress
it,
but
we
would
choose
the
most
worthy
part
of
the
file
to
compress
rather
than
the
whole
file.
So,
with
this
kind
of
method,
the
compression
benefit
is
the
largest
and
also
after
the
compression
it
will
be
cached.
So
we
just
need
to
compress
once
and
use
multiple
times
and,
of
course,
according
to
the
super
node
load,
we
would
decide
whether
to
compress
or
not.
Then
we
can
avoid
the
impact
on
the
super
node,
normal
scheduling
and
the
compressing
is
also
multi
thread
and
then
transmission.
B
That
includes
the
super
node
synchronization
image
and
also
the
client
and
it's
downloading
speed
for
blocks.
Then
the
size
limiter
that
is
the
size
of
the
block
for
different
files.
The
size
would
defer
and
then
data
security.
It
means
for
different
block.
We
would
use
the
dragonfly
designed
format
to
pack
them,
then
we
would
avoid
any
manipulation
during
the
transmission.
The
fourth
module
is
the
preheater
of
automation,.
B
B
We
have
different
kinds
of
data
to
be
stored,
for
example
with
block
data
and
the
metadata
the
metadata
for
different
block
and
also
the
MDU
code,
and
also
the
number
and
the
files
then
for
link
data.
This
is
the
soft
link
or
data
Y
soft,
because
with
soft
linker
we
can
go
cross
file
system,
so
we
can
access
to
different
driver
on
the
super
node
with
the
soft
linker
we
can
access
to
all
of
the
drivers
and
the
last
one
is
statistics
data.
B
It
includes,
for
example,
the
time
of
distribution
and
also
the
success
rate
and
for
each
node,
which
client
are
they
downloaded
from
and
so
on.
Then
it's
more
convenient
for
us
to
analyze
the
distribution,
efficiency
and
also
we
can
provide
some
data
support
for
the
later
improvement
on
the
bottom
layer.
That
is
the
memory
file
system.
We
have
the
memory
file
system
as
well
as
the
disk
file
system.
Why
do
we
have
the
memory
faster?
B
Because
we
want
to
solve
the
problem
of
large
file
distribution,
for
example,
10
G
10
Giga,
so
it
will
occupy
part
of
the
I/o
resource.
Then
it
will
cause
impact
on
the
downloading
of
other
files.
So
we
can
use
the
memory
file
system
to
avoid
this
kind
of
situation
and
we
can
solve
the
larger
file
resource
issue.
B
B
B
So
let's
take
a
look,
how
they
used
dragonfly,
so
it
deploys
the
super
node
config
API,
and
that
component
will
maintain
the
basic
configuration
of
super
node,
for
example,
then
list
and
also
the
rate
control
and
also
the
hardware,
control
and
so
on,
and
then
on
each
note
there
is
this
conflict
watcher
and
periodically.
It
would
put
down
the
configuration
and
save
it
locally.
Under
the
note
and
when
the
container
applies
pool,
then
the
pool
is
unto
the
TF
demon
and
then,
according
to
the
config
watcher,
it
will
pass
it
on
to
DF
get
client.
B
A
B
This
is
another
example
in
Georgia
mobile,
how
they
use
so
above.
There
is
the
ecosystem
of
judge
on
mobile.
They
have
this
client
in
charge
of
uploading
the
image
and
triggering
the
image
download
command
and
down
there.
The
light
blue
one
that
is
the
Dragonfly
component,
so
they
would
upload
the
image
to
hover
and
push
it
to
harbor,
and
this
is
the
back
a
storage.
B
After
the
push
there
is
just
distribute
command
to
the
API
gateway,
then
the
API
gateway
would
call
the
Conner
service
kernel
service
and
then
it's
pushed
to
the
duct
server
dock
server.
You
can't
understand
it
like
dark
demon,
the
API
interface
and
then
the
dock
demon
would
push
it
to
DF
client.
The
day
of
client
then
will
interact
with
the
df'
master,
which
is
the
super
node,
and
then
there
is
the
PDB
Network
and
there
is
a
p2p
transmission.
B
B
B
Then
let
me
introduce
to
you.
The
future
planning
of
dragonfly
talked
about
several
things:
for
example
the
CNF
Co
system,
since
F
ecosystem
we
would
corporate
will
integrate
deeper
with
harbor
and
also
we
want
to
support
the
clinic
D
and
that
is
already
being
carried
out.
Then
for
security
we
will
support
the
private
container
image
registry
and
also
that's
already
done,
and
then
in
encryption.
We
would
introduce
different
encryption
algorithm
for
the
customers
to
choose.
We
can
ensure
that
the
transmission
is
safer,
then
for
efficiency
of
distribution.
B
A
B
B
A
B
A
A
Image
you
need
to
push.
Every
image
registry
will
generate
request
to
a
super
node.
After
getting
the
request,
super
node
well
sync
up
image.
It's
a
local
locally!
Therefore,
super
node
don't
need
to
sync
half
image
from
remote
registry.
It
will
save
time
and
further
lower
bandwidth
cost
for
registry.