►
From YouTube: IETF110-ICCRG-20210309-1200
Description
ICCRG meeting session at IETF110
2021/03/09 1200
https://datatracker.ietf.org/meeting/110/proceedings/
A
All
right,
we'll
just
give
it
a
minute
more,
but
in
the
meantime,
if
anyone
would
like
to
volunteer
to
do
to
take
down
the
minutes
that
save
us
some
time
going
through
the
presentations
that
give
us
more
time
to
do
more
interesting
discussions,
let
me
put
it
that.
A
A
I
will
take
that
as
richard
is
volunteering?
Aha,
there
you
go
well
done.
Thank
you,
sir.
So
I'll
call
you
out
in
a.
A
A
Better
all
right,
let's
get
started
it's
a
few
minutes
past
the
hour
and
welcome
to
iccrg
everyone,
it's
ridiculously
early
for
me,
it's
four
in
the
morning
for
me.
Hopefully
it's
a
more
sane
time
for
many
of
you,
but
I'm
looking
forward
to
this
session.
Hopefully
this
will
wake
you
up
and
hopefully
you'll
be
engaged
as
we
go
through
this.
A
So
let's
just
start
off
with
a
couple
of
notes
at
a
high
level.
Just
note:
well,
oh,
hang
on.
Let
me
fix
this,
so
you
can
actually
see
it
there.
You
go.
A
A
A
All
right
we'll
go
through
that
in
a
second.
So
that's
the
note.
Well,
if
you
haven't
taken
a
look
at
it,
please
do
please
read
it
before
you.
You
participate
at
the
mic,
I'm
not
gonna
walk
through
all
of
it
in
in
in
detail,
but
that
and
then
you
note
on
the
code
of
conduct
that
we
are
bound
by
the
iit
of
code
of
conduct.
A
If
this
is
important,
please
make
sure
you've
seen
this
as
well
and
read
this
as
well
before
you
participate
and
here's
the
high
level
goals
for
the
irtf
and
reminded
the
folks
here
that
the
irtf
is
not
a
standards,
development
organization.
It
is
a
research
organization
and
with
that
I
will
jump
into
the
agenda
for
today.
A
So
we
have
a
several
presentations
today
and
we
have
a
an
excellent
agenda
here
with
with
events
which
are
we
I'll
talk
about
the
first
one
in
a
moment,
but
we
have
a
number
of
presentations
and
I
hope
that
everybody
stays
engaged
to
the
entire
meeting.
A
I
I'll
thank
richard
for
for
taking
down
the
minutes
and,
as
chris
pointed
out
on
the
chat,
if
somebody
wants
any
comments,
my
any
comments
to
be
voiced
at
the
mic.
Please
prefix
it
with
mike
in
this
chat
and
chris
has
offered
to
kindly
deliver
them
at
the
mic
and
I'll
take
him
up
on
that.
A
That
is
about
all
from
from
me.
I
am
going
to
now
kick
off
the
first
presentation
from
rui
now
this
is
a
slightly
unusual
presentation
in
that
we
are
actually
going
to
be
talking
about
a
congestion
control.
That
is
more
it's
more
geared
towards
data
centers.
We
will
kick
it
off
with
that
presentation.
Rui.
Do
you
want
to
I'm
looking
here
to
see?
Do
you
have
to
come
up
and
ask
for
camera
and
mike.
C
Can
you
hear
me
now?
Yes,
we
can
hear
you,
okay,
let
me
get
you
going.
A
C
C
So
this
is
our
work
we
publish
in
sitcom
2019
and
then,
after
that,
we
actively
deployed
this
technique
into
alibaba
cloud
and
with
many
other
vendors.
So
by
that
time
we
think,
like
we
are
working
with
many
vendors
to
want
to
standardize
this
algorithm
and
this
design,
so
we
can
be
aligned
in
the
in
the
same
setting.
C
Okay,
so
so
the
motivation
for
our
design
is
like
today
in
the
clouds
we
design
very
high
performance
networking.
So
there
are
a
couple
of
motivation
for
our
design.
First
is
the
first
application
in
the
cloud
right
now
is
very
sensitive
to
latency
inflation,
for
example,
in
the
ebs,
elastic
block
storage,
the
sra
guarantee
on
the
latency
is
200
microseconds
and
some
other
like
key
value
application
memory.
Id
application
also
requires
some
millisecond
latency,
so
we
we
think
that
makes
the
network
become
one
more
important.
C
The
second
is
that
this
many
applications,
such
as
high
performance
storage
and
the
distributed
deep
learning
application
that
use
a
device
hardware
that
can
generate
the
data
orders,
magnitude
faster
and
also
require
outer
low
latency
for
those
applications,
so
that
network
become
a
new
bottleneck
in
our
system
and
the
last
part
is
the
new
architecture
is
happening
in
the
cloud,
for
example,
the
resource
disaggregation,
which
separates
the
compute
memory
and
the
disk
into
separate
resources
pools.
So
even
memory
access
will
go
through
a
network
in
the
future.
C
C
So
to
support
those
applications,
the
silicon
manufacturer
generate
fast
asics
because
the
traditional
softwares
that
cannot
keep
up
with
the
speed,
so
the
hardware
offloading
is
inevitable.
So,
however,
using
faster
hardware
can
generate
traffic
more
aggressively.
The
network
congestion
becomes
a
severe
problem
for
our
clouds.
For
example,
we
running
rdma
in
our
network.
We
we
find
that
there
are
lots
of
pfc
store
and
also
data,
lock
events,
so
we
find
it's
necessary
to
solve
the
conjugation
control
problem
in
this
case.
C
So
we
identify
a
couple
of
key
issues
in
our
conjunction
control,
er
high
speed
network,
the
first
issue
that
we
observe
lots
of
pfc
store
and
the
deadlock
event
in
our
rdma
network,
and
we
we
find
that's
a
cause
of
stability
issue
in
our
network.
However,
we
cannot
disable
pfc
because
that
will
affect
the
application
performance
as
well.
So
the
key,
the
key
insight
for
this
limitation
is
that
the
current
congestion
control
algorithm
is
has
has
a
very
slow
convergence
speed.
So
so
that's
the
essential
part.
C
We
want
to
address
the
second
issue
that
we
want
to
run
application
mix
of
application.
In
the
same
cluster,
however,
we
cannot
achieve
both
high
throughput
and
the
low
latency
for
different
applications.
So
the
essential
reason
for
that
is
the
traditional
condition.
Control
algorithm
rely
on
the
standing
queue
so
when
the
queue
build
up,
the
condition
control
works
start
to
work
to
react
to
the
congestion.
However,
when
the
standing
cube
build
up,
the
application
latency
has
already
been
affected.
C
The
last
part
is
we,
we
run
dcqc,
which
is
a
state
of
art,
conjunction
control,
algorithm
for
the
rdma
network,
and
that
we
find
that
those
those
algorithms
rely
on
the
heuristic
to
configure
their
algorithms,
so,
for
example,
in
dcqcn
they
use
at
least
15
parameter
so
to
to
to
work
with
the
congestion
control.
So,
in
our
case,
it's
really
time
consuming
to
tune
those
parameters
to
work
for
different
workloads.
C
So
to
allow
us
to
calculate
precisely
the
the
extent
of
the
congestion,
so
the
packet
continues
forward
around
the
pass
and
each
switch
will
attach
the
terminal
tree
of
its
own
into
the
package.
When
the
package
arrives
at
the
destination,
the
reservoir
will
generate
acknowledgement.
Packets
back
to
the
sender,
while
putting
those
telemetry
pack
information
into
the
act
packet
to
allow
the
sender
to
adjust
the
scenery
based
on
telemetry
information.
C
So,
as
I
said,
the
the
most
important
telemetry
information
we
need
is
the
key
lens
and
the
tx
rates
so
which
it
tells
precisely
if
the
buffer
build
up
for
this
particular
egress
port
and
more
important
that,
if
there's
no
queue
buildup
in
in
a
in
the
eagles
port,
we
still
can
use
the
txt
rate
to
quantify
the
the
occupancy
of
the
link.
For
example,
we
can
control
the
link
to
you
use
as
a
95
percent
utilization.
C
C
So
we
we
want
to
compare
our
inbantanometry
mechanism
with
the
traditional
isin
marking
easy,
as
we
know
that
a
single
bit
notification
of
condition
is
a
simple
and
efficient
being
supported
by
many
vendors.
C
We
viewed
that
inventory
is
an
advanced
version
of
ecn
which
provides
a
fun
green
network
load
information,
for
example,
q
lens
precisely
q
lens
in
terms
of
bias
or
in
terms
of
sales,
and
also
transmit
bias,
which
allows
to
generate
on
the
calculate
the
tx
rates
on
the
egress
port,
based
on
the
tx
byte
and
the
timestamp
information.
C
The
link
capacity.
Actually,
we
use
to
con
to
differentiate
a
different
link.
Either
is
100g
link
is
25g,
link
the
benefit
or
the
key
insight
we
get
is.
First,
we
can
convert
to
appropriate
just
just
a
one
round
trip
ton.
We
can
adapt
to
the
corrector
rates
to
avoid
congestion,
while
traditional
condition
control.
C
C
That's
a
ideal
case,
but
in
reality
we
we
prefer
to
query
inbound
animation
information
per
round
trip
time.
In
this
case,
we
use
a
standard
rfi
1.0,
which
is
iotf
standard,
int
format
and
for
those
standard
we
have
200
bytes
metadata
for
the
for
the
int,
which
is
account
for
only
2.5
percent
of
the
binaries.
C
So
in
this
case,
we
we
see
that,
because
bandwidth
is
the
generally
abundant
in
the
data
center
and
we
we
treat,
we
use
the
those
small
bandwidths
for
for
low
latency.
I
think
we
think
it's
a
good
trade-off.
C
C
Oh
ifrs,
I
I'm
not
quite
sure
the
detail,
but
it
seems
like
a
another
standard
in
the
tsv
vwg
right
so
yeah.
They
proposed
to
use
the
fundraising,
I'm
not
sure
how
to
compare
it,
because
I
remember
ps
l4s
targeting
on
the
wide
area
network
and
it
focused
on
the
the
switch
support
to
to
improve
the
traditional
ecn.
C
While
we're
focusing
more
on
the
data
center
environment,
where
the
the
ion
team
features
can
be
supported
in
those
switches
that
can
give
us
more
information
and
especially
for
the
for
the
for
the
data
center,
the
bandwidth
is
abundant.
So
we
can
use
the
those
boundaries
to
send
the
telemetry
information.
C
If
the
protocol
intend
to
be
used
on
the
internet,
so
right
now,
we
are
focused
more
on
the
on
the
data
center
environment
because
you
know
the
because
the
telemetry
information
is
more
directed
to
the
data
center
or
cloud
provider,
so
those
low
level,
congestion,
information
or
traffic
load
information
is
very
sensitive
to
to
the
to
the
network
operator.
C
A
Through
just
a
high
level
point,
the
chat
will
keep
going
on
you
may
you
may
want
to
you,
want
to
finish
your
presentation
and
then
see
if
people
want
to
answer
questions
at
the
mic
or
you
can
continue
the
conversation
on
the
chat
after
your
presentation
as
well.
Okay,.
C
C
First,
the
first
convergence,
so
the
sender
know
exactly
how
to
react
to
the
condition
and-
and
it
can
adapt
to
the
precise
rate
in
just
one
round
trip
time.
C
C
C
So
this
is
the
preliminary
result
from
our
smarnic
implementation
we
deployed
and
as
we
see
that
in
the
in
the
left,
figure
showed
that
the
axis
is
the
different
flow
size.
The
y-axis
is
a
99
percentile
normalized
flow
composition.
Time
actually
in
this
figure,
the
lower
the
better
the
lower
means,
the
lower
latency,
and
we
compare
our
design
with
dcqcn
the
the
purple
curve,
which
is
the
default
condition
control
in
rdma.
C
C
We
showed
that
overall
in
the
experiment,
the
distribution
of
the
q
length
in
the
switch
and
as
we
see
that
in
the
hpcc
plus
plus,
we
say
that
in
even
in
the
99
percentile,
the
q
lens
is
only
23
kilobytes,
which
translate
to
seven
micro,
second
and
q,
delay
in
25
g
link,
which
demonstrates
that
we
can
achieve
ultra
low
latency
while
maintaining
near
zero
q.
A
A
D
D
Thank
you,
rui
for
the
great
presentation.
I
just
have
a
question
in
the
graph
that
you
were
showing.
C
D
C
D
C
A
C
In
this
case,
we
have
a
five
percent
bandwidth
header
room
to
absorb
some
more
bursts.
So
in
the
case,
the
shuffles,
like
a
the
the
number
in
the
in
the
in
the
left,
have
very
low
latency
because
we
don't
build
up
the
queue.
However,
if
you
look
at
very
tale
of
this
figure,
we
actually
perform
worse
than
the
dcq
scene
for
those
long
flow
because
they
can
use
hundred
percent
throughput.
We
only
use
a
9
percent.
D
F
Hi
we
we
just
comment
on
the
chat
that
this
idea
is
similar
to
a
protocol
discussed
in
ippm
group,
the
er
aom
protocol,
it's
different,
because
here
you
are
measuring
the
the
cues
which
is
directly
related
to
traffic
and
congestion,
and
but
there
is
several
issues
that
you
can
find
when
you
try
to
do
this
in
every
in
every
step
or
or
you
you
flow
and
the
the
question
is,
is
if
you
already
knew
this
protocol
that
we
are
discussing
ippm
and
how
do
you
think
that
this?
F
C
So
to
answer
a
question
I
think
for
for
hpcc,
plus
plus
the
up,
we
need
a
separate
queue
for
our
protocol.
Just
like
ecn,
you
need
a
separate
queue
for
easy
and
for
tcp
traffic
as
well
right.
So
we
need
to
have
a
separate
queue
for
hpc,
plus
plus.
A
Let's
continue
this
one
on
that,
if
you
would
like
ignacio
stuart
you're
up
next.
H
Hi,
hey
just
a
quick
comment.
I
think
I
understand
the
confusion
that
led
to
vidi's
question
and
this
happens
a
lot
when
people
use
the
terms,
latency
and
delay
interchangeably.
H
This
graph
is
not
showing
the
latency
in
the
sense
of
the
per
pack
round
trip
time
it's
showing
the
total
completion
time
for
the
transfer.
So
when
you
have
really
large
transfer
many
many
round
trips,
the
actual
per
packet
latency
becomes
less
important.
C
Yeah,
the
left
figure
shows
the
trans
the
flow
combination,
time,
which
means
the
translation
delay
for
this
flow,
the
hot.
How
much
time
for
the
network
transmit
this
particular
flow.
Yeah
certainly
need
a
couple
of
round
triple
times
for
the
for
the
figure
in
the
right.
It
actually
shows
the
the
the
delay
of
the
network,
because
the
left
side
shows
the
cdf
of
q
lens,
and
so,
if
you,
if
you
add
about
the
transmission
delay,
this
is
the
queuing
delay
right.
So
this
is
a
queuing
delay
distribution.
You
can.
H
J
C
There
are
a
couple
of
different
first
one
is
we
use
per
package
int,
but
in
the
production
once
we,
because
once
we,
after
the
published
on
sitcom,
we
we
work
on
the
vendor
to
deploy
this
in
in
our
production
and
during
this
procedure,
we'll
accumulate
a
couple
insights
or
experience
from
deploying
this
protocol.
That's
come
with
hpcc
plus
plus,
so
it's
more
production
ready
and
it's
a
standardizable
protocol.
C
Now
some
of
the
design
is
too
artificial
in
the
second
paper,
so
in
the
htc
plus,
plus
it's
more
practical
and
design,
and
it's
it's
good
to
align
with
different
vendors,
because
we
when
we
deploy
this
protocol
and
then
we
actually
talk
to
different
vendors
to
implement
these
protocols.
We
talk
to
the
unique
vendors
and
also
talk
to
the
switch
vendors
individually.
So
that's
the
common
ways
that
we
want
to
standardize
so
that
we
can.
We
don't
have
to
talk
to
each
one
individually,
but
we're
talking
a
common
language.
C
There
are
a
couple
of
a
detailed
design
in
in
the
draft.
That's
quite
different
with
the
paper
version.
K
C
C
In
the
paper
version,
we
consider
the
multiple
bottleneck
design
and
we
actually,
we
carry
int
information
for
each
hop
so
that
we
can
actually
know
where
is
the
conjunction
and
we
have
of
a
complete
design
to
consider
all
the
bottleneck,
and
we
actually
can
change
the
scenic
ray
so
both
based
on
different
policy.
So
we
can
see
like
under
the
multiple
bottleneck
we
can
achieve
maximum
fairness
or
proportional
fairness
or
something
between
that.
C
It's
our
fairness
we
can
achieve,
but
the
later
in
the
in
the
production
deployment,
we
found
that
the
simplest
way
is
just
to
do
the
the
the
maximum
fairness
we
don't
store,
because
in
multiple
bottleneck
cases
we
need
to
stop
all
the
information
for
each
hub.
That
actually
causes
more
resources,
but
the
the
the
benefit
is
very
limited.
The
benefit.
Just
how?
How
fairness
we
can
achieve,
what
what
fairness
we
can
achieve
and
how
quickly
we
can
adapt
to
the
the
fairness
point.
C
But
actually
we
we
we
care
more
about
ut
utilization
convergence
which
which
can
avoid
congestion,
but
we
don't
care
much
about
fairness,
convergence
so
because
the
flows
are
actually
very
short.
In
our
case,
we
don't
consider
so
that
that's
why
we
we
make
a
very
simple
design
that
that
works
with
the
multiple
personality
case.
Yeah
thanks.
A
All
right,
thank
you,
so
much
rui
for
that
I'll
encourage
this
is
this
is
a
presentation
which
was
in
part
there's
some
discussion
on
the
iccrg
mailing
list.
People
seem
to
be
interested
in
discussing
and
engaging
in
data
center
construction
controllers.
A
C
Yeah
thanks.
I
have
already
copied
these
comments.
I
will
reply
in
the
email.
Thank
you.
Everyone.
Thank
you.
So
much
roy.
A
With
that,
we
will
move
on
to
the
next
presentation
of
the
day.
Let
me
see
if
I
can.
E
Yeah
mike
is
on
yep
all
right
so
for
those
people
who've
seen
this
yesterday,
don't
worry!
It
contains
a
lot
of
new
information,
so
you
don't
get
too
bored
next
slide.
Please.
E
So
qlog
stands
for
quick
logging
and
its
project
started
about
two
years
ago,
when
we
identified
that
the
new
protocols
would
probably
get
quite
complex
to
actually
analyze
and
debug
in
practice,
and
we're
going
to
need
some
advanced
tooling
to
do
that,
which
would
typically
do
for
something
like
tcp,
for
example.
Next
slide
is
we
take
a
packet
capture
somewhere
in
the
network
right
and
then
analyze
that
using
something
like
wireshark,
for
example.
E
This
is
still
possible
for
quick,
but
quite
a
bit
more
difficult,
because
next
slide
quick
of
course
encrypts
most
of
its
transport
level
metadata
as
well.
So
to
do
this,
you
would
have
to
store
the
entire
packet
capture,
including
the
very
large
payloads,
and
also
the
tls
decryption
secrets,
leading
to
obvious
privacy
and
scalability
issues.
There's,
however,
a
second,
more
long-standing
problem,
with
the
typical
way
that
we
do
this
next
slide,
which
is,
of
course,
that
a
lot
of
protocol
information
isn't
always
reflected
on
the
wire.
E
That's,
of
course,
always
the
case
for
congestion
control
information
like
the
congestion
window,
which
I
don't
have
to
tell
all
of
you,
and
so
to
solve
these
problems
or
to
try
and
solve
them
for
quick,
we
proposed
a
different
approach,
next
slide,
where
the
idea
is
instead
of
doing
a
network
based
capture,
let's
log,
this
information
at
the
endpoints
directly
from
the
implementations,
obviously
or
have
all
the
salient
data
right
there
available
and
they
can
easily
leave
out
the
privacy
sensitive
parts.
E
E
On
top
of
that,
and
so
the
next
slide
qlog
really
is
not
rocket
science.
It's
relatively
simple.
Currently,
we
just
map
this
into
json
and
we
define
how,
for
example,
I
receive
packets
containing
an
acknowledgement
frame
should
look
like,
or
indeed
on,
the
right
side,
what
you
should
call
the
variables
related
to,
for
example,
just
control
updates
using
this
type
of
log
as
input.
We
were
then
able
to
create
quite
a
few
indeed
reusable
tools
next
slide,
which
are
available
in
the
queue
this
tool
suite.
One
of
those.
E
The
second
major
tool
that
we
have
in
cubist
is
is
a
kind
of
you
can
call
it
the
tcp
trace
for
them
for
quick.
Where,
because
of
this
approach,
we
we
don't
just
show
the
data
and
the
acknowledgements
and
the
flow
control.
We
can
also
show
just
the
window
bytes
in
flight
and
also
the
various
round-trip
time
measurements
that
are
used
internally
immediately
and
so
for
me,
as
definitely
a
non-expert
in
congestion
control.
E
E
E
But
it's
also
going
beyond
initial
deployment.
Initial
implementation,
debugging,
for
example,
what
facebook
has
done
is
they
have
deploys
q
log
at
scale
in
their
data
centers
as
well,
and
so
they've
been
able
to
find
quite
a
few,
relatively
quick,
specific
bugs
in
their
setup
that
were
previously
missed
in
lab
testing.
They
only
really
surfaced
during
their
deployment.
E
For
example,
one
of
the
things
they
had
was
that
they
were
underestimating
bandwidth
during
the
zero
rtt
phase,
because
they
were
relying
on
previously
acknowledged
packets,
which,
during
zero
rtt,
you
do
not
have
previously
knowledge
packets,
at
least
not
once
containing
application
layer
data
another
one
that
is
in
a
very
interesting
paper
that
I'll
link
to
the
bottom.
There
was
that
in
quick,
you
change
encryption
levels
and
that
is
accompanied
by
you
can
do
that
with
implicitly
acknowledging
all
the
packets
from
the
previous
encryption
level
and
the
way
they
did,
that
was
they.
E
E
Okay,
so
the
thing
is
that
no
no
previous
slide,
please.
E
So
the
thing
is
that
this
has
been
used
not
just
by
the
experts
or
the
people
that
knew
that
they
were
doing
to
great
effect,
but
I
think
it
quick.
E
We
have
this
very
special
situation
where
you
have
a
lot
of
quick
implementers
that
are
really
not
congestion,
control
experts
by
themselves
and
they
have
to
try
to
make
something
working
based
on
the
drafts
and
looking
at
other
people's
code,
and
I
think
this
kind
of
tool
and
approach
has
really
helped
them
to
get
their
things
at
least
somewhat
working
next
slide,
and
I
think
that's
that's
one
of
the
main
reasons
that
qlog
has
found
quite
a
bit
of
support
within
the
quick
community.
E
It's
because
of
this
that
the
format
has
shown
promise
and
the
tools
seem
useful
that
we
are
now
moving
for
adoption
of
this
work
by
the
quick
working
group
next
slide,
which
is
intended
to
happen
somewhere
over
the
next
months
as
part
of
the
quick
recharger
one
of
the
goals
there
is
to
flash
out.
All
of
this,
for
for
quick,
we
have
some
basic
adjust
control
stuff
in
there,
but,
for
example,
facebook.
They
have
added
a
lot
of
custom
events
as
well
to
help
them
better
debug
their
their
custom
setups.
E
E
We
also
have
a
few
proof
of
concepts
projects
around
that,
for
example,
for
tcp
we've
been
playing
around
with
ebpf
and
using
k
probes
to
get
this
kind
of
information
bubbled
up
from
the
kernel.
We
then
combine
these
with
the
raw
packet
captures
from
wireshark
to
get
like
a
full
view
of
what
the
tcp
stack,
for
example,
in
linux,
is
doing,
which
has
led
to
some
some
interesting
observations.
E
Next
slide,
we've
also
been
playing
around
with
this
for
multi-parts
quick,
and
we
have
an
ongoing
project
looking
at
it
from
multiple
tcp
as
well,
where
we
tag
each
event
with
a
specific
path
id.
So
we
can
split
them
out
later
and
have
a
comparison
between
the
different
paths,
of
course,
so
it
shows
definite
promise.
It
looks
like
we
will
be
able
to
do
this.
However,
we
are
also
running
against
a
few
challenges
or
bottlenecks.
Next
slide.
E
For
example,
we've
recently
started
a
major
push
towards
actual
performance
testing
of
quick
stacks
and
qlo.
Currently,
as
I
said,
is
json
based,
which
is
flexible,
but
it's
also
not
super
optimized,
for
example,
in
terms
of
file,
size
and
so
testing
performance
on
gigabit
networks
has
turned
out
to
be
quite
difficult
to
scale.
E
There
are,
however,
other
people
like,
for
example,
big
banks
from
microsoft,
they're
using
a
custom,
similar
approach,
but
they
are
using
a
more
optimized
format
and
custom
tools
where
they
are
able
to
ingest
much
much
larger
log
files
and
much
much
more
information
to
help
debug
this
kind
of
high
performance
scenario,
so
I
think
the
general
approach
is
is
good.
We
just
need
to
look
at
how
we
implement
it.
E
Specifically,
that's
one
of
the
discussions
that
we
will
definitely
have
for
q
log.
One
of
the
problems
that
I
have
there
is
kind
of
the
main
puller
of
this
thing.
Next
slide
is
that
again
I
am
not
a
I'm,
not
an
expert.
I
don't.
I
don't
implement
these
things
myself.
I
don't
test
them.
I
don't
do
research,
so
I
don't
know
what
what
like?
What
the
the
typical
way
of
working
for
this
is
either
research
or
actual
deployment.
I've
seen
things
like
this
come
along
here
in
this
research
group.
E
E
The
fact
that
I
need
more
feedback
became
obvious
earlier
in
the
early
days
when
I
talked
to
jonah
asking
you
know
how,
how
can
we
improve
this
tcp
trace
for
quick,
and
he
said
the
simplest
thing
you
could
do
is
add
a
type
of
thing,
a
ruler
where
I
can
just
drag
and
drop
across
the
data
line
here,
and
it
shows
me
about
what
the
data
rate
was
and
how
how
long
it
took,
which
is
really
really
simple
to
implement
it
did
in
about
one
hour,
and
I
would
have
never
thought
of
that
myself.
E
E
Most
of
this
work
for,
for
practical
reasons,
will
happen
in
the
quick
working
group,
even
though
we're
also
looking
for
other
protocols
as
well,
and
we
will
of
course
have
some
very
good
input
already,
because
there
are
a
lot
of
knowledgeable
people
involved
in
quick
with
this
as
well.
But
extra
feedback
and
opinions
are
always
welcome.
E
A
Thank
you
so
much
for
the
presentation
robin
we
have
time
for
just
a
couple
of
quick
questions,
but
I
want
to
quickly
say
that
I'm
I'm
grateful
for
robin
for
presenting
this
here
and,
of
course,
for
the
work
that
he's
doing.
This
is
super
super
useful
for
quick,
but,
as
he
points
out,
it's
super
helpful
for
for
for
transports
as
well.
People
who
wanted
to
do
and
use
tooling,
that's
that's
more
modern,
so
to
speak,
will
find
this
to
be
extremely
useful.
Okay,
john
john
you're
you're
you're
up.
I
Thanks,
I
just
wanted
to
ask
what
sort
of
compression
algorithm
you're
using
to
get
the
18
megabytes?
Oh
that's
just
normal,
zip!
Yesterday,
okay,
so
presumably
a
more
advanced
conversion
would
get
it
down
smaller
sure.
E
And
for
storage
and
transfer-
that's
fine
for
now,
but
it's
mainly
if
you
want
to
load
this.
For
example,
rqvis
tools
are
all
web-based
and
your
browser
really
doesn't
like
you
uploading
a
gigabytes,
json
file
or
or
or
downloading
the
compressed
version
and
then
having
to
unzip
it
in
the
browser
is
also
not
something
it
likes
to
do
so.
L
Yeah
hi,
I
I
do
think
this
looks
very
interesting.
Has
anybody
started
doing
anything
with
this?
For
for
tcp
like
this
seems
like
it
does
have
obvious.
E
Not
not
concretely
like
we
have
the
proof
of
concepts.
Those
are
more
like
research
projects.
I
do
know
that
people
at
facebook
and
google
have
shown
interest
in
applying
this
to
tcp
as
well,
and
I
think
even
apple
as
well
to
doing
that.
But
I
don't
know
of
any
concrete
efforts
yet,
but
I'm
very
interested
in
hearing
from
people.
L
M
Thanks
so
I
would
just
wanted
to
basically
chime
into
the
same
horn
here,
and
I
think
it's
very
interesting
to
have
a
standardized
way
to
lock
these
congestion
control
internal
events,
especially
useful
for
tcp,
obviously,
because
right
now
at
least
I
am
faced
with
at
least
three
different
approaches:
how
to
diagnose
and
troubleshoot
tcp,
and
none
of
them
are
standardized,
so
they
all
need
their
specialized
tool
sets.
They
may
need
the
recompilations
of
the
stack
and
so
forth.
A
Thanks
for
that
comment,
richard
I'll
say
that
at
high
level,
just
two
quick
points
before
I
let
robin
go
one
is
that
the
q
log
tool,
the
q
log
schema,
is
different
from
the
visualization
tool
and
that's
actually
quite
quite
an
important
distinction
here
to
your
point
there
richard,
which
is
that
if
you
want
to
extend
the
with
the
schema
to
include
more
events
in
tcp
and
so
on
and
so
forth,
that
is
an
excellent
that
that
would
be
very
useful
work.
A
All
right,
I
will
thank
robin
for
his
presentation
and
martin.
You
get
to
jump
into
the
queue
and
say
something.
A
Thank
you,
martin,
and
thank
you
robin
to
the
next
one
anna
I
think
europe.
I.
O
So
you
have
one
dccp
tunnel
per
path
and
the
two
main
use
cases
that
we
have
in
mind
for
this
is
in
the
context
of
3gpp
and
access
traffic,
steering,
switching
and
splitting
architecture
that
is
being
defined
there
for
combining
cellular
and
wi-fi
networks,
and
also
for
the
hybrid
access
use
case
in
in
home
networks,
where
you
combine
the
fixed
and
a
cellular
link
for
better
performance
next
slide.
Please.
O
So
then,
let's
look
at
the
the
problem
that
we
then
encounter.
If
we
have
this
type
of
multipath
framework
and-
and
this
is
generic
for
the
protocol,
as
I
said
so,
this
tunneling
solution,
of
course,
results
in
nested
congestion
controls,
and
this
can
also
be
a
problem
for
for
single
bath,
a
single
path
tunneling.
But
when
you
move
into
the
multipath
context,
this
encounters,
of
course,
an
added
complexity
and
in
our
work
we
are
using
uncoupled
congestion
control
over
the
the
two
parts.
O
In
all
the
cases
as
the
use
cases
here,
we
don't
see
the
need
for
coupling
the
congestion
control,
as
the
idea
is
to
to
use
the
two
parts
and
to
be
able
to
to
aggregate
them
and
use
them
for
the
better
performance,
and
we
don't
expect
the
fairness
issues
to
come
into
play
here.
So
you
see
the
general
setup
on
the
picture
here
you
have
a
ue
and
then
you
have
a
proxy
or
a
tunneling
endpoint,
where
you
in
this
figure
have
a
downlink
transfer.
O
Slide
so
this
scheduling
and
reordering
components
will
have
a
very
large
impact
on
the
performance
in
this
scenario.
So
in
this
results
that
I
will
talk
about
today
and
also
for
a
lot
of
the
the
work
we
have
done
here,
we
have
looked
particularly
as
at
the
cheapest
path,
the
first
scheduler
or
the
the
strict
priority
scheduler.
O
So
you
have
a
prepare
preferred
path
and
you
send
data
on
this
path
whenever
you
have
a
space
available
in
your
congestion
window,
and
it's
only
if
this
preferred
path
is
not
available
that
you
start
to
use
your
your
other
paths-
and
this
is
also,
of
course,
we
think
a
quite
reasonable
scheduler
for
the
scenario
that
we
are
targeting,
because
you
may,
for
instance,
want
to
use
your
your
wi-fi
network
and
then
only
use
your
cellular
quota
if
you,
if
the
wi-fi
network
does
not
provide
the
sufficient
performance
or
you
use
your
your
fixed
network
in
the
hybrid
access
scenario
and
then
use
your
cellular
network
for
improved
performance
when
needed.
O
The
reordering
component
can
also
have
different
functions,
and
I
think
marcus
will
talk,
particularly
about
reordering
in
the
the
next
talk,
but
in
the
work
I'm
presenting
here
we're
using
an
adaptive
time
limit
for
for
the
reordering
to
determine
if
to
pass
the
packets
on
or
not
as
we
may
not
have
a
reliable
transfer
also
over
the
the
tunnel
paths.
O
So
I
mentioned
that
if
we
move
to
the
multipath
domain,
we
have
some
additional
challenges
for
the
congestion
control
in
congestion
control,
and
particularly
the
challenge
in
in
this
scenario,
is
to
be
able
to
aggregate
the
capacity
over
the
two
parts,
so
you're
using
one
path
as
your
preferred
path,
and
the
challenge
then,
is
to
actually
be
able
to
also
use
the
second
path
when
needed
and
in
particular
the
end-to-end
congestion
control
may
react
before
and
slow
down
before.
You
are
able
to
actually
start
to
utilize
that
path.
O
And
here
we
have
a
example
of
this,
so
what
you
see
in
this
graph
is
a
time
sequence
of
four
different
transmissions
stacked
on
top
of
each
other
and
for
each
one
you
see
the
the
green
throughput
is
for
the
preferred
path
and
the
red
throughput
is
for
the
second
path.
So
up
at
the
top
of
the
the
topmost
scenario,
you
see
that
things
work
as
you
would
like
it.
You
have
capacity
using
both
the
first
path
and
the
second
path.
O
The
second
scenario,
from
the
top,
on
the
other
hand,
does
not
work
well
at
all
you're
not
able
to
use
any
of
the
second
path,
so
only
the
capacity
on
the
first
part
is
actually
utilized,
and
then
we
have
two
examples
where
you
kind
of
manage
to
use
some
of
the
the
capacity
of
the
second
path
and,
in
general,
the
the
use
here
is
going
up
and
down
a
bit,
and
this
is
experiments
with
four
different
configurations
that
shows
how
different
the
outcome
can
actually
be
here,
depending
on
a
number
of
parameters
that
impact
this
performance
next
slide.
O
O
So
the
results
that
I'm
showing
here
is
using
a
user's
base
implementation
of
this
multipath
framework.
So
there
is
also
a
kernel
level
implementation
developed
by
the
colleagues
at
deutsche
telekom
available
that
mike
has
presented
the
last
ietf
in
the
tsvwd
session,
some
measurements
from
there.
But
here
we
use
a
user
space
implementation
which
of
course
offered
and
quite
a
lot
of
flexibility
in
trying
out
different
protocols
and
different
configurations,
and
this
usage
base
program.
O
Okay,
something
seemed
to
have
happened
there
with
good
luck,
I'm
back!
Okay!
When
did
you
lose
me?
So
I
was
explaining
that
we
are
using
a
multi,
the
user
space
framework
for
this
experiments,
and
this
offers
them
a
lot
of
flexibility
in
how
to
try
out
different
scheduling
methods
and
and
different
protocols.
So,
in
this
framework
the
packets
are
captured
through
the
linux
tune
device,
and
then
the
framework
encapsulates
the
packets
with
information
like
the
path
sequence,
numbers
and
the
timestamps
of
the
packets,
and
then
packets
are
scheduled
over
two
single
path
sockets.
O
So
this
means
that
the
framework
as
such
is
not
tied
to
dccp.
It
also
allows
us
to
use
other
protocols
as
tunnels,
so
in
the
experimental
results
we
have
both.
O
Okay,
so,
as
I
said
for
the
tunnel,
we
are
using
using
both
tcp
tcp
neurino
and
tcp
bbr,
and
then
we
are
using
a
dccp
with
the
ccid2
which
corresponds
to
neurino
and
also
a
new
ccid
ccid5,
which
is
a
implementation
of
bbr
style,
congestion,
control
for
dccp,
and
we
have
some
base
delays
for
the
the
two
different
paths
on
the
multipath
and
for
the
added
delay
to
the
server
20
milliseconds
on
on
the
preferred
path
and
initial
20
milliseconds
to
the
server
as
a
baseline
and
a
similar
trick
path
as
a
baseline,
with
40
milliseconds
to
the
second
over
the
second
path
and
the
same
bandwidth
and
same
q.
O
So
now
some
example
results.
So
short
here
stands
for
the
basic
configuration
and
what
we
see
here
is
the
combination
of
different
congestion,
controls
on
the
end
to
end
path
and
then
in
the
the
tunnel,
and
the
results
are
relative
to
the
performance
over
a
single
path.
So
you
see
a
percentage
of
the
the
flow
completion
time
for
downloading
a
large
file
here.
O
So
if
you're
below
100,
you
have
gained
performance
as
compared
to
the
single
path
case,
and
you
can
see
here
that
the
the
performance
varies
a
lot
depending
on
what
congestion
control
combinations
you
use
in
general.
Bbr
is
performing
better
as
a
tunnel
protocol,
and
this
is
as
bbr
reacts
faster
when
you
start
to
experience
congestion,
and
there
is
also
less
loss
over
the
tunnel
with
bbr,
and
you
can
also
see
that
the
bbr
at
endpoints,
with
reno
in
the
tunnel,
performs
very
poorly
here,
because
bbr
reacts
before
the
the
second
path
is
used.
O
O
Next
slide,
please,
okay.
So
to
summarize,
the
congestion,
control
and
congestion
control
aspect
has
a
lot
of
impact
on
the
performance
on
the
the
multipath
tunneling
problem
and
there's
a
lot
of
factors
that
interact
here.
So
the
results
that
we
saw
here
were
particular
to
the
the
scheduling
mechanism
of
cheapest
pipe
first,
and
it
also
has
a
large
impact.
What
the
ordering
mechanism
you
use
the
different
congestion
controls
have
quite
the
different
interactions.
O
The
placement
of
the
proxy,
of
course,
is
also
quite
important,
as
well
as
the
the
path
characteristics
and
if
we
should
have
some
some
first
general
conclusions
from
the
the
work
we
are
going,
we
are
working
on.
We
can
see
that
overall,
bbr
performs
better
than
reno
as
the
congestion
control
for
the
tunnel
and
having
the
proxy
close
to
the
user
as
expected
is
typically
beneficial
and
we
are
actively
working
on
this
playing
with
the
different
parameters
and
analyzing
the
the
various
interactions.
O
A
Thank
you
anna.
Let's
keep
this
quick
because
we
have
questions
after
marcus's
presentation
next,
as
well
I'll
close
the
mic
line
here
after
gauri,
but
martin
you're
up.
N
Thank
you
anna.
This
is
very
interesting.
How
applicable
do
you
think
these
results
are
to
non-multi-path
scenarios
that
have
congestion
control
in
congestion
fall
like
mask.
O
So
I
mean
the
the
particular
problem
of
of
using
the
second
path,
which
is
the
main
challenge
here
will
not
happen
unless
you
have
have
multiple
paths.
Of
course,
some
of
the
some
of
the
aspects
that
we
see
here,
for
instance
the
relation
between
the
different
control
loops
in
terms
of
of
rtt
and
the
impact
of
what
what
the
congestion
controls
you
use
in
the
tunnel
versus
what
you
use
and
will
also
come
in
to
play
in
the
single
path
context.
But
the
results
are
not
directly
transferable.
Of
course,.
P
On
side
11,
I
was
curious
with
your
heat
map
and
how
this
would
play
out.
If
there
was
a
much
larger
rtt
at
place,
you
know,
is
it
so
so
can
you
just
talk
me
through
a
little
bit
more
about
what's
going
on.
O
O
O
P
O
That,
I
think
that's
just
a
random
effect,
so
I
mean
what
we
have
done
here
is
that
we
have
sampled
this
space
with
the
number
of
measurements,
and
so
each
measure
of
each
measurement
point
here
is,
is
you
know
not
repeated
many
times
it's
like
400
samples
over
the
space
to
create
this
heat
map.
So
there
is.
O
There
is
some
noise
in
in
this
graph,
of
course,
but
you
can
see
quite
clearly
that
the
yeah
that
the
blue
space
is
quite
separated
from
the
red,
so
you
can
see
quite
clearly
the
impact
of
the
two
control
loops
and
and
the
difference
between
the
rtt's
in
the
two
cases.
A
Thank
you
so
much
anna
we'll
move
on
to
the
next
presentation.
This
would
be
the
same
space,
but
I'm
gonna
bring
this
up
marcus.
Q
Q
Yes,
starting
with
how
multi-path
typically
works,
which
components
are
usually
employed,
so
let's
have
first
to
look
into
the
the
picture
so
on
the
left.
There's
a
sender
on
the
on
the
right.
You
have
a
receiver,
and
in
between
that
is
now
very
much
related
to
multiples.
You
have
at
least
two
communication
paths
and
on
center
side
you
have
a
scheduler
which
is
responsible
to
distribute
traffic
across
the
multiple
paths,
so
you
can
have
different
logics
for
this
and
you
will
find
the
number
of
logics
already
described
in
the
iccrg
scheduler
draft.
Q
Q
A
multi-pass
network
protocol
takes
responsibility
to
allow
the
communication
between
sender
and
receiver,
and
typically
in
such
scenarios
when
it
comes
to
heterogeneous
environments,
you
have
a
latency
delta
between
the
multiple
paths
employed
in
the
communication,
and
that
is
exactly
the
issue
where
I
want
to
talk
about
so
with
this
latency
delta.
Q
You
can
imagine
when
we
simultaneously
use
the
two
parts,
so
maybe
we
send
the
packets
in
a
round
robin
fashion
that
will
cause
out
of
order
delivery
on
receiver
side
and
with
that
it
completely
differs
from
the
characteristic
of
a
single
path
communication,
because
in
a
single
past,
communications
you're
only
dependent
from
from
from
the
latency
of
this
single
path.
Q
If
you
look
at
itf,
we
see
there
are
multiple
multi-pass
protocols
defined
or
available
as
a
draft
that
is
multiple
tcp,
multi-pass,
dccpa,
multi-pass,
quick,
cmt,
sctp
and
so
on,
and
would
be
interesting
to
see
in
the
next
slide
how
they
behave
when
it
comes
to
out-of-order
delivery
and
which
mechanisms
they
have
implemented
to
to
overcome
this.
Q
Nevertheless,
typically
scheduling
and
reordering
are
not
part
of
the
protocol
specification
and
it's
left
with
the
implementers
to
to
take
care
and
that
might
cause
trouble
if
an
implementation
is
dependent
on
protocol
mechanisms.
Q
Q
Q
So
from
from
today's
perspective,
I
think
they
should
consider
sequence
maintenance
in
their
design,
but
I
from
my
feeling
that
is
not
complete
completely
given
right
now,
so
for
the
three
protocols
I
mentioned
on
this
slide,
the
dccp
http
and
quick
when
it
becomes
combined
with
the
unreal
of
unreliable
transmission,
then
strict
reliability
is,
is
not
an
option.
Obviously,
and
the
solution,
if
we
talk
about
sequence,
maintenance
will
be
a
trade-off
between
maintaining
the
data
sequence
without
interrupting
the
data
flow.
Q
A
special
challenge
in
this
scenario
is
when
it
comes
to
to
packet
loss,
and
I
think
it's
likely
when,
when
we
talk
about
multi-path
transmission,
that
we
will
see
packet
loss
to
some
extent
and
then
this
is
all
always
combined
with
the
question.
So
how
long
do
we
have
to
wait
in
a
potential
reordering
mechanism
on
receiver
side
and
at
one
point
at
which
point
in
time
we
can
assume
that
the
packet
is
really
lost
and
we
have
not
to
wait
for
it
any
longer.
Q
So,
during
the
multi-pass
protocol
design,
this
has
to
be
considered
and
possible
measures
like
different
sequencing
schemes,
so,
for
example,
sequencing
scheme
for
path
and
separately
for
the
multi-pass
connection
or
a
sender,
receiver
signaling
have
to
be
taken
into
account.
I
will
come
to
this
in
the
next
slides
yeah.
With
this
draft
we
have
available
as
icci
the
multi-pass
reordering
craft
now,
in
version
two,
we
claim
to
cover
all
these
aspects
and
provide
guidelines
for
design
and
implementers
of
multi-parts
protocols
next
slide.
Please.
Q
Yeah,
okay:
let's
have
a
deeper
look
into
what
we
have
specified
so
far
in
the
draft,
so
we
discuss
several
mechanisms
to
support,
smooth
and
adjustable
in
order
delivery
for
multi-path
communication,
and
this
draft
scheduling
is
out
of
scope.
So
scheduling
maybe
also
provide
some
some
measurements
to
to
out
to
overcome
out
of
order
delivery
on
receiver
side.
But
we
see
is
this
rather
as
part
of
the
scheduler
draft
available
at
iccrg.
Q
So
we
discuss,
for
example,
resequencing
mechanism
with
which
we
want
to
keep
the
generated
sequence
of
data
at
receiver
side
and
there
we
discussed
multiple
functionalities
or
multiple
logics.
So
the
first
is
the
passive
one.
So
we
trust
forward
packets
as
arrived
for
sure
that
will
not
reorder
at
all.
Q
Q
Adaptive
expiration
is
similar
to
the
static
expiration,
but
we
do
not
have
a
fixed
time
threshold.
So
we
dynamically
adjust
the
time
on
how
we,
how
long
we
wait
for
a
missing
packet,
depending
on
the
rtt,
for
example,
or
the
latency
on
the
part.
Q
Then
we
have
delay
equalization
strictly
spoken.
That
is,
is
not
a
reordering
mechanism
that
just
delay
the
faster
packets
to
match
the
latency
of
the
slower
part.
And
last
but
not
least,
we
discuss
fast
packet
loss
detection.
That
is
probably
something
which
can
be
combined
with
the
other
logics
and
that
level
the
part
and
connection
sequencing
to
to
have
a
very
early
idea
about
if
a
packet
is
lost
or
delayed
next
slide.
Q
We
see
that
we
can
overcome
packet
loss
when
we
spend
redundancy
and
in
that
area
we
see
forward
error
correction
but
also
network
coding,
and
when
it
comes
to
re-transmission
so
overcome
packet
loss
by
by
re-transmission.
We
also
see
you
see
three
different
mechanisms
which
can
help
here.
On
the
one
hand,
we
have
the
signaling,
which
is,
for
example,
used
in
in
tcp
or
dccp
to
signal
outstanding
packets
to
to
the
sender
anticipated
means.
Q
We
predict
a
beneficial
early
retransmission
for
for
the
reordering
reordering
purpose,
and
last
but
not
least,
we
have
the
flow
selection,
so
the
abil
ability
to
retransmit
packages
on
a
path
different
to
the
original
one.
If
this
is
if
this
supports
the
reorder,
as
I
said,
combinations
of
mechanisms
are,
in
principle
possible
and
most
probably
useful
next
slide.
Q
Please
yeah.
First
of
all,
I
want
to
invite
you
to
to
contribute
to
this
draft,
where
I
think
that
this
is
a
applicable
to
many
of
the
multi-pass
protocols
discussed
or
standardized
at
iitf,
so
that
the
draft
itself
is
still
under
development
and
some
content
is
not
finalized.
Q
However,
you
have
got
an
idea
today
on
which
mechanisms
yeah
about
which
mechanisms
we
think
and
my
question
today
is:
is
there
any
mechanism
which
is
missing
so
far,
and
my
second
question
is
how
to
proceed
with
this
trough?
Do
you
see
any
value
in
that
yeah
and
with
that
I'm
ready
for
today
and
now,
I'm
would
like
to
see
some
feedback.
A
I
don't
see
anybody
at
the
queue
and
I
don't
want
to
spend
a
lot
of
time
waiting,
but
I'll
give
it
just
30
seconds
if
somebody
wants
to
come
up
to
the
mic
and
offer
feedback
in
the
meantime,
I'll
I'll
offer
marcus
that
your
question
about
how
do
you
want
to
proceed
with
this
draft?
I
think
the
one
thing
I
would
recommend
is
is
is
engaging
the
group
on
the
mailing
list
and
and
seeing
if
we
can
generate
discussion.
A
That's
always
a
good
way
to
to
to
to
get
people
interested
andre
is
in
the
queue
europe.
A
R
Okay,
sorry,
andre
bondi
software
performance
and
scalability
consulting
in
new
jersey.
My
question
for
marcus
is:
how
do
you
go
about
setting
the
adaptive
expiration
time?
What
are
the
variables
that
go
into
that?
Is
this
something
that
you're
going
to
do
at
connection
setups,
that
you
can
establish
what
a
floor
on
the
latency
would
be?
And
how
do
you
go
about
increasing
the
expiration
time?
Because
if
you
make
it
too
large,
there's
going
to
be
an
issue
and
then
there's
a
problem
of
degraded
throughput.
Q
Yeah
very
good
question.
Thank
you
for
that
short
answer.
Here.
We
have
some
implementation
in
the
multi-pass
gccp
available
for
that
and
there
we
continuously
update
the
timer
with
rtt
information
we
get
from
the
sender
during
transmission,
so
we
have
some
signaling
mechanism
implemented
for
that
and
second
point:
you
are
totally
right.
We
have
to
set
some
boundaries,
so
it
doesn't
make
sense
to
let
the
reordering
queue
grow
to
an
infinite
value
that
would
slow
down
the
total
throughput.
So
one
measure
could
be
here
to
set
a
manual
limit.
Q
Yeah
yeah.
That
is
exactly
why
I
think
we
need
this
draft,
and
that
is
a
question
which
should
should
be
mentioned
there
and
then
it's
on
on
the
particular
protocol
to
think
about.
Should
this
be
something
which
is
in
or
out
of
bed,
so
we
we
would
not
give
a
final
or
I
don't
see
that
we
have
to
give
a
final
recommendation
to
a
particular
protocol.
That
is,
we
just
give
some
guidelines
and
and
such
questions
you
have
put
now,
should
this
be
something
protocol
specific
or
out
of
band
or
whatever.
Q
That
then
has
to
be
decided
within
the
particular
working
groups.
Q
Oh,
that
is,
that
is
now
very
specific.
Ss7
from
my
knowledge,
uses
stdp
right.
A
G
Okay,
great
all
right,
so
thanks,
jana
yeah.
We
just
wanted
to
give
a
quick
update
on
the
bbr
related
work.
That's
going
on
in
our
team
at
google,
and
this
is
joint
work
with
my
colleagues
listed
here.
G
G
So
I
think
this
will
be
a
a
shorter
talk
than
many
of
our
recent
talks
at
the
itf
just
wanted
to
give
a
quick
update
on
the
deployment
status
at
google,
where
we're
nearing
completion
for
internal
tcp
traffic,
give
a
quick
update
on
the
alpha
open
source
release
on
github
talk
about
some
plans
with
respect
to
internet
drafts
and
then
also
talk
about
our
continued
work
on
what
we're
calling
pbr
netswift
and,
as
always,
you
know
we're
just
offering
this
in
in
spirit
of
offering
our
experience
with
these
kinds
of
experiments
in
deployment.
G
And
of
course
we
are
always
looking
for
feedback
or
test
results.
Issues
people
run
into
any
ideas
or
patches
folks
want
to
contribute
would
be,
would
be
great.
Next
libraries.
G
So,
in
terms
of
the
ongoing
deployment
of
the
bbr
v2
algorithm
within
google
for
google
internal
traffic,
this
is
coming
pretty
far
along
right.
Now,
we're
in
the
process
of
deploying
vbr
v2
as
the
default
tcp
congestion
control
for
internal.
B
G
Traffic
and
we've
gotten
to
a
point
recently
where
it's
used
as
the
congestion
control
algorithm
for
over
98
of
the
internal
tcp
traffic,
as
measured
by
traffic
raid
or
traffic
volume,
just
to
be
clear
here
so
for
this
internal
traffic,
we're
using
a
number
of
different
congestion
signals.
G
You
know
we're
using
the
core
bbr
approach
of
modeling
the
bandwidth
and
running
around
trip
time,
but
we're
also
using
ecn
and
loss
as
signals
as
well
and
as
we
deploy
this
we're
seeing
some
latency
reductions
at
the
tail
for
rpc
traffic,
and
this
is
as
compared
to
the
previous
congestion
control,
which
for
internal
traffic,
was
based
on
a
shallow
threshold,
ecn
algorithm
and
then
we
also
have
ongoing
work
for
looking
at
bbr
v2
for
google,
external
traffic,
so
basically
youtube
and
google.com
traffic
over
the
public
internet
to
end
users,
and
that
work
continues.
G
We're
seeing
some
reduced
camera
delays
and
reduced
packet
loss
versus
vbr
v1,
but
we're
still
not
quite
where
we'd
like
to
be,
and
so
we're
continuing
to
work
on
that
and,
of
course,
we're
continuing
to
iterate
in
internal
lab
tests
and
experiments
as
well.
Next
slide,
please.
G
So,
in
terms
of
the
status
of
the
algorithm
and
code,
as
we've
mentioned
at
the
ietf
a
few
times
before,
we've
got
a
release
of
the
both
the
quick
and
the
linux
tcp
code.
That's
available.
The
linux
tcp
code
is
on
github
and
we've
made
a
couple
of
recent
minor
updates
to
that
code.
Rebased
it
onto
a
more
recent
version
of
linux.
For
those
who
are
interested
and
posted
a
few
minor
bug
fixes,
and
we
think
the
the
pbrv2
alpha
releases
is
ready
for
experiments
and.
G
Reports,
you
know
over
the
past
year
or
two
from
other
folks
in
industry
and
academia
who
have
taken
a
look
at
it,
both
in
production
settings
and
in
lab
settings,
and
there
are
just
some
links
in
the
slides
to
previous
talks
where
we've
given
more
details
about
the
algorithm
and
the
code
and
how
it
behaves
next
slide.
Please.
G
So
we
there
have
been
a
number
of
requests
for
updating
the
apr
internet
drafts
they
the
ones
that
are
out
there
currently
document
the
version
one
of
the
algorithm,
and
we
are
now
planning
to
go
ahead
and
update
those
to
reflect
bbr
v2
and
our
goal
is
to
get
those
out
there
by
the
july
ietf.
G
So
we
can
present
those
and
discuss
those,
and
you
know
the
idea,
I
think,
is
to
just
replace
the
drafts
that
are
up
there
with
drafts
that
are
targeting
bbrv2,
hoping
that
that
will
make
it
more
clear
that
these
are
sort
of
replacing
the
the
earlier
drafts
next
slide.
G
G
So
we
did
want
to
mention
briefly
that
we
are
also
continuing
another
thread
of
research
work
that
we
discussed
briefly
at
the
november
ietf
on
a
an
approach,
we're
calling
for
now
bbr.swift,
which
basically
leverages
approaches
from
the
swift
congestion
control
algorithm,
which
was
presented
at
sitcom
2020,
where
the
approach
basically
uses
the
network
round
trip
time
as
the
primary
congestion
signal
and
there
the
the
main
motivation
is
that
it
gives
you
for
environments
where
it's
available.
G
It
can
give
you
a
richer
signal
with
more
information
about
the
current
degree
of
cueing
along
the
path
which
has
sort
of
two
advantages.
One
is
that
it
allows
faster
reaction
if
there
really
are
long
queues
right
now
and
then
the
other
is
that
it
allows
you
to
help
avoid
overreaction
when
the
cues
may
be
persistent,
but
might
be
short.
For
example,
and
you
know,
there's
ongoing
work.
G
Preparing
for
production
testing
doing
some
lab
testing
right
now
is
the
main
focus,
as
we
mentioned.
Tcpm
part
of
this
research
and
development
effort
includes
work
to
provide
time
stamp
information
in
the
tcp
options
to
provide
the
sort
of
detailed,
fine
grain
and
more
accurate
round
trip
time
measurements
that
you
need
for
a
scheme
like
this.
So
there's
a
link
to
the
extensible
time
stamp
draft
that
we
put
out
last
fall
and
ultimately,
the
goal
is
to
allow
this
as
an
optional
approach
for
contexts
where.
R
G
For
example,
if
you
have
separate
quality
of
service
cues
and
you
can
isolate
this
traffic
to
its
own
queue,
to
avoid
interaction
with
other
classes
of
algorithm,
and
we
do
ultimately
want
this
to
be
usable
for
physical
machines
and
virtual
machines,
which
will
take
some
work
to
plumb
the
time
stamp,
information
up
and
down
the
stack
to
make
these
timestamps
available,
but
that
is
kind
of
the
long-term
goal
next
slide.
Please.
G
So,
just
in
conclusion,
wrapping
up
you
know
we're
continuing
to
work
on
both
bbr
v2
and
this
newer
approach.
Bbr
swift
and
we're
finishing
the
rollout
for
internal
tcp
traffic
for
bbr,
v2,
continuing
to
iterate
on
external
traffic
or
public
internet
performance
and
are
hoping
to
release
an
internet
draft
in
july
and
as
always,
we
invite
feedback
or
test
results,
issues
patches.
Anything
like
that
and
next
slide.
Please,
and
I
think
that's
that
wraps
it
up.
Also,
if
there
are
any
questions.
A
Questions
I
am
going
to
try
a
new
experiment
this
time,
which
is
that
I
will
cut
off
the
q
and
a
at
at
in
about
in
just
under
five
minutes,
doesn't
matter
who's
in
the
line.
So
I'm
not
going
to
cut
the
mic
line,
but
I'm
going
to
cut
off
the
q
a
all
right
omer.
You
are
up.
S
Can
you
hear
me
yes
thanks
dan?
Can
you
share?
Do
you
have
any
data
on
what
kind
of
end
devices
bbr
v2
is
deployed
on,
for
the
external
users,
are
chromebooks
androids
or
something
else,
and
but
interests
me
the
masters,
how
bb2
interacts
with
the
modern,
modern
cellular
networks?
G
Sure
yeah
so
to
add
some
details
there
so
where
we're
deploying
bbr
v2
in
for
the
types
of
traffic
you're
talking
about
is
for
the
google.com
and
youtube
servers
that
are
sending
traffic
out
over
the
public
internet
to
end
users,
and
so
since
this
is
basically
talking
about
all
of
the
users
using
youtube
and
google.com
or
currently
just
some
a
small
percentage
of
them
for
testing.
G
This
basically
should
be
a
cross-section
of
of
every
kind
of
device
that
connects
to
google
and
youtube,
and
in
our
experience
of
course,
that's
a
pretty
diverse
set
that
largely
these
days,
I
think
largely
the
the
dominant
bottleneck
technology
is
wi-fi,
but
obviously
we
do
have
a
lot
of
cellular
users
as
well.
G
On
the
cellular
side,
I
think
it's
still
mostly
3g
and
4g.
Although
we're
starting
to
see
you
know
obviously
some
5g
trickling
in
now,
and
so
the
I
don't
have
any
numbers
to
share
with
you
today,
mainly
because
this
the
public
internet
aspect
is
still
a
work
in
progress,
but
I
can
share
that.
G
A
All
right
well,
I
will
thank
neil
then
for
his
presentation,
and
I
will.
I
won't
speak
for
everybody
here,
but
I
speak
for
many
people
that
very
excited
about
the
the
update
to
the
draft.
Looking
forward
to
reading
updates
to
the
draft
to
the
pbr
draft
view.
A
B
P
Excellent
okay,
so
this
short
talk
is
going
to
look
at
zero
rtt
parameters
for
quick,
basically
to
exchange
transport
parameters,
to
let
you
do
something
different
in
congestion,
control
and
there's
a
draft.
It's
a
revision,
seven,
and
it's
with
these
people
on
the
side,
nicholas
emile
tom
and
me
next
slide.
Please.
P
So
this
is
a
draft
that
tries
to
deal
with
paths
which
are
not
typical,
so
we're
talking
about
paths
that
have
something
that's
different
in
them
and
maybe
they're
slightly
higher
in
delay.
That
could
be
many
tens
of
milliseconds.
It
could
be
many
seconds.
They
maybe
have
a
very
large
bandwidth
delay
product.
P
P
Our
focus
was
primarily
on
satellite
paths
and
that
now
covers
a
very
wide
range
of
paths,
and
we
focus
just
on
geo
in
in
the
initial
work
here
and
but
you
might
see
other
paths
that
have
similar
needs
and
I
think
that's
one
of
the
important
things
I'd
like
to
kind
of
bring
up
here
is,
if
other
people
are
seeing,
paths
that
have
other
needs,
maybe
slightly
different
characteristics.
We'd
love
to
talk
to
you,
because
we'd
love
to
make
sure
whatever
we
propose
here
actually
works
on
across
a
variety
of
different
paths.
P
The
context
is
to
define
some
transport
parameters
as
extensions
to
quick,
and
these
are
shared
during
the
zero
rtt
phase,
basically
allowing
resumption
using
additional
transport
and
connection
properties
discovered
from
a
previous
successful
connection
and
what
this
is
a
lot
like.
Tcp
control
block
sharing,
but
it's
also
different,
because
it's
designed
for
quick.
P
The
core
thing
we're
going
to
talk
about
is
using
the
information
to
make
a
jump
in
the
sea
wind,
the
congestion
window,
so
that
you,
instead
of
starting
a
session
with
a
configured
large
initial
window
or
with
a
normal
small
initial
window.
You
choose
something
which
is
based
on
previous
history
and
use
that
to
initialize
a
safe
sea
wind
and,
like
any
other
method,
that's
used
with
tcp,
for
instance,
we'd
like
this
information
to
be
shared
across
multiple
connections.
P
P
And
why
is
this
important?
If
you
have
the
sort
of
bdp
and
delay
that
you
might
see
in
a
satellite
geo
environment,
then
it
might
take
you
many
seconds
to
download
something
which
you
could
actually
send
using
tcp
and
in
maybe
a
small
number
of
seconds.
P
So
maybe
we
can
do
much
better,
and
this
this
slide
shows
two
methods
which
could
be
used
to
improve
performance.
That
we've
tried,
in
a
spreadsheet
analysis,
using
a
a
little
tool.
We
have
to
look
at
different
combinations
of
parameters
and
we
see
the
orange
one
is
a
jump
to
25
of
the
last
window
and
then
two
rounds
of
rtt
to
get
to
the
full
size
window
and
green,
a
high
jump
method
where
we
delay
it
by
one
rtt.
So
we
make
the
jump
more
conservatively.
P
This
range
of
options
in
the
orange
green
area
all
the
way
across
the
glue
area,
where
you
can
trade
performance
against
conservative
congestion,
control,
behavior,
and
that's
the
good
reason
for
presenting
this
in
iccrg,
because
a
lot
of
the
issues
are
concerned
with
how
best
to
adapt.
But
before
you
adapt,
you
need
a
method
of
having
the
data
about
the
previous
connection.
P
P
P
P
The
most
dangerous
is
the
change
of
the
path,
but
the
method
we
propose
will
validate
the
rtt
against
the
previous
rtt
and
we're
suggesting
that
we
include
some
form
of
pacing
when
we
initially
start
a
new
higher
congestion
window,
and
in
this
way,
if
there
is
a
big
rtt
change
on
the
path,
then
the
damage
that's
created
is
very
much
limited
and
we
believe
could
be
made
safe
for
wide-scale
deployment
and
therefore
something
that
might
be
interesting
to
standardize
next
slide.
P
Please
this
is
the
set
of
metadata
we
expect
to
put
in
the
bdp
metadata
three
parameters:
bytes
in
flight
minimum
rtt
encountered,
which
is
partly
to
configure
as
a
safeguard,
but
also
to
configure
a
pacing
interval
and
one
of
the
issue.
One
of
the
problems
is,
is
initializing
the
rtt
of
the
pacer,
so
this
information
is
quite
helpful
in
getting
a
good
response
and
the
maximum
packet
number
encountered.
P
Well,
we've
approached
this
in
various
ways.
One
way
was
to
perform
some
implementation
work
in
pico,
quick
and
there's
a
github
that
you
can
use
to
access
this.
This
is
primarily
focused
on
the
exchange
of
the
cryptographic
information
at
the
start
of
quick
of
a
quick
connection,
so
that
you
can
actually
get
the
bandwidth
parameters
exchanged
and
it
focused
on
a
very
simple
easy
to
implement
change
to
the
congestion
controller.
We
have
a
more
advanced
version
of
how
we
expect
that
congestion
control
update
to
occur
in
the
draft.
P
So
please
leave
there
to
find
more
details
about
what
we
actually
suggest,
but
it's
clear
even
from
these
three
simple
results:
that,
with
out
the
option,
it
took
four
point
three
seconds
to
exchange
the
two
megabyte
chunk
of
data
on
average
as
the
median
value
with
the
zero
rtt
enhancement
3.4
and
with
the
zero
rtt
vdp
and
2.9
seconds,
so
at
least
saving
what
we
would
see
as
people
evaluating
satellite
links,
a
significant
proportion
of
the
download
time-
and
these
are
for
modern
satellite
length
running
at
50
megabits
per
second
next
slide.
P
Please,
with
also
looking
at
how
the
client
might
use
this
information,
because
the
server
can
always
optimize
and
it
be
nicely
optimized
in
a
predictable
way.
So
the
user
didn't
have
to
be
concerned
about
it,
but
also
there's
a
possibility
to
optimize
the
client
if
it
knew
about
the
likely
bdp
aspects
of
the
path
it's
using
and
we
did
some
work
in
2018,
which
is
published
in
the
netsat
days
conference.
P
We
could
take
this
information
as
one
of
the
inputs
to
adapt
the
results
to
produce
a
much
more
predictable
performance
in
this
case,
trying
to
avoid
the
strange
behaviors
that
happen
when
your
predictor
gets
the
capacity
wrong
and
therefore
vastly
under
underestimates
the
amount
of
capacity
you've
got
because
you've
got
a
larger
rtt
and
there
was
talks
in
the
irtf
open
about
the
various
ways
in
which
dynamic
adaptive
streaming
players
can
play
out,
and
this
is
kind
of
like
one
of
the
input
parameters
and
a
good
example
of
how
knowing
something
at
the
client
can
help.
P
P
P
P
That's
25
of
the
previously
used
capacity.
That
means
the
following:
rtt:
there
is
a
step
up
to
use,
half
the
capacity
and
then
another
step
up,
etc.
Until
after
several
rtts,
we've
used
the
whole
capacity
now,
it
would
be
possible
to
jump
immediately
to
use
the
full
capacity,
but
if
you're,
a
congestion
control
person
that
might
frighten
you
and
because
it
would
cause
severe
congestion
against
any
other
flaws.
That
happened
to
be
present
at
the
time
when
your
new
flow
started,
rather
than
when
you
previously
measured
your
available
capacity.
P
P
P
P
P
P
P
There's
some
interaction,
emil
is
our
tls
person
and
there
are
probably
things
that
we
should
discuss
between
quick
and
tls
to
see
how
this
initial
exchange
of
data
should
best
be
handled,
and
I
think
some
form
of
synchronization
between
the
two
working
groups
is
important.
P
P
Congestion
control
safety
is
the
thing
which
probably
is
most
important
for
this
group,
and
if
we
can
standardize
or
adopt
a
way
of
exchanging
this
information,
we
also
need
to
adopt
a
way
of
safely
using
it
and
we're,
assuming
that
any
method
we
use
here
will
have
a
way
of
backing
out
quickly
and
efficiently.
As
soon
as
there
is
loss
detected
and
the
cw
d
has
been
artificially
increased,
so
we
will.
We
will
expect
to
quickly
back
out
of
any
problem,
but
do
we
need
a
draft
on
congestion
control
safety
that
updates
6928?
P
We
could
use
a
new
bdp
extension
specified
in
quick,
and
we
are
wanting
to
do
that
part
of
the
work
in
quick,
but
there's
obviously
also
a
congestion
control
piece,
which
is
why
we're
trying
to
bubble
this
up
here
in
this
group
and
attract
some
comments
so
trying
to
stick
roughly
to
time.
I'd
like
to
take
comments
now,
if
possible,.
A
A
I
want
people
to
engage
on
this
question
and
I'll
take
my
my
my
moment
to
basically
say
that
this
is
something
that
we
expect
the
the
the
us
at
a
specific
mechanism,
but
the
idea
of
recording
and
reusing
constitutional
control
information
is
something
that
we
expect
will
happen
in
quick,
because
there
are
places
to
store
this
information
at
the
client.
This
is
something
that
is
different
from
pcb,
where
a
server
can
actually
encode
this
information
and
ship
it
off
to
a
client
and
then
use
it
on
the
next
connection.
A
When
connection
is
established,
which
makes
it
much
more
likely
that
something
like
this
will
get
deployed
by
quick
implementations.
So
it
is
much
more
important
now
that
we
actually
engage
on
this
topic.
Iccrg
is
the
right
forum.
Perhaps
we
can
continue
this
discussion
on
the
list
and
at
subsequent
at
the
next
meeting
as
well,
but
I'll
thank
gauri
also
for
putting
this
together
very
quickly.
At
short
notice,
do
you
want
to
say
something
cody
yeah.
P
I'd
just
like
to
say
that
we
are.
We
are
super
interested
in
not
doing
this
as
a
group
of
satellite
engineers,
who've
worked
on
peps
and
enhancement
and
modelling
of
really
long
delay
paths,
but
to
do
it
within
the
ietf
where
we
can
get
other
people
involved
in
this.
This
is
a
super
interesting
place
where
we
can
actually
collaborate
between
different
people
and
we'd
really
love
to
get
feedback
on
this.
A
Thank
you
so
much
gauri
again
for
the
presentation.
One
quick
announcement
bob
just
announced
that
a
new
draft
describing
product
construction
control
has
been
posted.
Please
take
a
look.
We
might
end
up
discussing
that
at
the
next
idea
at
the
next
iccrg
meeting.
Thank
you
again
everybody
and
enjoy
the
rest
of
the
itf.
We'll
see
you
next
time.