►
From YouTube: IETF 111 IRTF Open
Description
The Internet Research Task Force (IRTF) Open session will be held at IETF 111 on 26 July 2021 at 21:30 UTC. It will include the Applied Network Research Prize (ANRP) presentations by Rüdiger Birkner for his work on network specification and verification ( “Config2spec: Mining Network Specifications from Network Configurations”, Proceedings of USENIX NSDI 2020) and Sadjad Fouladi for his work on low-latency video streaming (awarded in 2020) (“Salsify: low-latency network video through tighter integration between a video codec and a transport protocol”, Proceedings of USENIX NSDI 2018).
A
A
A
All
right,
in
that
case,
we
should
get
started.
Welcome
everybody.
This
is
the
irtf
open
meeting.
My
name
is
colin
perkins
from
the
university
of
glasgow,
I'm
the
irtf
chair
and
welcome
to
the
irtf
sessions
at
itf
111.
A
So
to
start
with
the
the
usual
set
of
reminders,
we
have
the
the
no
well
the
intellectual
property,
a
reminder
that,
by
participating
in
the
irtf,
you
agree
to
follow
the
irtf
processes
and
policies
and
in
particular,
you
agree
to
disclose
any
ipr
that
you're
aware
of
following
the
the
terms
on
the
that
are
linked
from
the
slide.
So
this
is
the
the
usual
itf
intellectual
property
disclosure
rules.
A
In
addition
to
that,
the
irtf
routinely
makes
available
recordings
of
the
these
online
meetings,
including
audio
video
and
photographs,
and
we
record
and
publish
those
meet
those
those
recordings
online,
we're
being
streamed
in
in
mitako
and
we're
also,
I
believe,
going
out,
live
on
youtube
for
this
session.
A
A
In
addition
to
that,
we
have
a
a
code
of
conduct,
we
have
a
privacy
policy
and
so
on.
A
If
you
release
any
personal
information
to
the
irtf
to
be
handled
in
accordance
with
the
ietf's
privacy
policy
by
participating,
you
agree
to
work
respectively
respectfully
with
the
other
participants,
and
if
you
have
any
concerns
about
that,
the
itf
onpads
team
can
hopefully
help
you
out
there
and
we
have
a
code
of
conduct
in
the
ante
and
a
set
of
anti-harassment
procedures
for
the
ietf
and
those
also
apply
to
the
irtf
sessions.
A
So
what
is
the
the
irtf?
The
I
rtf
is
the
the
the
research
arm
of
the
ietf.
It
focuses
on
longer-term
research
issues
relating
to
the
internet,
while
the
itf
focuses
on
on
shorter
term
issues
relating
to
engineering
and
standards.
Making
the
irtf
is
is
a
research
organization,
it's
not
a
standards,
development
organization
and,
if
you're
not
familiar
with
the
way
the
irtf
works.
A
If
you
want
to
stay
informed
about
what
we're
doing,
we
have
a
website,
we
have
the
usual
social
media.
We
have
a
reasonably
active
twitter
account
and
much
less
active
facebook
accounts
and,
I
believe,
also
a
linkedin
account.
Now.
We've
also
got
a
low
volume
announcement
list.
If
you
want
to
know
what
the
irtf
is
doing
and
there's
a
general
discussion
list.
In
addition
to
the
research
group
lists,
the
main
activities
of
the
irtf
are
organized
as
a
set
of
research
groups.
A
The
the
current
set
of
research
groups
is
listed
on
the
slide
of
these
groups.
The
the
privacy
enhancements
and
assessments
research
group
met
in
the
previous
session.
The
the
rest
of
the
groups,
which
are
highlighted
in
dark
blue
on
this
slide,
are
meeting
later
in
this
week.
So
do
do
look
out
for
those
and
do
join
the
sessions
if
you're,
if
you're
interested
in
any
of
those
topics.
A
The
iota
irtf
can
publish
rfcs,
we,
we
don't
publish
standards
track
rfcs,
but
we
can
publish
informational
and
experimental
rfcs.
We've
had
two
rfcs
published
since
the
last
meeting,
the
the
privacy
enhancements.
Sorry,
the
the
pathway,
networking
research
group
published
the
the
path
aware,
networking
obstacles
to
deployment
rfc,
which
is
a
nice
summary
of
some
of
the
issues
with
trying
to
deploy
pathway,
networking
technologies
over
the
years
and
trying
to
learn
some
lessons
from
those
experiences
and
dave
iran.
A
In
the
information
center
centric
networking
research
group
had
an
rfc
looking
at
considerations
on
the
deployment
of
quality
of
service
architectures
for
ccnx,
like
intermissions
and
networking
protocols.
A
In
addition
to
the
research
groups,
there's
two
other
activities
which
we
do.
The
first
of
these
is
the
applied
networking
research
prize,
which
will
be
the
the
main
focus
of
this
this
session.
Today,
the
the
prizes
is
awarded
to
recognize
the
best
results
and
the
the
most
recent
results
in
applied
networking.
A
It's
awarded
to
to
recognize
interesting,
new
research,
ideas
of
potential
relevance
to
the
internet
standards
community
and
to
recognize
upcoming
people
that
are
likely
to
have
an
impact
on
internet
standards
and
technologies
and
we're
trying
to
bring
in
people
who
who
not
otherwise
get
much
exposure
would
not
have
always
participate
in
the
the
ietf
and
the
irtf
and
try
and
get
a
dialogue
going
between
the
research
community
and
the
irtf
community
and
the
standards
development
community
in
the
atf.
A
The
main
focus
of
this
meeting
will
be
that
we
have
two
a
nrp
prize
winning
talks
to
get
today.
Rudiger
will
talk
about
his
work
on
network
specification
and
verification.
First
of
all,
and
then
in
the
second
half
of
the
meeting
said
we'll
talk
about
low
latency
video
streaming.
We've
got
some
really
nice
talks
coming
up
on
those
topics
and
again
thank
you
to
the
the
internet
society.
Thank
you
to
comcast
and
to
nbc
universal.
A
A
In
addition
to
the
applied
networking
research
prize,
we
also
run
the
applied
networking
research
workshop,
and
this
is
in
conjunction
with
acm
sitcom
and
that's
running
concurrently
with
with
the
itf
this
time.
The
first
of
those
sessions
was
in
the
the
previous
slaughter
over
the
last
couple
of
hours.
A
The
the
the
applied
networking
research
workshop
is
an
academic
workshop,
provides
a
forum
for
researchers,
vendors
network
operators
and
the
standards
community
to
present
and
discuss
emerging
results
in
applied
networking
and
the
that.
A
As
I
say,
the
the
this
year's
workshop
is
running
concurrently
with
with
this
itf
and
the
program
and
all
the
papers
and
the
recordings
are
on
the
website
linked
and
I'd
like
to
thank
andra
and
nick
who've
been
organizing
the
the
workshop
this
year
and
thank
you
to
akamai
and
comcast
and
acm
sitcom,
who
are
the
sponsors
of
the
workshop.
A
So
that's
that's
about
all.
I
have
to
say
what
we
have
for
the
remainder
of
this
session
is
the
two
applied
networking
research
price
talks,
starting
with
rudiger
berkner
he'll,
be
talking
about
conflict
to
spec
and
then,
following
on
with
subject
for
lady,
who
will
be
talking
about
sulsify
before
we
get
to
that,
though
I
I
believe
matt
ford
from
the
internet,
society
wants
to
to
say
a
few
words
to
the
to
the
prize
winners.
B
Thanks
very
much
colin,
I
just
I'll
keep
this
brief.
B
Listen
to
the
award-winning
talks,
it's
a
great
honor
and
privilege
for
the
internet
society
to
work
with
the
irtf
to
deliver
the
applied
networking
research
prize,
as
colin
mentioned,
there's
sort
of
two
purposes
to
this
program
really
to
get
exciting
new,
applied
networking
research
in
front
of
the
irtf
and
ietf
community,
and
also
to
introduce
new
upcoming
researchers
to
to
the
community
as
well.
And
if
that
sounds
like
something
that
that
you
and
your
your
day,
job
company
would
like
to
support.
We
very
much
welcome
additional
sponsors
for
this
effort.
B
The
only
other
thing
I
have
to
say
is
I'll
be
trying
to
take
notes
in
the
kodi
md.
So
if
you
do
speak
at
the
microphone,
you
might
want
to
check
that
to
make
sure
I've
captured
your
name
and
your
comment
correctly.
Thanks
very
much.
A
Okay,
thank
you,
matt!
Congratulations
again
to
the
two
presenters
we'll
have
the
the
first
video
now,
if,
if
meet,
echo
can
play
the
video
which
is
by
rudiger
berkner
rudiger,
is
a
finally
a
phd
student
at
btu
zurich
advised
by
laurence
van
ber
that
van
beaver
and
martin
and
his
work
is
centered
around
developing
methods
to
improve
network
understanding
and
help
the
adoption
of
new
network
analysis
and
synthesis
tools.
A
If
you
have
questions
we'll
get
a
couple
of
minutes
after
the
talk
or
put
them
into
the
chat
and
I'll
encourage
rudiger
to
join
the
chat
as
well.
Okay,
thanks,
if
we
can
play
the
talk.
C
Hi
everyone,
my
name,
is
rudiger
bergner
and
today
I'm
going
to
talk
about
convict,
spec,
a
tool
to
facilitate
intent-based
networking
or
at
least
a
tool
to
take
some
steps
towards
it.
For
quite
some
years,
intent-based
networking
or
short
ibn
has
been
on
everyone's
mind
and
the
idea
of
it
sounds
great.
You
specify
your
intent
and
the
network
just
behaves
accordingly,
no
more
cumbersome
configurations,
you
need
intent
and
you're
done.
C
C
For
example,
there
are
net
network
configuration,
synthesis
tools
that
allow
operators
to
automatically
produce
network
configurations
according
to
their
intent,
and
there
are
many
configuration,
validation
and
verification
tools
that
allow
to
check
whether
your
configuration
satisfies
your
intent
or
not.
Some
of
these
tools
made
it
even
and
are
commercially
available.
C
C
C
C
C
C
C
So,
for
example,
our
specification
could
consist
of
the
following
policies
for
the
given
network.
But
what
does
it
actually
mean?
These
policies
should
hold?
The
network
can
be
in
different
states,
routers
and
links
can
fail,
and
the
network's
behavior
will
change
for
sure.
The
policies
will
all
hold
when
all
the
links
are
up,
but
what,
if
router,
r1
or
some
link
fails
or
what?
If
all
the
link
fails,
do
the
policy
still
have
to
hold
so
clearly,
just
having
a
set
of
policies
is
not
enough.
C
We
also
need
a
context,
a
context
under
which
these
policies
have
to
hold.
So
we
can
just
extend
our
definition
of
the
specification
by
not
only
requiring
the
set
of
policies,
but
also
by
requiring
that
the
set
of
policies
have
to
hold
under
a
given
failure
model.
Now
this
failure
model
provides
the
missing
context.
C
C
So,
therefore,
you
can
actually
capture
such
a
failure
model
in
two
parts,
in
with
a
symbolic
environment
and
a
failure
bound,
the
symbolic
environment
is
basically
the
network
topology
in
which
every
link
is
assigned.
One
of
three
states
up
down
or
symbolic
and
symbolic,
simply
means
that
the
link
can
be
either
up
or
down
and
the
second
part
the
failure
bound
simply
provides
a
bound
on
the
maximum
number
of
links
that
can
or
are
expected
to
be
down
at
the
same
time.
C
Unfortunately,
this
is
not
the
case,
while
the
definition
of
a
specification
is
actually
quite
simple.
Writing
down.
The
complete
specification
is
actually
extremely
difficult,
and
this
is
not
just
the
network
in
the
case,
but
in
general,
as
this
tweet
of
a
well-known
researcher
shows,
the
need
for
specification
often
prevents
people
from
using
verification
tools.
C
You
might
still
wonder
and
ask
yourself:
is
it
really
that
hard
to
write
down
how
you
want
your
network
to
behave?
Well,
just
take
internet
2
as
an
example.
As
part
of
this
project,
we
analyze
the
publicly
available
configuration
from
2015
and
found
that
its
specification
is
made
up
of
almost
4
000
policy
predicates.
Now.
Imagine
writing
this
specification
by
hand
and
doing
this
without
mistake,
or
if
you
are
a
network
operator,
would
you
know
all
the
policies
of
your
network
or
could
you
fully
describe
its
behavior?
C
C
So
all
the
policies
that
your
network
enforces
in
the
following,
we
will
first
consider
two
strawman
approaches
that
individually
explore
one
dimension
of
the
problem
at
the
time.
Then
I
show
you
how
we
combine
these
two
approaches
to
leverage
their
individual
strengths
and,
in
the
end,
we'll
talk
about
config
to
specs
performance,
so
how
long
it
actually
takes
to
compute
such
a
specification.
C
So,
let's
get
started
with
the
straw
man
solutions
when
you
think
back
to
the
specification,
you
remember
that
we
only
want
to
have
those
policies
from
the
space
of
all
policies
that
hold
for
every
single,
concrete
environment
in
the
failure
model.
So
we
kind
of
have
two
dimensions
or
we
have
kind
of
two
search
spaces.
One
is
the
failure
model,
so
all
the
concrete
environments
that
it
contains
and
the
other
one
are
all
the
policies.
So
it's
a
space
of
all
possible
policy
together
in
combination
this
becomes
untractable.
C
However,
we
can
look
at
the
two
in
isolation
when
we
look
at
it
from
the
perspective
of
the
failure
model.
So
when
we
look
at
all
the
concrete
environments,
we
can
make
use
of
data
plane
analysis
and
when
we
look
at
it
from
the
perspective
of
the
policies,
we
can
make
use
of
control
plane
verification.
C
So
what
do
I
mean
by
that?
Let's
first
look
at
data
plane
analysis
at
the
high
level.
We
can
break
down
the
problem
of
finding
the
specification
for
a
failure
model
and
the
configuration
into
finding
the
specification
for
each
concrete
environment
within
the
failure
model,
and
then
we
just
combine
the
outputs.
C
So
we
start
with
one
concrete
environment
from
the
failure
model
and
together
with
the
network
configuration
we
compute,
the
forwarding
state
using
data
plane
analysis
from
the
forwarding
state.
We
can
easily
determine
whether
two
nodes
are
reachable
isolated
and
so
on
so
below
here
I
show
you
kind
of
a
abstract
illustration
of
this.
The
blue
area
represents
the
space
of
all
possible
policies
after
running
data,
plane,
analysis
and
finding
all
the
policies
for
this
one,
concrete
environment.
C
C
So
when
we
now
look
at
the
problem
for
from
the
perspective
of
the
policies
and
we
use
control,
plane
verification,
we
have
again
the
same
situation.
We
have
a
large
number
of
candidate
policies,
we
have
the
failure
model
and
the
configuration
and
with
modern
verification
tools.
We
can
easily
check
if
a
single
policy
holds
for
the
entire
failure
model
or
not.
C
C
Now
we
can
do
this
for
every
single
possible
policy
separately,
verify
them
one
by
one
and
ultimately
obtain
the
full
specification
by
taking
the
union
of
this.
So
here
we
kind
of
under
approximate
the
specification
and
increase
it
step
by
step
until
we
reach
the
full
specification
so
to
quickly
recap:
both
techniques
have
their
pros
and
cons
data
plane
analysis
can
find
all
the
policies
that
hold
for
a
single,
concrete
environment,
whereas
control
plane
verification
can
check
whether
one
policy
holds
for
the
entire
failure
model.
C
So
the
strength
of
data
plane
analysis
is
actually
to
quickly
cut
down
the
candidate
space.
As
with
checking
one
single
concrete
environment,
one
can
already
rule
out
a
lot
of
policies
and
control
plane.
Verification
is
very
good
at
verifying
a
small
set
of
candidate
policies
and
quickly
identifying
which
belong
to
the
specifications
and
which
don't
so
the
natural
question,
or
actually
I
also
I
already
led
to
this-
is
why
don't
you
combine
the
two
approaches,
and
this
is
exactly
what
we
do
so
internally.
Config
to
spec
consists
of
multiple
modules.
C
C
C
C
It
will
check
whether
these
policies
hold
for
the
fader
model,
some
won't
some
will
and
in
continuous
one
policy
after
another,
until
we
have
checked
them
all
and
config
to
spec
terminates,
and
we
have
found
the
full
specification
so
to
make
convicted
spec
really
work.
We
optimized
both
approaches
following
I
will
just
present
the
intuition.
C
We
do
that,
based
on
the
remaining
candidate
set,
so
we
always
pick
the
next
environment
that
maximally
disrupts
the
forwarding
state
with
respect
to
those
policies
to
speed
up
the
verification.
We
can
kind
of
do
three
things.
First
of
all,
we
can
group
policies
so
instead
of
checking
them
one
by
one,
we
check
them
in
small
groups.
C
For
example,
if
we
check
a
reachability
between
two
points
and
we
see
that
it
doesn't
hold,
we
don't
need
to
check
the
way
pointing
policy
between
those
points,
because
we
know
if
the
reachability
doesn't
hold
waypoint,
they
can't
hold
either
and,
as
a
last
part,
we
can
trim
policies
without
even
verifying
them.
So
we
can
trim
policies
which
cannot
hold
because
of
topological
constraints,
just
think
of
a
failure
model
that
allows
for
two
link
failures.
C
C
Finally,
let's
look
at
how
config
to
spec
performs
to
this
end,
we
have
fully
implemented
config
to
spec
in
about
5000
lines
of
python
and
java
code,
and
we
rely
on
two
state-of-the-art
tools:
bad
fish
for
data,
plane,
analysis
and
minesweeper
for
control,
plane,
verification
to
test
config
to
spec.
We
have
generated
configurations
using
a
tool
called
netcomplete,
and
we
did
this
for
three
networks:
a
small
and
medium
and
large
one.
C
Let's
look
at
the
runtime
of
config
spec.
Let's
look
at
how
long
it
takes
to
actually
find
such
a
specification.
Now
in
the
following.
I
will
only
show
you
the
results
for
the
largest
topology,
which
has
roughly
160
routers
for
all
the
other
topologies.
The
results
were
similar.
Obviously
it
was
faster,
but
they
were
qualitatively
similar
on
the
y-axis.
We
showed
the
runtime
in
hours.
C
C
Interestingly,
we
never
use
verification
in
cases
where
we
only
allow
up
to
one
link
fader-
and
this
is
due
to
the
fact
that
there
are
only
very
few
concrete
environments,
but
a
lot
of
candidate
policies.
So
the
predictor
decides
that
it
is
faster
to
just
iterate
through
all
of
the
environments
instead
of
trying
to
verify
the
policies.
C
On
the
other
hand,
the
other
special
case
are
situations
where
we
allow
for
a
lot
of
link
failures,
because
then
the
trimming
is
extremely
efficient
and
they're,
actually
not
that
many
candidate
policies
left
that
we
have
to
check.
So
it's
faster
to
actually
just
go
through
the
policies
and
check
them
one
by
one.
Instead
of
going
through
all
the
environments.
C
Obviously,
if
you
have
a
larger
topology,
it
will
take
longer,
however,
note
that
you
don't
run
config
to
spec.
Whenever
you
perform
a
configuration
change,
you
run
config
to
spec
once
or
twice
to
learn
a
specification,
and
then
you
can
use
that
specification
for
verification,
synthesis
and
so
on
with
that
we're
already
at
the
end
of
this
talk,
so
I
presented
to
you
conflict
spec,
a
system
that
takes
on
the
challenging
task
of
automatically
mining
a
specification
from
the
configuration
in
the
failure
model
and
after
all
this
you
might
think
great.
C
C
So
if
I
raise
your
interest,
I
really
invite
you
to
check
out
our
nsti
20
paper,
as
there
are
more
details
to
convict
to
spec
than
what
I
cover
today.
In
addition,
we're
still
working
on
config
to
spec
trying
to
improve
it.
So
if
you
have
any
opinion
on
it,
if
you
think
this
is
useful,
please
let
us
know
so,
please
reach
out
to
us
either
to
my
email,
address
or
visit
our
website.
A
Yes,
so
are
there
any
questions
for
rudiger?
I
see
jonathan
had
something
in
the
chats
jonathan.
Do
you
want
to
bring
that
to
the
microphone.
D
So
I
was
just
wondering
if
we
you
said
at
the
beginning
of
the
talk
that
the
currently
you
know
you
might
have
to
write
a
4
000
line
spec
and
then
verify
and
write
in
that
spec
is
very
hard.
So
if
we
assume
that
that
4009
spec
is
the
intended
one,
then
your
tool
will
generate
me
a
4
000
line
spec.
D
C
Yes,
so
this
is
a
very
good
point,
so,
of
course
we
will
not
learn
the
intended
spec,
but
the
spec
that
your
config
implements
and
then
you
say
well,
four
thousand
predicates.
It's
way
too
much.
If
I
go
through
it,
maybe
I
forget
what
was
at
the
beginning,
how
it
interacts-
and
this
is
really,
I
think,
one
thing
that
we're
working
on
so
kind
of
what
I
hinted
at
at
the
end.
One
of
the
things
is
that
we
can
do
anomaly
detection
on
the
specification.
C
C
A
So
so
I
had
one
so
the
the
generated
specifications.
You
said
they
were
quite
low
level.
Are
they
basically
human
readable
or
are
they
too
low
level
to
be
usefully
editable
or
usefully
readable.
C
So
I
think
that
they're
usefully
readable,
but
for
a
trained
it's
so
let's
say,
like
a
network
configuration
right
that
it's
quite
low
level,
but
if
you're
trained
you,
you
know
what
it's
doing.
Yes
exactly.
It
depends
on
the
human,
but
it's
also
what's
current
specification
tools,
expect
you
to
provide
as
an
input.
So
I
hope
yes,
a
human
operator
will
be
able
to
work
with
it.
A
Sure,
okay,
okay,
is
it
possible
to
then
feed
them
into
a
synthesis
tool
and
generate
the
configuration
and
and
complete
the
loop.
C
Yes,
yes,
so-
or
this
is
one
of
the
ideas
right
that
what
they
pointed
kind
of
hinted
at
at
the
end.
So
maybe
you
want
to
switch
to
a
different
routing
protocol
switch
from
ospf
to
isis
and
want
to
make
sure
that
it
behaves
the
same
way.
Obviously
you
need
to
have
a
synthesis
tool
like
netcomplete,
for
example,
that
is
able
to
do
that.
But
then
you
can
feed
in
the
specification.
C
However,
what
is
really
kind
of
a
challenge
is
the
policy
language
or
kind
of
the
predicates
that
we
support,
because
maybe
we
cannot
cover
all
the
behaviors
that
you
want
or
the
certain
the
predicates
that
we
have
at
the
moment
are
kind
of
data
plane
policies,
but
maybe
you
have
more
control
plane
policies
like
prefer
the
routes
from
this
neighbor
over
this
neighbor.
If
we
talk
bgp
or
don't
provide
transit
for
a
certain
labor-
and
this
is
at
the
moment-
not
supported
in
conflict
is
back
sure.
Okay,.
E
C
Yes,
yes,
I
invite
you
to
take
a
look
at
it
and
also
just
reach
out
to
us
with
any
feedback
or
any
ideas
or
yes,.
E
A
Okay,
thank
you,
I'm
conscious
that
we,
we
don't
have
too
much
time
today.
So
I
see
there's
one
more
question
in
the
chat,
so
maybe
you
could
answer
in
the
chat.
What.
F
A
Thank
you
again
if
you're
around
for
the
rest
of
the
week,
I
would
perhaps
encourage
you
to
have
a
look
at
the
network
management
research
group,
which
is,
I
guess,
working
on
some
more
topics
yeah.
So
thank
you.
Thank
you.
A
Okay
and
the
the
the
next
talk
today
is
by
sajad
fuladi
who's.
A
phd
candidate
in
computer
science
at
stanford
university
advice
by
keith
winston
sajad
is
broadly
interested
in
computer
systems,
focusing
on
distributed
applications
massively
burst
parallel
computing
and
video,
and
today
he
will
be
talking
about
celsify,
which
is
a
low
latency
network,
video
codec,
which
achieves
low
latency
through
tighter
integration
between
the
codec
and
the
transport
protocol.
G
G
So
the
problem
that
we
tackled
in
salsa
was
how
these
systems
react
to
variations
in
the
network.
So
here
I'm
going
to
show
you
one
example:
we
are
doing
a
video
call
with
this
very
nice
guy
at
nasa
using
one
of
these
real-time
video
systems.
Webrtc-
and
here
you
can
see
the
network
throughput
graph-
that
shaded
region
shows
the
network
capacity
and
the
line
shows
the
amount
of
data
that
the
program
tries
to
send
over
to
network.
In
this
scenario,
the
capacity
is
slowly
going
to
go
down.
G
G
So
if
you
look
at
this
part
of
the
graph,
you
can
see,
webrtc
actually
tried
to
send
more
than
the
network
can
handle.
You
know
those
packets
are
either
going
to
be
dropped
or
cued
both
will
cause
stalls
and
glitches,
and
then
it
had
to
spend
several
seconds
to
recover
from
the
mistake
that
it
made
earlier.
You
know
a
one
second
outage
here
caused
webrtc
to
spend
several
seconds
recovering
the
video.
G
Okay,
before
I
get
to
salsa
vs
architecture,
let
me
tell
you
a
little
bit
about
the
conventional
design,
so
the
current
systems,
they
have
two
important
components.
One
is
a
video
codec
responsible
for
compressing.
The
video
and
one
is
the
transport
protocol
responsible
for
sending
that
video
over
the
network
and
these
two
components
talk
to
each
other
through
a
very
narrow
interface.
G
You
know,
the
transfer
protocol
has
some
notion
of
the
network
capacity
like
a
target
bit
rate
and
it
communicates
that
information
to
the
video
codec,
occasionally
like
every
second
or
every
two
seconds
and
the
video
codec
adjusts.
Its
internal
parameters
tries
to
target
the
bitrate
that
it
received
from
the
transfer
protocol
by
setting
things
like
quality
and
frame
rate,
and
it
produces
frames
that
the
transport
protocol
has
to
send
over
the
network
to
the
receiver
side,
and
you
know
there
has
been
decades
of
research
and
development
on
these
components.
We
have
all
these
video
codecs.
G
Let
me
give
you
one
example:
like
researchers
actually
knew
about
this
problem
that
I
showed
you
in
the
beginning.
You
know
they
studied
the
skype
and
saw
that
skype
in
face
of
network
variability
doesn't
really
do
well.
You
know
like
here
skype
tried
to
send
more
data
than
the
networking
handle
similar
to
webrtc
and
it
caused
a
very
huge
spike
in
delay.
G
So
they
said.
Okay,
it
seems
that
the
skype
has
a
problem
predicting
the
variable
network
capacity,
so
they
built
a
better
transport
protocol
that
accurately
can
match
these
variable
network
capacity
great,
but
unfortunately,
just
improving
the
network
components
didn't
save
the
day,
because
we
totally
forgot
about
the
video
codec.
You
know,
and
the
video
codec
has
a
very
big
flaw
and
that's
the
fact
that
the
codec
can
only
achieve
that
bit
rate
on
average.
G
You
know
like
here
we
ask
the
vpx
encoder
to
target
two
megabits
per
second,
you
know,
and
it
produced
a
video
that,
on
average,
is
at
two
megabits
per
second,
but
then
you
look
at
the
size
of
individual
frames.
They
are
all
over
the
place
and
even
a
frame
like
this,
that
is
over
the
network
capacity
can
still
cause
packet
loss
and
congestion
in
the
network.
G
So
the
problem
is
that
we
have
a
codec
that
can
only
respond
to
changes
in
the
target
bit
rate
over
course
time
intervals.
So
even
if
we
have
a
transport
protocol
that
can
have
a
very
accurate
estimate
at
any
point
in
time,
we
don't
have
a
codec
that
can
respond
to
that
high
resolution
prediction
from
the
transport,
so
in
salsafee
we
explored
a
more
tightly
integrated
design
where
the
transfer
protocol
and
the
video
codec
work
together
within
the
same
control.
G
So
I'm
going
to
start
by
telling
you
about
the
transport
particle
in
salt
sphere.
The
transfer
protocol
in
philosophy
only
answers
to
one
question:
what
should
be
the
size
of
the
next
frame?
So
there
is
no
notion
of
average
bit
rate
or
network
capacity
here.
The
transport
tells
me
if
you
want
to
hit
your
target
latency.
G
Your
next
frame
should
not
be
over
a
certain
size
like
10,
kilobytes
or
50
kilowatts
based
on
the
network
conditions.
So
now
the
codec,
the
video
codec,
might
be
able
to
use
that
information.
But,
as
I
told
you,
the
codec
has
trouble
hitting
a
certain
size
for
a
single
frame.
It
tends
to
overshoot
or
undershoot
that
target.
You
know
the
transport
tells
me
the
network
can
handle
a
frame
of
maximum
10
kilobytes
right
now,
but
the
codec
may
produce
a
frame
that
is
larger
or
smaller
than
10
kilobytes.
G
G
So
if
you
look
at
the
video
encoder,
you
know
the
app
the
program
responsible
for
compressing
the
video
it
receives
frames
and
produced
the
compressed
bit
string.
When
you
look
inside
that
black
box,
it's
actually
stateful.
So
when
you
encode
a
frame,
the
encoder
goes
through
a
state
transition
and
that
frame
becomes
a
part
of
the
video
stream
and
now
it
has
to
be
sent
over
the
network
and
it
has
to
be
received
by
the
receiver,
because
the
future
frames
are
going
to
be
dependent
on
the
state
that
is
produced
by
that
frame.
G
G
G
So
using
this
also,
we
can
explore
different
execution
paths
without
committing
to
them.
So
for
every
frame
in
the
video
salsafee
produces
a
version
with
a
slightly
higher
quality
than
the
previous
one,
one
with
a
slightly
lower
quality
and
thus
smaller
size,
and
it
also
gives
the
transport
the
option
to
discard
the
frame.
If
the
transfer
doesn't
like
that
frame,
the
codec
can
silently
just
go
back
the
previous
thing.
G
So
this
is
how
these
components
come
together
in
a
single
control.
The
transport
has
some
estimate
for
what
the
network
can
handle
right
now.
It
says
the
network
can
handle
30,
kilobytes
great
and
the
codec
gives
transports
two
options:
one
with
a
slightly
better
quality
one
with
a
slightly
worse
quality
than
the
previous
frame,
and
the
transport
picks
the
one
that
doesn't
go
above
that
estimate
and
it
tells
the
codec.
This
is
the
option
that
I
picked
and
the
codec
will
base
the
next
frame
based
on
that
exiting
the
state.
G
Now
this
the
transport
repeats
the
same
thing
and
sometimes
the
situation
is
that
none
of
the
options
fit
the
network
like
here
the
target
is
5
kilobytes
and
the
options
are
70
kilobytes
and
50
kilobytes.
So
the
transport
tells
the
codec
just
discard
those
frames
and
then
base
the
next
frame
on
that
exiting
state.
So
the
frames
are
discarded
and
now
the
codec
moves
on
to
the
next
frame
and
the
cycle
continues.
G
G
Okay,
let's
take
a
look
at
the
results
and
before
I
go
any
further,
let
me
show
you
a
comparison
between
between
philosophy
and
webrtc
in
the
video
that
I
showed
you
in
the
beginning
of
this
talk
so
same
situation,
for
both
the
capacity
is
going
to
go
down
and
then
go
back
up
again.
That's
how
solicifi
reacts
so
on
the
left
side.
G
Second,
let's
see
how
webrtc
and
also
react
in
this
situation
left
side,
philosophy,
right
side
by
words.
G
And
no
webrtc
again,
the
network
is
out,
salsa
v
has
recovered
and
now
webrtc
this
works
out.
Salsafee
has
recovered
and
again
it
takes
webrtc
several
seconds
to
recover
from
that
one
second
outage,
and
when
we
look
at
the
network
throughput
graph,
we
see
that
bill
rtc,
basically
disregarded
whatever
was
happening.
You
know
in
the
network
and
just
kept
sending
data
at
the
old
rates
here
in
order
to
evaluate
salsa
fee,
we
built
a
measurement
testbed
that
is
capable
of
basically
testing
any
real-time
video
system
as
a
black
box,
without
requiring
any
modification
to
that.
G
You
can
read
all
about
this
measurement
test
that
in
our
paper,
just
one
thing
I
want
to
mention
is
that
this
measurement
test
that
uses
a
barcoded
input
fit
each
frame
in
this
input.
Video
has
a
unique
barcode
and
using
this
poor
code
we
can
actually
match
frames
at
the
sender
and
the
receiver
side,
and
we
can
see
what
was
the
change
in
the
frame
quality
and
how
long
did
the
frame
take
to
reach
from
the
sender
side
to
the
receiver
side?
So
we
are
capable
of
measuring
delay
and
quality
on
a
paraframe
basis.
G
G
We
wanted
to
see
how
the
ideas
that
we
had
in
salsa
fee
actually
contributed
to
the
final
results.
So
we
started
by
implementing
a
real-time
video
system
that
works
very
similar
to
the
conventional
design.
You
know,
and
it
ended
up
kind
of
close
to
the
other
systems,
then
in
the
conventional
design
we
went
and
just
replaced
the
transport
protocol.
With
this
video
ever
transport
protocol
that
tries
to
take
control
of
the
video
on
a
per
frame
basis,
we
still
have
a
conventional
codec.
G
It
has
to
come
up
with
the
right
parameters
up
front,
but
now
the
transport
tries
to
set
a
size
for
each
frame.
As
you
can
see,
the
delay
was
improved,
but
the
quality
it
actually
dropped
a
little
bit,
but
now
in
the
full
version
of
salsa
fee,
when
we
put
together
all
these
components,
you
know
the
video
of
our
transport
protocol
and
the
functional
codec,
that's
when
we
can
achieve
better
delay
and
better
quality
than
the
other
systems.
So
these
are
the
results
on
verizon
lte
trace.
G
Here
we
have
the
results
on
atnt
lte,
trace
where
again,
salsa
v
achieves
better
delay
and
better
quality
than
other
systems,
and
here
we
have
the
t-mobile
umts
trays,
which
is
a
very
troubled
lick.
You
can
see
delays
up
to
18
seconds
and
again.
Salsa
v
achieves
lower
delay
and
higher
quality
compared
to
other
systems,
and,
of
course
there
are
situations
where
solicity
doesn't
really
offer
an
advantage
over
other
systems.
G
Let's
just
take
a
step
back
and
see
what
happens
so.
The
individual
components
that
we
have
in
salsa
video
are
not
exactly
new,
like
the
transfer
protocol
is
a
very
dumbed
down
version
of
packet
pair
and
sprout.
The
video
format
that
we
are
using
is
like
13
years
old
and
the
functional
video
codec
that
I
told
you
about
was
built
for
a
different
purpose.
You
know,
and
because
it
was
built
by
one
person
in
three
months,
its
compression
efficiency
and
speed
is
way
lower
than
commercial
codex.
G
G
So
philosophy
is
a
new
architecture
for
real-time
internet
video.
It
tightly
integrates
a
video
ever
transport
protocol
with
a
functional
video
codec,
allowing
the
system
to
respond
quickly
to
the
changes
in
the
network
conditions.
Sorcery
achieves
lower
delay
and
higher
quality
when
compared
to
facetime,
hangouts,
skype
and
libraries.
A
Okay,
thank
you
excellent
talk.
There,
hey
are
there
any
questions
I
see,
hesham
is
in
the
queue
and
then
janna
hashem
go
ahead.
G
G
To
be
honest,
that's
the
first
time
that
I'm
hearing
the
term,
so
I'm
definitely
going
to
look
it
up.
Holographic.
A
Okay,
china.
I
Thank
you
so
much
for
that
lovely
talk
sergeant.
It
was
lovely
to
see
salsa
presented
here.
I've
known
this
work
for
a
little
while
and
I'm
curious
to
to
know
if
there's
been
more
progress
on
in
this
direction,
either
in
terms
of
deployment
like
in
terms
of
you
know,
people
picking
this
up
and
and
trying
to
run
with
this
or
in
terms
of
your
own
work
on
because,
as
you
said,
it's
really
lovely
the
the
architecture
is
fairly
simple,
but
it's
effective
and
it's
that's
what
you're
demonstrating
in
my
opinion
here.
I
Well,
it's
it's.
It's
a
lovely
little
piece
of
work
that
shows
how
far
apart
these
two
worlds
of
video,
codex
and
and
transport
are,
unfortunately,
despite
so
many
years
of
work,
and
so
I
want
to
thank
you
for
for
for
revealing
that.
But
I'm
curious
to
know
if
there's
more
work,
that's
happened
since
the
paper
was
published
a
couple
of
years
ago.
G
Thank
you.
Thank
you.
So
much
for
the
question
I
mean,
I
think
the
biggest
hurdle
around
deploying
salsafee
in
real
life
is
that
you
know
before
we
didn't
have
support
for
this
kind
of
encoding.
This
stateless
codec,
you
know
in
hardware-
and
you
know
if
you
can't
have
a
video
conferencing
application
on
your
phone,
that
you
can't
really
expect
something
like
that
to
be
deployed.
G
The
good
news
is,
I
think,
like
about
a
year
ago,
stateless
codex
actually
found
their
way
into
linux
kernel,
and
I
think
today
you
can
go
and
actually
buy
chips
that
have
stateless
codecs
that
basically
offer
the
same
functionality
that
we
had
to
like
implement
from
scratch
in
software.
So,
in
terms
of
you
know,
seeing
more
and
more
things
like
that,
you
know
availability
in
the
hardware.
I
think
there
is
actually
a
path
forward
to
bring
architectures
like
salsa
v
to
production.
G
But
until
that
happens
you
know-
and
we
can
actually
have
like
those
kind
of
hardware
in
our
phones-
the
big
idea
in
salsa.
I
don't
think
that
we
can
actually
like
and
get
that
to
implement
in
implement
deploy
widely.
G
You
know
I've
spoken
to
people
at
whatsapp,
google,
all
these
companies-
and
there
are
these
ideas
that
we
are
trying
to
communicate
and
you
know,
have
them
implement
in
their
systems,
and
I
think
a
lot
of
them
are
actually
useful.
But
if
you're
talking
about
the
overall
architecture,
I
think
we
are
still
a
few
years
behind.
Until
we
have
these
kind
of
interfaces
available
widely
in
hardware.
I
I
would
I
would
encourage
people
in
the
room
to
go
talk
to
just
because
there
are
people
in
this
room
from
the
same
from
who
worked
on
or
who
do
work
on
the
products
that
you
mentioned,
they're
probably
sitting
around
here,
but
yeah.
I
definitely
encourage
them
to
go
properly.
Thank
you
again
for
the
talk.
It
was
lovely.
Thank
you.
Thank
you
very
much.
Thank
you.
A
F
Yeah
thanks
for
the
talk,
I
put
my
question
in
the
chat
window,
but
let
me
since
we
are
just
talking
here.
Let
me
let
me
just
rephrase
the
question
again,
so
I
what
I
feel
like
in
terms
of
status
encoding,
what
you're
doing
is
mostly
you
know,
all
inter
coding,
and
then
in
that
case,
obviously
a
lot
of
things
get
really
simplified,
and
but
the
quality
overall
will
suffer
very
very
greatly
in
my
opinion.
F
So
I
you
know
looking
at
the
results
in
the
paper
and
then
in
your
slides,
I
feel,
like
you
know,
using
the
ssim
metric
is
not
really
the
correct
term
here,
because
when
face
time-
or
you
know
some
other
applications
that
you
showed
in
your
results
when
when
they
show
when
they
use
a
much
better
codec
than
a
vp8,
I
mean
it's
really
unreasonable
to
expect
that
their
quality
is
going
to
be
less
unless
they
suffer
really
very
significant
and
very
long
freezes.
F
Now
you
know
I
I
feel
like
what
you
are
doing
with
vpa
is
mostly
in
recording
all
inter
coding,
and
that
simplifies
the
frame
level
rate
control,
as
I
as
I
mentioned
earlier,
but
that
is
going
to
be
detrimental
to
your
overall
video
quality
when
compared
to
other,
you
know
more
professional
video
codecs
that
use
inter
predictive
coding
as
well.
So
I
I
I'm,
surely
going
to
look
in
more
detail
in
your
tests
and
results
and
simulations,
but
I
just
wanted
to
point
it
out.
F
You
know
it's
not
very
fair
to
compare
and
all
internet
coding
with
your
you
know,
video
predictive
video
quickly,
so
I
just
wanted
to
make
that
comment.
But
if
you
have
any
follow-up
comments
on
that,
I
will
be
glad.
G
G
Thank
you.
Thank
you.
So
much
for
the
question.
First
of
all,
only
the
first
frame
in
the
video
is
intra-coded
in
salsafee.
All
the
other
frames
are
inter-coded
they're
all
motion
compensation
compressed
and.
F
Then
but
then
let
me
ask
this
very
quickly,
like
you
showed
an
example
where
you,
for
example,
to
transport,
says
the
next
frame
size
should
be
this,
and
then
you
obviously
need
to
pick.
I
mean
you
know,
quantization
factor
or
whatever,
to
satisfy
that
size
requirement
right.
That
means
you
are,
you
might
be
significantly
reducing
the
bitrate
for
that
frame.
Now,
any
frame
who
is
dependent
on
that
frame
will
also
be
low
quality,
because
you
know
the
source,
the
the
frame
that
they
are
using
as
the
source
will
be
low
quality.
G
So
when
we
change
the
quality
between
frames,
actually
we
do
that
relatively
so
we
basically
copy
the
quantizer
from
the
previous
frame
and
just
fill
that
a
little
bit.
So
if
you
look
at
like
the
output,
video
at
the
difference
between
two
subsequent
frames
is
not
really
visible
even
to
human
eye.
You
know
and
the
changes.
G
The
most
important
thing
is
that
the
situations
that
we
actually
go
and
drop
a
frame.
So
when
you
get
in
a
situation
when
there
is
a
sudden
drop
in
the
capacity
and
just
decreasing
the
quality,
a
little
bit
won't
help
you
that's
the
situation
where
you
go
and
drop
the
frame,
because
later
you
know
you
have
to
like
pay
a
very
expensive
price.
G
If
you
cause
packet
loss
or
any
sort
of
packet
queuing
and
as
you
mentioned,
increasing
the
quality
and
bringing
the
quality
back
is
going
to
be
difficult
again,
because
you
know
your
quality
has
dropped
exactly.
A
A
I'm
conscious
that
we're
holding
a
little
bit
over
time
here
I
don't
want
to
cut
the
discussion
I've
met,
maybe
maybe
people
could
join
the
the
gather
town
afterwards
and
try
and
catch
up
there.
Perhaps
if
there's
more
to
discuss
but
again.
Thank
you
really
interesting.
Thank
you.
Rudiger
again
also,
two
two
really
interesting
talks.
The
talks
are
available
on
the
irtf
website.
A
If
you
want
to
look
at
them,
I
I
hope
both
sajad
and
and
rudiger
will
hang
around
for
for
some
of
the
rest
of
the
the
sessions
later
later
this
week
anyway,
we
have
the
the
audio
video
transport
working
group
in
in
about
20
minutes.
So
if,
if
subject
is
able
to
hang
around-
and
he
may
find
that
interesting,
thank
you
everyone.
A
A
And
that
that's
that's
all
we
have
for
this
session,
so
look
out
for
the
the
rest
of
the
irtf
sessions
later
in
the
week
and
thank
you,
everybody
for
your.