►
From YouTube: Smart Records
Description
Recorded at DataSystems September 2021 Colo. Join us in #retrieval-market on filecoin slack to ask questions or build an open data transfer stack with us!
A
So
the
idea
of
this
stuff
is
to
introduce,
like
smart
records
and
like
a
multiple
routing
framework
that
we
were
working
on
a
few
months
ago,
just
in
case
it
gives
you
like
ideas
or
a
framework
for
how
to
work
on
the
retrieval
client,
or
at
least
like
I
don't.
I
we
don't
need
to
implement
like
this
all
the
way,
but
maybe
give
you
some
inspiration
on
how
to
approach
the
problem.
A
So,
basically,
and
where
I
view
the
compostable
routing
is
composed
by
two
different
frameworks
or
two
different
pieces,
which
is
the
smart
records,
which
is
the
persistence
and
the
data
the
hour
metadata
that
is
stored
in
the
industry,
for
instance,
that
then
clients
will
be
able
to
process
and
understand
to
run
the
their
output
logic.
So
the
loading
yeah.
A
If
we
put
the
indexer
example,
smart
records
will
be
stored
in
the
indexer
so
that
when
the
retrieval
client
fetches
this
this
for
cid,
this
actually
wouldn't
be
like
that,
because
actually,
you
would
send
a
full
expression
to
the
industry.
A
So
we'll
return
the
metadata
and,
according
to
that
metadata,
your
universal
friend
will
perform
additional
requests
or
orchestrate
everything
that
you
need
in
order
to
find
the
content
you're
looking
for
and
in
the
routing
behavior,
I'm
not
going
to
go
in
depth
there,
but
like
we
have
links,
we
have
context
and
we
have
the
routing
logic.
So,
according
to
the
data
that
we're
receiving
and
the
links
that
we're
receiving
from
the
from
the
systems,
the
context
of
my
notes.
So
if
I'm
in
a
private
network,
if.
A
So,
instead
of
like
an
http
request,
where
you
just
send
a
request
and
get
the
answer
back,
you
will
be
able
to
send
a
routing
expression
which
the
other
side
will
be
able
to
populate
all
that
it
knows
so
that
you
get
it
back.
If
you
have
all
that,
you
need
in
order
to
fix
the
concept,
that's
great.
If
this
is
not
the
case,
you
can
send
it
back
to
another
system
like
linux
or
just
part
of
it.
Then
you
send
it
to
the
dht
note.
A
A
We
didn't
have
our
standard
way
of
sharing
the
same
information
between
all
protocols,
so
the
dhc
has
its
own
format
and
stored
some
information.
Then
in
industry
we
are
storing
metadata
pipeline
tax
vouchers,
so
there
were
a
lot
of
pieces
of
information
that
were
not
standard.
We
didn't
have
a
way
so
that
overwhelming
systems
took
the
same
language.
A
A
A
But
there
is
no,
so
you
only
can
have
conflicts
with
yourself,
and
this
is
really
interesting
because
then,
according
to
what
tiers
have
in
store
with
data,
you
can
trust
it
more
or
less,
and
also
we
introduced
what
we
call
smart
tags,
which
is
whenever
we
store
data
or
we
modify
data
in
some
clear
that
is
storing
a
smart
record.
A
We
can
trigger
some
operations
to
mutate
that
data,
so,
for
instance,
one
thing
that
well
that
we
tried
is
that,
if
we
store
in
appear
a
smart
record
with
with
a
reachability
tag
and
a
multi
address,
the
peer
will,
once
this
big
smart
record
store
locally.
It
will
check
if
it
has
reachability
so
that
you
have
additional
information
of
if
this
viewer
is
able
to
reach
that
guy
or
not,
and
when
you
fetch
back.
This
metadata
is
interesting.
A
If
you
have
additional
information
that
we
don't
have
right
now-
and
this
is
how
bizarre
like
to
refer
to
the
to
the
smart
records-
it's
like
dht
values
that
become
publicly
updatable
system
documents
so
like.
Instead
of
providing
resources,
you
can
store
arbitrary
data
and
we
can
add
additional
logic,
and
I
was
going
to
say
something
else
about
smart
requests,
but.
A
So
what
kind
of
use
cases
we
were
thinking
initially
with
smart
records
like
ways
of
upgrading
the
network
without
having
to
make
a
lot
of
changes
for
a
lot
of
purple
upgrades
a
good.
I
like
an
example
that
entire
food
I
haven't,
been
maintaining
like
the
phd,
so
I
don't
know
but
like
apparently
updating
the
the
format
in
which
we
are
storing
provider
records,
it's
kind
of
a
nightmare,
and
with
this,
as
we
can
store
data,
we
would
have
a
way
of
like
making
updates
without
having
to
to
change
the
language.
A
A
A
Once
you
put
data-
and
we
have
a
few
examples
on
this
and
and
of
course,
for
our
indexing
matters,
it
would
be
great
because
this
data
could
be
encoded
with
smart
records
so
that
we
have
a
standard
way
of
referring
to
and
in
the
end,
like
smart
record,
something
that
would
use
iple.
A
A
Use
I
killed
these
because
max
knight
will
be
the
key
always
has
to
be
a
string,
and
we
wanted
the
keys
to
be
any
type
of
our
data
mode.
D
A
A
So
yeah
because
actually
yeah
there's
another
thing.
All
of
this
is
not
only
for
small
records
but
also
for
the
possible
routing
language.
So
we
have
brackets
because
we
have
functions
and
we
have
expressions
and
links
and
so
on
here
so
explain.
D
A
D
A
You
use
your
query.
Yes,
and
we
can.
I
mean
I
have
an
example.
We
can
jump
right
into
that
yeah
because
yeah,
we
have
different
kinds
like
a
link
interfaces
which
the
interface
is
always
the
same.
A
I
receive
an
expression
and
they
give
you
an
expression
populating
all
that
I
know
we
talked
about
subnets
because,
according
to
the
net,
that's
the
context,
so
I
may
be
part
of
this
is
related
to
what
hadeem
was
thinking
of
having
a
federation
of
supernodes
or
or
like
pinatas
and
gateways
that
talk
to
each
other
to
accelerate
contemporary.
It
would
be
a
subnet,
but
we
could
be
part
of
several
subnets,
so
the
dhc
could
be
a
subnet.
A
We
could
have
several
different
dhts
that
have
a
different
like
protocol
and
but
they
could
leverage
the
same
language
and
and
like
the
interface
for
composition.
Writing
is
really
simple,
because
you
send
an
expression
you
get
back
the
expression,
populated
and
according
to
what
knowledge
you
have
you
decide
to
go
right
away
and
request
the
content
or
keep
populating
until
you
have
enough
information
and
the
ones
that
start
so
every
peer
would
have
their
middle
words,
which
are
the
policies
that
we
were
mentioning
yesterday.
A
Configure
the
the
different
middle
words
that
you
want
in
your
notes,
so
that
when
you
receive
an
expression,
you
start
populating
with
default
in
this
graph,
so
first
is
batching,
which
is,
if
I'm
receiving
requests
for
for
certain
content.
That
is
a
list
of
cids
needs.
What
we
will
do
back
and
forth.
We
may
like
populate
all
of
the
cids.
We
know
them
already
and
the
providers
that
have
them.
Then
we
have
caching
if
I've
seen
this
cab
already.
I
can
like
send
the
expression
back
without
having
to
do
additional
processing.
A
Then
we
could
have
a
prefetch,
which
is
this
radar,
that
aden
has
introduced
in
manifest
ideas
to
have
different
middleware
so
that
you
could
choose
the
when
you
send.
So
I
have
my
peer.
I
say
that
I'm
gonna
serve
and
like
answer
to
routing
expressions
using
batching,
then
caching
them
prefetch
and
then
either
like
a
radar
or
a
dht
lookup.
This
is
a
demo
of
a
person
responding
receiving
yeah
responding
so.
A
A
A
Every
period
so
every
the
idea
is
yeah.
For
instance,
an
indexer
will
only
have
maybe
batching
caching
and
then
the
index
of
core,
but
a
nike
phase
output
has
patching
caching
prefetching
and
then
the
radar
and
the
dhc
and
the
dht
would
like
itself
send
a
request
to
another
peer
in
the
phd
that
could
have
like
this,
their
own,
like
policies
or
their
own
logic.
So
all
of
us
have
these
logics
to
answer
to
routing
expressions
and.
C
Which
is,
I
think,
essentially
what
you're
saying
is:
you've
got
some
arbitrary
data
structure,
so
we
can
think
of
it
as
an
ideality
or
a
json
wall
and
or
this
one
that
you
guys
have,
but
we
think
of
it
as
that,
and
then
each
node
has
some
set
of
these
nodes
that
might
be
found
in
there
that
they
know
things
they
can
do
for
the
process.
So
it's
like,
oh
when
I
see
a
node,
that's
of
this
schematic
that
happens
to
be
in
here.
I
know.
A
Yes,
so,
and
the
idea
was
to
like
decouple
the
actual
content
routing
from
the
transport
layer,
so
I
can
send
you
these
requests
to.
I.
A
Was
the
idea
and
like
this
is
an
example
of
what
we
all
mentioned
so
imagine
that
we
want
to
provide
some
content.
I
do
like
route,
which
is
the
like
the
expression,
and
I
want
to
provide
this
cid
into
this
miner
and
I
send
this
data
to
an
indexer
so
that
the
indexer
stores
this
metadata
there's
nothing
to
do
for
the
indexer.
It
can't
like
fulfill
the
full
request,
so
it
just
sends
okay
and
then
some
virtual
miner
wants
to
get
data
from
like
from
the
network.
A
A
So
I
just
forward
the
data
that
I
don't
have,
which
in
the
address
I
want
to
find
pcid
in
the
dht
so
that
I
get
the
key
record
and
when
you
get
to
the
dht
you
may
have
not
found,
and
this
means
that
it's
not
fully
fulfilled.
So
you
could
keep
going
and
answer.
Others
or
you
could
like
get
an
okay
with
the
multi
address
of
the
app
or
fetch
the
other,
so
that
you
go
there
to
meet
the
others
or
something
like
that.
A
So
the
idea
is
yeah
that
you
keep
going
with
all
of
these
middlewares
so
middle
wheels.
Ideally,
ideally,
there
are
reference
implementations
that
you
could
like
plug
in
your
in
your
node
and
that's
it,
and
you
would
be
populating
the
expressions
until
you
you
manage
either
to
timeout
or
because
you
haven't
found
the
content
likes
in
this
way,
or
it
gives
you
all
the
information
that
you
need
to
fetch
the
content.
From
the
specific
view.
D
D
A
A
A
A
There
are
some
proposals
in
the
so
in
this
repo
I
mean
we
left
it
halfway,
but
in
this
repo,
which
is
a
go
routing
language,
the
language
that
will
exchange
like
all
this,
there's
a
proposal
that
I
made
for
middlewares,
because
I
wanted
to
try
if
we
could
do
like
the
index
like
the
client
side
for
the
index,
or
with
this
when
I.
A
So
here
are
it's
a
proposal.
How
would
the
middleware
thing
work,
which
in
the
end,
is
just
like
a
chain
of
middle
words
that
keep
like
propagating
routing
as
our
syntax
expression
until
it's
managed
to
fill
everything
and
if
not,
if
it
receives
the
populated
one,
the
other
way
and
so
yeah
open
to
a
discussion?
Yeah
yeah?
So
it's
just
a
middleware
that
has
the
root,
which
is
how
to
process
the
expression
and
then
some
middle
word
context
to
propagate
between
the
different
with
the
chain
of
of
little
words.
A
And
then
this
is
how
the
routing
language
looks,
which
in
the
end
would
have
the
predicate
with
tax,
which
is
the
the
function
name.
And
then
we
have
key
values
elements.
So
so
we
can
have
positional
or
or
name
named
arguments
and
it's
up
to
the
middleware
to
to
know
how
to
process
it
and-
and
so
in
that
sense,
total
flexibility.
A
A
This
now
that
I
know
a
bit
of
ikeli
and
then
the
parser,
so
that
this
is
what
the
middlewares
use
in
order
to
know
like
in
the
routing
expressions.
If
this
is
the
tag,
what
the
the
learning
expression
is
asking
for,
and
but
the
actual
operational
thing
is
my
records.
There
are
a
few
examples
and
we
built
like
a
chat
application
that
used
so
that
used
the
smart
record
as
a
server
to
store
information
in
smart
ground.
Like
all
the
chatting
all
the
messages,
we
can
try
that
and
also
like.
A
There
are
a
few
examples
with
how
the
the
actual
phd,
how
would
be
used
in
the
dhc
in
which,
like,
for
instance,
we
may
store
this
primary
record.
A
Id
and
the
multi
address,
and
we
add
additional
so
we
add
a
connectivity
tag,
an
available
tag
to
check
if
the
peer
the
dhcp
were
in
the
end
of
where
I
end
up.
Storing
the
data
has
the
connectivity
to
that
here
or
not
so
when
the
when
the
phd
peer
that
needs
to
store
this
information
receives
the
the
smart
record.
A
The
first
thing
that
it
will
do
is
execute
the
this
smart
tag
to
check
it
has
connectivity
and
it
will
mutate
the
the
smart
record
to
say
here,
so
it
will
muted
this
this
tag
into
diode
if
it
was
able
to
dial
or
not
so
that
we're
not
available
so
that
when
someone
fetched
that
that's
my
record,
you
have
additional
information.
That
is
not
only
like
the
multi
address,
but
also
this
is
my
multi
address
and
I
was
able
to
die
on
this
guy
or
I'm
not
going
to
discuss.
A
A
D
A
D
Sorry
question,
I
don't
think,
did
they
keep
coming
close
to
my
legs?
I
just
want
to
like
understand
things
so
with
the
example
that
this
swap
run
have
right.
Obviously
you
could
change
the
protocol
so
that
you
know
it
actually,
instead
of
having
this
like
one
half
or
less
actual
routing.
Yes,
yes
right,
but
that
requires
changing
the
particle
right.
So
sure
enough,
could
you
you
could
if
I'm
reading
your
middlewares
right
directly
like
write
a
local
middleware
that
translated
into
a
routing
request
into
like
a
regular.
A
D
A
That
yeah
and
finally,
like
we
even
like
embedded
this
in
ikea
fest,
so
there's
a
branch
somewhere
in
goei
cafes,
where
you
can
like
ask
for
smart
records
for
other
peers
that
are
running
the
smartphone,
so,
in
the
end,
what
it
does
is
just
find.
The
key
is
that
embedded
into
the
actual,
like
regular
matching
code
like
this?
Yes,
no!
No!
No!
No,
like!
This
is
a
parallel
protocol
that
you
could
use
to
test
that
and
somewhere
over
here.
A
A
I
don't
what
was
this
but
like
it
gives
you
a
cli
comment
so
that
you
could
ask
like
if
the
other
iqfs
had
smart
records,.
B
A
That's
it!
That's
why
we
wanted
to
copy
it
from
the
transport,
because,
right
now
you
only
you
can
only
fetch
prior
records
with
the
vhc,
but
ideally
you
could
use
like
a
gossip
subchannel
to
or
even
like
direct
requests
by
http
to
get
the
same
private
record
that
you
have.
A
So
you
could
like
store
and
like
create
new
smart
attacks
and
process
them
and
send
them
and
there's
a
link
to
the
message
request.
But
you
would
have
to
do
like
the
rest,
which
is
figuring
out.
Middlewares,
probably
like
we
have
to.
I
have
to
check
the
code
because
working
a
bit
more
than
me
on
that,
like
after
I
switched
to
this
team,
but
like
he
had
parsers
so
that
you
can
pass
functions
and
like
run
the
logic
the
right
way.
So
only
the
smart
record
part
is.
D
B
D
A
A
D
I
didn't
really
know
where
they
provide
this
interesting
symmetry
on
a
point
where,
like,
if
you
really
wanted
to
like
make
like
immediately
the
client,
have
a
top-level
smart
writing
language.
You
just
write
a
whole
bunch
of
compatibility
layers
that
are
calling
out
to
existing
yeah,
which
is
interesting.
A
A
A
A
D
I
just
say
cuz
like
if
that
existed,
then
if
that
were
done-
and
I
certainly
will
be
hopeful
at
some
point-
then
the
path
implementation
that
avoids
all
complex
you
know
networking
protocol
changes
is
to
just
implement
it
all
locally,
with
like
translation
layers
before
it
goes
to
the
output.
D
B
A
D
So
well
controversial
suggestions.
C
So
we
need
to
define
what
the
actual
concrete
interface
of
what
one
of
these
records
looks
like
in
terms
of
a
thing
that
is
used,
and
probably
we
have
some
schema
known
as
that
recess,
rather
than
the
current
abstract
virtual
machine
center.
So
there's
one
level
more
concreteness
that
we
need
to
scope
this
to
something
that's
doable,
which
is
we
have
like
a
specific
set
of
things.
C
C
C
But,
but
having
that
in
a
constrained
way
of
what
the
initial
parameter
is
rather
than
the
abstract
grammar
is
going
to
make
this
much
more.
So.
A
The
grammar
is
like
the
challenge
is
that
this
is
a
general
framework
in
the
sense
that
you
have
a
general
language.
You
have
only
one.