►
From YouTube: RETMKT Builders - Lotus Node Architecture
Description
Raúl presents on the evolution of a lotus node at the Retrieval Market Builders Mini-Summit in April 2021.
A
Welcome
to
day
two
today
we're
gonna,
as
david
said,
we're
gonna
cover
some
some
basics
about
about
some
changes
that
we
plan
to
to
make
to
to
to
the
lotus
runtime
architecture.
I
think
I
was
told
that
juan
presented.
I
wasn't
here
for
day
one,
but
I
was
told
that
that
juan
kind
of
like
presented
a
little
bit
of
what
we're
calling
the
architecture
for
for
the
golden
path.
And
you
see
that
there
are
two.
Let
me
change
one
thing
on
my
screen
because
I
think
it's
a
little
bit
yellow.
A
I
don't
know
if
you,
if
you
saw
that
that
tin,
we
don't
we
don't.
We
don't
see
the
plot
right.
You
don't
do
that
all
right,
okay,
cool!
That's
that's
great
yeah
because
I
had
flux
and
night
shift
enable
both
at
the
same
time,
so
they
were
compounding,
and
it
was
really
you
know
anyway,
yeah.
So
we're
gonna,
we're
gonna
focus
on
on
two
elements
of
this
overall
architecture
diagram
have
we
has
this
been
covered
in
this
session
and
we.
B
We
briefly
showed
this
diagram
earlier.
I
think
what
the
maybe
maybe
it
is
useful
to
have
a
little
bit
of
context,
which
is
so
this
service
thinking
about.
Okay,
we've
got
existing
lotus
clients
and
minors,
and
there's
work
to
do
especially
on
retrieval
side
to
make
it
more
easy
to
get
that
content
right,
and
that
is
what
eventually,
that
that's
that's
this
path
towards
a
retrieval
market,
and
so
in
the
shorter
term.
B
What
what
are
the
you
know,
next
steps
that
we
should
be
thinking
about
evolving,
more
concretely
to
get
towards
this
market
that
we're
that
we're
headed
towards,
and
so
I
think
what
the
next
piece
is
is
sort
of
more
core,
like
next
steps
that
we
can
imagine
on.
These
two
highlighted
portions
of
this.
So
what?
B
How
is
a
file
point
miner
going
to
change
and
what
needs
to
happen
there
to
make
itself
more
easy
for
external
clients
to
get
stuff
out
of
it
and
then
introducing
the
start
of
some
interfaces
and
then
implementations
of
this
network.
Indexer
thing:
that's
here,
which
is:
we
need
some
way
to
to
quickly
learn
which
miners
have
which
content,
and
so
how
do
we?
How
do
we
start
evolving?
That
subsystem.
A
Exactly
and
we
can
like
potentially
dive
into
other
areas
of
this
diagram
as
well,
but
the
areas
that
we're
focusing
on
today
are
the
ones
that
we
believe
are
highly
relevant
to
the
retrieval
work
here,
because
there
are
things
that
are
going
to
be
that
we
plan
to
change
on
the
architecture
that
are
going
to
off
of
lotus
itself
and
those
principles.
We
propose
that
other
clients
also
adopt
them
and
of
course
this
is
an
open
conversation.
A
So
if
you
think
that
you
know
there
is
we're
going
in
the
wrong
direction
or
there
are
things
that
we
haven't
considered,
then
this
is
you
know
a
really
good
moment
to
to
actually
voice
that
out.
There
are
gonna
be
more
discussions
in
the
community
around
around
these
changes
that
I'm
gonna
that
I'm
gonna
be
presenting.
Today,
we
are
within
kind
of
like
the
lotus
maintainers
team,
doing
a
little
bit
of
like
the
initial
work
of
getting
the
rough
ideas
getting
an
internal
consensus
before
we
actually
start
engaging.
A
A
It's
a
really
good
opportunity
to
do
that
now,
as
retrieval
markets
is
really
the
the
the
the
initiatives
that
that
you
guys
are
going
to
be
working
on
really,
you
know
potentially
need
to
need
to
evolve
the
larger
scope
of
of
what
a
lotus
client
is
not
just
a
minor
part
of
it,
but
kind
of
like
the
whole
architecture
of
the
client.
So
just
to
give
you
a
little
bit
of
context,
project
bedrock,
retrieval
markets
and
other
projects
are
intending
to
introduce
new
capabilities
to
the
platform
network.
A
A
These
are
really
good
indicators
that
the
product
itself
should
be
giving
miners
like.
There
should
be
some
more
embedded
modularity
within
the
product
itself
within
the
the
the
code
itself.
So
we
also
detected
that
monolithic
lotus
architecture
is
quite
limiting
in
terms
of
like
many
things
that
I'm
gonna
that
I'm
gonna
be
touching
on.
So
just
to
give
you
kind
of
like
the
initial
picture
of
what
the
status
quo
is
of
of
lotus
itself.
First
of
all,
lotus.
A
We
consider
it
largely
monolithic,
but
it
does
have
like
some
small
sprouts
of
like
a
little
bit
of
physical
segregation
of
responsibilities.
Concretely,
you
would
be
familiar
with
two
with
two.
There
are
two
entry
points
into
the
lotus
binary
itself
right
now
as
it
stands
today,
we've
got
the
demon
and
we've
got
the
minor,
and
these
two
have
some
these
two
perform
entirely.
A
You
know
different
responsibilities,
so
the
demon
is
owning
chainsync,
it
is
offering
the
full
json
rpc
it's
offering
wallet
functionality
and
it's
offering
a
bunch
of
a
bunch
of
other
things,
and
the
miner,
on
the
other
hand,
is
taking
care
of
ceiling.
Running
storage,
deals,
retrieval
deals
and
and
a
few
more
things
now.
There
are
also
a
few
commands
that
also
encapsulate
specialized
small
services
like,
for
example,
there
are
commands
to
or
lotus
shed
tools
that
run
the
off
chain
window,
post
challenger
or
the
consensus
port
reporter.
A
So
just
to
kind
of
like
outline
what
the
responsibilities
of
lotus
miner
the
binary
itself
are-
and
this
is
like
a
non-comprehensive
comprehensive
list.
We've
got
participating,
storage,
mining,
onboarding,
storage,
coordinating
the
ceiling
activity
in
the
pipeline
sector,
proving
block
mining,
and
then
we've
got
evaluating
and
accepting
storage
storage
deals.
Conducting
the
data
transfer,
assigning
deals
to
sectors,
blah
blah
blah
taking
care
of
serving
those
those
deals
and
a
bunch
of
other
things.
A
The
mining
operation
of
a
miner
is
really
the
right
now,
the
most
profitable
activity
of
the
miner,
and
it
actually
can
be
entirely
private,
so
lotus
miner
is
an
entirely
private
thing
that
interacts
with
a
demon
to
interact
with
the
network
right,
but
in
but
the
mining
activity
within
lotus
miner
could
stay
entirely
private.
However,
markets
is,
by
definition,
entirely
public
facing
right.
So
now,
you've
got
these
two
like
disjoint,
like
critic,
critical,
like
two
pieces
that
have
different
criticality
that
also
have
different
levels
of
exposure.
A
Right
now,
two
should
be
having
different
levels
of
exposure
to
the
network,
but
are
being
bundled
into
one
thing
that
makes
it
like
more
fragile
than
it
should
be.
So
precisely
what
are
the
problems
with
this?
With
this,
with
this
architecture,
the
one
hand
I
just
talked
about
that
fragility
bugs
in
one
area
of
of
the
code
can
impact
other
areas,
and
this
means
that
non-critical
processes
or
what
miners
can
consider
today
to
be
non-critical
processes,
can
end
up
affecting
the
operation
of
critical
processing
and
potentially
even
crashing
the
inspects
right.
A
That
is
doing
the
public
interfacing
with
the
network
right,
like
the
lotus
demon
and
also
like
it
might
not
just
be
attacks,
it
might
be
legitimate
traffic
spikes
right,
so
we've
got
different
workloads
here
that
have
different
levels
of
predictability
and
when
you're
serving
deals
and
potentially
retrieval
deals,
you
can't
really
plan
for
the
amount
of
traffic
that
you're
gonna
that
you're
gonna
receive
right.
So
any
activity
on
that
endpoint
could
potentially
deschedule
or
displace
or
impact
other
critical
processes.
A
A
That's
that's
really
hard
to
achieve,
because
it's
all
bundled
into
into
a
single
process
and
also
like,
as
I
said,
load
balancing
firewalling
as
well.
If
you
want
to
like,
if
you
want
to
protect
these
public-facing
endpoints
to
to
like
prevent
denial
of
service
attacks
or
other
things,
it's
hard
to
just
achieve
that
level
of
like
protection
for
this
particular
type
of
workload
right.
A
That
is
completely
isolated
from
everything
else
right
because
they
are
experimental.
So
you
don't
want
to
you,
don't
you
don't
want
to
affect
your
production
production
setup
right?
So
what
we've
been
talking?
What
we've
been
thinking
of
is
kind
of
like
already
paved
the
way
and
like
set
the
stage
for
for
this.
We
are
right
now
thinking
about
separating
the
lotus
miner
process
into
the
lotus
markets
process
and
the
lotus
mining
process.
A
A
So
one
idea
that
I
wanted
to
introduce
is
you
know
that
you've
seen
that
kind
of,
like
my
whole
talk,
has
revolved
around
what
we
want
to
do
in
the
lotus
minor
itself
right,
but
there
is
opportunity
to
as
we
dive
into
into
projects
and
completely
retrieval
markets,
which
is
like
going
to
be
composed
of
a
lot
of
services
that
that
provide
functionality
and
like
together,
make
up.
You
know
the
the
ideal
retrieval
experience
for
users.
A
Do
we
want
to
consider
file
coin,
potentially
as
a
layer,
one
protocol
that
offers
foundational
primitives
on
on
which,
on
top
of
which
a
plethora
of
services
and
capabilities
can
be
built?
So
do
we
want
to
like
move
towards
a
capability
like
a
capability
oriented
deployment
right
so
like
users
and
clients
would
be
able
to
start
lotus,
and
just
you
know,
run
the
chain
capability.
A
If
all
they
want
to
do
is
follow
the
chain
and
interact
with
the
chain,
maybe
potentially
start
just
a
lotus
wallet
capability
or
stride,
a
retrieval
capability,
if
you're
a
miner
right
or
like
how
do
we
want
to
kind
of
make
the
ability
to
select
what
features
we
wanna
run
from
from
kinda
like
the
filecoin
network,
easy
from
from
the
user's
perspective
right,
but
we
gotta
be
careful
because
capabilities
can
be
stacked
or
interdependent.
So,
for
example,
storage
deal
making
depends
on
the
chain.
A
Indexing
operates
on
storage
deals
right
like.
If
you
don't
have
a
storage
deal,
then
then
you
have
nothing
to
to
index.
Really.
Retrieval
can
only
happen
if
there
are
storage
deals.
So
like
all
these
things
are
like.
If
you
break
apart
all
these
services,
then
the
reality
is
that
these
services
will
end
up
having
dependencies
or
some
services
would
be
downstream
consequences
of
of
offering
another
service
right.
So
these
are
things
to
take
into
account.
I
just
wanted
to
like
put
out
there.
A
Some
of
the
design
principles
have
been
out
of
the
office
for
one
week,
but
the
team
has
made
a
lot
of
progress
and
I've
kind
of
like
picked
up
on
on
the.
What
we're
considering
what
the
team
is.
Considering
the
current
design
principles
of
how
we
want
to
conduct
this,
this
potential
re-architecture
of
of
lotus.
A
First
of
all,
we
are
explicitly
staying
away
from
microservice,
architectures
or
microservices
like
architectures,
and
when
I
talk
about
this
I
mean
you
know,
service
meshes
and
things
that
are
super
super
complex
to
operate
in
practice
because
they
end
up
be
like
enabling
or
permitting
patterns
of
like
spaghetti
interactions
between
a
lot
of
pieces,
creating
webs
of
interdependencies
that
are
really
really
hard
to
debug
and
like.
We
think
that
that
is
the
wrong
level
of
complexity
to
introduce
in
in
lotus,
so
definitely
staying
away
from
from
those
patterns.
A
We
think
that
the
like
one
right
approach,
one
potential
right
approach
here-
could
be
to
have
supervisor
instances
which
know
which
capabilities
are
provisioned
in
the
deployment.
So
if
you
have
a
miner
that
is
offering
that
is
obviously
doing
ceiling
and
is
running
a
farm
of
retrieval
nodes,
a
farm
of
storage
nodes
and
it's
running
as
well
indexing
nodes
and
a
bunch
of
other
things
that
might
come
that
might
come
down
the
pike.
The
supervisor
supervisor
instances
would
know
which
capabilities
are
provisioned
by
which
nodes
in
the
network
right.
A
And
once
you
have
this
kind
of
like
level
of
of
oversight
of
insight
into
the
network,
the
supervisors
themselves
can
serve
as
the
external
json
rpc
api
entry
points
so,
rather
than
now
that
you've
like
divided
your
functionality
in
into
many
components
that
are
placed
in
different
machines.
Now,
do
you
wanna,
like
we
don't
wanna?
We
don't
want
the
user
to
have
to
face
or
the
the
end
line
to
have
to
face
the
ultimate
like
complexity
of
dealing
with.
Where
is
each
thing?
A
So,
if
I
want
to,
I
want
to
invoke
a
particular
json,
rpc
method.
We
don't
want
the
client
to
have
to
like
figure
out,
which
is
the
instance
where
they
need
to
invoke
that
method,
so,
supervisors,
knowing
which
capabilities
are
provisioned
by
which
nodes
could
also
serve
as
the
as
the
json
rpc
api
endpoints,
and
at
this
stage
we're
thinking
that
routing
could
be
done
in
a
reverse
proxy
manner.
So,
basically,
the
the
request
comes
into
the
supervisor,
supervisor,
load,
balances
and
then
just
does
does
a
reverse
proxy.
A
One
really
key
thing
here
is:
I
mentioned
it
before.
There
are
dependencies
between
these
services
so
supervising
the
pros.
The
process,
life
cycle
of
everything
that
is
running
in
the
deployment
is
really
is
really
critical,
because
if
something
fails-
and
that
means
that
you
know
downstream-
other
services
are
going
to
fail.
Then
we
must
be.
The
entire
system
must
be
able
to
notice
and
react
right.
A
So
one
potential
idea
here
to
simplify
the
operational
side
of
things
is
instead
of
having
like
each
of
these
potential
processes,
be
a
separate
physical
process.
We
just
have
lotus,
be
lotus
with
a
start
command
to
which
you
to
which
a
operator
can
provide
a
services
stack,
so
you
provide
a
flag,
and
you
say
I
want
to
run
markets
and
miner.
A
Here
are
some
some
resources.
So
there
is
everything
most
of
what
I've
covered
today
is
discovered
here
in
the
mind,
minor,
runtime
segregation
proposal.
I
know
I'm
a
little
bit
over
time,
so
I'll
just
I'll
just
wrap
up
quickly,
so
make
sure
to
to
to
to
read
that
if
you're
interested
and
then
there
is
also
a
draft
prototype
of
the
separation
and
lotus
of
markets
and
the
miner
kind
of
like
as
a
as
a
starting
point
and
as
an
as
an
initial
as
an
initial
stab
to
this.