►
From YouTube: Build a Kubernetes Based Cloud Native Storage Solution From Scratch - Sheng Yang, Rancher Labs
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Build a Kubernetes Based Cloud Native Storage Solution From Scratch - Sheng Yang, Rancher Labs
When it comes to cloud-native and container-ready storage, most industry discussions focus on how to make existing storage systems work with Kubernetes/Docker. But the rapid development of SSD technology, along with container and Kubernetes, make it possible to create a more elegant solution from scratch, compared to the storage technologies exist tens of years ago.
So they have created a storage solution based on Kubernetes
In Longhorn, control flow is driven by Kubernetes's controller model; deployment is done using Kubernetes workload; HA is backed by Kubernetes's HA capability. Though sometimes it's easy to say than done. As a result, lots of insights were gained on how to write an application based on Kubernetes. In this talk, Sheng Yang will give a talk about the experience and insights he has gained as a part of building a Cloud Native storage solution on Kubernetes.
https://sched.co/MPXK
A
A
So
Longhorn
is
the
open
source
distributed
block
storage
software
for
kubernetes.
You
can
find
more
information
about
Longhorn
in
the
github
links
now
and
what
long
run
do
is
very
easy
way
to
add
persistent
storage
support
to
any
kubernetes
cluster,
no
matter
where
the
cluster
is
being
hosted.
So
you
can
add
the
Longhorn
to
your
system
just
using
this
one
line
of
code
control
command,
but
many
people
will
raise
questions
there's
already
so
many
existing
storage
solution
there.
Why
we
need
Longhorn,
let's
compare
Longhorn
to
the
legacy
story,
software's.
A
So
normally
the
legacy
storage
software's
implements
sophisticated
storage,
stack,
that's
including
disk
management,
storage,
visualization
file
system
and
the
storage
protocols
and
amendment
many
of
them
may
be
Patera
touring
and
also
they
have
h8a
storage
controller.
That
can
those
worry,
hi
I
will
opt
as
throughput
across
the
cluster,
the
which
can
be
a
bottleneck
in
real
use.
At
the
last
their
most,
the
storage
system
appeared
upon
the
hundreds
of
thousands
lines
of
code
or
is
not
more
and
is
very
complexed,
so
the
approach
we
take
with
Longhorn
is
different.
A
We
leverage
it
modern,
high
speed,
high
capacity
SSD
an
nvme,
and
we
use
proven
Linux
storage
features
like
sparse
file
and
the
seat
groups,
and
we
are
using
well.
Each
volume
in
Longhorn
is
a
set
of
independent
microservices
and
we
utilize
in
communities
and
other
existing
technology
to
orchestrate
that
that's
reduced
the
complexity
in
the
long
run
itself.
In
the
end,
Longhorn
I
mean
currently
have
about
30,000
live
a
code.
We're
writing
mostly
in
go
and
broke
into
clear
pieces.
A
You
can
easily
find
you
can
easily
read
the
code
and
understand
it
and
work
with
it.
So,
in
the
end,
Longhorn
is
the
micro
service
basis,
community
basis,
storage
solution,
so
our
latest
release
is
long-form
delta,
v,
dot,
o
and
currently
Longhorn
still
in
the
alpha
stage
and
as
I
said,
Lancome
provides
enterprise
grade
this
you
will
distributed
block
story
software
for
kubernetes
and
we
support
a
warning,
snapshot,
backup
and
restore
the
difference
between
the
snapshot
and
the
backup
is
the
snapshot
was
taking
inside
your
kubernetes
cluster.
A
So
data
doesn't
leave
your
cluster,
but
the
backup
is
the
way
we
provide
a
way
to
migrate.
You
backup
your
data
into
someone
else
like
AWS,
s3
or
ending
s3,
compatible
type
object
stores
or
the
anther
server.
So
in
that
way
you
have
another
failsafe
for
your.
If
anything
goes
wrong
with
your
cluster.
A
A
So
we
provide
a
UI
as
a
complement
of
the
compliment
of
the
function
of
the
kubernetes,
like
besides
the
create
and
delete
warnings,
a
snapshot
UI
provide
the
full
functionality
like
a
backup
and
other
operations
for
your
long
form
systems,
and
we
support
one-click
installation,
as
we
mentioned
before,
you
can
use
the
coop
control,
apply
a
Yama
file
or
you
can
use
a
helmet
chart
or
you
can
use
the
rancher
catalog
apps
to
use
to
a
long
haul,
with
zero
configuration.
Of
course,
more
features
are
coming.
A
A
So
here
we
have
two
nodes.
As
an
example,
each
node
has
some
hard
drives
and
the
matron
and
the
member
CPU
and
memory
so
when
kubernetes
create
a
part
which
would
request
the
system,
volume
Longhorn,
received
a
request
and
will
create
queue,
replicas
on
three
different
nodes
and
it
probably
different
discs.
A
So
the
with
reason
why
we
separate
two
replicas
is
because,
if
one
node
goes
down,
we
still
have
one
replicas
running
and
as
long
as
the
warning
has
one
replicas
exist
and
running,
the
warning
will
be
fine.
So
whatever
happens,
you
have
multiple
parts
as
you
can
you
match
the
same
things
happens
very
simple:
we
created
multiple.
We
create
the
two
more
replicas
at
one
more
engine
for
the
second
volume
at
the
third
one.
A
So,
in
fact,
this
approach
has
a
worried,
desirable
effect,
that
the
data
path
of
volume
is
isolated
from
each
other,
as
you
see
that
if
one
woman
goes
down
due
to
whatever
reason,
there's
no
reason
for
other
volume
to
be
affect
as
well,
because
their
data
path
is
completely
separate.
It's
nothing
interact
with
each
other,
but
now
we
have
a
question:
how
can
we
manage
all
this
engines
replicas,
so
they
are
managed
by
the
cube?
Ladies,
so
that's
the
answer.
A
A
So
we
build
the
local
manager
based
on
kubernetes.
Here's
the
extraction
of
the
local
manager
still
assume
we
create
a
new
part.
We
want
some
warnings.
Kubernetes
is
going
to
talk
with
the
Longhorns,
the
SI
plugin,
csi's,
container
storage
interface.
A
single
starter
has
to
mention
that
in
his
keynote
and
his
world,
chameleon
way
for
the
storage
winter
you
hook
into
the
kubernetes
and
the
Longhorns
are
plugging-
will
talk
to
the
local
manager
through
the
local
capi.
A
In
fact,
the
local
manager
is
just
like
another
another
controller
inside
kubernetes
the
local
manager
will
create
new
CRD
object.
Customer
resource
definition
object
inside
kubernetes
api
server,
so
we
don't
have
the
burden
to
have
another
data
store,
so
the
change
of
say
local
manager
create
a
new
volume.
Crd
inside
connects
api
server.
That's
one
where
we
pick
up
by
the
other
controllers.
A
The
volume
control
inside
local
manager
and
that
controllers
see
that
ok
now
I
have
a
new
warning,
so
I
need
to
create
a
new
engine
and
replace
that
sorry
replicas
to
to
implement
that
wording
for
the
user.
If
you
have
multiple
requests
to
see
the
same,
so
the
local
manager
is
implemented
on
top
of
the
kubernetes
cluster.
To
do
all
this
kind
of
obvious
works.
A
Another
way
to
interact
with
local
manager
is
used
in
the
long
haul
UI
as
I
as
I
mentioned
before
so
local
UI
provides
backup
a
lab
upgrade
cross
cluster
disaster
recovery.
These
kind
of
features
the
additions
to
come.
Let's
provide
create,
delete,
volume
and
snapshot.
If
you
have
external
snapshot,
plugins.
A
So
that's
about
local
manager,
so
this
is
really
good.
Interesting.
If
you
haven't
know
about
what
is
the
controller
pattern
or
operator
pattern
inside
kubernetes
I
can
show
you
the
example
here
so
on
the
right.
You
can
see
that
we
have
a
warning
object.
We
have
an
engine
object.
Each
of
them
have
two
fields.
Wine
is
called
spark
another
one
is
called
the
status,
so
basic
concept
of
controller
pattern
is.
A
Let's
take
an
example
that
how
long
who
are
handling
the
self-healing
say:
I
have
three
replicas
of
three
node
at
one
engine
talking
to
all
of
them,
and
the
engine
company
talk
to
the
manager
now
suddenly
notes
reach
down
low
an
engine
will
tilak
detect
that
zero
immediately
and
Mark
the
critical
three
as
red
as
arrow
in
the
in
the
engine
status
field,
and
that
information
was
picked
up
or
going
to
be
pick
up.
Immunity
by
the
manager
so
manager
found.
Ok,
there's
the
bad
replica
I.
Don't
you
remove
it?
A
A
But
as
you
notice,
the
top,
the
member
of
current
healthy
replicas
in
the
warning
status
has
been
reduced
to
two
because
we
lost
one
healthy
replica,
so
many
to
design
that
yeah.
We
see
that
this
is
a
one
replica
difference
between
the
world.
We
have
right
now
at
what
we
desired,
so
we're
going
to
create
another
replica.
A
The
manager
found.
Another
note
note
4,
which
is
available
and
the
space
is
suitable,
so
they
create
a
replica
for
on
the
note
4
and
after
the
replica
for
get
start
up.
The
replica
for
we
are
planning
here
we're
at
replica
for
in
the
engines
back
at
winching
will
receive
the
spec
say:
ok,
so
I
suppose
we
have
a
replica
for,
but
I
don't
have
now
so
I'm
going
to
add
a
replica
for
in
my
status
and
that
engine
are
going
to
connect
two
replicas
for
and
start
process.
We
call
the
rebuilding.
A
The
rebuilding
process
is
basically
copy.
The
data
from
existing
housing
replicas
to
the
new
replicas
and
to
make
sure
that
we
have
the
correct
residency
of
our
warnings
once
rebuilding
is
down.
The
replica
floor
will
be
marked
as
the
brain,
and
you
see
the
currently
housed
replicas
count
on
top
right,
well,
update
to
3
and
then
use
this
our
stack.
A
A
All
right,
okay,
so
what
I'm
looking
right
now
is
a
rancher.
Our
cook,
that
is
management
platform
and
I,
have
a
demo
class.
The
setting
up
here,
I'm
going
to
currently
this
is
the
storage
project
project
is
the
concept
of
rancher,
which
is
just
a
bunch
of
namespace
nosing
fancy,
and
we
are
going
to
deploy
a
long
word
using
the
Rancher
apps.
A
So
here
is
the
long
current
catalog
in
the
Rancher
apps
and
you
can
just
click
view
details
and
then
you
can
start
in,
but
now
we
are
going
to
I'm
going
to
demonstrate
how
life
upgrade
works
later
so
I'm
going
to
choose
a
slightly
older
version.
I
want
to
just
want
to
name
it
as
the
local
system,
let's
click
Launch,
so
the
app
is
the
way,
also
a
way
to
aggregate
all
the
related
components
like
a
different
deployment,
different
services
into
one
catalog.
A
A
As
you
see
here,
party
forge,
we
have
a
pass
which
we
use
going
to
use
as
store
the
data
in
the
replicas
and
for
each
node.
You
have
add
more
disk
to
it
through
this
way
or
disable
it
or
doing
something
like
say,
disable,
scheduling
for
the
nodes
or
disable
scheduling
for
the
disk
and
okay.
You
can
you
also
say
that
how
many
volumes
has
been
allocated
in
this
node?
This
basically
says
space
take
by
the
replicas
and
was
this
real-time
usage
of
this
disk
and
this
node.
That
was
the
total
size
of
this
disk.
A
The
next
half
is
the
volume
which
we
don't
have
any
run
so
I'm
going
to
skip
that
for
now,
the
next
one,
okay,
you
see
the
arrow
here
why
this
one
is
the
back
half.
So
in
order
you
have
a
backup,
as
I
mentioned,
it's
basically
a
defined
place.
You
can
backup
your
snapshot,
you,
but
you
need
to
find
that
place
first.
So
let's
go
to
the
settings
as
set
a
backup
target.
A
A
Okay,
so
let's
take
a
look
at
back
at
the
long
run
dashboard.
So
now
the
WordPress
application
has
been
installed.
It
should
be
it's
being
installed,
so
the
lungworm
should
have
worried.
Yeah
see
so
well
France,
as
I
said
before,
requires
Tim
Williams
and
the
WordPress
is
creating
that
inside
long
can
receive
that
request
and
creating
volumes,
and
you
can
see
that
this
color
means
that
we're
in
progress
is
either
creating
attaching
or
detaching,
and
if
you
take
it
volume
page
now,
you
can
see
that
two
morning
has
been
attached.
A
A
Ok,
so
the
Mary
DB
stuff,
but
this
hasn't
been
completed.
Yet,
let's
take
a
look,
a
look,
okay,
now,
I
think
he's
going
to
be
out
very
soon.
Yeah
you
see,
narrative
is
running
so
the
WordPress.
It
depends
on
Mara
TV.
So
you
have
to
be
married.
We
have
to
drown
first,
okay,
I
think
yeah.
The
WordPress
will
be
running
soon.
A
A
Hopefully,
you
got
your
brain.
Yes,
all
right,
let's
add
posts,
a
could
come
demo,
what's
not
very
exciting,
though
okay.
So
now
this
beta
has
been
written
into
the
long
haul
system
and
now
I'm
going
to
try
what
I
mentioned
before
a
life
upgrade
the
first
part
that
will
be
I
just
clicked
upgrade
here.
I'm
going
to
up
this
one
will
operate
the
local
manager
and
just
click
upgrade
all
right.
I
guess
I
need
to
refresh
the
page,
and
during
this
time
of
course,
the
wordpress
you're
walking
no
problem,
you
can
click.
A
A
Okay,
now
it's
back
now,
you
can
see
that
the
version
number
has
been
changed
to
zero
five
zero
and
the
next
step
we're
going
to
do
an
upgrade.
You
can
see
that
because
the
local
manager
is
different
from
the
orchestration
level,
a
local
engine
is
the
one
provided
data
pass.
So
that's
your
upgrade
on
engine.
This
is
far
more
risky
than
what
I
did
before,
but
let's
try
it.
If
it
works.
A
Okay,
let's
go,
have
NJ
upgraded,
a
doing
that
right
now
turns
out
what
Fry's
need
upgrade
as
well.
So
I
click
this
one
this
to
process.
It
happens
simultaneously,
we'll
press
finish
very
fast,
but
that
I
guess
they
have
a
back
yeah.
It's
supposed
to
be
five
to
one
and
let's
take
a
look
at
the
engine
yeah
the
engine
works
now
because
you
didn't
you
see
that
green
arrow
in
front
is
that
it's
gone,
and
you
can
see
that
yeah
and
immersion
has
been
upgraded
to
the
latest
one.
A
So
now,
you're
looking
at
is
the
volunteer
health
page.
So
it
gives
you
the
every
teacher
of
this
volume.
You
can
see
that
where
the
replicas
resign
at,
what's
notice,
running
on
and
even
was
the
data,
was
the
location
of
the
data
is
stored.
On
that
note,
so
you
can
take
a
snapshot
or
create
backup
easily.
A
Another
feature
we
have
is
we
have
a
recurring
snaps
on
the
backup
schedule,
which
means
that
you
can
make
you
can
decide
that
when
you
would,
you
want
to
take
snapshot
on
the
backup
doing,
say
now
low
traffic
hours
like
3:00
in
the
morning?
Let's
try
that
yeah.
In
fact,
I
don't
have
that
much
time,
so
I
just
take
a
backup.
Every
minute,
I
want
to
return
the
last
four
or
five
of
them
and
save
so.
Here's
how
we
implement
this
feature.
A
Okay,
it's
down
by
using
the
cuvee
nice
chrome
job,
it's
hundred
percent,
just
chrome
job,
all
right!
So
now
you
can
see
that
this
now
the
scheduling
has
been
scheduled
and
you
can
see
the
last
backup
is
be
created.
Okay,
it
was
a
minute
ago.
Now
it's
a
few
seconds
go
means,
and
now
the
backup
is
still
going
on
and
also
now
it's
in
the
new
version.
You
can
see
that
do
I
have
more
information
present
in
this
dashboard.
A
You
know
who
the
PvP,
who
the
PVC
is,
that
was
named
in
stage
name
space
Liang
at
what
part
they
are
using
using
the
warning.
This
is
one
purely
based
on
the
users
refer
feedback
that,
because
the
kubernetes
provides
storage
vendor
only
a
worried,
random
PVC
name,
they
have
no
way
to
associate.
That's
warn
you
mr.
actual
workload,
so
we
added
the
feature,
make
it
easier
for
the
user.
A
A
Yeah
so
I,
just
like
I
just
demonstrated
kubernetes
August
rate
the
data
plan,
because
the
engine
part
and
the
replica
part
are
scheduled
to
the
Associated
nodes
by
the
kubernetes
and
also,
as
you
said,
as
you
see,
each
replica
contains
the
full
copy
of
the
volume
and
you
can
see
easily
well
that
data
resides
on
your
nodes.
So
that
means
only
one.
Correct
replica
is
all
that
neat.
If
you
really
have
word,
disaster
happens
and
try
to
recount
recover
your
whole
volume.
A
That's
this
worry
transparent
compared
to
many
others,
to
which
windows
approach,
which
they
just
have
a
blog
presenting
you.
But
you
don't
really
know
where
the
data
is
that
or
not
also
kubernetes
helped
the
increase
resiliency
like
what
I
mentioned
the
node
status
was
we
called
from
kubernetes.
So
if
any
node
is
down,
we
are
going
to
skip
that
node
for
the
scheduling
and
do
some
fabric
failover
recoveries.
But
in
fact
the
core
logic
of
the
data
plan
not
really
relay
on
that.
That
depends
on
that,
because
community
is
it
kind
of
slow
and
reflecting?
A
What's
the
power
statuses,
so
we're
the
constant
energy
in
the
self
will
decide
if
it
replicas
is
bad
purely
based
on
its
connection
to
the
replicas,
so
that
is
rich,
so
that
is
makes
it
very
faster
and
the
port
status
monitoring
yeah.
We
still
rely
on.
We
can
some
information
for
the
kubernetes
if
something
goes
wrong
and
we
automatic
connect
lock.
If
something
unexpected
happens
to
that
replicas
or
ng
part.
Another
feature
is
we
can
automatically
retouch
the
volume.
If
you
reboot
your
communities
book
node,
we
use
the
special
field
called
boot.
A
A
Of
course,
there's
always
a
few
issues
when
you
working
with
kubernetes.
Well,
I
don't
know
if
everybody
agrees,
but
at
least
as
a
storage
vendor
there's
two
major
things
we
had
the
first
one
is
the
driving
interface.
In
fact,
it's
keeping
changing
for
almost
every
other
major
version
before
the
community
has
a
standard
quarter.
Flax
volume,
which
missing
probably
not
going
to
DGA
but
in
the
end,
is
winter
TGA
and
become
one
of
the
standard
and
the
things
went
out.
A
11
the
CSI
served
Austria
was
out
as
he's
in
the
beta,
so
we
start
started
using
si
si,
and
then
we
found
the
CSI
two
out
of
four,
which
is
one
dollar
off
brake
cam
ability
upstairs
at
one
tom,
is
a
third
of
three
and
the
CSI
wonder
all
into
one.
Thirteen
yeah
brakes
complete
backwards
completely
again,
so
the
long
haul
adapts
by
using
a
once
more
to
we
call
the
driver
deployer
as
I
show
you
before.
That's
one
where
automatic
act
was
the
environment.
This
was
recruiting
area
settings.
A
What
your
automatic
version
they
decide
was
the
proper
way
to
deploy
your
driver.
It
wasn't
correct
configuration
for
that
another.
What
issue
is
related
to
the
controller
pattern
were
using.
So
if
you
have
been
fully
minion
with
the
combat
is
control,
we
know
that
we're
normally
using
something
called
informal,
easter-egg
former
and
the
informer
contains
another
object
called
Lister
which
is
easily
used
to
access
the
cupola
in
CRD
objects
in
the
cache.
A
That's
why
it's
much
faster
than
using
the
API
directly,
but
we've
found
out
the
Lister
can
stamp
I
retain
the
style
information
even
with
OneNote.
So
here's
the
some
example.
The
correct
the
correct
sequence
is
something
like
this
client
issue.
A
request
to
the
API
server
create
a
new
object,
API
server
using
the
sed
watch
mechanism.
Now
the
Lister
yeah,
usually
you
now
you
need
to
update
your
internal
cache
and
the
object
will
in
the
cache
and
the
client
will
go
to
the
listed
say.
Ok,
what's
my
new
object?
A
It's
there
and
this
one
is
correct,
but
sometimes
the
race
condition
happens.
The
first
time
the
same
client
issue
a
new
required
to
create
a
new
object,
but
before
API
server
notified
the
Lister
to
update
it's
the
internal
cache
trying
to
make
the
query
for
the
new
object.
Of
course
object
is
not
there.
The
problem
is
the
process
of
client
initial
request
and
the
Lister
updating
internal
cache
is
a
synchronized
process.
They're,
not
really
secretive
sequentially.
So
the
client
will
see
this
kind
of
strange
result.
I
create
object.
Well,
it's
down.
A
There
I
update
the
object,
but
the
change
is
now
there
so
long,
we'll
have
to
implement
additional
levels
of
a
cache.
Validation
basically
means
that
whenever
we
do
a
modification
to
your
object,
we
need
to
check.
We
need
to
wait
for
the
Lister
to
return
us
a
newer
version.
If
we
can
check
you
can
check
that
by
using
the
reverse
version
inside
that
object
as
long
as
after
the
after
we
can
confirm
that
least
the
return
last
a
new
version.
A
We
cannot
continue
all
right,
so
the
upcoming
local
release
is
where
we
stare
at
all
things
that
all
is
well
be
the
one
we
announced
beta
and
yeah.
We
need
to
go.
Do
some
react
active
there
of
the
engine
level,
the
the
engines
and
replicas
will
be
wrong
and
the
process
instead,
the
demoiselle
part
instead
of
each
part
for
each
engine
replicas.
A
Now,
that's
give
us
a
few
battery
result
that
the
current
solution,
so
we
should
be
able
to
speak
out
warning,
attach
and
detach
process,
because
whenever
you
start,
even
though
the
container
on
the
part
of
the
starts
and
stop
work
as
it's
not
as
fast
as
you
start
to
stop
the
process
another.
Why
is
pretty
major?
Because
kubernetes
has
the
part
of
limitation
of
110
so
basically
means
that
you
can
only
have
100
10
parts
per
node
and
we
already
have
some
users
hitting
that
because
we're
using
pretty
much
many
parts
per
volume.
A
So
we're
going
to
avoid
that
by
running
just
the
demoiselle
part
on
each
node
and
the
third
reason
is
now:
we
can
get
guaranteed
the
CPU
resources
for
the
daemon
set
apart
before
we
can
really
not
do
that,
because
the
Canadian
scheduling
is
doesn't
know
about
storage
and
the
local
scheduler
doesn't
really
know
about
CPU.
So
now
we
basically
we
the
schedule,
the
demon
center
part.
C
A
So
yeah
everybody
know
about
rooks.
Brookshire
is
the
position
to
be
storage
operator?
It's
not
a
really
a
full
stack
storage
solution.
Rooks
will
lay
on
the
staff
and
our
because
I
works
for
telecom
and
other
campaigns
before
we
know
that
staff
there's
many
users
the
complaints
that
is
way
too
complex.
It's
very
hard
to
maintain
you,
some
kinda
know
where
you
practices
and
some
time
you
probably
configure
your
crash
map
wrong.
So
we're
not
looking
to
replace
route,
but
we
are
lists.
A
D
A
A
Alright,
so
in
fact
I
have
backup
just
for
that.
So
in
fact,
we
have
tried
a
few
ways
to
expose
the
block
device.
We
try
the
MPD,
which
is
very
easy,
but
not
not
reliable
at
all.
It's
easily
close
to
Colonel
crash
and
we
try
to
use
the
TCM
you
and
spend
quite
diverse
to
that
and
I
contribute
you
through
kernel
patches
to
TCM
you
to
make
to
spire
to
increase
the
performance.
A
But
in
the
end
we
found
the
limitation
that,
because
it's
kernel,
so
it's
really
taking
a
long
time
to
get
down
to
the
end
users.
So
I
don't
want
to
limit
it
out
user
because
that
we
realize
TCM
you
and
also
we
try
to
fuse
well,
it's
too
slow
and
in
the
end,
the
block
device
were
using
the
tea,
GTE
and
ice
classic
combination.
The
tea
GTD
is
the
kind
of
worried
old
ice
guards,
the
use
of
series
user
space
framework,
but
it's
very
stable.
A
Its
I
didn't
see
crash
the
curl
anytime
at
all,
and
the
process
is
most
likely.
We
ever
get
a
little
bit
complicated
here,
but
basically
the
engine
will
start
a
GGG
target
inside
container
and
then
engine
will
talk
to
the
host
and
using
the
ice
cassia,
the
meaning
of
the
host,
to
connect
you
that
you
digitize
the
TTT
target.
The
reason
is
ice
custody
which
is
organized
gauzy.
It's
not
really
namespaced.
A
A
In
long
form,
every
replica
is
treated
the
same,
so
we're
not
really
prioritized
anyone.
Basically,
when
you
do
a
write,
you
write
to
the
old
replicas
simultaneously.
When
you
read
you.
Basically,
we
basically
put
employee
round-robin
mechanisms
that
you
just
read
another
replicas
and
whatever
replicas
is
the
readwrite
position.
We
can
read
from
it.
Yeah.
A
A
Why
yeah
sure
sure
you
can
remove
the
from
the
cluster
yeah
because
after
you
remove,
is
to
basically
chance
that
the
worst
case
scenario
is
one
replicas
gone
and
then,
as
I
demonstrate
on
that
controller
pattern,
slide
we're
going
to
review
the
replicas
for
another.
You
know
another
node,
so
the
warning
will
stay
healthy.