►
From YouTube: Zero Downtime Deployments: Controlling Application Rollouts and Rollbacks - Christopher Hanson
Description
Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Zero Downtime Deployments: Controlling Application Rollouts and Rollbacks - Christopher Hanson, RX-M LLC
Kubernetes provides several types of automation controllers to deploy and manage applications at scale. Each exposes features that provide for zero downtime upgrades and rollbacks but not all controllers are created equal. Deployments provide the widest application compatibility but work best with stateless applications; the aptly named StatefulSet controller provides guarantees that benefit stateful applications, but how do the features of these controllers behave in practice? Learn about these and other Kubernetes controller types through a series of demonstrations that will reveal their features, benefits, and drawbacks. The session will also explore the primitives that enable granular control over rollouts so that you can choose the settings that work best for your application.
https://sched.co/ZeoV
A
Hi
and
welcome
to
zero
downtime
deployments,
where
we're
going
to
take
a
look
at
application,
rollouts
and
rollbacks.
I'm
going
to
be
your
speaker,
hey
there.
My
name
is
chris.
I
am
a
consultant
and
advisor
and
sometimes
speaker
you
can
see
some
of
my
accolades
there.
So
without
further
ado,
let's
get
started
controllers
in
kubernetes
are
control
loops,
that
watch
the
cluster
state
and,
if
necessary,
they're
going
to
introduce
some
sort
of
change
right.
So
the
control
loop
holds
desired
state.
A
It's
querying
the
cluster
for
actual
state
and
if
those
things
don't
match,
then
it
takes
action.
So
we're
going
to
focus
on
the
deployments
in
the
staple
sets
specifically
because
they're
primarily
for
long-running
applications,
and
we
want
to
look
at
their
behaviors
when
it
comes
to
rolling
updates.
So
first
look
at
deployment
here.
A
It
describes
the
desired
state
for
an
application
right,
so
we're
going
to
take
the
image
that
we
want
to
run
the
affinities
and
anti-affinities
all
those
things
and
put
them
in
the
deployment
manifest
and
the
deployment
isn't
alone
in
making
sure
that
the
application
gets
deployed.
It's
actually
a
wrapper
for
another
controller,
called
a
replica
set
which
enforces
the
replication
factor.
Replication
factor
is
of
pods,
of
course,
and
then
within
pods
are
application
containers.
A
But
what
that
means
is
that
if
we
create
a
deployment,
then
in
the
cascade
we're
going
to
create
the
replica
set,
the
pods
and
the
containers,
and
if
we
delete
a
deployment,
we
do
that
in
a
cascade
as
well.
The
rolling
update
feature
that
we
get
with
deployments
will
change
the
actual
state
of
the
cluster
to
the
desired
state,
with
zero
downtime
right,
no
no
client
will
be
sent
into
any
black
hole
right.
A
That's
the
goal
here
deployments:
do
support
a
wide
array
of
application
types,
but
really
do
have
a
feature
set
that
works
best
with
stateless
apps,
so
pod
host
names
are
not
really
predictable.
They
have
hashes
that
we
can't
really
predict
ahead
of
time
so
talking
to
an
individual
or
sending
client
traffic
to
an
individual
pod
replica
is
a
little
bit
more
difficult.
A
If
you're
going
to
embed
a
persistent
volume
claim
within
a
deployment
template
for
a
pod,
remember,
this
is
a
template
right,
you're
going
to
have
many
replicas,
and
so
pvcs
typically
have
unique
identities.
I
can't
parameterize
the
name
of
the
pvc
within
the
template,
so
if
I
have
it
in
rewrite
many
mode,
then
that's
great
otherwise.
Well,
that's
what
the
stable
stable
set
is
for,
which
we'll
look
at
later
so
replica
sets
are
a
top-level
citizen
in
the
kubernetes
api.
A
Their
primary
job
is
really
to
reconcile
desired
state
versus
actual
replication
factor
right
so
somewhere
on
the
cluster.
There
are
a
number
of
pods
actually
deployed,
and
the
replicas
add
needs
to
reconcile
that.
So
within
the
replica
set
object
here
we
see
two
important
things
desired:
replicas
and
then
the
label
query:
the
label.
Query
is
going
to
be
used
to
do
that
reconciliation
and
then,
when
the
answer
comes
back,
a
decision
has
to
be
made.
Are
there
seven
pods
great,
do
nothing
right?
No
action
needs
to
be
taken.
A
If
there
are
six,
then
we've
got
to
add
a
new
one
right,
similar
to
cube
ctl,
run
to
add
a
new
pod.
If
there
are
eight,
then
the
replicas
that's
going
to
kill
one,
and
so
this
guarantees
the
availability
of
n
number
of
application
replicas
across
nodes
in
the
cluster
across
availability
zones
and
such
right
they
can
be
used
independently
of
deployments,
but
if
you
do
use
them
independently,
they
lose
the
rolling
update
feature
that
we're
going
to
look
at.
A
However,
there
are
some
cd
platforms
like
spinnaker,
which
you'll
use
replicas,
that's
directly
for
other
patterns
like
red
black.
You
will
see
them
used
in
some
cases
now.
The
relationship
between
deployments
and
replica
sets
is
typically
a
one-to-many
right.
Where
we've
got
a
single
deployment,
that's
got
multiple
rss
underneath
it.
This
is
because
an
rs
is
a
snapshot
of
an
application
version.
So
when
we
change
the
image,
what
happens?
Is
we
hash
the
the
new
manifest?
A
This
allows
us
to
roll
out
with
zero
downtime
right,
so
it
lets
us
essentially
increment
the
desired
replicas
of
the
new
replica
set
and
decrement
the
desired
replicas
of
the
old
at
the
cadence
that
we
specify.
It
also
allows
us
to
have
number
robots,
because
if
I
want
to
roll
back
to
an
earlier
version,
I
can
just
do
that
same
pattern
right
and
I've
rolled
back
my
application.
It's
also
kind
of
lightweight
tracking
revisions
of
an
app.
So
we'll
look
at
the
history
feature
as
well.
A
So
the
idea
behind
rolling
updates
is
we're
going
to
roll
out
those
changes
at
the
control
grade
right.
New
pods
are
rolled
out
before
old.
Pods
are
terminated
because
we
want
to
have
that
new
replica
up
before
any
of
the
old
ones
come
down.
Now
the
the
new
pot
doesn't
go
to
the
exact
same
node.
It
is
independently
scheduled
and
gets
scheduled
on
a
node
that
has
available
resources
based
on
the
resource
request
and
other
factors,
and
the
default
setting
ensures
that
at
least
75
of
pod
replicas
are
available
throughout
the
update
right.
A
So
we're
not
going
to
tear
down
below
75
availability.
You
know:
that's
tweakable,
that
we'll
look
at
in
a
subsequent
slide.
Client
traffic
is
then
load
bounce
across
all
available
pods
as
well.
If
you
want
to
see
the
live
in
progress,
roll
out,
you
can
like
anything
else
in
kubernetes.
You
can
watch
that
for
updates.
A
A
We've
got
some
some
settings
here
that
will
give
us
10
replicas
and
give
us
some
time
in
between
the
deployments
for
minimum
ready
seconds
and
then
also
the
default
settings
for
max
urgent
max
unavailable,
which
we'll
talk
about
going
forward.
They're
explicitly
set
here,
but
those
will
be
the
defaults
anyways
and
we've
got
a
application
here
that
simply
outputs
its
ip
and
its
hostname,
so
that
it
will
show
us
like
which
one
is
being
communicated
with
from
a
client.
A
So
we
will
first
set
up
a
watch
and
so
in
the
interest
of
time
I'm
just
going
to
grab
some
commands
that
I've
got
going
so
we're
going
to
watch,
keep
ctl,
get
pods
and
then
we're
going
to
use
the
key
value
pairs
here
for
the
for
the
deployment
right.
So
this
is
going
to
tell
me
no
resources
found
right
because
I
haven't
deployed
anything
yet,
but
let's
go
ahead
and
keep
ctl
apply.
That.
A
And
we'll
see
that
we've
got
10
pods
running
now.
What
I
want
to
do
is
run
a
client
on
this
other
terminal
here,
and
so
I'm
just
going
to
run
a
client
pod
with
busybox
1.27,
so
that
I
can
run
a
loop
and
so
my
loop
is
going
to
do
a
wget
against
the
service
and
what
that
means
is.
I
also
need
to
create
the
service
so
going
back
over
here
we're
going
to
expose
our
deployment.
A
So
we
expose
that,
and
now
I
can
run
my
loop
and
what
will
happen
is
you'll,
see
that
it
should
query
and
see
that
we've
got
the
certain
rs
rs
hash,
but
we've
got
different
endpoint
pods
that
we're
connecting
to
right.
As
we
see
change,
we're
going
to
see
this
rs
hash
change
in
each
one
of
the
requests
so
we'll
let
that
go,
and
so
we've
got
our
watches
established.
A
So
let's
introduce
a
change
so
here
we're
going
to
set
a
new
image
that
just
uses
a
different
tag
same
so
up
here,
host
info
latest
we're
going
to
go
to
the
alpine
version
and
we're
going
to
record
that
change.
So
we
can
actually
see
it,
and
so,
let's
take
a
look
at
the
behavior
that
happens
out
of
the
box.
A
So
we
get
a
number
of
new
pods
that
are
created
immediately
and
one
terminated
immediately
and
so
most
of
our
load.
Balancing
just
right
now
is
still
going
to
the
same
pods,
except
for
there's
our
first,
I'm
just
clicking
on
it,
but
there's
our
first
request
to
the
new
set
of
pods
and
obviously
we'll
see
that
increase
over
time,
so
lots
of
change
happening
in
the
cluster
right.
A
We've
got
25
surge,
25
surge
and
25
unavailable,
so
we're
going
to
see
a
quarter
more
pods,
each
iteration
through
and
so
there's
some
more
terminations
cleaning
up,
pods
after
they've
been
replaced
with
new
ones,
and
so
now
all
of
our
requests
are
going
to
the
new
set
of
pods
based
on
the
new
hash,
the
566
hash
over
here
with
our
client
all
right.
So
now,
let's
look
at
undoing
that
because,
let's
say
you
know,
we
don't
want
this
version
of
the
application
something's
wrong
with
it.
A
A
So
it's
going
to
give
us
and
we
can
see
right
away
that
you
know
it
started
with
three
and
then
jumped
really
quickly
to
five
and
we
got
a
slight
pause
there
and
then
now
we've
got
a
few
more
old
replicas
pending
termination,
but
pretty
much
we've
got
everything
that
we
need
rolled
out
already
so
fairly
quickly
because
again
the
application's
lightweight.
So
it
doesn't
really
need
to
pull
it
now.
A
If
we
look
at
the
history
once
this
is
all
the
way
rolled
out
right,
it
tells
us
it's
rolled
out
and
we
can
look
at
the
history
for
our
deployment
right.
So
the
first
rollout
just
says:
essentially
none
so
sorry,
the
the
undo
essentially
says
none,
because
we
didn't
really
record
the
undo,
but
that's
our
undo
and
then
there's
our
change,
so
we're
back
to
the
old
application
revision.
So
if
we
wanted
to
interrogate
version
three,
we
could
roll
a
history
and
then
use
revision.
Three.
A
So
one
of
the
things
to
note
the
record,
as
you
saw
in
the
command
that
we
used
actually
records
an
annotation
for
the
change
cause,
but
in
the
case
of
the
undo,
just
as
not
so
it
doesn't
mean
there's
no
information
available.
It
just
didn't
set
an
annotation
by
interrogating
it.
We
can
actually
see
that
the
image
changed
as
part
of
revision
number
three
right.
We
did
the
undo,
so
it's
still
possible
to
get
the
change
cause.
A
Okay,
so
that's
just
kind
of
the
default
behavior
of
rolling
updates
and
interacting
with
them.
There
are
a
couple
other
features
that
we
can
leverage
when
it
comes
to
rolling
updates
pausing.
So
we
saw
that
once
we
introduced
the
change,
it
then
immediately
triggered
the
rollout
right,
but
deployments
rollout
trigger
can
actually
be
positive
any
given
time.
A
So
if
you
do
it
before
any
sort
of
changes
introduced,
then
you
can
issue
several
commands
to
make
different
changes
and
it
won't
trigger
anything
until
you
resume
it
and
then
all
the
changes
made
prior
to
resuming
are
rolled
out.
Also
in
the
middle.
You
can
pause
the
rollout
and
do
something
like
confirm
settings
before
the
whole
thing
completes,
and
so
then,
in
that
case,
resuming
just
simply
finishes
the
rollout
that
was
stopped
in
progress.
A
So
let's
introduce
the
alpine
revision
again
right
in
this
case,
we're
again
updating
the
version
to
alpine
and
we're
going
to
see
nothing
happens
right,
no
rollout
was
triggered
our
client's
still
hitting
the
same
demo
or
sorry
the
same
revision.
Let's
introduce
a
second
change,
though,
just
to
show
that
that's
happening.
So
in
this
case,
we're
going
to
add
cpu
limit
right,
so
that
will
let
us
set
resources
for
that
for
the
container
right.
A
A
Pod,
we'll
do
minus
oh
yaml,
and
we
should
be
able
to
see
resources,
so
we've
got
the
resources
that
was
not
in
our
original
manifest
right
now.
One
couple
of
ways
that
we
can
control
rolling
updates
is
through
the
max
origin
max
unavailable.
So
we
have
the
default
settings
of
25
to
25,
but
what,
if
we
change
those
things
right,
the
the
max
surge
is
something
that
controls
the
number
of
pods
in
addition
to
the
desired
number
that
can
be
scheduled
during
the
rolling
update.
A
This
essentially
allows
us
to
break
up
the
iterations
into
larger
waves
right
we've
seen
in
previous
cases
that
there
were
a
small
number
of
pods.
We
can
break
this
up.
A
high
search
percentage,
however,
is
going
to
mean
more
resources
right,
because
each
application
replica
is
going
to
need
cpu
and
memory,
and
so
you
can
just
need
more
of
those
during
the
rollout
max
unavailable
ensures
that
a
minimum
number
of
pods
are
always
available
right.
The
idea
being
that
we
want
to
guarantee
that
that
client
traffic
is
delivered
through
pods
throughout
the
rolling
update.
A
A
A
Now,
let's
do
the
opposite,
where
we're
going
to
patch
again,
except
for
this
time
we're
going
to
set
max
unavailable
to
90
to
see
how
that
affects
again.
We
should
expect
the
opposite
where
we
don't
have
a
lot
of
surge,
but
we
have
a
lot
of
pods
removed
again,
we'll
introduce
another
change
and
see
what
happens
there
again.
Hopefully
that
meets
our
expectations,
so
we've
got
a
lot
of
containers
terminating,
but
then
also
because
we
need
to
replace
them.
A
Okay
moving
on
to
staple
sets,
so
staple
sets
are
intended
to
support
staple
applications.
These
have
some
other
features
regarding
stable
network
identities,
so,
unlike
deployment
based,
pods
staple
sets
have
unique
ordinals
so
that
we
can
actually
communicate
with
individual
members
a
little
bit
easier
and
persistent
storage.
So
there's
an
embedded
pvc
template
that
creates
a
persistent
volume
for
each
pod.
Now,
when
we
do
the
deletion
deleting
a
staple
set
will
delete
the
pods,
but
not
the
pvcs,
because
what
we
want
to
do
is
make
sure
that
it
was
a
mistake
right.
A
What
if
I
accidentally
deleted
my
staple
set
and
all
my
pvcs
went
with
it.
Well,
usually
in
the
scenario
where
the
persistent
volumes
are
dynamically
provisioned,
the
thing
that
keeps
them
from
being
returned
to
the
storage
pool
is
the
binding
between
the
pvc
and
the
pv.
So
this
allows
us
to
make
a
deliberate
decision
when
we
want
to
delete
the
pvcs
making
sure
that
if
we
accidentally
deleted
this
table
set,
we
don't
have
any
data
loss.
A
The
pod
identities
are
tied
to
volumes,
so
failed
pods
are
replaced
by
pods
with
identical
identical
identifiers,
which
means
that
existing
volumes
that
were
tied
to
a
given
pod
replica
easily
bind
back
to
that
new
pod.
That
represents
that
same
replica
right.
So,
even
though
it's
a
it's
a
new
pod,
that's
replaced
the
old
one.
It
gets
the
same
storage
that
the
old
pod
had
now
the
ordinals
give
us
some
features,
ordered
or
sequential
deployments
in
scaling
right.
So
you
can
see
here
we
start
with
pod
ordinal
zero.
A
Once
it
passes
its
readiness
check,
then
we
can
move
on
to
pod
one
once
it
passes
its
readiness
check.
We
move
on
to
pod
two
and
so
on
and
so
forth
when
we
scale
out
for
initial
deployments
and
then
when
we
add
more
replicas,
this
allows
us
to
have
applications
that
communicate
with
each
other
as
peers
right.
A
So
what's
the
point
of
a
readiness
jackets
to
indicate
when
the
application
is
ready
to
receive
requests
on
the
network,
and
so
when
we
have
peer-to-peer
communication,
we
don't
want
to
roll
out
a
second
replica
until
the
first
one
is
ready
to
communicate
with
it
so
that
they
can
join
consensus.
Algorithms
and
things
like
that.
A
No
guarantee
during
termination,
though
you
can
see
at
the
bottom
here
termination
happens
all
at
the
same
time.
If
you
need
an
ordered
termination,
the
workaround
is
just
to
scale
to
zero
first
and
then
delete
the
staple
set,
and
that
will
give
you
again.
It
will
tear
down
scaling
down
from
the
highest
ordinal
down
to
zero,
for
rolling
updates.
Pods
are
deleted
and
recreated
or
replaced
on
the
same
node.
A
So
unlike
deployments
where
we're
deploying
applications,
application
replicas
to
potentially
new
workers,
we're
going
to
deploy
a
new
version
of
the
application
to
the
exact
same
marker.
This
eliminates
that
need
to
detach
and
attach
network
volumes
from
the
old
node
to
the
to
the
new
node
right
that
could
be
a
heavyweight
operation.
A
Rolling
updates
will
support
the
undo
feature,
so
all
the
imperatives
we've
been
using
with
deployments
and
the
status
commands.
The
history
is
not
super
functional,
unfortunately,
and
pause
and
resume
is
not
supported
at
all.
So
it
does
have
some
overlap
in
terms
of
features
for
the
imperatives,
but
not
everything,
and
occasionally
you
may
run
into
a
scenario
where
you
have
to
force
a
rollback.
A
A
So
let's
look
at
a
scenario.
First
things.
First,
let's
get
the
storage
classes
just
to
show:
we've
got
two
storage
classes,
we've
got
the
local
storage
and
then
we've
got
an
nfs
client,
which
is
the
default
storage
class.
We're
going
to
use
for
the
network
attached
storage
scenario
in
this
case,
we'll
take
a
look
at
the
staple
set,
although
it's
rather
large
you'll
see
that
the
volume
claim
template
isn't
using
a
lot
of
data
and
it
doesn't
have
a
storage
class
request.
So
it's
just
using
the
default
storage
class.
A
A
And
we've
got
our
service
and
our
staple
set
created
you're,
going
to
see
the
rollout
right
so
initial
rule
app.
We
start
with
replica
0
and
it'll
go
to
1
and
up
to
4.,
and
while
that's
going,
I'm
going
to
start
a
client
over
here
again
in
this
case
we're
going
to
just
run
lcd
of
the
same
version,
but
just
call
it
ctl
and
that's
all
we're
going
to
use
is
the
ncd
cuddle
client
here
to
run
some
things.
A
A
And
then
we
get
a
member
list
just
to
see
that
they're
all
up
and
running,
we
can
see
that
one
is
just
creating.
So
remember
list
should
give
us
a
member
list
of
0
through
4
there.
Now
I'm
going
to
put
a
value
just
to
show
that
we
can
set
and
retrieve
data
and
then
we'll
start
a
loop.
So
in
this
loop
we're
going
to
get
the
value
that
we
set
and
then
also
get
the
endpoint
statuses
to
see
the
different
versions
that
have
been
deployed.
A
A
So
what
we'll
see
is
it's
again
just
like
the
deployment
update
in
terms
of
the
updates
that
we're
getting
from
rollout
status?
But
you
can
see
the
tear
down
happens
from
the
ordinal
four
now
on
to
ordinal
three
and
we'll
go
all
the
way
down
to
zero.
Now
what
you'll
see
over
here
with
the
client
is
that
we're
going
to
get
some
error
messages,
because
certain
endpoints
are
not
going
to
give
us
their
status.
A
But
what
you'll
see
every
single
time
is
that
we're
still
getting
the
data
right,
because
the
ncd
application
supports
that
the
various
members
can
still
answer
the
query
that
we're
sending
them
right.
We
may
not
be
able
to
query
individual
members
for
their
status
because
they're
in
the
middle
of
being
down,
but
the
data
is
always
there
right
and
that's
the
important
part,
and
so,
as
you
see
the
clients
answering
over
and
over,
they
are
giving
us
the
data
every
single
time
right
so
we'll.
A
And
it's
up
and
running
so
everything's
been
updated
right
and
so
the
last
two
members
here
in
the
very
last
query
from
our
client
didn't
answer,
but
now
we're
on
to
the
fact
that
they're
all
answering
again
and
of
course
the
data
is
still
there.
So
let's
go
back
right.
We
something's
wrong
with
this
version.
So,
let's,
let's
roll
it
back
again,
so
behavior
again
is
the
same
where
we're
tearing
down
from
the
highest
orbital
down,
which
is
what
we
expect
right.
So
in
this
case
I
didn't
ask
for
live
update
status.
A
So
let's
take
a
look
at
the
history,
as
this
thing
rolls
out
the
or
reverts
the
change,
rather
so
note
that
we
did
on
the
first
change
right
going
back
up
here.
We
did
record
it
but,
as
we
said,
the
history
is
not
very
workable
or
not
very
valuable.
The
second
change
we
didn't
set
a
record.
We
did
an
undo
and
started
to
roll
things
back,
but
still
didn't
get
anything
so
again.
If
I
want
to
interrogate
a
particular
revision
right,
I
don't
get
a
lot
of
information
here.
A
So
let's
take
a
look
at
that,
but
what
we're
going
to
do
is
perform
that
with
local
storage
volumes
and
we
still
have
the
client
running
we'll
just
make
some
changes
to
which
endpoints
it's
connecting
to,
and
we
want
to
start
a
new
watch
with
the
new
app
ncd
local
key
value
pair.
The
only
difference
between
the
old
old
network
attached
storage
version,
which
uses
the
default
storage
class
and
this
new
one
is
that
it
has
the
storage
class
name
here
so
that
we
definitely
get
the
local
storage
class.
A
Which
we
see,
if
I
can
spell
we've
got
our
watch
established,
let's
go
ahead
and
deploy
our
application,
and
so
now
it's
grabbed
a
local
pv.
So
if
I
do
cube
ctl
get
pv,
you
can
see
that
we've
got
several
being
bound
already
right.
So
our
level
storage
volumes
are
getting
used
now
again
we're
going
to
export
in
our
client,
some
endpoints,
but
different
names
right,
sts,
local
and
scs
local
is
the
service
name,
so
I'll
export
those
endpoints
and
again
we'll
do
a
put.
A
A
Okay,
so
the
staple
set
is
patched
now:
let's
introduce
the
change
again:
updating
the
minor
version
to
dot
2
4
and
witness
the
behavior
that
happens
here.
So
again
we
see
that
the
version
of
replica
4
changes,
the
version
of
replica
3,
is
going
to
change,
and
then
we
stop
right.
So
we're
going
to
see
when
the
members
start
to
report
their
values
over
here
that
the
last
two
are
24
and
the
other
three
are
18..
A
It's
going
to
terminate
that
pod
and
it
should
allow
us
to
create
a
new
one
to
replace
it
and
once
the
client
catches
up
and
can
query
all
the
endpoints
once
again,
we'll
see
that
the
version
remains
18
right,
even
though
it
was
deleted.
The
version
comes
back
as
the
old
version.
Now,
let's
say
I'm
doing
some
sort
of
phased
rollout
and
I'm
happy
with
the
phased
rollout.
A
Now
I
can
update
my
partition
to
0
and
that
will
let
me
roll
out
the
rest
of
that
change.
So
we
see
that
version.
2
now
updates
itself
version,
one
is
or
a
replica
one
I
should
say,
is
now
gonna
update
itself
and
then
we'll
see,
go
all
the
way
down
to
zero.
So
we
see
the
entirety
of
the
rolling
rolling
updates
happen
right.
So
once
we're
done
with
our
face
rollout,
we
everything
updates
and
now
everything's
up
to
3.24,
once
everything's
up
and
running
again,
our
client
throughout
has
availability
to
the
data.