►
From YouTube: TGI Kubernetes 028: Exploring CockroachDB on Kubernetes
Description
Index
0:00:00 - Intros
0:02:50 - Container World
0:04:00 - How to deploy web applications on K8s with Countour + LetsEncrypt
0:07:28 - Kubernetes in CNCF incubation
0:09:40 - Debugging scratch containers
0:13:19 - Stateful Sets and Stateful Workloads in Kubernetes
0:14:20 - Defining Stateful Workloads
0:20:15 - Understanding StatefulSets
0:20:00 - Jetstack Navigator
0:39:06 - Launching CockroachDB
0:40:04 - Inspecting the Cockroach Manifests
0:50:30 - Applying CockroachDB
0:51:18 - Readiness checks with CockroachDB
0:55:15 - Observing CockroachDB coming up in Kubernetes
0:56:39 - Approving the CSR
0:59:27 - Secrets
1:02:27 - Common name "root" with CSRs in CockroachDB
1:05:04 - Creating a database called "bank"
1:06:24 - Port forwarding and accessing the UI
1:11:33 - Getting kuard running
1:12:45 - DNS with CockroachDB
1:14:35 - Advanced features of the CockroachDB UI
1:16:49 - CockroachDB and etcd
1:20:30 - Taking down an instance of the cluster
1:28:02 - Debugging pods and pvcs after an outage
1:34:06 - Tearing down the cluster
1:40:16 - Q&A
Come hang out with Joe Beda as he does a bit of hands on exploration of Kubernetes and related topics. Some of this will be Joe talking about the things he knows well. Some of this will be Joe exploring something new with the audience. Ask questions, comment and help decide where things go.
Stateful service and applications are a hot topic in the Kubernetes world. In this episode I'll talk in general about how I view state on Kubernetes and then we'll explore a system built for a dynamic Kubernetes like environment -- CockroachDB
Links:
* Dave's guide to HTTPS Ingress. Contour + Let's Encrypt + cert-manager: https://blog.heptio.com/how-to-deploy-web-applications-on-kubernetes-with-heptio-contour-and-lets-encrypt-d58efbad9f56
* Blog post based TGIK 027 on securing the k8s dashboard: https://blog.heptio.com/on-securing-the-kubernetes-dashboard-16b09b1b7aca
* Kubernetes is graduating! https://lists.cncf.io/g/cncf-toc/topic/vote_kubernetes_moving_to/12935280?p=,,,20,0,0,0::recentpostdate%2Fsticky,,,20,2,0,12935280
* Debugging "From Scratch" containers: https://ahmet.im/blog/debugging-scratch/
* StatefulSet basics: https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/#writing-to-stable-storage
* Jetstack Navigator for managing Cassandra and Elasticsearch: https://github.com/jetstack/navigator
* Metacontroller catset: https://github.com/kstmp/metacontroller/tree/master/examples/catset
* Secure install of Cockroachdb: https://www.cockroachlabs.com/docs/stable/orchestrate-cockroachdb-with-kubernetes.html
* "Scaling Raft" talks about raft in Cockroachdb (from 2015 -- may be out of date?) https://www.cockroachlabs.com/blog/scaling-raft/
A
A
So
yes,
I'm
really
excited
about
this
episode.
Well,
first
of
all,
so
my
name
is
Joe
Beda
I'm,
a
CTO
and
founder
of
this
company
called
hep
to
you,
we're
helping
to
bring
kubernetes
and
cloud
native
thinking
to
more
companies
and
more
enterprises,
and
just
more
people
in
general
and
TG
I
kubernetes
is
a
chance.
Every
week,
every
Friday,
where
I
play
with
some
technology,
explore
some
aspect
of
kubernetes
I.
Do
it
live?
A
There's
people
in
the
chat
over
here
and
it's
scrolling
away
on
me
there
that
that
join
me
ask
questions,
and
so
I
really
like
to
make
it
as
interactive
as
we
can
with
the
the
time
delay
that
we
see
with
the
chat
with
YouTube
so
yeah,
so,
okay.
So,
first
of
all
so
Lamanna
good
to
see
you
again.
We
have
Bismarck
and
Jay
Hernandez
and
partha
bond
from
Atlanta
Alex.
So
so
sill
you
from
Basel
Hans.
A
Is
he
like
last
week's
episode?
Okay,
that's
great,
because
I
have
a
blog
post
him
to
talk
about
Alex
good
to
see
you
lien
in
New,
York
yeah,
Seattle,
weather
I
was
down
in
San
Francisco
yesterday
and
it
was
raining.
It
was
hilarious.
All
the
San
Francisco
people
can't
handle
rain
David
Kevin
from
Toronto
Carlos
hola
Eddie
from
Guatemala
Syed
in
London
wali
in
Saudi,
Arabia
Saudi.
Is
it
Saudi,
Arabia's,
correct,
I
or
do
people
just
say
Saudi
I
have
no
idea
you
can
tell
I'm
clueless,
but
but
what
thanks
for
joining
us?
A
We
have
Christian
from
West
Orange,
New,
Jersey,
Erin
and
ilish
Alpesh
Steve
Bob,
let's
see
and
then
my
hip,
all
from
Sweden
/
India
I'm,
not
sure
how
that
works.
Your
you
got
to
be
in
one
of
those
right
now
or
maybe
you're
in
the
air
I.
Don't
know
I
mean
you
could
be
flying
from
Sweden
India,
well,
it'd,
Kyle,
Mike,
good
to
see
hoho
Stefan,
alright,.
A
Because
this
is
but
but
thank
you
all
for
joining
me-
I
really
appreciate
it
and
I
know
that
hefty
owe
was
at
a
container
world.
This
week
we
had
a
booth
and
I
know
there
were
a
bunch
of
folks
from
lieutenant
Samir
and
his
crew
that
stop
by
the
booth
I
wasn't
there,
but
they
told
Sarah
who
was
there
that
they
get
together
in
a
conference
room
and
watch
every
time.
So
you
know
if
you're,
watching
this
one
welcome.
If
not,
hopefully
you
can
catch
it.
A
A
So
thank
you
for
joining
me.
I'm
really
excited
about
this
episode.
So
this
week
we're
going
to
be
covering
stateful
sets
and
in
cockroach
DB,
and
talking
about
some
of
the
approaches
to
to
running
stateful
workloads
on
top
of
kubernetes.
What
works,
what
doesn't
work
and
and
hopefully
give
a
little
bit
of
context
there
and
then
play
with
cockroach,
because
it's
a
it's
really
an
interesting
system
and
and
the
person
who's
doing
all
of
the
all
of
these
sort
of
kubernetes
worked
for
cockroach
DB.
A
A
So
the
first
thing
last
week,
I
teased
this
blog
post
that
Dave
Cheney
was
going
to
be
writing
about
how
to
deploy
web
applications
with
kubernetes
and
contour,
and
let's
encrypt
and
jets
tax
cert
manager.
That
would
have
been
a
long
title.
I'm.
Sorry
we
we
didn't
put
that
in
the
title
there,
but
this
goes
into
a
lot
of
detail
about
how
to
use
all
these
things
together
and.
A
So
a
lot
of
good
progress
there,
thanks
for
writing
that
Dave
and
then
and
then
I
took
the
last
episode
where
we
talked
about
securing
the
dashboard
and
using
the
I'm
using
the
OAuth
proxy
and
all
that
stuff
and
I
turned
that
into
a
blog
post
also,
and
so
just
so
it
to
be
more
of
a
reference.
I
found
some
of
my
own
details
that
I
chase
down
in
there
some
bugs
and
stuff.
So
that
was
really
good.
A
One
of
the
things,
though,
if
you
go
down
in
the
comments,
if
you
show
responses,
Joe
here
was
saying
that
there's
a
bunch
of
Forks
out
there
at
the
auth
to
proxy
and
sort
of
the
the
bitly
code
is
not
super
maintained
right
now
and
so
really
interested
to
to
see.
You
know
if
anybody
has
any
details
on
those
that
you
know.
A
B
A
See
so
we
have
John
from
Wyoming
Ibrahim
good
to
see
you
also
Ralph
down
in
rainy
San,
Francisco
I,
assume
I,
don't
know
where
Ralph
is
these
days
with
a
drink
in
his
hand
as
well.
It's
past
1:00,
let's
see
Chris
and
Schad
from
Cincinnati
good
to
see
y'all
yeah
all
right.
So,
let's
see
so
other
stuff,
that's
going
on
Tom
from
New
Zealand
how's
it
going
you're
living
in
the
future.
It's
already
Saturday
morning
there
right
and
Brad
from
Madison
all
right.
A
A
The
different
classes
of
projects,
and
so
there's
there's
graduated
projects,
there's
incubating
projects,
and
then
there
were
these
things
called
inception
projects,
but
they're
re
working
that
and
there's
gonna
be
like
sandbox
projects,
and
it
turns
out
that
technically
kubernetes,
even
though
was
a
founding
project,
was
a
incubating
project
in
the
CNC
F.
Well,
we
we've
got
enough
act
together
that
we
can
meet
all
the
criteria.
A
We
have
governance
and
all
that
stuff
that
that
we
finally
are
moving
towards
making
kubernetes
a
full
project
to
the
CNC
f,
it's
kind
of
a
little
bit
of
a
you
know,
ceremony
type
of
thing,
but
it's
kind
of
exciting
to
see
that
happen.
Also
yeah,
so
yeah
so
Ralph
says
it's
raining
in
San
Francisco,
but
it's
not
5:00
yet
so
he
doesn't
have
a
drink
in
his
hand,
all
right
so
Jason
good
to
see
and
then
jacob
in
fremont
is
that
fremont
seattle
or
is
that
fremont
california
I'm,
not
sure
which
Fremont?
A
You
know
you
you're
walking
and
you
get
to
a
stoplight
and
and
like
anybody
who
like
lives
anywhere,
there's
rain.
You
know
you
take
like
a
couple
steps
back
from
the
curb,
because
if
you're,
not
careful
with
some
car
will
drive
by
and
give
you
a
big
ol
splash,
but
everybody
in
San
Francisco
is
like
right
up
on
the
curb
you
know,
ready
to
for
the
walk,
like
I
thought.
That
was
kind
of
funny
there,
their
total
total
noobs
when
it
comes
to
rain.
A
Okay,
it's
from
Berlin
good
to
see
ya,
yeah
umbrella
warfare,
exactly
alright,
so
other
interesting
stuff
that
I
saw
and
again
I
put
all
these.
All
these
links
in
the
show
notes
is
a
met.
Who's
I
believe
at
Google,
I
I
should
yeah
so
he's
he's
a
Googler
wrote
up
a
blog
post
that
covers
another
Google
engineer.
A
This
is
this
is
Tim
all
Clair's
wrote
this
sort
of
like
crazy
script
for
essentially
doing
debugging
of
a
scratch
container,
because
what
we
find
is
that
a
lot
of
folks
create
these
sort
of
super
bare
minimum
containers
that
are
optimized
for
running.
But
if
you
have
to
like
get
into
those
things
are
start
to
bugging
or
start
poking
around.
A
That's
really
interesting
and
I
think
you
know,
but
this
is
this
is
to
get
you
working
right
now,
but
there's
also
been
a
lot
of
talk
of
this
feature
called
debug
containers,
which
is
this
idea
of
launching
new
containers
to
be
able
to
do
debug
type
of
stuff.
So
so
this
is
a
really
great
tip
if
you're
gonna
be
figuring
out
how
to
how
to
debug
containers
that
are
from
scratch
or
sort
of
super
amenable
containers.
A
So,
let's
see
so,
we
have
a
DES
lawyer
in
sunny
La,
Jolla,
La,
Jolla
Mateus
and
from
Berlin
yeah
pros.
When
it
comes
to
ring
yeah,
you
guys
have
any
Hills,
though
Berlin
superflat,
it
changes
things
and
then
lo
Madi
says
do
you
happen
to
know
if
the
kubernetes
debug
container
project
will
make
110
well,
110
is
in
you
know,
is
in
code
freeze
now
so
I,
don't
think
that's
the
case.
A
Let's
see
is
this
just
clicking
through
here
I,
don't
see
yeah,
it
says,
okay,
so
the
so-so
verb
here
who's
this
I
don't
even
know.
This
is
a
Google.
Livre
burn,
apparently,
is
been
working
on
this
as
we
made
significant
progress,
but
could
not
agree
on
the
API
in
time
for
110.
This
slips
to
1
111,
and
this
is
actually
some
kind
of
somewhat
fascinating.
A
For
you
know
a
long-term
sustainable
project
is
that
is
that
as
it
as
it
gets
older,
you
don't
necessarily
see
the
future
cadence
that
you
might
like,
because
people
are
trying
to
be
be
more
careful
so
yeah.
So
that
looks
like
it
did
not
make
it
for
110
all
right,
and
then
we
have
a
neuro
t1
Cal,
saying
Dreier
Jerry
Woodinville
yeah,
so
it's
pretty
dreary
actually
was
really
nice
really
right,
but
blinds
are
down.
I
can't
see
it,
but
we
totally
saw
the
mountain
earlier.
A
Is
you
know
so,
let's,
let's
start
jumping
in
to
sort
of
steep
'full
workloads
on
top
of
kubernetes
and-
and
I
know
chris-
is
writing
something
on
this-
I'm
not
quite
sure
how
it's
gonna
make
its
way
out,
but
but
I
want
to
get
crisp
about
what
we
talked
about
when
we
say
a
stateful
workload,
cuz
and
cuz
this
you
know
I'll
confess
this
actually
confused
me
a
little
bit
because
you
know
when
you
think
about
when
you
think
about
like
from
a
sort
of
computer
science.
Ii
point
of
view
state
is
anything.
A
That's
like
a
state
machine
right.
It's
like
it's!
It's
it's
state
is
a
sort
of
principal
part
of
sort
of
a
von
Neumann
architecture
right
of
a
Turing
machine.
Is
this
idea
of
state
and
position
and
all
that,
and
so
you
really
don't
have
any
program
out
there
that
doesn't
have
any
state
and
if
you
really
talk
about
a
stateful
program,
what
you
mean,
though,
is
that
it
doesn't
store
any
data
right.
A
So
so,
if
you
want
to
be
SuperDuper
pedantic
about
this
stuff,
if
you
want
to
be
really
really
pedantic,
you
know
state
every
program
is
essentially
stateful.
What
people
mean
when
they
say
stateful
workloads,
at
least
with
respect
to
containers
with
respect
to
this
stuff
is,
is
long-term,
State
things
that
are
being
stored
on
disk,
really,
you
know,
and
and
it's
stuff,
that's
directly
managing
so
so
I
think
instead
of
stateful
like
like
a
better
term,
might
be
like
you
know,
I,
don't
know
workloads
with
disk
with
files
right.
A
That's
actually
I,
think
more
accurate
in
terms
of
this
stuff
and
and
sometimes
people
talk
about
stateful
applications
and
and
and
I,
don't
think
that
makes
a
ton
of
sense
either
because
a
lot
of
times,
especially
like,
especially
with
micro
services,
but
even
with
more
traditional
multi-tier
applications,
would
you
tend
to
do?
Is
you
tend
to
concentrate
a
lot
of
your
state
into
into
like
a
database,
and
so
so
and
you'll
see
this
pattern
with
kubernetes
itself,
but
this
is
SuperDuper
common.
A
Is
that
the
the
workloads
that
are
running
are
relatively
stateless
but
the,
but
all
the
long
term
state
gets
stored
in
a
database
there's
kind
of
an
in-between
there
and
I
think
this
is
something
that's
interesting.
Is
that
and
you'll
see
this
with
more
legacy
applications
where
you
know
you'll
have
a
web
session
right
so,
like
you'll
connect
to
a
browser
that
browser
will
actually
return,
you
a
cookie
and
then
every
time
you
connect.
It
will
actually
use
that
cookie
as
key
into
a
map
right
and
more
modern
architectures.
A
That
map
gets
stored
into
a
database,
which
means
that
your
front
ends
are
truly
stateless.
It
doesn't
matter
which
fernand
you
hid.
It
can
actually
read
the
state
for
your
session
out
of
the
database
when
you're
doing
session,
state
based
type
of
stuff,
but
a
lot
of
legacy
applications
they
weren't
built
to
scale
out,
and
they
didn't
assume
that
they
would
ever
be
restarted,
because
why
would
you
do
that?
And
so
then
they
actually
end
up
story
in
that
state
in
memory
right
and
so
that's
a
type
of
stateless,
equal
application.
A
That
I
don't
think
anybody
has
really
talked
about
handling
well
with
with
something
like
kubernetes.
And
so
when
you
start
having
people
talk
about
things
like
session
stickiness,
it's
like
hey
when
a
request
comes
in.
You
want
to
route
it
to
the
right
back
end
so
that
you
can
actually
maintain
that
that
level
of
in
memory
state
that
in
memory
session
stuff.
A
So
what
we
mean
here
in
kubernetes
in
the
container
world,
with
with
stateful
workloads,
means
workloads
that
access
a
disk
that
have
files
and
I
say
workloads
instead
of
applications
again
because
it's
it's
usually
part
of
an
application
into
your
database
or
it's
an
external
service
right.
So
if
I
have
something
that
talks
in
stores
and
manages
files
in
s3,
that
I
wouldn't
consider
that
to
be
a
stateful
application,
because,
like
you
know,
you
can
shut
it
down
and
start
it
up
and
like
everything
like
can
rehook
itself
up
and
everything
sort
of
recovers
right.
A
Whereas
if
you
totally
eliminate
everything
with
your
database
and
try
and
break
the
database
back
up,
obviously
it's
not
going
to
be
where
you
wanted
it
to
be
all
right.
So
once
you
start
dealing
with
this
idea
of
disks
and
volumes
and
stuff
like
that,
what
you
find
is
that
a
lot
of
the
primitives
that
we
built
with
kubernetes
things
like
deployment
and
replica
sets
really
aren't
built
to
deal
with
this
stuff.
Very
well,
specifically,
if
you
look
at
persistent
volumes,
they
the
most
common.
A
This
means
that
only
one
thing
can
be
reading
and
writing
to
this
thing
at
a
time,
and
there's
these
like
really
esoteric
topologies,
where
you
can
have
like
I
scuzzy
with
multiple
people
talking
to
it.
At
the
same
time,
it
takes
a
very
sophisticated
and
special-purpose
type
of
workload
to
access
that
most
of
the
time.
What
happens
is
that
you
know
the
the
file
system
driver
that's
running
in
your
kernel.
A
That's
talking
to
the
block
device
assumes
that
it's
the
sole
writer,
and
so
you
know
to
that
particular
block
device,
and
so
that's
something
that
that
you
should
keep
in
mind.
There
are
some
features
like
like
this
is
one
of
the
things
that
we
built
into
Google
container
engine
is
that
a
network
block
device
on
Google
container
engine
can
either
be
mounted
read/write
to
one
VM
or
you
can
mount
it
as
read-only
to
a
bunch
of
amps,
and
so
that's
a
feature
that
fell
out
of
the
way
that
we
did.
A
A
You
can
pair
a
pod
with
a
persistent
volume,
either
through
directly
or
through
a
persistent
volume
claim
and
and
those
things
get
paired
one-to-one
and
when
you
delete
the
thing
it
actually
detaches
and
then
you
can
reattach
that
volume
someplace
else,
and
so
this
is
really
similar
to
to
you
know
a
virtual
machine
in
some
ways
right
you
bring
up
a
VM,
you
attach
a
disk,
you
shut
down
the
VM.
Maybe
you
delete
the
disk,
maybe
you
actually
leave
it.
A
Parthiman
says:
do
you
foresee
deconstruction
of
stateful
set
primitive
incoming
release?
I
haven't
been
tracking
it
that
carefully
I,
wouldn't
imagine
that
I
haven't
heard
any
rumors
of
anybody
doing
any
sort
of
big
reinvention
or
rewrites
of
this
stuff.
So
I,
don't
I,
don't
think
there's
any
big
changes
coming
down
the
pike
but
I,
but
I
could
be
wrong.
Okay,
so,
but
the
problem
now
is
that
now
you
have
a
pod
know
how
to
pair
that
with
a
persistent
volume.
But
now
you
get
to
replica
sets
right
and,
and
so
replica
sets
have
this
nice.
A
A
If
you
missed
the
beginning,
you
know
you
can
you
can
go
back
and
try
and
watch
and
catch
up
or
you
can.
You
know,
maybe
catch
up
with
the
recording.
So
sorry
about
that,
for
those
of
you
watching
the
recording
ignore
what
I
just
said,
everything
when
it's
planned
all
right,
Scott,
all
right
so
replica
set
so
before
we
had
stateful
sets,
you
had
a
replica
set
and
you
do
a
replica
count
of
one
and
then
you
test
the
persistent
volume
for
that.
A
If
you
did
a
replica
count
of
two
like
bad
things
would
happen
because
you'd
schedule
two
pods
that
would
then
reference
the
same
persistent
volume
and
then,
like
the
you
know,
kubernetes
would
be
like
that
things
already
in
use.
I
can't
launch
it.
So
that
means
that
your
second
pod
would
never
actually
launch,
and
so
so
the
problem
is
that
we
didn't
have
a
replicating.
A
You
know
we
didn't
have
a
controller
that
was
similar
to
a
replica
set
that
also
took
into
account
the
fact
that
you
wanted
to
manage
not
just
an
array
of
pods,
but
you
wanted
to
manage
an
array
of
persistent
volume,
also
right
or
provost,
persistent
volume
claims,
and
so
that's
really
where
stateful
set
came
through.
It's
like
okay.
What
I
want
to
do
is
I
want
to
launch.
You
know.
A
A
This
data
I'm
gonna
go
to
instance,
number
three
of
my
array
of
stateful
pods,
whereas
if
it's
a
stateful
thing
I
just
want
to
go
to
one
of
them
right,
you
don't
care,
so
you
get
sort
of
more
of
a
spraying,
type
of
load
balance
or
typically,
when
you're,
when
you're
running,
stateless
and
and
so
predictable.
Naming
of
the
things
that
you're
managing
is
so
so
with
a
replica
set
when
it
actually
creates
replicas.
It
just
gives
a
random
name
to
each
of
the
ten
replicas
that
you're
creating
and
there's
no
implied
order.
A
These
things
are
all
part
of
essentially
what's
an
unordered
set
and
and
there's
no
guarantees
about
the
order,
but
with
stateful
sets.
What
you'll
see
here
is
that
when
you
start
them
up,
it'll
be
like
the
name
of
the
stateful
set
and
then
they'll
be
ordered,
they'll
have
an
index
0,
1,
2,
3,
4,
etc.
A
And
then,
when
you
scale
it
down,
you
go
in
reverse
order
right,
so
you'll
always
have
essentially
dense
numbering
from
0
to
up
to
how
many
replicas
you
have
or
replicas
minus
1,
and
so,
and
so
once
you
introduce
that
ordinal,
you
really
want
to
have
some
expectations
around
sort
of
how
these
things
come
up,
and
you
know
if
you
scale
down,
you
have
predictability
which
one
you're
scaling
down
that
type
of
thing.
Now.
This
is
actually
how
Borg
works
inside
of
Google.
A
Everything
inside
of
Google
essentially
has
an
index
and
what
it
means
is
that
there's
a
little
bit
less
flexibility
when
it
comes
to
being
able
to
scale
things
up
and
down
inside
of
Borg,
and
so
so
this,
the
sort
of
very
deliberate,
ordered
scale
up
scale
down,
is
actually
more
typical
of
the
way
that
board
works
inside
of
Google
and
so
yeah.
So
these
things
are
stable
names,
stable
Network
identities
and
then,
and
then
we'll
once
we
get
bring
up.
A
A
Also
that
deleting
the
stateful
set
does
not
necessarily
delete
the
the
volumes
that
are
sitting
behind
it
and
yeah,
and
so
there's
some
other
sort
of
you
know,
options
and
stuff
like
this
idea
of
rolling
out
a
canary
and
there's
rolling
update.
So
one
of
the
things
that
I
think
is
is
I,
don't
know
if
it
would
be
possible
to
break
it
out.
B
A
Strict
ordering
about
how
you
actually
start
these
things
up
and
down
you
could
wrestle
from
the
Netherlands.
Welcome,
welcome
alright,
so
yeah,
so
there's
all
sorts
of
strategies
around
rolling
out
and
then
deleting
you
know,
there's
a
note
here
where
it
leaves
the
the
PVCs
around
oh
yeah,
so
yeah
so
there's
other
details.
There
I'm
not
going
to
sort
of
go
through
all
of
this
and
then
there's
some
examples
here
in
the
documentation.
A
So,
preventing
volume
deletion
might
become
handy,
yeah,
so
I
think
by
default
it
just
doesn't
delete
the
volumes
which
is,
which
is
really
nice.
There's
examples
of
running
like
like
my
sequel
or
Cassandra
zookeeper
with
these
things,
because
Gamble's
are
useful,
but
I
don't
know
if
they're
as
useful.
A
But
what
you
see
here
is
that
there's
like
if
we
start
digging
into
this
example,
there's
like
a
Rhonda
sage,
and
so
it's
doing
some
stuff
to
essentially
adapt
Cassandra
to
how
the
stateful
set
works
right.
So
it's
doing
a
whole
bunch
of
goop
before
it
actually
launches
Cassandra,
and
so
what
this
shows
here
and
again
I,
don't
think
the
examples
go
through
a
lot
of
do
they
make
it
look
easier
than
it
really
is.
B
A
Most
of
the
time
you
need
some
level
of
code,
an
adapter,
that's
going
to
be
doing
some
level
of
management
outside
of
the
the
database
itself,
and
so
stateful
sets
end
up
being
a
a
useful
building
block
for
doing
stateful
workloads,
but
it
doesn't
solve
the
entire
problem.
You
still
need
to
do
some
management
and
some
plumbing,
and
so
when
I
was
when
I
when
I
was
gonna,
be
talking
about
this,
the
the
jet
stack
folks,
our
friend,
has
just
wrote
in
and
wanted
me
to
bring
up.
This
is
really
interesting.
A
This
is
this
thing
called
navigator,
which
again,
which
is
a
system
that
using
things
like
stateful
set,
manages
databases
right.
So
this
adds
a
lot
of
that
extra
knowledge
that
domain-specific
knowledge
about.
Okay,
for
you
know,
Cassandra,
for
example,
you
you
it's
not
enough
to
just
sort
of
launch
a
bunch
of
processes.
You
also
have
to
sort
of
stitch
them
together
and
poke
them
in
the
right
way
and
manage
them
in
the
right
way
to
make
them
work
well
inside
of
kubernetes,
and
so
this
is
all
of
the
automated
code.
A
Now
I
haven't
used
this
yet
I'm
really
interested
in
checking
it
out
at
some
point,
but
I
figure.
This
is
a
good
example
of
how
you
do
need
some
relatively
sophisticated
adapters
to
make
a
traditional
database,
or
at
least
a
scale
out
database
or
something
that's
doing
doing
a
replication
to
make
it
work.
Well.
A
Inside
of
inside
of
kubernetes
yeah
so
check
this
out,
it
looks
really
interesting
again:
I
haven't
I,
haven't
used
it,
but
but
it
looks
really
cool,
and
this
is
also
an
example
of
something
using
the
the
cube
aggregator,
which
is
a
sort
of
more
full
way
of
extending
kubernetes.
It's
also
more
difficult
to
managed
to
so
yeah.
A
So
abdel
says
I
love
the
stateful
set,
but
why
doesn't
it
delete
the
PVCs
I?
Think
it's
just
a
matter
honestly
of
being
paranoid
right,
because
you
know
if
you
have
state
it's
because
you
want
to
actually
keep
stuff
around
and
it's
too
easy
to
accidentally
delete
something
and
then
have
that
go
ahead
and
delete
the
PVCs
on
you.
So
so
really
I
think
it's
it's
just
a
matter
of
it's
it's
just
a
matter
of
being
paranoid.
A
I
I
could
see
you
know,
let's
see,
I
could
see
and
maybe
I'm
sure
somebody
has
proposed
this
I'm,
not
aware
of
it,
of
having
some
sort
of
option
on
a
stateful
set.
That
says,
delete
the
PVCs
when
you're
when
you're
done
with
them,
but
the
scaling
down
section.
What
does
it
say
here?
It
says:
ordered
pod
termination,
stable
storage,
so
I'm
just
reading
through
the
I.
Don't
know.
My
guess.
Is
that,
like
err
on
the
side
of
not
surprising
people
and
not
yeah,
so
again,
not.
A
So
that's
where
we're
at
so
I
think
that's
my
intro
to
stateful
sets
but
I
think
it's
it's
useful
to
see
this
stuff
in
action,
and
so
I'm
gonna
be
playing
with
this.
This
project
Anthony
says:
if
you
prefer
to
delete
PVCs,
you
could
try
cat
set.
Oh
it's
this
alright.
So
this
is.
This
is
a
bad
joke
that
Anthony's
making
here,
because
because
the
the
for
stateful
sets
for
stateful
sets,
they
were
actually
called
pet
sets,
which
is
playing
on
the
whole
sort
of
pets
vs.
A
cattle
terminology,
whereas
you
know
each
of
the
items
in
a
stateful
set
are
not
sort
of
purely
cattle
because
you
do
want
to
actually
manage
them,
so
they
really
are
pets,
but
it's
a
set
of
pets
right,
but
then
Anthony
has,
and
we
did
the
episode
on
the
meta
controller
and
he's
continued
to
work
on
this
and
I
haven't
looked
at
elite
recently.
This
is
yes
when
now
it's
in
the
KS
temp,
is
this
because
figuring
out
how
to
actually
get
it
into
the
according
okay?
A
So
now
it's
in
the
KS
temp
org,
which
I'm
not
sure
why
it's
there
but
I
think,
let's
see
yeah
so
Anthony
has
an
example
of
a
thing
called
cat
set.
We
essentially
reimplemented
stateful
sets
using
the
meta
controller,
which
I
think
is
a
good
proof
point
for
the
for
the
meta
controller,
and
you
can
tell
that
it's
actually
not
not
as
hard
as
you'd
want
it
to
be.
I
don't
know
if
it's
totally
feature
to
feature
thing,
but
it
definitely
hits
the
same.
A
It's
the
same
usage,
Jay
Hernandez
says
what
about
multi
AC
deployments
I
believe
that
stateful
sets
tracks
zones.
I
don't
know:
do
they
track
zones,
it's
not
in
the
documentation
here.
My
guess
is.
That
is
that!
Well
usually,
what
happens
is
that
when
you
actually
create
a
P
V,
whatever
zone,
it
lands
in
that's
actually
where
the
stateful
set,
where
the
the
pod
gets
scheduled.
Also
yeah
these
this
pay,
no
attention
of
the
weird
organy
and
the
grande
I
think
I
know
what
happened
there.
A
Anthony
I'm
not
gonna,
I'm
not
going
to
go
into
the
details
there,
so
I'm
not
exactly
sure
about
what
happens
with
multi
AZ
deployments
and
stateful
sets.
I
mean
the
truth
of
the
matter
is:
is
that
if
you're
managing
state,
you
probably
want
to
be
very
zone
aware
and
so
I
would
be,
I
would
probably
run
a
stateful
set
per
zone
is
the
way
that
I
would
start
looking
at
it
and
then
I
would
have
zone
constraints
on
the
stateful
side.
A
If
you're
doing
things
across
zones,
then
then
you
know
do
that
expansion
there
sandy
says
I
think
there
were
some
special
cases
put
into
the
scheduler
look
at
stateful
set
PVCs
differently.
Perhaps
I
mean
perhaps
they
try
and
get
some
zone
zone
spread
across
these
things.
That
would
make
me
sad
if
there
was
actually
special
cases
put
into
the
scheduler,
because
that's
a
layer
violation
that
would
that
is
a
lazy
shortcut
but
yeah
all
right.
So
so
Bruce
says:
are
there
any
performance
numbers
with
respect
to
the
PV
disk
IO?
A
Whatever
it's
the
underlying
cloud
platform
I
mean
there's
like
a
gazillion
different
sort
of
adapters
for
how
you
do
PBS,
you
know.
Typically,
they
work
over
the
network
and
so
you'll
definitely
see
performance
issues
like,
however
fast
your
network
is,
there
is
a
feature
that
I
believe
is
alpha
and
1:9
I'm,
not
sure
I'd
have
to
look
it
up
for
essentially
having
a
persistent
volume.
That's
local
and
so
I
think
that's
going
to
be
really
interesting.
A
When
you
combine
from
you
know:
semi
persistent
but
non
replicated
local
disk
with
with
stateful
sets
and
things
like
Cassandra
or
cockroach
TV
that
do
their
own
replication.
So
so
right
now
what
happens
if
you
run
something
like
Cassandra
on
top
of
kubernetes
and
you're,
using
something
like
EBS
volume
is
behind
the
scenes,
you're
doing
like
double
replication
right,
because
Cassandra
itself
will
do
replication,
but
then
the
persistent,
the
block
storage,
that
it's
using
none
of
the
covers,
like
the
AV,
the
EBS
volume,
is
also
doing
replication,
so
that
ends
up
being
slower
and
wasteful.
A
If
you,
if
you
look
at
folks
who
are
doing
sort
of
like
really
big
Cassandra
deployments
on
top
of
Amazon,
like
you
know,
say
like
Netflix,
they
will
always
use
like
these.
These,
you
know
local
flash
devices
that
are
on
these
big
instances
and
they'll
buy
the
whole
instance
that
type
of
thing-
and
that's
one
of
the
big
reasons
why
you
end
up
with
with
like
local
flash
on
GCE
also,
so
the
local
PV
volumes
are
really
really
interesting,
so
Bismarck
says:
okay,
so
are
there
any
tools
out
there
to
audit
PVC
for
orphans?
A
I,
don't
think
that
there
are
well
what
you
can
do
we'll
play
around
with
it
when
we
delete
the
cockroach
DB
stuff,
I
believe
what
you
can
do
is
you
can
list
your
PVCs
and
you
can
actually
see
whether
they're
attached
to
a
pot
or
not
I,
think
you
can
do
that
I.
Don't
let's
check
that
out
and
see
what
happens
well,
I'm
and
then
Bismarck
says
that
apologia,
where
volume
scheduling
is
going
into
beta
and
110,
so
hopefully,
hopefully,
that'll
get
rid
of
any
of
this
sort
of
lair
violation.
A
A
That's
working
with
it,
so
I
think
a
this
is
a
pattern
that
we're
going
to
see
over
time.
Is
that
we're
gonna
see
more
and
more
systems
like
cockroach
that
are
built
to
take
advantage
of
distributed
platforms
from
the
get-go?
So,
instead
of
having
to
build
up
their
own
sort
of
like
distributed
paradigms,
they
can
take
advantage
of
an
underlying
platform
like
kubernetes
and
so
I.
Think
that's
gonna
be
a
really
exciting
development.
A
It's
gonna
lower
the
bar
for
people
doing
more
and
more
stuff
like
this
all
right,
so
we're
gonna,
we're
gonna,
launch
cockroach,
DB
and
there's
this
really
great
tutorial
that
that
they
have,
and
so
it
starts
with
starting
kubernetes.
But
I
already
have
a
cluster
up
and
running,
and
since
we're
gonna
be
playing
around
with
stuff
I
launched
a
cluster
with
I
believe
like
five
nodes,
five
worker
nodes.
A
Yeah,
so
we
have
we
have
enough
nodes
so
that
we
can
start
playing
around
with
stuff
saying
get
from
Naperville
good
to
see
you
all
right.
So
that's
our
thing
so
so
it
turns
out
that
this
is
actually
really
really
easy.
It's
not
going
to
take
us
that
long
I
did
this
while
I
was
waiting
for
the
delayed
plane
last
night.
A
So
the
first
thing
it
does
is,
it
says,
like
you
know,
do
a
cute
control
create
dash
F,
which
we
know
is,
like
you
know:
pseudo
piped,
a
pseudo
curl
pipe
to
bat
or
curl
pipe
to
bash
type
of
thing,
but
so
we're
gonna
we're
going
to
under
the
covers
there
and
we're
gonna
do
curl
and
we're
gonna
download
that
thing
and
that
got
us.
Cockroach
DB
stateful
set
secure,
yeah
mo,
and
so
let's
look
to
see
what's
in
these
in
this,
what
we
happen
to
my.
A
A
So
that's
really
cool.
The
next
thing
that
I
think
is
is
is
interesting.
It
and
we'll
get
to
this
is
we're
doing
the
secure
deployment
of
cockroach
DB.
It
actually
uses
some
other
api's
that
the
certificate
signing
API
in
kubernetes
to
create
certificates
that
it
can
use
to
talk
between
the
instances
of
cockroach,
and
this
starts
to
hint
at
the
types
of
things
that
we'll
see.
A
So
we
have
a
service
account
called
cockroach
DB,
it's
not
talking
about
any
specific
namespace,
so
this
will
run
in
whatever
namespace
you
give
it
so
we're
gonna
be
running
in
the
default
namespace.
We
have
a
role.
So
this
is
our
not
a
cluster
role.
This
is
a
regular
role.
So
this
is
something
that's
local.
To
this
specific
name,
space
that
we're
deploying
this
to
and
I
love
seeing
this,
because
that
means
that
it's
very
easy
to
isolate
it.
A
B
A
Narrow
set
of
permissions
that
we're
giving
cockroach
so
we're
not
giving
it
a
lot
of
a
lot
of
permissions
on
the
cluster,
and
the
next
thing
here
is
that
there's
a
service
over
this
thing-
and
let's
see
says
this
service
is
meant
to
be
used
by
clients
of
the
database.
It
exposes
a
cluster
IP
that
will
automatically
load
balanced
connections
to
different
database,
pods
and
so
apparently
and
I,
don't
know
the
details
because
I'm
not
an
expert
on
cockroach,
but
when
a
client
connects
it
doesn't
matter
which
of
the
nodes.
A
It
does
not
create
a
load
balanced
cluster
IP
and
should
not
be
used
directly
by
clients
in
most
circumstances,
because,
most
of
the
time
you
don't
need
to
be
able
to
talk
to
the
clients
directly
and
then
there's
an
annotation
here,
and
this
is
a
hint
to
the
DNS
system.
That's
actually
reading
these
these
services
that
says
that
we
want
to
tolerate
unready
endpoints,
which
means
that,
even
if
a
particular
end
point
is
unready
still
put
it
into
DNS,
and
so
this
changes,
because
okay
little
bit
of
a
backup.
A
So
when
you
do
a
service,
you
actually
say:
ok,
all
the
pods
that
meet
this
particular
criteria.
I
want
to
publish
those
things
and
give
them
a
name.
So
that's
the
the
core
thing
that
you
do
and
then
what
happens
is
that
kubernetes
goes
through
and
actually
creates
an
a
buddy
endpoint
object
that
sort
of
hits
buddy
buddy
to
the
service.
That
endpoint
object
contains
details
about
all
of
the
pods
that
that
you
identified
with
that.
A
With
that
thing,
so
one
way
to
think
of
a
service
is
it's
a
standing
label
query
one
of
the
details
that
gets
surfaced
in
that
end
points
is
which
of
the
endpoints
are
ready
versus
on
ready.
Now,
there's
also
one
of
the
things
that
cube
dns
to
us
is
it
takes
these
things
and
it
exposes
them
via
DNS.
Now,
if
you're
using
a
cluster
IP
dns,
is
that
cluster
IP,
so
it
essentially
becomes
a
very
simple
mapping.
A
If
you're
not
doing
a
cluster
IP,
then
what
you'll,
see
and
we'll
play
around
with
this
is
that
the
the
the
DNS
name
for
the
service
ends
up
being
a
list
of
a
entries,
and
it
will
typically
only
include
the
endpoints
that
are
already
there,
and
this
is
say
no
I
want
to
include
all
of
the
endpoints
there.
This
is
an
alpha
I.
A
Think
I
saw
with
at
least
the
latest
releases
that
has
this
has
been
promoted
to
GA,
and
so
now
it's
not
an
annotation
anymore,
but
there
also
looks
like
there
might
be
some
bugs
with
that.
Let's
see
some
annotations
on
here
to
help
Prometheus
work
and
then
those
same
ports
that
we
had
before
we're
exposing
those
in
this
headless
service.
But
here
we're
doing
cluster
IP
equals
none,
which
essentially
says
don't
give
it
a
cluster
IP
and
it
becomes
a
non
non
cluster
IP
service.
A
A
A
This
thing
is
actually
going
through
and
creating
a
certificate
request
for
us
we'll
get
into
some
details.
There
Caleb
good
to
see
you
it's
doing
some
downward
API
stuff.
Where
we're
actually
populating
some
environment
variables
from
other
things
in
the
in
the
in
the
in
the
the
pod
specification,
and
then
it's
mounting
this
this
this
volume
called
certs
and
then
oh
and
then
here's
something
that's
interesting.
A
It
has
a
pod
anti
affinity
to
say
that
as
much
as
possible,
we
want
to
make
sure
that
these
things
don't
land
on
the
same
node,
so
preferred
says
like
hey.
If
you
can't
do
anything
else,
go
ahead
and
do
that,
but
all
things
being
equal,
we
want
to
make
sure
that
we
spread
these
things
across
host
names
and
then
there's
the
main
cockroach
TV
container.
B
A
It
looks
like
they
have
two
service
definitions.
Could
they
have
created
the
DNS
entries
with
just
the
first
one
I
believe
that
when
that
that
the
DNS
system
acts
differently,
when
you
do
cluster
IP
and
when
we
spin
this
up,
what
I'm
gonna
do
is
I'm
gonna
launch
a
corgi
which
has
a
interactive
DNS
agent
and
we'll
start
exploring
what
DNS
looks
like
it.
A
So
we'll
look
at
the
differences
in
terms
of
how
these
different
services
get
exposed
into
DNS,
yeah
and
then
and
then
here's
the
the
volume
claim
template
that
we
have
okay
and
it
says
we're
going
to
do
one
gigabyte
of
storage
for
these
things.
Alright,
so
that's
super
cool.
Let's
go
ahead.
Cube
control
applied
a
chef,
cockroach
DB
stateful
set
secure.
This
is
going
to
do
its
thing.
Oh
and
let
me
do
I
always
forget
the
shortcut
key
for
this
view.
B
A
Alright,
so
we're
going
to
be
watching
the
pods.
Actually,
let's
and
then
let's
do
PVC
alright.
So
what
we
see
here
so
this
is
the
watch
command,
which
you
can
install
on
your
Mac
with
brew,
says:
execute
this
command
every
second
and
then
give
us
the
output,
and
so
we're
going
to
be
watching
sort
of
what
the
state
of
the
the
pods
and
the
the
PVCs
as
we
go
through
here.
A
A
A
Kyle
says:
if
you
ever
make
a
TGI
Cooper
nature,
it
should
have
that
shortcut
on
it,
which
shortcut
the
watch
shortcut
yeah
I,
do
like
I
do
want
to
make
t-shirts
at
some
point.
I
think
we
should
totally
do
that:
okay,
so
okay,
so
now.
What
we
have
here
is
so
alex
is.
Okay,
alex
is
the
one
who
did
a
bunch
of
this.
No
readiness,
aliveness
checks
at
the
moment,
will
be
added
after
the
next
release
of
cockroach.
Okay,
so
those
are
coming.
A
That's
a
it's
a
good
point,
though
Alemanni
good
eye
they're
coming
all
right.
So
if
we
go
back
to
the
tutorial
here,
you'll
see
that
there's
this
thing
saying
we
want
to
approve
the
certificates.
So
what
we
have
here
is
we
nuke
you
controlled
get
CSR
that
stands
for
a
certificate
signing
request,
and
this
is
the
way
that
the
way
that
public
key
encryption
should
work.
A
But
you
want
to
also
make
sure
that
that
none
of
your
secrets
and
under
the
secret
stuff,
actually
flows
across
the
network
or
leaves
your
machine,
and
so
what
you
do
is
you
create
a
public/private
key
pair?
So
these
are
the
sort
of
two
big
prime
numbers.
Typically
with
RSA,
you
take
the
private
thing
and
you
keep
that
local
on
the
machine.
So
you
never
wanted
to
leave
the
machine
if
you
can
help
it,
you
take
that
public
key
and
that's
the
thing
that
you
want
somebody
to
sign.
A
What's
happening
is
that
the
individual
pod
for
cockroach
DB
is
creating
a
certificate
signing
request,
keeping
the
private
key
local,
putting
the
public
key
up
to
an
API
and
then
and
that
API
happens
to
be
embedded
in
the
kubernetes
api
server.
And
then
it's
the
kubernetes
api
server,
that's
playing
the
role
of
certificate
authority,
it's
gonna
again
sign
this
thing
and
return
it,
and
so
this
is
typically
used
for
bootstrapping
a
cluster
using
it.
This
way
is
honestly
a
little
bit
of
an
abuse
of
this
API.
A
It's
not
what
it
was
attended
for,
but
it
totally
works
and
it's
a
great
thing
to
do.
Well,
we
wait
around
for
the
rest
of
the
workload
identity
stuff
to
actually
get
its
act
together
and
get
done.
So
what
we'll
do
is
we'll
do
get
csr
and
what
you'll
see
here
is
that
we
have
CSRs
for
each
of
the
nodes.
So
this
is
like
when
I
talked
about
bootstrapping.
A
This
is
how
individual
nodes
actually
sort
of
bootstrap
their
authentication
and
their
credentials
for
talking
to
the
API
server,
but
we
actually
see
one
for
cockroach
DB
here
also,
and
we
can
inspect
that
so
I
can
do
cube,
control
describe
CSR
and
then
we'll
look
at
that
and
what
you'll
see
here
is
that
this
thing
has
who
requested
it.
So
this
is
the
service
account
that
was
used
to
request
it,
and
then
what
are
the
names
that
we
want?
A
This
thing
to
be
able
to
work
under,
and
so
there's
localhost,
which
is
kind
of
a
little
dicey,
cockroach
DB,
both
a
short
and
a
long
name,
and
then
cockroach
DB
public,
which
is
the
that
common
service
that
people
are
going
to
be
using
to
talk
to
it,
and
this
should
probably
use
the
long
name
to
also
and
then
IP
addresses
that
you
want
this
thing
to
be
able
to
assert.
Also,
and
so
then,
what
we
can
do
is
we
can
do
Cube
control
certificates,
certificate,
approve.
A
Now
we
do
that
and
now,
when
we
describe
the
certificate,
we
can
see
that
it's
approved
and
issued
and
if
we
go
through
and
you'll
see,
that's
there
too.
Now,
if
we
look
at
this
so
now,
what
you'll
see
is
that
because
we
approved
it
instant,
zero
finished
launching
and
now
instance,
one
is
in
that
same
state,
but
if
we
go
ahead
and
if
we
actually
get
get
CSR
this
and
we
dump
the
yamo,
what
you'll
see
is
here
is
the
request.
A
This
is
the
CSR
and
then,
after
it's
approved
as
part
of
the
status,
there's
the
certificate
that
was
actually
issued,
and
so,
if
you
know
all
the
SuperDuper
cryptic
flags
for
the
OpenSSL
command,
you
can
like
decode
this
thing
and
figure
out
what
it
is
actually
saying
and
so
yeah
the
CSR
was
approved
by
cube
control
certificate
approved
because
that's
we
manually
approved
it.
Now
you
can
write
API,
driven
things
that
look
at
it
and
say:
hey
that
looks
good
enough
I'm
going
to
go
ahead
and
approve
it.
A
Now
we
could
keep
the
private
key
local,
but
then,
if
something
died,
if
that,
if
that
that
workload
died,
then
what
you
would
find
is
that
that
private
key
would
be
lost
and
we'd
have
to
go
through
this
dance
again.
So
one
of
the
things
that
you'll
see
here
is
that
the
the
bootstrap
stuff
for
a
cockroach
TB
also
takes
that
private
key
and
then
stores
that
into
a
kubernetes
or
a
kubernetes
secret,
that's
accessible
by
the
service
account
so
we're
not
getting
as
much
sort
of
individual
individual
sort
of
identification
as
we'd
want.
A
A
Here's
the
here's
the
certificate,
so
this
is
the
same
thing
as
we
had
up
here
that
was
issued
and
then
here's
the
private
key
that
matches
that
certificate
all
right.
So
that's
the
flow.
So
let's
do
cute
control,
get
CSR
and
then
approve
and
we'll
do
approve
one
and
now
what
happens
that
you'll
see
this
pod,
initializing
it'll,
complete
and
then
and
then
we'll
see,
cockroach
DB
to
actually
complete,
so
we'll
ask
for
a
certificate,
we'll
go
ahead
and
we'll
promote.
No,
it's
still
getting
us
act
together
to
even
submit
the
CSR.
B
A
A
So
there's
another
yamo
file
that
we're
gonna
take
a
look
at
so
do
curl,
we'll
take
a
look.
This
is
cluster
and
it's
secure
and
what
this
is
doing
is
it's
running
a
job
that
job
actually
is
using
another
cert,
so
it's
going
to
request
another
certificates,
and
so
that's
that's
the
NIC
container
and
then
that's
going
to
run
and
that's
going
to
call
using
the
cockroach
client
and
that
certificate,
it's
going
to
say
in
its
this
particular
host
thing,
and
then
that's
going
to
be
enough
for
these
things
to
stitch
themselves
together.
A
A
This
is
saying:
hey
you,
common
name
root
is
now
approved
for
using
the
the
the
CA
for
your
entire
cluster,
which
means
that
I
could
then
go
ahead
and
take
the
certificate
to
use
to
talk
back
to
my
cluster
and
and
if
I
have
our
back
set
up
to
actually
recognize
that
user
names
root,
which
is
not
insane.
Then
this
club,
this
certificate
be
used
to
talk
to
that
so
I'd.
A
But
to
do
so
I
assume
that
a
common
name
of
root
carry
special
meaning
with
the
cockroach
DB
code,
and
so
that
means
that
probably
would
require
another
flag
in
the
cockroach
DB
code
to
recognize:
hey,
which
identity
do
what
you
do
want
to
give
read
to.
But
for
now
we'll
just
go
ahead
and
ignore
that
and
we'll
approve
this
thing.
Anyways.
A
A
B
A
B
A
Does
nothing
but
sleep
for
essentially
forever,
and
so
this
will
sleep
for
a
long
time.
It's
interesting
that
it
yeah
and
then
and
then
and
then
so
you're
like
well.
Why
would
you
want
to
do
that,
and
so
this
is
so.
You
have
something
that
you
can
quickly
do
a
cube,
control,
exact
into
and
start
working
interactively.
So
it's
essentially
as
sort
of
like
looking
a
little
footprint
on
the
cluster.
A
A
You
can
probably
do
like
a
port
forward
type
of
thing
to
one
of
the
nodes
and
then
and
then
have
the
client
run
locally
and
using
the
certificates
that
way
it
might
be
cool
if
the
cockroach
DB
client
actually
was
kubernetes
aware
enough,
that
it
could
actually
go
through
and
use
your
cube
control
and
actually
create
its
own
tunnel,
its
own
port
forward
tunnel
through
the
things,
so
that
would
be
kind
of
cool
too.
But
so
what
we're
gonna
do
is
we're
gonna
go
ahead
and
run
this
thing.
A
And
so
this
thing
is
called
client,
secure,
yamo
and
you'll
see.
When
we
get
this
thing
up
and
running
there
we
go
and
now
what
we
can
do
is
we
can
do
this
cube
control,
exact
thing,
we'll
copy
this
line,
and
this
says
okay,
I,
wanted
to
view
an
executive.
This
client
secure
thing
and
the
command
I
want
to
run
as
cockroach
sequel.
It
can
we're
tell
it
where
the
certs
are
and
where
to
actually
find
the
cluster.
So
we
go
ahead
and
run
this
and
boom
we're
at
a
prompt.
A
We
can
start
doing
cockroach
sequel
with
this,
and
so
they
have
some
example.
Sort
of
commands
that
we
may
want
to
run
here.
So
I
can
go
well
if
I
copy
does
it,
oh
that
okay,
so
we
created
a
database
called
Bank.
Oh
man,
I'm
gonna,
be
so
rich.
We're
gonna
create
a
table
called
accounts
which
has
both
an
ID
and
the
balance
you
probably
in
the
real
world.
You
want
more
than
that
and
then
we'll
enter
some
values.
Their
account
number
one
has
one
thousand
and
fifty
cents.
A
Let's
do
one
hundred
thousand
boom
alright,
and
so
this
is
decimal,
so
yeah,
so
we
actually
have
like
real
banky
types
like
decimals
in
there,
which
is
cool,
and
then
we
can
actually
query
this
too,
and
we
say:
okay,
we
can
do
a
query
and
I
love.
How
it
tells
me
how
long
each
of
these
things
took
as
we
go
so
yeah.
So
we
have
a
database
and
it
looks
like
so.
If
I
try
and
insert
it
again,
it's
like
I'm.
A
Sorry,
you
have
a
duplicate
primary
key
value,
so
this
is
looking
very
database
II
and
apparently
it
talks
sort
of
like
Postgres
compatible
API.
So
you
can
reuse
a
bunch
of
the
the
clients
that
you
have,
and
this
thing
is
is
distributed
across
a
bunch
of
nodes,
all
right,
so
that's
kind
of
cool
and
then
let's
see
and
then
we
can
do
this
port
forward
to
one
of
these
things
and
we'll
hit
will
hit
8080
and.
A
So
Christa
says
this
is
looking
very
database
II
all
right,
so
we're
gonna
do
this
this
fascinated
me,
so
we
hit
8080
HTTP,
but
then
it
like
upgraded
us
to
HTTPS
and-
and
so
we
have
to
say-
well
trust
this
anyway.
So
it's
actually
not
a
not
an
HTTP,
it's
htps
there
and
then
this
is
actually
a
really
cool
dashboard
that
that
cockroach
has,
and
so
we
can
tell
sort
of
like
how
much
capacity
were
using
latency.
So
there's
a
bunch
of
really
good
information
here.
A
A
A
These
must
be
get
written
into
the
database,
also
interesting,
LeMat
II.
What
options
are
available
to
interface
with
cockroaches
before
outside
the
cluster
would
be
a
good
use
case
for
ingress
I.
You
know
I
mean,
since
everything
is
nice
and
TLS
II
right
and
encrypted.
It's
not
crazy
to
actually
sort
of
like
make
this
available
over
something
like
ingress
to
the
rest
of
the
world,
but
it's
generally
not
what
you
do
I
mean
like
it
smells
really
bad.
A
Even
if
you're,
you
know,
you
know,
even
if
you're
you're,
protecting
it
with
TLS
to
have
your
database
exposed
to
the
world
in
any
way,
shape
or
form,
and
the
reason
being
that,
like
there's,
still
an
opportunity
for
somebody
to
do
sort
of
have
didn't
denial,
a
service
type
of
thing
directly
to
your
database
and
so
I,
wouldn't
I,
wouldn't
go
through
and
and
expose
this
directly
to
the
to
the
Internet.
What
I
would
do
that,
though,
is
is
for
sort
of
interactive
things.
A
The
stuff
that
we're
doing
here
is
great
for,
like
you
know,
manual
type
of
interactions
with
the
thing
port
forwards
really
appropriate.
Maybe
if
you're,
you
know,
if
you
have
a
VPN
into
the
network
where
your
clusters
earning
you,
could
do
a
node
port
for
the
thing
that
doesn't
seem
crazy
to
me,
but
I
wouldn't
expose
it
to
the
internet
with
something
like
ingress.
A
So,
if
you're
doing
like
an
internal
in
Grasse,
because
you
can
do
an
ingress
for
internal
versus
external,
then
maybe
that
makes
sense
but
typically
you'd
be
accessing
this
from
within
the
cluster.
So
you'd
like
bring
up
like
you,
know
a
Python
program,
or
you
know
node
program
that
sits
right
next
to
cockroach
and
actually
talks
to
that.
To
that
directly,
let's
see
and
so
part
of
mine
is
asking,
does
cockroach
DB
and
spanner
share
some
the
same
API
and
sequel
version
support.
A
Think
that's
a
really
smart
thing
for
the
cockroach
folks
to
be
doing
because
they're
you're
gonna
find
out
heck
of
a
lot
more
post
quest,
sequel
clients
out
there
and
you
know
or
Em's
and
all
that
stuff.
Then
you
will,
then
you
will
the
the
spanner
G
RPC
stuff
alright.
So
we
have
a
database
up
and
running,
or
at
least
I
think
we
do.
It
looks
really
reasonable
to
me.
You
know
we
haven't
like
beat
up
on
it
too
hard,
so
there's
a
couple
of
that
I
want
to
do
here.
A
A
Alright,
so
Alex
here's
a
bug
report
for
you.
If
you
see
somebody
coming
in
on
localhost,
do
a
302
redirect
not
a
301,
okay
and
then
Bruce
says
there's
more
stuff
that
cuz
now
this
thing
is
actually
being
like:
hey
I
know
that
8080
is
always
gonna
go
to
so
I'm
gonna
do
80
81.
So
now,
like
my
browser,
is
tainted
and
I
can't
do
regular
old
HTTP
on
8080.
So
that's
annoying.
A
A
Cockroach
TV
public.
So
if
I
do
a
query
on
that,
what
you'll
see
is
we're
going
to
get
the
the
regular
old
cluster
IP
if
I
do
a
serve
record
on
this
same
old
thing,
we're
gonna
get
the
cluster
IP,
and
so,
when
you
have
a
cluster,
IP
DNS
will
only
ever
show
the
cluster
IP.
So
this
is
why
we
have
two
different
services.
Now,
if
I
go
through
and
if
I
do
like
an
a
record
on
cockroach
DB,
what
you'll
see
is
I
get
records
back
when
I.
A
Do
that
a
query-
and
these
are
the
three
individual
instances
of
this
thing
now,
when
you
do
an
a
record
you're
actually
going
to
be
getting,
they
all
have
the
same
name.
So
all
we
know
is
that
this
name
resolves
to
any
three
any
of
these
three
IP
addresses
and
there's
limits
to
how
many
of
these
things
you
can
return,
because
if
you
had
like
I,
don't
know
what
thousand
you
know
at
some
point:
they
don't
fit
into
a
response
from
DNS.
A
Now,
if
I
change
this
to
a
serve
record,
what
you'll
see
here
is
now
the
serve
record,
says:
hey
this
thing
actually
resolves
to
to
the
individual
names
and
then
I
can
go
ahead
and
here's
the
IP
addresses
of
those
individual
names.
In
addition,
if
I
want
to
talk
to
those
things
directly,
I
actually
the
DNS
will
I
can
do
like
like
cockroach
DB,
0
dot
cockroach,
this
thing
should
be
longer
DB
and
it'll.
Actually,
so
not
only
do
we
now
have
on
that
internal
facing
service.
A
And
so
this
is
what's
happening
with
the
DNS,
and
this
is
why
cockroach
DB
needs
to
DNS
to
services,
because
it
has
the
external
service
which
has
everything
all
the
details
getting
behind
the
cluster
IP
and
then
the
internal
thing,
because
it
doesn't
have
a
cluster
IP.
All
the
details
get
exposed,
but
that's
what
we
want
so
that
the
internal
parts
can
talk
to
each
other.
Alright,
so
let's
go
through
and
let's
do
the
port
forward
back
to
cockroach
DB.
Oh.
B
A
A
So
one
of
the
fascinating
things
about
cockroach
DB
is
that
it
does
synchronous
replication
across
nodes,
and
so
you
get
a
certain
level
of
predictability
and
it
actually
uses
the
same
algorithm
of
as
at
CD.
But
it's
built
for
sort
of
scaling
out
is
a
more
realistic
database,
and
so
it
shares
some
of
the
some
of
the
underlying
sort
of
algorithms
and
even
there's
a
shared
library
between
at
CD
and
and
cockroach
DB.
And
that's
this
algorithm
called
raft
and
so
raft
is
a
consensus
algorithm.
It's
a
way
that
you
can.
A
So
what
you
say
is
that
the
databases
is,
it
provides
a
sort
of
an
absolute
sort
of
sequence,
order
of
what
happens
when
and
where
and
the
more
things
that
you
put
in
there
the
more
difficult
it
is
to
actually
fit
all
that
into
one
sequence
and
I
may
be
screwing.
This
up,
I'm,
not
a
database
expert
and
so,
and
so
what
cockroach
DB
does
is.
It
runs
a
whole
bunch
of
sort
of
raft
algorithm,
so
you
can
think
about
it
as
like.
A
A
bunch
of
vet
CDs
all
running
side-by-side,
but
one
of
the
things
that
you
get
out
of
that
now
is
that,
like
it,
Kent's
gets
a
lot
chatty
if
you're
doing
like
I,
don't
know
100
CDs
that
are
synchronizing
with
each
other.
That
gets
really
chatty,
and
so
these
guys
did
an
algorithm
called
I
believe
it
was
like
multi
raft
where
they
find
ways
to
compress
and
sort
of
minimize
the
amount
of
chattiness
after
doing
sort
of
a
parallel
set
of
raft.
Algorithms.
So.
A
B
A
So
that's
interesting,
and
so
you
can
see
that,
like
okay,
you
can
look
at
some
of
the
status
around
this
I.
Don't
know
any
of
this
stuff
means
Prometheus
metrics,
so
this
is
ready
to
be
consumed
by
Prometheus.
Here
are
some
other
metrics
about
sort
of
the
GC
metrics
for
for
ago.
So
all
sorts
of
good
stuff
going
on
here.
So
yeah
super
interesting.
Alright,
alright,
let's
go
through
and
let's
kill
a
node,
because
that's.
A
Note
see
what
they
all
have
these
green
dots
here
for
those
who
are
red-green
colorblind
I
promise
you
they
are
green
and,
let's
so
the
what
so.
The
way
that
this
has
us
do
this
is
it
has
us
just
go
ahead
and
on
with
a
background
and
delete
a
pod.
So
when
we
delete
a
pod,
it's
gonna
actually
do
a
clean
termination
of
that
pod,
because
you
can
see
it'll
take
upwards
of
60
seconds
for
this
thing
to
exit.
But
then
what
will
happen
is
the
state
will
settle,
be
like
hey.
A
This
pod
disappeared,
just
like
a
replica
said
it
will
actually
spin
up
a
new,
a
new
version
of
that
thing.
Solo
Maddie
says:
do
all
the
nodes
in
a
cockroach
DB
state
will
set,
handle
right
simultaneously
or
is
one
node
elected
master
and
the
other
slaves
are
replicas
I
believe
it's
like
a
multi
master
type
of
thing,
but
Alex
would
be
able
to
tell
you
more.
A
So,
let's
see,
if
alex,
has
any
details
on
this,
so
this
can
take
because
we
have
that
termination
grace
period
of
60
seconds.
This
could
take
up
to
a
minute
to
actually
have
this
thing.
Grace
Lee
shut
down,
gracefully
shut
down,
and
then
what
happens
is
kubernetes
takes
care
of
making
sure
that
it
detaches
that
EBS
volume
from
the
node
that
this
particular
pod
was
running
on
and
then
reattaches
it
to
the
replacement.
Oh
yeah,
and
now
we
can
see
that
we
have.
The
replacement
is
getting
a
knitted.
A
Let's
see,
and
so
if
avec
says
it's
multi
master,
so
you
can
read
right
from
any
node,
oh
and
now
we're
back
up
and
running.
So
what
you
can
see
is
that
this,
the
others
have
been
up
for
17
minutes.
This
one
is
a
few
seconds,
and
it's
still
coming
up
to
speed
here
and
so
I
believe
that
there's
enough
replication
between
these
things
that
the
database
would
still
be
accessible,
you
wouldn't
have
any
downtime,
even
though
one
of
those
nodes
went
missing
now,
if
we
took
down
two
nodes,
bad
stuff
happens
now.
A
This
was
a
graceful
shutdown
of
the
thing.
Let's
see
if
we
can
be
a
little
bit
meaner
to
the
thing
cube
controlled,
get
right.
So
now,
let's
see
so.
This
thing
now
landed
on
this
particular
node,
which
is
where
Mike
Ward
is
running
to
which
is
okay.
Now,
let's
go
into
let's
go
into
ec2
and
let's
see
if
we
can
be
ruder
about
taking
down
that
instance.
So
this
thing
was,
it
was
5.1.
B
B
B
A
I
said
five
dot
174,
so
this
guy
all
right,
so
this
guy
here
is
running.
What
can
we
do?
Can
we
instance
state
I
want
to
do
like
a
hard
terminate,
but
I
think
this
still
does
a
somewhat
graceful
terminate
well
Chris
says:
I
can
just
pasted
in
the
search
there
yeah,
okay,
so
we're
going
to
terminate
that
and
we're
gonna
say:
I
want
to
delete
the
I.
A
A
Let's
see
so,
this
is
ready.
This
will
take
it
a
few
minutes
to
actually
sort
of
recognize.
What's
going
on
here,
as
this
thing
shuts
down
again,
is
this:
this
is
the
sort
of
the
nice
way
of
shutting
the
thing
down
low
Maddie
says
echo
CSIS
request
trigger
just
disconnect
the
network
yeah
exactly
oh.
A
Down
now
so
now
that
got
removed
from
here
and
it's
terminated,
and
what
should
happen
here
is
that
this
is
a
an
auto
scaling
set
and
so
Amazon
should
launch
a
replacement
for
this
thing.
Okay,
now
what
we
see
is
that
cockroach
db2
actually
got
rescheduled
and
run,
so
the
database
is
up
and
running.
B
A
Them
so
eventually
that'll
happen,
but
and
then
we'll
see
that
Corgi,
because
I
just
launched
party
as
a
pod,
not
as
a
replica
set,
it's
gone
right,
and
so,
if
I
wanted
to
actually
do
that,
I'd
have
to
delete
and
relaunch
Corgi
explicitly
if
I'd
launched
it
under
a
replica
said
I'd
be
I'd,
be
ok,
it
would
have
gotten
restarted.
Also,
alright,
so
refresh,
let's
see
refresh
that's
not
happening
well,
I'm
doing
I'm
doing
a
watch
here,
so
that's
happening
every
two
seconds.
A
Okay,
let's
see
is
this
I
think
I
now
have
I
think
it's
it's
probably
launched.
Ok,
so
it's
now
launched
the
replacement
in
that
replacement
is
actually
initializing
here,
and
so
what
this
is
going
to
do
is
it's
going
to
rejoin
the
cluster
using
because
we're
using
cube
admin
here,
as
that
thing
comes
up
cube,
admin
join,
will
rejoin
the
cluster
using
the
the
token
that
we
have
alright.
B
A
A
A
A
A
A
This
is
actually
much
harder
for
kubernetes
to
actually
recover
from
and
I'll
tell
you.
Why
is
well
number
one.
Is
that
notice
that
this
thing's
not
ready
and
then
you'll
see
that
these
things
are
now
error
down?
Okay,
so
that's
all
expect
what
we
expected
to
happen
here
now.
What
should
happen
here
is
that
we
want
to
relaunch
this
thing
onto
another
node
right,
and
so
what
will
happen
is
that
we're
doing
in
a
knit
onto
another
node.
So
now
this
is
running
on
250
1.16.
A
A
Well,
how
forcefully
do
what
I
want
to
the
the
volume
from
the
previous
guy,
because
he
didn't
do
a
clean
shutdown,
another
thing
right,
and
so
it
had
to
unmount
to
then
remount
it,
and
there
might
be
some
issues
around
like
running
like
you
know,
you
know,
checking
the
disk
for
integrity
or
whatever,
but
but
the
idea
is
that
eventually
yeah.
So
now
we
have
an
it
error
here,
and
my
guess
is
that
this
error
is
around
being
able
to
being
able
to
go
ahead
and
deal
with
unmounting
the
volume.
A
So
so
this
thing
is
in
a
crash
loopback
off
and
this
thing's
in
an
error,
so
we
actually
an
it
crash,
loopback
off
and
so
something's
not
going
right
here
now.
What
we
do
have
is
that
it's
running
on
68
or
no
that's
running
on
21
one
one
one.
So
this
thing
is
ready.
It's
been
running
for
a
while.
Is
that
the
one
that
I
killed?
No,
that's,
not
the
one
that
I
killed
and
then,
if
we
refresh
this
what'll
happen
is
eventually
I.
Think
Amazon
figures
out
that
things
are
busted
too.
A
A
Oh
and
then
so
so
the
init
container
is
failing,
and
so
this
is
a
trick.
Also,
is
that
if
you
want
to
get
the
logs
for
the
net
contract
container,
you
have
to
specify
the
name
of
the
unit
container
directly,
so
the
name
of
that
and
it
containers
and
it's
certs.
So
so,
if
we
do
in
its
certs,
the
older
write
files
could
not
create
sim
link
from
the
secrets
to
oh.
So
this
is
interesting.
Is
that
because
this
thing
was
getting
restarted
in
some
way
all
right?
So
this
is
probable.
A
bug.
B
A
And
so
this
was
the
iniquity
net
container
called
yeah.
So
alex
is
not
good.
I
broke,
it
I
knew
I
was
gonna
break
it.
Alex
I
was
gonna,
find
a
way
yeah,
so
I
think
what
happens
is
that
something
got
mixed
up
in
the
the
thing
that's
trying
to
initialize
sorts
like
didn't,
find
things
as
pristine
as
it
thought
it
would
and
and
then,
and
so
now
it's
the
NIC
container
can't
complete.
There's,
no
reason
why
I
shouldn't
be
able
to
I
think
there's
just
a
little
bit
of
a
bug
there.
A
A
So
we
get
rid
of
that
one
yeah!
It's
odd,
though,
since
the
cert
volume
is
an
empty
durval
um--.
So
I'm
not
sure
why
that
happened.
It
could
be
that
it
ran
once
and
then
it
crashed
for
some
reason
and
then
I
ran
again
and
it
crashed
for
another
reason
or
maybe
like
there's
some
sort
of
weird
behavior
going
on
there.
That's
not
what
I
would
have
expected
to
have
happen
there,
so
I'm
not
sure.
What's
going
on
there
I,
like
sorry.
A
If
you
can,
if
you
can
track
that
down,
we
might
have
found
a
bug
in
kubernetes
itself,
all
right.
So
now
this
thing
is
doing
the
init
thing
again
and
then
the
pot
is
initializing
and
now
our
clusters
back
up
to
full
strength,
so
we're
in
good
shape,
so
I'm
not
sure
what
happened
in
that
particular
case.
Meanwhile,
nodes
we
have
all
full
nodes
and
then
did.
A
One
two
three
four
five
yeah
and
so
Amazon
recognized
that
that
node
died.
Also
so
I
think
we're
in
good
shape,
so
Abraham
says:
is
it
mounted
into
containers?
Well,
the
empty
jurors
should
be
should
be
local
to
that
pod
and
so
should
first
get
mounted
into
the
anit
container,
which
is
going
to
do
all
this
sort
of
certain
magic
and
then
it
should
get
mounted
into
the
main
runtime
container
and
I
think
what
alex
is
saying
and
I
agree
with.
Is
that
it's
odd?
Because
how
could
that
sibling
say?
A
A
Yeah
I,
don't
know
that
that
would
be.
My
guess
is
that
the
init
container
failed
and
so
the
read
the
init
container
got
restarted
multiple
times,
and
so
maybe
oh
you
know
what
okay
here's,
what
we're
gonna
do
we're
gonna
debug
this
a
little
bit.
So
if
we
do
we're
gonna
do
we
have
logs
right
the
init
container?
What
we
can
see
is
that
now
every
oh,
no
I
deleted
the
thing
shoot
because
there's
an
option
in
cube
control
logs,
which
is
previous,
and
so
so
that
was
probably
the
last
run
of
the
unit
container.
A
Probably
if
we
looked
at
previous
we'd
actually
see
that
there
was
a
previous
run
of
the
unit
container.
That
failed
for
some
other
reason
and
that's
probably
the
root
thing,
and
then
that
caused
the
emptied
everything
but
unfortunately
lost
that
state.
So
we
can't
figure
that
out
now
so
we'll
have
to
try
and
Reaper
alright.
So
that's
all
I
had
for
today.
I
think
this
is
a
real.
We
were
going
to
tear
it
down,
let's
tear
it
down,
let's
go
ahead,
and
so
you
can
scale
the
cluster
up
and
down
and
like
magic
things
happen.
A
I!
Think
we're
going
to
have
time
to
do
that
and
then
we're
gonna
stop
the
cluster.
So
so
we
can
do
it
like
this,
where
we're
deleting
all
this
stuff
like
that,
but
what
I'm
gonna
do
is
I'm
going
to
do
a
cube.
Do
like
the
Omega
cube,
control,
delete,
dash,
F
and
then
we're
gonna
delete
the
job
cuz.
Why
not?
Okay!
So
now,
if
you
play
with
this
with
a
toy
cluster,
make
sure
that
you
tear
things
down
cleanly,
because
if
you
don't,
then
those
volumes
will
get
left
along.
A
So
if
I
were
gonna
go
and
actually
delete
the
cloud
formation
stack
for
the
cluster,
it
doesn't
actually
know
that
hey
these
volumes
were
logically
part
of
that
cluster
and
so
that'll
over
orphan
those
volumes
and
then
and
then,
let's
see
and
then
well
that's.
Actually
a
nice
thing
about
as
yours
is
er.
A
Has
this
idea
of
sort
of
like
resource
groups,
and
so
the
resource
groups
in
Azure
are
in
some
ways
a
little
bit
like
projects
and
in
in
GCE,
but
essentially
you
can
create
a
resource
group
for
your
cluster,
and
then
that
includes
all
the
things
that
that
cluster
creates.
Also,
so
it's
a
flexible
way
to
sort
of
group
things
and
then
tear
them
all
down
so
Alex
says
I.
Think
the
pod
got
rescheduled
to
the
node
that
it
crashed
originally,
not
sure.
If
that
means
anything
it
could
be.
A
So
I'm
not
sure
what
happened
there.
All
right
so
now
what's
happening
is
that
we
see
that
because
of
the
when
you
scale
down
a
stateful
set
it
scales
down
in
a
very,
very
deliberate
way
also,
and
so
what
we're
seeing
as
it's
taking
its
time
to
actually
terminate
each
one
of
these
things,
wait
for
it
to
do
a
clean
termination
and
then
bring
the
stateful
set
down
now.
A
There's
options
with
stateful
set
that
you
can
say
you
know
what
don't
worry
about
the
ordering
just
blast
everything
up
and
down,
but
but
we're
not
using
those,
but
we
can
see
that
our
PVCs
are
still
here
and
we're
gonna.
So
let's
go
through
and
we're
gonna
do
and
for
those
that
we're
asking
earlier
about
PVCs.
A
So
now
we're
done
so
I,
don't
know
let
Maddie.
Let's
look
so
if
we
look
at
the
PV
cute
control
describe.
What
we
can
see
is
that
we
got
some
metadata
about
it.
Read-Only
partition,
there's
events
and
stuff.
It
may
show
up
in
the
event
the
last
access
time,
but
it
doesn't
look
like
it.
Stat
is
bound,
so
it's
bound
to
a
PVC
and
bound
by
controller
provisioned.
I,
don't
see
anything,
that's
like
last
access
time
on
the
PV,
but
what?
If
we
actually
look
at
the
PVC
itself,.
A
That
actually
has
when
it
was
provisioned
annotations
bound,
but
it's
not
in
use.
Oh
we're
gonna
check
to
see
what
does
it
look
like
when
it's
in
use
I?
Don't
see
that
so
I
don't
know
if
there's
a
way
so
Han
senses
our
max
unavailable
in
the
deployment
also
yeah.
So
the
max
I'm
available
I
believe
Hans
is
for
when
you're
doing
an
upgrade
of
a
stateful
set
or
in
the
deployment
yeah
yeah
in
a
deployment.
The
same
thing
same
way:
yeah.
A
Okay,
so
now
we
have
to
go
through
and
let's
delete
those
PVCs
cube,
control,
delete
TVC,
all
that
deleted
those
and
then
that'll
go
ahead
and
delete
the
EBS
volumes
that
are
behind
the
scenes.
Also
David,
thank
you
for
sending
that
fixed
for
the
302,
because
whenever
you
I
would
only
do
the
perp
I
mean
like
maybe
if
you're,
maybe,
if
you're
hitting
it
and
it's
not
coming
from
localhost.
A
Not-
and
you
know
if
the
host
header
is
not
localhost,
then
do
a
then
do
a
301,
but
if
you're
doing
localhost
it's
kind
of
bad
form
to
do
a
302
on
localhost,
because
you
would
use
that
stuff
all
over
the
over
the
place
so
yeah.
So
now
we're
cleaned
up,
I,
think
yeah.
So
I
am
really
impressed.
This
is
some
really
slick
stuff.
Obviously
we
just
scratched
the
surface
in
terms
of
using
cockroach,
DB
and
I.
You
know
and
I
think
over
time.
A
What
we're
gonna
see
is
that
you
know,
especially
with
things
like
you
know,
including
the
meta
controller,
but
also
like,
like
the
jet
stack
navigator
stuff,
we're
going
to
see
more
and
more
opportunities
for
people
to
build
for
people
to
build.
First-Class
sort
of
you
know:
stateful
set
management
on
top
kubernetes
and
so
I
think.
The
end
state
here
is
that
we
get
things
that
feel
a
lot
like,
say
something
like
Amazon
RDS
but
actually
have
sort
of
the
extensibility
and
the
portability
of
running
stuff
on
top
of
kubernetes
and
I.
A
B
A
Those
were
in
the
roll
bindings
yeah
and
in
the
cluster
roll
bindings,
because,
okay,
because
whenever
you
specify
a
service
account,
you
have
to
specify
the
namespace
for
the
service
account.
You
can't
leave
that
blank,
whereas
if
you
leave
the
namespace
blank
in
the
metadata
for
everything
else,
then
then
queue
control,
sort
of
stamps.
A
The
the
current
service
account
on
that,
but
that
doesn't
work
when
you're
talking
about
the
subject
for
for
a
role
binding
on
cholesterol,
minor
solo,
Maddie
says
on
the
point
of
storage:
do
you
happen
to
know
if
anyone
is
using
Gluster
set
for
Poquette
e
in
production?
I
know
a
lot
of
the
open
shift
deployments,
use
SEF
I,
don't
have
any
experience
with
it.
I
know
that
you
know
people
both
swear
by
and
swear
at.
It
I
think
the
interesting
sort
of
you
know.
Upstart
in
this
world
is
Thomson,
so
it
was
locked
to
default.
A
There
wouldn't
work
if
you
yeah,
that's
it's
an
easy
thing
to
modify
I
think
it's
a
reason
to
do
some
sort
of
level
of
templating
on
this
stuff,
whether
it
be
helm
or
something
like
case
on
it.
So
it's
a
good
point,
though,
that
that
this
would
not.
This
would
not
apply
out-of-the-box
to
a
different
namespace.
A
So
so
the
upstart
in
the
storage
world
is
rook
and
so
rook
is
really
interested
because
it
uses
a
lot
of
the
sort
of
data
plane.
Parts
of
Ceph,
but
it's
it's
more
like
cockroach
in
terms
of
the
management
stuff,
is
built
to
be
run
dynamically
in
a
system
like
kubernetes
I.
Think
the
larger
issue
is
that
there's,
like
you
know
the
whole
idea
that
you're
going
to
be
reusing,
the
same
infrastructure
for
both
storage,
with
like
block
devices
over
the
network
and
you're,
going
to
be
using
it
for
your
workloads.
A
That
whole
scheme
is
what
people
called
hyper-converged
storage,
because
you're
running
both
your
storage,
your
sand
and
your
compute
all
on
the
same
same
system.
That
is
some
really
interesting
stuff,
but
I.
Think
a
lot
of
people
underestimate
the
you
know
how
that
type
of
system
acts
it
load.
The
reason
being
is
that
it's
very
easy
to
start
saturating
your
network
when
you're
running
both
workloads
and
storage
over
it
and
when
you
end
up
with
a
saturated
network
distributed
systems
like
kubernetes,
have
some
really
wacky
failure
modes,
because
what
happens?
A
Is
that
hey
I
can't
go
ahead
and
and
talk
to
this
node
I
can't
do
a
health
check
to
the
cubelet.
So
therefore
the
cubelets
down,
let
me
start
re
replicating
everything
and
you
end
up
in
a
cascading
failure
situation,
and
so
the
advice
that
I
give
folks
is
that
I
wouldn't
even
touch
any
sort
of
hyper-converged
stuff,
where
you're
going
over
the
same
network
for
both
storage
and
compute.
A
A
So
that's
all
I
got
for
today.
Thank
you,
everybody
for
joining
me.
Hopefully
you
learned
a
little
bit.
I
definitely
did.
This
is
a
really
interesting,
exciting
system,
I'm
glad
to
see
that
the
cockroach
is
embracing
cockroach
TVs,
embracing
kubernetes.
There's
a
lot
of
good
work
being
done.
There
I
think
it's
going
to
get
even
better,
as
we
start
seeing
the
the
local
volume
support
happen,
because
you
won't
be
doing
that
that
double
replication
that'll
probably
improve
performance
in
these
worlds,
quite
a
bit
all
right
so
tune
in
next
week.
A
I
have
no
idea
what
I'm
going
to
cover,
but
but
we'll
do
something
fun
and
yeah,
and
if
you
have
any
comments
on
the
video
you
know
leave
them
in
the
the
YouTube
comments,
give
it
a
like
or
subscribe
or
reach
out
to
me
on
Twitter.
If
you
have
other
stuff
yeah,
so
echo,
c2,
proc,
sis
cute,
request
trigger
and
then
watch
the
world
burn.
That's
exactly
what
we're
doing
all
right!
All
right!
I'll
talk
to
you
all
later.