►
From YouTube: OCB: Database Transactions in Kubernetes and OpenShift - Spencer Kimball (Cockroach Labs)
Description
Kubernetes and Red Hat OpenShift gives organizations the flexibility to run workloads on-premise, or in any public or private cloud. It is allowing all of us to speed development and become more efficient.
But what about the database? We’ve shifted our monolithic applications to distributed services, but what about the databases that support them? The monolithic legacy database simply does not “fit” this modern distributed environment and that is why we created CockroachDB. It’s a highly-evolved cloud-native, distributed OLTP database that helps companies of all sizes—and the apps they develop— scale fast, survive anything, and thrive everywhere. In this session, we’ll talk about CockroachDB and how it was built for Kubernetes with Spencer Kimball, Co-Founder & CEO and Jim Walker, VP of Product Marketing at Cockroach Labs.
A
A
series
we're
doing
with
folks
who
have
built
operators
that
run
on
openshift,
but
we're
not
necessarily
going
to
talk
about
the
operators.
Specifically,
we
might
talk
about
them
a
little
bit
and
get
you
guys
to
tell
us
what
you
did
and
what
you
learned
building
them,
but
today
we're
going
to
do
something
kind
of
special.
Actually
we're
going
to
the
title
is
database
transactions
in
kubernetes
and
openshift,
but
the
guests
are
from
cockroach
labs.
A
We
have
with
us
spencer,
kimball,
the
ceo
and
co-founder
and
jim
walker
who's,
the
vp
of
product
marketing
at
cockroach,
and
we're
going
to
have
an
interesting
conversation
with
them,
because
spencer
comes
out
of
google.
Has
a
lot
of
the
genesis
of
cockroachdb
is
comes
out
of
google,
we're
going
to
talk
a
little
bit
about
that
we're
going
to
talk
about
distributed,
sql
we're
going
to
talk
about
working
with
openshift
to
make
all
that
work
for
you
and
kind
of
the
impact
of
cloud,
5g,
serverless
and
all
those
kind
of
things.
A
B
That
is
great
and
thank
you
diane.
Thank
you
for
having
us
sincerely.
This
is
a
bit
of
a
little
bit
of
a
homecoming
for
me
as
well,
because
I
was
as
a
member
of
the
core
os
team
as
well,
and
so
you
know,
coreos
being
part
of
red
hat
and
and
part
of
openshift,
and
this
whole
shift
to
kubernetes
and
everything
else.
You
know
I'll,
introduce
spencer
I'll,
let
him
introduce
himself,
but
I
gotta
say
the
very
first
time
I
ever
met
spencer.
C
B
A
B
C
B
You
know
yeah
I've
been
in
the
industry
now
wrestling
with
databases
for
about
30
years.
I
started
even
when
I
was
at
university.
I
wasn't
very
interested
in
databases,
but
the
reality
is.
If
you
try
to
build
anything,
you
run
head
first
into
them,
and-
and
I
did
a
con
startup
and
then
ended
up
at
google
in
2002,
and
I
was
there
for
10
years
and
it
was
really
a
front
row
seat
on
the
evolution
of
google's
interest
in
databases.
B
One
of
the
first
things
I
worked
on
was
around
the
adwords
product
and
they
were
using
my
sql
and
they
ended
up
having
to
shard
it,
which
just
means
that
you
have
some
customers
on
one
mysql
instance,
but
it
gets
too
big.
So
then
you
employ
another
mysql
instance
and
then
another
and
then
another
as
you
grow
by
the
time
I
left
that
project.
It
was
about
32,
shards
and
eventually
went
to
about
a
thousand
stars
before
they
replaced
it.
Then
google
built
bigtable
just
super
interesting
that
was
like
about
hey.
B
We
don't
want
sql
or
we
do
want
sql.
It's
not
really
the
point
we
want
scale.
We
want
elastic
scale
and
then
they
built
something
called
megastore
started
to
introduce
transactions
a
couple
years
later
and
then
finally
spanner
so.
I
actually
worked
on
some
of
that
distributed
infrastructure.
A
product
called
colossus
which
is
a
hexascale
distributed
file.
Storage
is
kind
of
the
successor
to
gfs,
but
also
worked
on
applications
in
between,
and
so
I
got
this
it's
a
really
nice
cycle
right.
B
You
work
on
applications,
you
decide
well,
there's
some
things
that
are
really
missing
from
what
the
infrastructure
is
providing.
Then
you
work
on
infrastructure
and
you
get
to
provide
those
things
and
then
you
go
back
to
applications.
You
realize
that's
great.
I've
got
that
now,
but
I
need
these
other
things,
and
so
the
cycle
continues.
When
we
left
google,
I
say
we
because
I've
got
two
co-founders
of
cockroach
and
all
of
us
worked
at
google
for
those
ten
years
together.
B
B
It
started
as
sort
of
a
manifesto
we
weren't
trying
to
build
a
database
when
we
left
google
we're
trying
to
build
a
private
photo
sharing
application,
but
it
became
pretty
obvious
that
what
google
had
evolved
internally
was
exactly
what
we
needed
for
our
startup
and
then
we
got
acquired
by
square
and
we
realized
wow
square
needs
a
lot
of
the
same
things
that
we
needed
at
viewfinder
and
that
google
needed
for
the
10
years.
We
were
at
google
and
then
you
look
around
people.
B
I
knew
at
dropbox
and
pinterest
and
then
yelp
and
everyone
needed
a
database
like
this.
So
that's
where
we
started
the
open
source
project
and
you
know
the
rest
is
sort
of
history.
It's
been
five
and
a
half
years
now.
Cockroach
and
you
know
we
have
some
of
the
world's
biggest
companies
and
some
of
the
world's
smallest
companies
is
our
customers.
It's
very
exciting,
but
the
way
the
right
way
to
think
about
cockroach
is
it's
a
relational
sql-based
database.
B
In
fact,
it
looks
like
postgres,
but
instead
of
being
monolithic
in
the
tradition
of
systems
like
sql
server
and
db2,
and
oracle,
where
you're
really
scaling
up
an
instance
that
sits
in
one
location,
coach
is
fully
distributed.
You
know
from
the
the
the
sort
of
most
basic
parts
of
the
the
overall
system
is
fully
distributed,
and
what
that
really
does
by
you
is
you
know
what
we're
going
to
talk
about
fundamentally
in
this
interview,
but
it's
been
and
then
it's
been
a
long
journey
and
you
know
I'm.
B
I
never
was
interested
in
databases
when
I
was
at
university,
but
that's
what
I've
basically
spent
all
my
time
on
since
so
spencer.
Thank
you
for
that.
That
was
like,
basically
the
first
half
of
our
conversation.
In
a
nutshell,
I
love
it
buddy,
so
the
so
I
you
know
there
are
two
things
I
actually
want
to
actually
ask
you
about
too.
In
university.
You
also
had
a
pretty
popular.
I.
B
C
B
Were
you
know
coming
out
of
in
93
it
was
everyone
had
windows
or
mac
at
home
mac
os?
On
these
earlier
versions
of
my
quest,
we
had
windows
3.1.
I
can't
remember
exactly
what
it
was
at
that
time
and
we
got
to
berkeley
and
it
was
wow
look
at
these
unix
systems,
and
you
know
by
today's
standards
they
weren't
so
good.
C
B
Know
using
photoshop
and
the
like,
and
there
was
xv
and
there's
something
called
x
paint
and
they
were,
you
know,
pretty
pretty
sad
compared
to
what
photoshop
was
at
that
point
in
time.
So
we
we
kind
of
decided
one
night
after
a
couple
of
years.
I
guess
that
you
know
hey.
This
would
be
a
wonderful
thing
to
build
as
sort
of
free
software,
and
so
that's
where
the
idea
of
the
came
from
and
was
you
know,
titled
as
a
new
image
manipulation
program.
B
B
On
that,
for
basically
our
entire
undergraduate
career
is
often
skipping
classes
and
not
doing
a
great
job
on
some
of
our
projects
in
lieu
of
working
on
this
like
full-time
and
then
when
we
left
college.
The
beauty
of
it
is
that
we
stopped
working
on
it
in
97
and
it
still
exists
and
it's
going
strong
right.
So
it's
been
since
we
stopped
working,
it's
been
20
23
years.
That's
it's
hard
to
imagine,
but
23
years
and
the
open
source
community
has
inherited
it
and
has
promulgated
it
and
improved
it
continuously.
B
That's
the
beauty
of
open
source,
and
that
is
the
beauty
of
open
source.
It's
exactly
right!
It's
it's
years
and
years
later,
and
here
you
are
still
doing
open
source
as
well,
but
the
question
that
you
and
I
always
get
no
matter
where
we
go.
What's
with
the
name
right
so
cockroachdb,
I
think
you
kind
of
described
it,
but
like
a
little
bit
about
the
name,
I.
A
B
People
always
ask
us
that
right,
so
how
did
you
come
up
with
the
name?
Well,
as
you
can
tell,
I
think
peter
came
up
with
the
name
and
peter
and
I
happen
to
have
similar
senses
of
humor,
and
so
we,
like
things
that
are
a
little
bit
darker,
maybe
is
the
right
way
to
say
it
just
kind
of
tickles,
our
funding
bone.
B
You
know,
cockroach,
the
name
was
one
that
I
chose
and
it
was
really
because
when
we
were
desperately
figuring
out
what
kind
of
database
we
are
going
to
use
in
our
viewfinder
architecture,
which
is
the
private
photo
sharing
startup,
we
kind
of
hit
on
the
idea
after
examining
things
like
hbase
and
cassandra
and
mongodb,
and
you
know
all
the
normal
things
like
my
sql
and
postgres,
we
said
you
know,
we
want
something.
That's
like
google
spanner,
but
it
should
be
open
source,
and
you
know
if
we
really
think
about
how
it's
going
to
work.
B
It's
like
these
individual
sort
of
pods
or
nodes
that
are
all
greedily
optimizing,
but
making
sure
that
the
data
they
have
is
is
replicated
elsewhere,
sort
of
greedily.
Managing
that,
if
you
give
them
more
space,
they
colonize
it
right,
and
so
this
idea
of
you
know
just
kind
of
the
evocative
concept
was
a
bunch
of
cockroaches.
I
live
in
new
york
city
and
I
don't
like
cockroaches
believe
me.
I
hate
them.
I
mean
I
hate
them,
probably
more
than
most
people.
It's
a
little
ironic.
C
B
Well,
funny
enough,
I
think,
love
it
or
hate
it.
People
remember
it,
that's
for
sure
it's
it's
definitely
got
you
know
it
definitely
has
that
going
for
it.
So
let's
talk
about
spanner
and
google
a
little
bit
and
kind
of
agenda,
because
that
is
really
the
genesis
of
cockroachdb
was
I
mean
the
google?
You
know
the
google
cloud
banner
that
that
white
paper
that
they
published
right
and
and
some
of
those
sort
of
things
you
were
kind
of
front
lines
in
terms
of
like
they
had
bigtable.
You
mentioned
this
in
your
intro.
B
They
had
bigtable,
they
had
a
couple
other
solutions.
Why
did
google
need
to
build
yet
another
database
on
top
of
borg
and
everything
else?
At
the
time
I
mean
I,
I
know
you
weren't
on
that
team,
but
you
were
kind
of
adjacent
to
that
right
I
mean
I
think
you
were
working.
Were
you
working
within
like
the
classes
team
at
that
time,
like
the
the
file
system?
Correct,
that's
right!
That's.
B
B
Has
a
had
a
long
and
storied
history
of
databases,
I'm
sure
it's
continued
in
the
eight
years
I've
been
gone,
you
know,
google
is
not
afraid
to
to
do
pretty
involved.
R
d
and
and
part
of
the
reason
was
that
they
had
to
like
if
they
wanted
to
realize
their
ambitions,
and
if
you
think
about
google
in
2002
scale
was
a
you
know,
an
unbelievably
pressing
concern
for
them.
They
had
the
entire
world.
B
Basically,
starting
to
do
you
know
daily,
maybe
hourly,
maybe
by
the
minute
of
you-
know,
actions
that
involve
their
systems,
and
so
they
needed
very
large
scale
databases
and
at
the
beginning
it
was
really
read-only
databases
or
write,
wants
to
read
many
times.
They
had
these
systems
that
were,
you
know,
really
just
read-only
indexes,
and
then
they
started
moving
to
something
that
could
kind
of
gradually
layer
in
additional
changes
called
the
rt
server
and
then
the
the
the
indexing
pipeline
you
know,
was
kind
of
its
own
database
and
index
and
so
forth.
B
It
was
very
custom
purpose,
but
then
they
started
branching
off
into
other
things
like
adwords,
for
example
right.
So
they
needed
something
to
manage.
All
the
creatives-
and
you
know
the
places
where
people
come
in
and
that's
that's
where
they
started
using
mysql
and
but
at
the
same
time
they
started
building
other
things
like
started,
storing
data
for
every
you
know.
B
If
you
had
a
cookie
and
you
started
searching
on
google,
they
wanted
to
associate
data
with
that
so
that
they
could
build
better
search
right
so
that
they
knew
what
you're
what
you
were
probably
looking
for,
based
on
past
things,
and
and
so
there
was,
you
know,
a
huge
need
for
you
know
massive
scale.
Data.
What's
interesting.
B
It
he's
had
quite
a
career,
and
you
know
we
would
look
at
the
problems
that
that
happened
since
the
the
last
day
and
we'd
be
like
figuring
out
how
to
solve
them,
how
to
do
the
short-term
phase.
This
is
what
the
medium
long-term
fixes
were
and
that
that
went
on
for
months,
because
it
was
such
a
big
problem
and
we
we
got
it
somewhat
stable.
We
built
lots
of
interesting
things,
as
I
mentioned
before,
that
went
to
a
thousand
charts
by
two
thousand,
oh
wow,
so
it
went.
B
B
Not
a
thousand
hundreds
of
thousands
and
facebook
has
spent
engineering
millennia
knitting,
those
together
into
a
big
meta,
meta
database,
very
custom.
Super
snowflake
technology,
only
good
for
facebook,
but
you
know,
but
cockroaches
building,
probably
wouldn't
ever
be
able
to
apply
to
facebook
because
their
scale
is
so
enormous
and
they
have.
They
have
a
careful
balance
of
consistency
and
eventual
consistency.
It's
a
very
complex
system,
but
what's
interesting,
is
google's
experience
with
adwords
in
my
sequel
and
sharding
led
them
to
create
a
moratorium
on
that
kind
of
an
architecture.
They
said.
B
No
we're
not
going
to
do
this
again
right
once
is
enough
right.
So
if
you
want
to
build
a
large
scale
system,
you've
got
to
use
something
else,
and
around
2004
is
when
big
table
came
out
and
bigtable
is
really
geared
towards
that
problem.
I
was
talking
about
of
associating
data
with
cookies
and
search
they're,
just
massive
scale
and
bigtable
wasn't
like
you
know,
we
don't
like
sql,
we
don't
want
to
use
sql,
it
was
really
well.
Sql
is
really
complicated
and
we've
got
a
bigger
problem
right
now,
which
is
scale.
B
That's
big
enough
and
not
just
scale
but
elastic
scale,
and
so
big
table
was
a
direct
consequences
of
the
challenges
and
the
opportunities
that
google
was
facing,
like
how
do
you,
how
do
you
they
didn't,
buy?
Mainframes
or
you
know,
sun
micro
system.
You
know
huge
multi-processor
systems,
they
they
were
doing
commodity
hardware,
they're
kind
of
like
the
proto
cloud
right.
It
was
google's
cloud
and
it
felt
like
a
public
cloud
internally
right.
So
google's
like
10
years
ahead
of
the
the
wrestling.
B
They
started
a
lot
of
them
and
it
was
internally.
It
was
very
exciting
too.
It
was
an
amazing
piece
of
technology,
and
you
know
everyone
was
pretty
jazzed
about
it,
and
I
remember
there
was
an
ask
early
on
of
like
from,
I
think,
the
infrastructure
team-
okay
adwords,
you
know
you
guys
are
having
trouble
with
your
my
sequel,
sharding.
You
just
use
bigtable.
This
will
handle
your
scale
issues
and
there
was
a
huge
pushback
because
they
said
okay
guys
we
have
like.
I
don't
know
what
it
was
at
the
time.
B
Probably
like
a
couple
hundred
tables.
We
need
transactions,
we're
talking
about
people's
financial
information,
there's
essentially
a
ledger
happening
like
this
is
not
there's
a
huge
impedance
mismatch
between
us
coming
from
a
relational
database
system,
even
if
we
have
a
very
awkward
sharding
mechanism
to
go
into
bigtable,
which
is
just
not
set
up.
It's
like
bigtable
is
great
for
certain
kinds
of
tasks,
but
adwords
required
something
that
you
know
fundamentally.
B
Sql
relational
database
has
been
evolving
for
40
years
by
that
time,
40
years
right,
so
you
can't
just
replace
it
in
one
fell
swoop,
and
so,
interestingly,
google
responded
to
that.
The
infrastructure
teams
responded
back
and
they
created
something
called
megastore
and
megastore
came
out
two
years
later.
This
is
very
interesting
because
you
know
mongodb
didn't
get
transactions
until
like
2019
or
something
like
that
right.
Google
had
transactions,
limited
form
of
them
to
be
fair
in
2006,
because
the
big
table
kind
of
fell
flat
for
certain
use
cases
right.
B
We
need
to
have
some
kind
of
consistency.
The
big
table
megastore
introduced
that
they
also
introduced
the
idea
of
consensus-based
replication,
so
a
big
step
forward
in
megasword,
it's
two
years
after
big
table,
but
quickly
after
that
people
realize
you
know
what
there
are
more
things
we
need
to
do.
We
want
general
purpose
transactions.
It's
not
not
these
limited
ones
that
megastore
brought-
and
we
want
to
sort
of
in
some
ways
you
can
think
of
spanners-
is
sort
of
closing.
The
circle
like
the
circle
got
opened
right.
B
It's
like
we
need
to
deal
with
massive
scale
and
then
it's
like.
Okay
started
to
close:
we
need
to
get
some
kind
of
transactions
now
we
need
really
general
purpose
transactions
and
then
eventually,
they've
created
something
called
f11,
which
is
we
actually
want
to
bring
sql
back,
because
we
want
to
replace
adwords.
We
want
to
replace
the
mysql
sharded
mess
with
a
the
new
next
generation
system,
but
in
order
to
do
that,
we
can't
just
make
them
rewrite
everything
for
some
weird
new
api.
It's
actually
should
be
sql.
B
It
should
look
enough
like
sql
that
we
can
make
that
transition,
so
they
came
full
circle.
It's
very
interesting,
and
so
you
know
when
we
left
google,
we
realized.
Oh,
you
know
we
want
a
relational
database
too,
and
we
also
want
these
capabilities
and
we
know
it
needs
to
be
open
source.
So
we
had
the
enviable
opportunity
to
fast
follow
google,
which
is
a.
B
Than
doing
it
all
for
the
first
time,
well,
I
think
we're
all
benefiting
from
that
period
of
time
at
google
right
I
mean
today,
spencer,
you
and
I
we
get
in
conversations
with
customers
and
it's
like
they're
almost
like
your
conversation
right
now
is
some
of
the
stuff
that
I'm
hearing
in
some
large
organizations.
Well,
we've
scaled
this
database
and
it's
falling
over
and
we're
having
problems
with
it.
We
can
only
go
so
far
and
then
they're.
B
Legacy,
tech
and
knowing
that
well,
there
is
actually
prior
art.
There
is
actually
somebody
who
can't
sell
this,
and-
and
I
you
know
to
me
this-
the
you
know
board
turning
to
kubernetes,
that
that
whole
movement,
just
the
the
wherewithal
and
the
the
vision
at
google
to
actually
understand
what
the
future
of
of
compute
was
going
to
be.
You
know
resulting
what
are
we
15
17
years
later
or
12?
You
know,
20
years
later,
after
that
journey,
you
know
really
really
kind
of
took
off
right.
C
C
B
Thing,
that's
really
interesting
is
just
that
you're
mentioned
board
with
kubernetes.
I
mean
you
know
for
a
while
when
kubernetes
was
new
and
didn't
really
do
stateful
workloads,
there
was
like
a
non-trivial
contingent
of
people
that
would
opine
frequently
that
well,
kubernetes
is
really
just
for
stateless
services.
Faithfulness
isn't
going
to
work,
that's
not
what
it's
for!
B
Well,
you
looked
at
borg
back
in
2005
and
six
and
believe
me
borg
ran
everything
had
ran
big
table
around
megastore
I
ran
spanner,
ran
colossus
or
md,
it
ran
stateful
services
and
it
ran
them
with
ease
and
and
that's
of
course,
what
orchestration
needs
to
do.
It's
not
just
the
stateless
stuff
right.
It's
it's
everything!
So,
there's
a
there's,
a
there's,
a
really
good
match
between
orchestration
technologies
like
kubernetes
and
database
systems
like
cockroach.
They
too
are
made
for
each
other.
B
So
it's
good
to
see
kubernetes,
evolving
and
maturing
and
being
able
to
handle
that
that
much
wider.
You
know
spectrum
of
use,
cases
and
infrastructure.
Yeah!
That's
right,
and
it's
it's!
It's
brandon
phillips
a
long
time
ago,
once
quoted
this
out.
It's
like
it's
giphy.
It's
google,
google
infrastructure
for
everybody.
I
think
is
what
he
wanted
to
have
and
I
think
that's
where
we're
headed,
but
you
know,
but
one
of
the
things
that
had
come
out
of
that
whole
conversation.
B
I
remember
this
like
four
years
ago,
was
the
whole
concept
of
operators
and
actually
simplifying
the
life
of
how
all
this
stuff
works.
How
much
did
it
take
to
actually
make
all
this
work
back
then?
I
mean
just
sort
of
just
a
pure
resources
point
of
view
spencer,
like
if
I'm
gonna
have
okay,
I'm
gonna
go
from
four
shards
to
a
thousand
charts
at
google
or
at
facebook.
We
have
teams
and
teams
of
people
that
have
to
manage
this
right
is
that
I
mean
how
well
we're.
C
B
A
B
C
B
B
B
A
B
B
I
mean
you
know
we
have
a
customer,
that's
one
of
the
the
big
big
financial
services
companies
and
just
in
one
of
their
arms,
they
have
6
500,
just
like
one
of
their
business
units,
6
500
externally
facing
applications,
that's
just
like
mind-blowing,
yeah
yeah
and
the
amount
of
databases
that
you
hit
in
your
day,
thus
far
between
waking
up
till
this
moment,
and
it's
only
a
couple
hours.
It's
just
insanity
right,
like
I
joke
about
that
in
the
legacy
databases
all
the
time.
B
But
you
know
I
love
the
concept
of
engineering
millennium
and
it's
like
well,
there's
devops
millennium
and
there's
an
sre
millennium
as
well
that's
put
into
these
systems,
and
I
don't
think
people
really
understand
like
what
it
actually
takes
to
even
go
like
one
of
the
reasons.
I
love
that
I'm
on
here,
with
with
what
the
openshift
team
is
doing
at
red
hat
and
simplifying
the
operations
side
and
that
side
of
the
world
I
just,
but
we
talked
a
little
bit
about
how
you
know.
B
Cockroachdb
is
aligned
very
well
with
kubernetes,
spencer
and
just
being
a
distributed
database
on
top
of
a
you
know,
distributed
orchestration
platform
right
like
that.
It
is
absolutely
what
originally
attracted
me
to
this
company.
If
you
look
at
the
future
of
the
database
to
me,
I
believe
it's
got
to
be
distributed
sql,
but.
B
You
know
you
know,
I
think
it
was
is.
Is
it
stone
breaker
that
says
it
takes
seven
years
for
a
database
to
fully
gestate
mature
right,
yeah,
so
like
building
a
database
takes
seven
years
right?
Building
a
distributed
system
adds
a
holy
other
layer.
On
top
of
that,
right,
like
it
is,
that's
that's
complexity.
On
complexity.
B
You
know,
I
guess
going
back
to
the
the
beginning
of
of
what
we've
done
at
cockroach
and
architecting
from
the
ground
up.
What
were
the
biggest
challenges
to
actually
get
this
to
work
right,
like
I
mean,
I
think
we
had
a
really
good
sample
in
this
banner
white
paper,
but
you
know
technically
there's
some
pretty
pretty
hefty
challenges
that
we
faced
early
and
we
still
face
today.
Right
I
mean
we've
been
building
for
five
and
a
half
almost
six
years
now,
so
what
were
some
of
the
biggest.
B
At
the
very
beginning,
well,
the
distributed
transaction
model
is,
is,
you
know,
required
a
a
lot
of
work?
Also,
the
consensus
replication.
I
mean
people
yeah.
It's
not
unusual
for
undergraduates
to
implement
raft,
which
is
a
variant
of
access,
is
what
we
use.
You
know
one
of
my
co-founders
ben
darnell
likes
to
quit
that
it
took
him
a
week
to
implement
raft
in
you
know
probably
another
several
years
to
make
it
actually
work
in
production
properly.
So
that's
just
it's
it's
kind
of
shocking
to
hear
that.
B
These
things
and
the
transaction
model
is
another
fascinating
thing
that
we
worked
on
because
cockroach
isn't
just
distributed
in
terms
of
hey.
You
can
shovel
in
more
commodity
hardware
within
a
data
center
cockroach
is
meant
to
be
distributed
across
data
centers,
for
example,
within
a
region
like
different
availability
zones
on
the
us
east
coast.
That
would
be,
you
know
how
you
really
really
want
to
do
the
geo
replication,
so
you
have
low
latency,
but
consensus
replication,
but
it's
also
built
to
be
replicated
across
continents
like
vast
geographic
distances.
B
B
You
know
all
of
the
every
time
you
do
a
read
you're
actually
hitting
the
database
that
certainly
you'd
never
ever
want
that
chattiness
to
go,
for
example,
from
australia
over
to
virginia.
That
would
that
would
absolutely
destroy
your
your
transaction
time.
So
how
do
you
actually
build?
How
do
you
plan
for
that?
What
does
the
topology
look
like
of
the
cluster
and
how
does
the
transaction
model
work?
And
we
have
you
know
anyone?
B
That's
really
interested
in
the
deep
details
of
this
there's
a
lot
of
interesting
posts
on
our
blog
that
describe
sort
of
a
state
of
the
art
in
terms
of
how
we're
managing
that.
But
you
know
those
two
things:
transactions
and
replication
by
far
are
the
stickiest
points
and
they
can
continue
to
be
honestly.
There's
there's
a
lot
of
stuff
on
the
drawing
board
for
how
we're
gonna
really
make
global
transactions,
something
that
everybody
can
use.
Trivia.
That's
right!
B
That's
right,
and
I
think
it's
I
I
always
I
always
point
people
to
our
docs
as
well
spencer,
because
I
think
they're
just
like
if
you
really
want
to
get
into
it.
What
jesse
and
team
does
on
that
side?
Is
it's
tremendous
right
like
it's
there's
such
really
really
good
stuff
there
so,
but
coming
back
to
distributed
transactions,
and
you
also
mentioned
serializable,
you
know-
and
I
think
that's
a
that's
a
that's
a
big
task
to
take
on
right,
because
you
know
our
ultimate
competition
here
is
the
speed
of
light
right.
B
B
B
I
mean
look
at
building
a
distributed.
Transaction
model
from
the
ground
up
is
not
simple.
I
think,
there's
a
couple
companies
that
are
trying
to
do
this
as
well.
It's
not
a
simple
task.
It
took
you
know
how
long
at
google
I
go
through
the
beginning
of
everything
at
google.
How
long
it
took
to
get
to
spanner
spencer,
because
it.
B
It
is
a
lot
of
expertise,
because
these
are
not
simple
things
to
do.
You
know
building
oracle
from
the
ground
up
from
way
back
and
what
an
incredible
database
I
mean.
What
a
I
mean.
It
is
incredible
technology.
I
still
think
like
that
and
photoshop
I
think,
are
two
of
the
most
incredibly
complex.
I
mean.
B
C
B
Fantastic
dummies
system,
no
question:
I
mean
it's
running
most
of
the
world's
high-value
use
cases,
at
least
the
plurality
of
them.
It's
it's
definitely
an
extraordinarily
successful
company
as
well.
There's
there's
some
things
that
you
know:
people
don't
love
about
their
their
vendor
relationship
with
oracle,
but
you
can't
argue
with
their
success.
It's
a
great
product
yeah
I
mean
our
our
society.
C
B
You
know
a
lot
of
the
advances
that
we
have
are
due
to
that,
because
this
this
transactional
model
and
being
able
to
actually
implement
serializable
like
our
banking.
Everything
like
our
money
is
basically
run
through
systems
like
that
and
so
doing.
Serializable
isolation
in
a
in
a
distributed
transaction
model
presents
some
fairly
difficult.
B
A
B
B
A
B
You
know
one
thing:
that's
just
really
important,
there's
a
there's,
a
dichotomy
in
the
database
that
I
think
people
sometimes
conflate
the
two
things,
but
there's
geo-replication
and
then
there's
or
just
replication
and
generally
not
it
doesn't
have
to
be
geographic
but
and
then
on
top
of,
that
is
the
transaction
model.
So
when
you
write
a
single
key
or
even
a
group
of
keys
that
are
kind
of
close
together
and
they
they
can
all
be
written
atomically
as
part
of
the
wrath
or
paxos
doesn't
matter
replication
protocol.
B
The
the
problem
is
that
when
you
have
a
truly
distributed
system,
that's
quite
large
you've
got
lots
of
different.
You
know
for
lack
of
better
term
shards.
We
call
them
ranges
but
they're,
essentially
chunks
of
the
key
space
that
are
replicated
between,
say
any
three
nodes
you
have,
but
you
might
have
a
hundred
nodes
and
so
you've
got
lots
of
different
replicated
shards
of
the
key
space,
and
so
in
order
to
actually
write
to
multiple
parts
of
that
key
space
that
live
on
different
nodes
in
the
in
the
larger
distributed
system.
B
That's
where
a
distributed
transaction
comes
in,
so
at
the
low
level
you've
got
replication,
there's
something
like
paxios
or
wrap
and
at
the
top
level.
That's
where
you
have
a
distributed
transaction
protocol
and
those
two
things
have
to
work
very
well
together.
You
know
so
you
mentioned
serializable,
as
probably
most
of
the
people
on
that
are
listening
to
this
are
aware
from
their
experience
with
databases
in
the
sql
ansi
standard,
there's
a
there's,
a
number
of
different
isolation
levels.
B
If
how
much
latency
there
might
be
in
a
transaction
versus
you
know
how
much
isolation
that
you're
getting,
and
so
you
can
have
less
isolation,
and
you
know
really
let
the
database
blaze
through,
in
other
words,
transactions
that
might
be
overlapping
or
intersecting
in
some
way
they
don't
really
care
about
each
other.
Some
people
call
that,
like
the
the
I
shouldn't
use
the
word,
but
you
know
that
starts
with
an
f,
but
you
know
effort
mode
transactions.
C
B
These
different
levels
all
the
way
up
to
serializable,
and
when
people
talk
about
acid
transactions
that
I
isolation,
that
means
serializable,
let's
just
be
honest,
right,
like
all
those
other
modes
they're
about
trading
off
isolation
in
order
to
get
more
efficiency
in
the
database.
So
you
know
what
we
decided
to
do
like
one
of
our
our
sort
of
overall
mission
for
cockroach
labs
is
to
make
data
easy.
We
came
from
square
before
and
theirs
is
make
commerce
season.
B
We
kind
of
got
a
little
lazy
there,
but
it's
it's
actually
a
great
mission,
because
you're
only
ever
going
to
ask
how
to
be
approaching
data
is
not
easy
right.
It's
very,
very
much,
not
easy.
So
by
having
that
mission,
we
always
know
what
our
our
north
star
is.
Can
we
make
the
database
simpler
for
people,
and
I
talked
about
how
to
make
it
so
that
people
could
truly
do
global
transactions
easily?
That's
that's
kind
of
one
of
our
north
stars,
but
you
know
in
the
early
days
of
cockroach
it
was
like.
B
How
can
we
make
it
so
that
when
you're
using
distributed
transactions
you're
not
trying
to
understand
the
distributed
transaction
model,
so
you
don't
screw
things
up
and
still
make
it
efficient?
We
said
what
we
said:
there's
gonna
be
one
mode
of
isolation
in
cockroach
serializable,
and
we
started
with
two.
Actually
we
had
snapshot
isolation
as
well.
We
ended
up
with
just
one
serializable.
B
In
fact,
people
never
get
it
right,
and
there
was
a
paper
about
this
that
came
out
of
stanford.
It's
called
acid
rain
and
it
analyzes
available
open
source
e-commerce
applications
which
actually
happen
to
run
more
than
50
of
the
e-commerce
on
the
web
and
they
analyze
it
with
this
very
intelligent
tracing
mechanism
to
see
whether
there
are
what
are
called
anomalies
due
to
weaker
isolation
levels
and
the
transactions
that
the
e-commerce
systems
were
doing.
I
mean
some
of
these
e-commerce
systems,
weren't
even
doing
transactions
some
of
the
time.
B
So
it's
like
there's
some
pretty
shoddy
programming
out
there,
but
even
when
they
were
trying
to
do
the
right
things,
they
often
didn't-
and
this
thing
would
highlight
those
in
ways
that
you
could
take
these
e-commerce
systems
and,
for
example,
you
could
put
multiple
things
in
your
checkout
card.
Just
by
like
sending
a
concurrent
request,
because
those
concurrent
requests
would
come
in
and
the
database
wouldn't
have
the
right
isolation
level.
B
It
would
let
you
do
things
that
should
be
illegal
if
you
had
the
right
isolation
level
and
that
allowed
people
to
check
out
multiple
items
and
pay
once
or
to
use.
Coupon
code
codes
multiple
times,
and
so
the
reality
is
that
this
is
just
like
a
small
sampling
of
e-commerce
things
that
happen
to
be
very
heavily
used,
but
I
mentioned
before,
like
there's,
there's
probably
hundreds
of
thousands.
Maybe
millions
of
database
backed
applications
out
there,
where
people
just
use
the
default
isolation
level,
which
is
usually
read,
committed
right.
C
B
Are
basically
these
these
applications
out
there
are
just
filled
with
whole.
I
mean
they're
leaky
buckets,
so
any
you
know,
and
you
know
just
to
give
you
a
practical
example
of
the
risks
here
there
was
a
bitcoin
exchange.
I
I
don't
want
to
say
the
wrong
ones.
I
can't
exactly
call
what
it
was,
but
it's
in
the
paper
that
was
completely
emptied
out
and
it
was
a
it
was
a
currency
based
acid
rain
type
attack,
and
so
this
is.
This
is
actually
a
huge
risk
and
there's
really
money
in
it.
B
B
A
that's
kind
of
a
it.
I
think
it's
a
state
of
mind
that
we
enter
like
the
product
sort
of
ideation,
and
you
know
what
are
we
gonna
set
our
standards?
How
are
we
gonna
set
them?
And
it's
just
very
helpfully
aided
by
what
our
true
north
is,
which
is
really
to
make
data
easy,
that's
right
and
don't
make
compromises
because
of
we
want
to
actually
have
the
serializable
isolation
right.
B
A
B
They
want
they
want
it
to
work,
they
expect
it's
going
to
work.
I
mean
databases,
work
really
well
right.
The
problem
is
that
people
have
set
these
isolation
levels.
They
don't
want
people
to
do
a
load
test
and
you
know
if
they
defaulted
to
serializable,
which
would
be
the
safe
thing
to
do
and
let
people
opt
out
of
that.
Then
you
know
what
would
happen.
Is
people
like
this
database
sucks?
It's
really
slow.
B
And
so
it's
really
just,
I
know
it's
a
little
cynical,
but
in
the
end,
what
you
end
up
with
is
a
lot
of
people
that
don't
understand.
Isolation,
levels,
use
the
default
isolation
level
and
are
leaving
a
big
gap
in
the
security
of
their
system,
and
that's
that's.
I
think
a
fundamental
problem
is
something
you
know
we
wanted
to.
Let
users
you
know
off
the
hook
for
that
extra
knowledge.
You
know
we
do
want.
We
want
developers
to
do
what
they
do,
which
is
like
hey.
I've
got
to
solve
this
problem.
B
I
want
to
build
this
application.
I
expect
the
database
to
work
and
it
should
work.
Well,
that's
right!
That
is
exactly
what
they
should
think.
They
shouldn't
have
to
understand
isolation,
levels,
but
other
databases
either
force
you
to
understand
them.
If
you're
going
to
be
a
very
responsible
programmer
or
they
let
you
do
the
wrong
thing.
That's.
C
C
B
Mentioning
jim
that
there's
a
we,
we
just
had
a
paper
and
sigmod
which
actually
that's
right.
Is
it
it's
a
wonderful,
wonderfully
concise
description
in
10
pages
of
livestream
talking
about
some
people
that
are
interested
to
read
that
and
if
you
really
want
the
gory
details,
the
blog
posts
from
the
engineers
that
worked
on
these
things
are
are
really
explanatory.
B
C
B
And
and
if
not
anything
just
to
explore
kind
of
the
inner
workings
of
cockroach
database,
which
I
think
is
really
interesting,
but
as
people
go
and
build
distributed
systems,
there
is
some
really
good
examples
of
how
you
actually
think
about
these
things
about
a
software
engineering
point
of
view
as
well.
I
think
that's
one
of
the
reasons
that
you
know
being
part
of
an
open
source
company
contributing
back
to
a
community
is.
We
do
share
all
these
ideas
and
I
think
it's
some
really
kind
of
novel
ways
of
thinking
through
these
things
is.
A
A
B
Like
it's
only
right
that
we
also
publish
these
sort
of
things
back
so,
but
the
coming
back
to
that,
and
so
you
know
people
can
use
nosql
to
get
scale
across
the
planet
right,
but
they
can't
get
serializable
isolation
right
and
that's
one
of
those.
That's
when,
like
at
adwords
right
way
back
in
the
day
spencer,
that's
why
they
couldn't
use
bigtable
right,
and
so
you
know
how
are
you,
how
are
organizations
using
cockroachdb
today?
B
B
Well,
I
think,
there's
you
know
the
the
the
easiest
answer
is
the
cockroach
is
a
system
of
record.
I
said
it's
a
cloud
native
relational
database
and
it's
for
any
application
that
you
know
needs
a
system
of
record
which
is
most
of
them,
and
it's
particularly,
I
think,
a
a
an
appropriate
choice
for
workloads
that
are
very
high
value,
and
I
mean
there's
certain
workloads
where
you
can.
You
can
absolutely
use
something
like
cassandra:
it's
not
necessarily
a
bad
choice
or
mongodb
I
mean
these.
B
Are
these
are
good
systems
in
their
own
right,
but
you
know
relational
database,
high
value,
use
cases,
the
right
kind
of
consistency,
that's
what
cockroach
is
really
geared
towards,
but
there's
three
primary
capabilities
or
differentiators.
That
cockroach
is
bringing-
and
you
know
some
of
our
customers
need
all
three.
B
Just
kind
of
go
through
them
scale
is
pretty
simple:
to
understand
a
relational
database,
that's
monolithic,
you
can
scale
it
up,
but
it's
a
super
linear
cost
curve
with
a
ceiling.
That's
actually
not
very
high
when
you
look
at
sort
of
global
workloads
right,
so
you
know
as
an
example,
you
know
if
you're
a
food
delivery
service
in
covid
and
we
we
have
a
customer
that
moved.
You
know
they
were
using
amazon,
aurora
and.
C
B
B
Xxl
size-
and
you
know,
if
you
kind
of
start,
hitting
the
the
limits
there,
your
database
starts
to
cavitate
and
then
falls
over,
and
so
they
they
became.
It
was
you
know,
code,
red
and,
and
they
had
to
get
off
amazon
aurora
and
they
tried
cockroach
and
cockroach
can
scale
the
rights
out
arbitrarily
right
so
doing
that.
Getting
that
elastic
scale,
which
is
a
no
sequel
thing
that
nosql
brought
to
the
the
world
to
the
market
but
doing
it
in
the
context
of
a
relational
database.
B
That's
special
we're,
not
the
only
ones
that
do
it
right
now.
Obviously
spanner
can
do
it.
There's
there's
some
other
sort
of
smaller
competitors.
You
know,
but
it's
it's
it's
significant
and
it
really
matters
like
especially
like.
I
think,
we're
entering
the
era
of
what
I
call
transactional.
Big
data
right,
if
you
think
about
you,
know
the
the
history
of
what
databases
have
supported
it.
Sort
of
you
know
back
in
the
90s.
It
was
enterprise
scale
so
like
within
a
company.
B
What
are
the
kinds
of
customers
that
an
average
company
used
to
have,
but
then
the
web
kind
of
hit
this
web
scale
idea
and
that
was
kind
of
okay?
You
know
in
in
the
web
scale
you
can
get
100
million
customers
that
are
banging
on
this
every
day
you
can
get
a
billion
right
yeah.
You
can
even
go
beyond
that,
but
there's
only
so
many
human
minions
that
are
interacting
with
computers
and
interacting
with
the
databases
that
are
backing
services
and
applications.
B
I
think
the
next
era
that
we're
already
entering
is
really
when
you
have
non-human
entities
with
some
agency
that
are
connecting
to
services
as
an
example,
iot
right
or
virtual
agents
that
aren't
even
like
physical
pieces
of
hardware,
but
it's
just
virtual
things
that
are
checking.
You
know
the
prices
on
something
but
they're
doing
it
at
machine
speeds
and
now
you're
talking
about
not
like
sort
of
order:
10
billion
human
beings
banging
on
keyboards
and
mouses,
and
things
like
that
interacting
you're.
Talking
about
you,
know,
100
billion.
B
Maybe
a
trillion
like
you're
talking
about
orders
of
magnitude
that
are
going
to
increase
and
so
scale
is
going
to
become
a
a
much
bigger
problem
even
than
it
is
today,
and
it's
already
becoming
one
for
many
companies
resilience.
You
know
kind
of
mention
this
about
geo-replication.
Fundamentally,
you
want
to
be
able
to
think
of
a
data
center
going
away
not
as
a
disaster
recovery
scenario,
but,
as
I
t
resilience
like
a
data
center
going
away
should
be
okay.
B
We
had
three
seconds
of
additional
latency,
no
data
loss,
no
postmortems
for
our
application
team
everything's
coming
along
we'll
get
that
data
center
back
up
things
will
re-replicate
and
then
we'll
be
back
in
sort
of
the
nominal
green.
If
you
look
at
google
and
their
high
value
use
cases,
they'll
put
it
across
five
data
even
across
like
the
the
different
power
grids
in
the
united
states,
because
they
want
to
be
able
to
say.
Okay,
we've
got
five
things:
everything's
normal
everything's
up
and
running
lots
of
redundancy.
Here
things
are
working:
fine,
okay.
B
B
So
if
you
have
five
three
out
of
five,
if
you've
got
three
data
centers
for
replication,
that's
two
out
of
those
three.
As
long
as
you
have
the
majority,
then
you're
never
gonna
lose
data.
You're
gonna
have
forward
operation
in
your
system.
So
if
you've
got
five,
you
have
this
amazing
property
where
you
can
take
one
out
for
a
plan
maintenance
and
then
you
know
all
hellblade
breaks
loose
somewhere
else
and
you
lose
another
one.
Now
you've
only
got
three
of
your
replication
sites.
B
B
This
is
the
most
exciting
one.
This
is
like
2020..
This
is
going
to
be
on
everyone's
roadmap,
but
the
question
is:
how
do
you
build
a
global
data
architecture
and
what
I
mean
by
that
is:
you
have
even
as
a
startup
the
opportunity
to
have
customers
in
australia
customers
in
brazil.
How
do
you
give
those
customers
a
first
class
users
I'll
tell
you
what
you
don't?
Do
you
don't
put
your
entire
application
in
one
availability
zone
right.
B
Have
australian
users
hitting
virginia
every
time
they
want
to?
You
know,
do
something
on
their
app.
It's
going
to
be
like
a
half
second
lag
at
the
minimum
right
when
they
hit
a
button,
a
half
a
second.
You
can
notice
that
very
easily.
In
fact,
department
of
defense
figured
out
is
100
milliseconds
or
less
for
command
and
control
systems.
That's
the
the
threshold
for
instant
instantaneous.
B
But
I
think
it
also
matters
for
any
kind
of
application
going
forward
when
you
start
to
have
an
experience
with
your
mobile
app,
where
you
hit
a
button
and
instantaneously
you're,
seeing
interactions
with
other
users,
but
you
can
design
new
things
that
we
haven't
seen
yet
I
just
kind
of
push
the
state
of
the
art
forward
and
companies
that
are
kind
of
doing
things.
The
old-fashioned
way:
they're
users
in
brazil,
they're
users
in
australia,
they're
users
in
japan,
their
users
are
going
to
feel
like
this
is
a
kind
of
antique
app.
B
It's
kind
of
like
when
the
iphone
got
bigger
and
you
had
those
like
black
bars
on
the
side,
because
the
app
maker
hadn't
updated
their
application.
You're
like
well.
This
app
maker
is
kind
of
cheesy,
like
everyone
else
has
kind
of
gotten
with
the
program
these
guys.
Obviously
they
must
not
have
enough
engineers
because,
like
they're
not
using
my
full
screen
and
it's
a
it's
not
as
good
experience
as
it
could
be,
that
same
thing's
going
to
happen
and
part
of
that's
driven
by
5g.
B
You
mentioned
that
earlier,
but
the
other
big
part
of
it's
not
just
latency.
It's
about
data
sovereignty.
You've
got
things
like
gdpr.
You
even
have
states,
like
california
and
new
york,
threatening
to
create
data
privacy
regulations
that
are
different
from
other
states.
We
could
have
a
balkanization
of
data
privacy
laws
in
the
united
states,
but
you
know
vietnam
requires
you
to
keep
a
copy
of
the
data
there.
South
korea's
got
its
laws.
Brazil's
got
very
stringent
laws,
china
and
russia.
They
require
that
all
your
data
is
domiciled
there.
B
Obviously,
if
you're
facebook
you
can
fight
those,
but
smaller
companies
can
just
get
jumped
out
of
business
pretty
quickly.
So
there's
a
there's,
a
lot
of
liability
right
now
in
the
world
in
terms
of
hey,
I've
got
a
business
and
I'd
like
to
embrace
a
global
user
base.
But
can
I
do
that
responsibly?
Can
I
do
that
in
a
way
that
you
know
meets
any
kind
of
legal
requirements
that
I
have,
but
also?
What's
the
user
preference,
let's
say
you're
a
sas
business
right
and
you
would
like
to
have
customers
in
asia?
B
You
can
bet
that
their
preference
is
going
to
be
that,
like
if
you're,
storing
their
employment
records
or
their
sales
data
or
something
you
can
bet
that
they
want
that
store
in
their
legal
jurisdiction
and
if
you're
going
to
compete
with
a
local,
a
more
local
operator.
That's
a
asia
based
that
is
going
to
to
give
them
that
guarantee
you're,
not
going
to
have
a
very
good
story
to
tell
right
so
like
how
do
you
build
those
holistically
global
data?
Architectures
and
facebook
can
build
them.
Uber
builds
them,
netflix
builds
them.
C
B
C
B
Because
that's
not
their
expertise,
they
don't
hire
those
kinds
of
people
that
are
kind
of
on
that
cutting
edge.
So
what
we
fundamentally
think
of
ourselves
as
doing
is
okay,
it's
very
complex,
building
a
truly
global
application
or
service,
but
the
database
is
the
hardest
part
of
that
no
question:
we
want
to
make
that
easy
right.
We
want
to
make
that
at
least
if
it's
not
doesn't
start
easy,
it's
tractable,
and
then
we
make
it
easy
as
we
go.
So
those
are.
C
B
Scale,
resilience
and
global
the
big
yeah
and
and
that's
exactly
right,
spencer,
and
I
think
what
we're
looking
at
is
the
beginning
of
a
massive
huge
transformation
in
a
way
that
all
of
us
think
about
applications.
Right,
I
mean
it's
like
you
know
you
and
I
joke
about
5g.
It's
like
you
remember
when
we
went
from
dial
up
to
dsl
and
how
fast
the
internet
was
at
home.
B
It's
that
same
type
of
aha,
I
think,
is
going
to
happen
with
people,
but
it's
going
to
be
on
their
phone
and
it's
going
to
be
everywhere,
and
how
do
you
actually
meet
the
needs
of
your
consumers
from
sla
right
there?
What
do
they
expect?
Out
of
you?
I
think
the
100
millisecond
rule
is
is
hugely
important
for
organizations
as
they
think
about
applications
at
that
scale.
But
this
resilient
thing
too,
and
I
think
you
know-
I
think
this
is
why
a
lot
of
people
are
moving
towards
kubernetes
and
doing
this
whole
thing.
B
It's
like
this
whole
concept
of
having
this.
You
know
this
this
orchestration
platform,
that's
doing
that
for
you,
but
then
surrounding
it
with
the
with
the
enterprise
capabilities
that
they
need
to
basically
manage
and
monitor
these
things,
and
I
think
that's
where
you
know
our
our
hosts
red
hat
openshift
are
doing
such
a
great
job.
You
know
pushing
down
that
path.
How
do
we
make
it
easy
to
install
things?
How
do
we
make
it
easy
to
manage
things?
How
do
you
you
know?
B
That
is,
that
is
that
allows
everyone
to
do
more
with
less,
but
it
you
know,
ultimately
what
you,
what
you
find
is
is
pretty
interesting,
like
the
cloud's,
a
lot
cheaper,
right
and,
and
you
know,
taking
infrastructure
as
a
service.
The
total
cost
of
ownership
is
actually
less
expensive,
but
but,
ironically,
people
end
up
spending
more,
but
it's
not
necessarily
a
bad
thing.
A
B
The
whole
thing
worked
because
it
was,
I
was
used
to
the
old
world.
You
know
it
was
a
different
way
and
I
think
there
is
priority.
There's
you
know,
there's
organizations
I
google
and
others
that
have
kind
of
done
this
sort
of
thing,
but
there
were
a
couple
questions
that
were
in
the
chat
spencer
that
I
want
to
actually
come
back
to.
Some
of
them
have
to
do
with
this.
Like
you
know,
how
do
you
shift
you
know,
there's
things,
there's
concepts
that
we
had
in
the
enterprise
world.
B
A
B
C
B
Potentially,
you
know
250
nodes
and
they
might
be
in
different
countries.
Since
they've
got
this,
you
know
geopartition,
some
data
might
be
replicated
only
within
europe.
Some
data
only
replicated
you
can
control
all
that
at
the
row
level
and
a
table
in
cockpit,
so
yeah
that
that
vastly
complexifies
the
the
burden
on
the
backup
and
restore-
and
you
know
what
we
actually
end
up
doing
we
found
to
work
best
is
to
you
know,
make
the
backup
restore
policies
definable
on
the
same
level
as
we
actually
do
the
sort
of
domiciling
policy.
B
B
Only
in
europe
too,
you
don't
want
that
to
stream
over
to
the
us
for
lots
of
reasons,
and
so
you
know,
but
one
one
one
I
think
really
important
point
to
bring
up
here
is
that
we
don't
have
to
solve
all
these
problems
on
our
own.
You,
you
realize
that
in
this
new
ecosystem,
all
of
your
customers,
not
all
of
them,
but
most
of
them
have
access
to
all
the
other
cloud
infrastructure.
That's
out
there,
so
really
what
you
really
want
to
be
able
to
allow
them
to
do
as
an
example.
B
This
is
just
one
of
many
things
you
could
do,
but
you
can
have
for
the
european
data
that
could
be
backed
up
to
an
s3
bucket
that
is
sitting
in
europe
specifically
right
right,
and
so
you,
you
kind
of
you're.
Actually
using
am
what
aws
is
providing
geographically
spread
out
around
the
world
in
order
to
augment
cockroach's
capabilities
and
to
complement
them,
and
that's,
I
think,
that's
critical,
because
if
you
try
to
solve
this
all
you
know
in
the
database
the
the
the
amount
of
work
and
the
complexity.
B
It's
it's
counterproductive
right
and
it's
ultimately
not
what
customers
want
right.
You
kind
of
want
to
use
the
best
of
breed
approach,
so
we
do
try
to
integrate
with
everything
in
the
in
the
ecosystem
yeah
and
to
be
multi-cloud
and
run
everywhere
and
take
advantage
of
whatever
it
is
on
each
of
the
public
cloud
providers
that
they
use.
It
is
a
big
piece,
so
I'm
dan
mike
was
there
any
other
question
that
you
guys
wanted
to
throw
in
here.
C
Right,
like
I've,
been
capturing
them
on
youtube
and
pulling
down
from
twitch,
and
it
looks
like
you've
been
addressing
them
without
specifically
saying:
okay,
here's.
You
know
paris
lucas's
question.
So
if
there
are
any
other
questions
either
here
on
the
blue
jeans
session
or
from
youtube
or
twitch
or
any
others,
please
do
post
them.
We
have
our
video
producer
team,
grab
them
and
pull
them
into
the
chat
here.
So
we'll
get
them
addressed
for
you.
B
B
So
the
question
is
software:
like
lifecycle
support,
how
does
the
software?
How?
How
do
you
do
a
software
upgrade
without
downtime?
So
you
know
that
that's
a
good
question.
It's
actually
not
just
the
the
software
upgrade.
It's
also
things
like
backup
and
restore
you
don't
want
to
lock
table
schema
changes,
so
we
do
online
schema
changes.
It's
very
you
know,
cockroaches.
B
You
know
very
predicated
on
any
kind
of
thing
that
you
do
to
to
manage
the
database,
whether
it's
an
upgrade
or
a
backup,
or
you
know,
change
the
underlying
data
model.
All
of
those
are
do
not
require
downtime
and
including
scaling
the
database
up
scaling
the
database
down
moving
from
one
cloud
to
another.
All
these
things
can
be
done,
live
the
software
upgrade.
I
I
won't
go
into
too
much
detail,
but
basically
what
you're
allowed
to
do
is
have
multiple
nodes
and
successive
versions.
B
You
can
have
a
mix
of
versions
and,
like
certain
kinds
of
features,
don't
mind
that
there's
a
different
version
because
they're
not
really
predicated
on
the
version,
but
sometimes
you
have
a
new
feature
where
it's
not
going
to
ever
work.
If
there's
a
mixed
version
situation,
so
what
will
happen
is
when
you
you
have
that
mixed
thing.
You
can't
use
that
new
feature.
Yet
it's
available
on
the
nodes
that
that
are
the
newer
version,
but
even
on
those
nodes,
you're
not
actually
able
to
access
that
feature.
And
then
what
happens?
B
Is
you
kind
of
you?
You
get
to
all
of
the
nodes
being
the
newest
version
and
then
at
that
point
you'd
make
sure
that
the
system
works.
You
can
run
it
that
way
for
a
while.
If
it
doesn't
work
you
can.
You
can
downgrade
back
to
the
previous
version
and
again
it's
really
just
about
restarting
each
node
sort
of
in
a
rolling
restart
fashion.
But
if
things
are
working
nicely
and
you
don't
need
to
downgrade,
then
you
kind
of
flip
a
switch.
B
B
There
is
things
you
have
to
think
about
inside
the
application
itself
to
actually
make
that
work
and
work
very
well.
The
orchestration
platform
will
surely
do
this.
This
beautiful
rolling
upgrade
for
you,
which
is
like
what
a
huge
massive
awesome
kind
of
capability.
So
there's
another
question
in
here:
spencer:
it's
a
little
bit
in
the
weeds,
I
don't
know.
Is
there
one
you're
looking
at
in
particular.
C
B
B
Unfortunately,
so
I
can't
answer
that
part
of
the
question,
but
you
know
the
storage
we
prefer
to
run
on
top
of
the
are
the
ones
that
we
have
the
most
experience
with
and
right
now
that's
gcp's
persistent
ssd
and
aws
evs,
and
I
think
we've
tried
a
lot
of
the
different
ebs
ones
and
we,
you
know,
we've
had
different
experiences
with
each.
It's
usually
like
a
cost
versus
performance
trade-off,
those
those
persistent
ssds
and
the
ebs.
B
The
really
beautiful
thing
about
those
storage
solutions
versus
using
raw
ssd
is
that
if
you,
if
you
actually
experience
some
sort
of
downtime
from
one
of
your
nodes,
let's
say
you
just
lose
a
node.
You
know
that
that
would
be
attached
normally
to
raw
ssd.
That
usually
would
require
cross
data
center
bandwidth
in
order
to
re-replicate
that
missing
as
raw
ssd
device.
When
you
use
something
like
persistent,
ssd
and
gcp,
you
actually
rely
on
gcp
to
do
that
within
that
data
center.
So
it
saves
you
any
kind
of
cross
data
center
bandwidth.
B
So
it's
very
nice,
that's
how
we
run
cockroach
cloud,
which
is
our
hosted
service
when
com.
When
we
have
customers
that
sell
posts,
if
they're
running
it
in
aws
or
gcp,
they
can
do
the
same
thing
we're
doing
with
cockroach
cloud
if
they're
running
in
a
private
data
center.
I
actually
think
the
right
thing
to
use
is
probably
some
sort
of
kubernetes
storage
solution.
That's
doing
something
similar
right,
that's
actually
doing
a
distributed
file
system
that
can
handle
intra-data
center
re-replication
if
a
node
is
lost.
B
B
A
A
B
Yeah
I
mean
I
listen,
the
the
thing
that
is
my
guiding
north
star
for
where
the
ecosystem
is
going,
and
I
want
car
coach
to
play
very
nicely
in
that
ecosystem.
Is
you
know
as
a
developer
myself
and
I
I
would
like
to
write
an
application
next,
you
know
kind
of
my
earlier
point
about
iterating
between
infrastructure
and
applications.
B
I
want
to
be
able
to
write
something
on
my
laptop
and
then,
when
I'm
done
with
it
like
it's
working
when
I
test
it,
I
want
to
say:
okay,
I'm
going
to
push
this
to
the
cloud,
but
I'm
not
going
to
have
to
deal
with
vms
or
or
any
any
kind
of
infrastructure
myself.
Whatever
I've
works
on.
My
laptop
is
not
just
going
to
work
on
the
cloud
it's
going
to
scale
elastically.
B
If
I've
got
100
million
users,
it's
going
to
scale
to
100
million
users
if
I've
got
users
in
brazil,
all
of
a
sudden,
it's
going
to
start
storing
their
data
in
brazil
and
it's
going
to
do
that
in
serverless
fashion.
In
terms
of
my
my
my
relation
to
the
whole
project
as
a
developer
and
what
that
means
is
fundamentally,
yes,
I
don't
deal
with
vms
and
you
know
the
hands-on
infrastructure
and
monitoring
and
all
that
stuff.
B
None
of
that,
but
also
I
only
pay
for
what
I
use,
because
you
know
I've
done
a
number
of
startups
back
when
I
had
to
do
exodus,
co-location
and
build
my
own
servers
and
put
them
there
all
the
way
till
you
know
more
recently,
where
I
get
vms
in
ec2
or
in
gcp,
and
you
know
the
I'm
not
very
good
at
that
stuff.
But
worse
what
you
end
up
doing
when
you
launch
your
startups.
B
You've
got
like
10
people
poking
at
it
right
because
you
don't
have
product
market
fit
yet
and
you're
paying
for
these
vms
to
just
sit
there
get
a
bm
for
your
database,
which
is
a
big
one
right
like
because
you're
you're
planning
on
success.
If
you
start
with
all
these
things,
you
just
you're
just
paying
out
hundreds
of
dollars
a
month
and
just
immediately
for
you
know
what,
honestly
you
don't
you
don't
have.
You
should
be
paying
cents
per
month
right,
and
so
that
is
that's
the
goal
like
you.
B
You
should
have
consumption
based
billing
on
everything
and
it
should
scale
globally
to
any
size,
and
you
should
never
lose
your
data.
It
should
just
always
be
up.
You
don't
have
to
deal
with
any
kind
of
monitoring.
So
that's
the
future.
It's
like
a
truly
serverless
future.
Where
you
can
take
your
idea,
you
can
develop
it
and
then
it
just
becomes
a
global
reality
like
the
way
that
google
builds
things
right.
A
All
right,
so
you
heard
it
here.
First,
I
think
that's
like
if
you
read
lots
of
people's
marketing
messagings,
that's
already
here.
The
reality
is
it's
not
and
we
still
have
to
build
it
together.
So
it's
been
wonderful
getting
to
getting
to
hear
your
story,
hear
your
point
of
view
and
how
the
evolution
of
distributed
databases
came
about
and
where
you're
at
right.
Now
we
really
look
forward
to
having
the
cockroach
operator
working
on
openshift,
getting
more
use
cases
more.
A
You
know
feedback
to
you
guys
and
want
to
hear
more
someday
soon
about
cockroach
cloud
and-
and
I
love
cockroaches
in
a
way
that
you
wouldn't
expect
is
they
never
die
and
they
will
be
here
until
the
end
of
time.
So
I
think
your
naming
in
branding
is
brilliant,
so
you
know
keep
keep
going
whatever
the
next
product
is.
I
want
to
know
what
the
name
is
you
choose
because
you're
you're
dead
on
with
it,
with
your
your
naming
and
branding
as
well
as
your
technology.
A
So
thank
you
very
much
for
joining
us.
Michael
waite
has
got
a
number
of
other
operators
coming
up
folks
coming
in
the
next
couple
of
weeks
in
stana
and
a
few
others,
so
look
forward
to
more
conversations
around
this
and
learning
more
about
what
people
are
doing
in
building
that
runs
on
openshift.
So,
thanks
for
your
time,
spencer
and
your
wonderful
view
and
jim
for
your
your
great
q,
a
and
question
and
keeping
the
conversation
rolling,
so
thanks
again
guys.