►
From YouTube: CI/CD, Kubernetes, and Databases: Better Together - Niraj Tolia & Tom Manville, Kasten
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
CI/CD, Kubernetes, and Databases: Better Together - Niraj Tolia & Tom Manville, Kasten
Adding data to both your CI and CD pipeline is one of the last steps of the DevOps journey and probably the scariest given the impact of getting it wrong. This talk covers how one can safely combine Kubernetes, Databases, and the CI/CD pipeline to actually make the process safer and more stable that the status quo today and, in today’s rapid deployment culture, make databases “shift left” and reduces DBA burnout. This includes leveraging techniques and building an open-source toolkit to deliver automated schema changes, cloning, sandboxing, masking for production-like data in staging, and rapid data movement for fast database creation. More importantly, this talk will show how these benefits can help with internal culture shift by breaking down silos and bringing in a traditionally conservative database group more fully into the automation fold.
To Learn More: https://sched.co/GrSq
A
All
right
everyone
thank
you
for
coming
today.
I
know
it's
the
afternoon.
It's
been
a
long
day
for
a
lot
of
folks
in
this
room,
but
we
are
very
excited
to
talk
about
C,
ICD,
kubernetes
and
databases,
and
why
we
strongly
believe
they're,
definitely
better
together
in
terms
of
just
a
quick
round
of
introductions
for
us
I'm
Natalia
I'm,
the
co-founder
at
Caston.
We
are
startup
based
in
the
Bay
Area.
A
We
really
look
at
data
management
for
all
your
cloud
native
applications,
everything
from
backup
and
recovery
to
some
of
the
things
we're
going
to
talk
about
in
this
talk,
but
personally
I've
been
working
on
data
and
storage
systems
for
a
long
time
in
reverse
chronological
order,
as
at
EMC
responsible
for
the
cloud
was
family
of
products
ma
generics
where
we
did
distribute
file
systems,
next
generation,
storage,
class
memories
at
HP,
and
before
that
at
carnegie-mellon.
Where
did
my
PhD
on
destroyed
storage
systems?
All
that
Tom
introduced
himself
next.
B
A
Tom,
so
the
high
level
goal
behind
a
lot
of
what
we
do,
and
what
we're
going
to
talk
about
today
is
how
you
can
move
fast,
but
without
breaking
things
and
in
particular
tests
with
real
data
at
different
stages
of
your
pipeline,
and
some
of
the
benefits
that
come
about
from
that.
If
there's
one
thing
I
want
you
to
take
away
from
the
stock,
is
that
this
is
something
they'll,
be
very
valuable
for
you
and
will
give
you
a
number
of
benefits.
But,
more
importantly,
it's
not
that
hard
to
do
as
you
might
think.
A
There
are
a
few
things
that
we
will
not
cover
in
this
talk
today.
We
some
people
sometimes
have
doubts
about
whether
kubernetes
is
ready
for
production,
stateful
apps.
We
definitely
believe
it
is.
We
have
to
talk
earlier
this
year
at
sneer.
Storage,
developer
conference,
encourage
you
to
check
that
out,
and,
secondly,
we're
mostly
going
to
be
talking
about
data
usage
in
pre-production
stages
of
your
pipeline.
If
you're
looking
for
more
information
on
how
to
manage
and
protect
it,
either
in
your
production
settings
go
for
a
talk
tomorrow
at
2:35
p.m.
right.
A
That
should
be
equally
interesting,
so
to
jump
into
the
con
of
the
talk
really
when
we
go
look
at
databases
in
terms
of
how
people
are
using
them
in
the
cloud
Nate
a
well
sometimes
people
believe
that
is
fine,
but
from
our
perspective,
maybe
we
believe
there's
a
large
gap
between
where
we
are
today
and
where
I
think
we
should
be
for
things
that
we
care
about,
which
is,
at
the
end,
the
application
and
delivering
value
from
those
apps.
But
Before
we
jump
into
how
to
get
to
much
better
state
in
the
world.
A
It's
good
to
ask
ourselves:
why
is
there
a
fear
and
risk
in
some
of
these
environments
why
people
are
concerned
about
bringing
databases
into
your
pipeline
testing
with
data
things
of
that
sort?
And
some
of
it
is
because
of
the
reasons
you
see
on
screen.
Databases
have
traditionally
been
snowflakes
more
pets
versus
cattle,
if
you're
familiar
with
that
analogy
and
they
would
set
off
on
the
side
somewhere
outside
of
your
application
clusters,
potentially
even
manually
changes
manual
changes
applied
to
them,
which
led
to
a
lot
of
risk
in
terms
of
making
changes
to
that.
A
It's
also
been
significant
automation
gaps
as
what
we've
discovered
with
the
customers.
They're
not
built
in
early
into
your
CI
CD
pipeline
testing
might
happen
with
synthetic
data
or
data
that
could
be
stale
multiple
months
out
of
date.
And,
finally,
it's
not
about
always
a
technical
issue
with
the
DevOps
shift.
We
are
seeing
Developer
and
operations
definitely
coming
closer
together,
but
we
still
sometimes
see
data
based
engineering
somewhat
setting
off
on
the
side,
and
this
is
another
silo
we
need
to
break
down
and
to
bring
it
in
into
earlier
stages
of
an
EC
ICD
pipeline.
A
You
bring
up
to
reduce
all
this
risk
and
fear
that
we're
talking
about
in
terms
of
what
we
believe
that
the
future
should
look
like
wait
at
the
very
high
level.
It
is
simple
you
keep
calm
and
automated
as
much
as
you
feasibly
can.
But
more
concretely,
we
believe
that
to
increase
agility
with
databases
in
your
cloud
native
environment
revolves
around
three
things:
the
first
one
is
source
control,
and
now
this
is
common
for
a
lot
of
our
application
code.
But
there
are
a
couple
of
important
things
to
note
as
to
what
we
propose.
A
You
all
do,
first
of
all
include
all
schema
changes,
any
upgrade
tools,
you
use
modifications,
etc
into
your
source
code
into
a
source
control
system,
but
not
any
source
control
system.
You
want
us
to
be
tightly
integrated
with
the
application
code
that
the
database
is
serving
now,
it's
not
just
about
schemas.
It's
also
about
infrastructure,
where
you
want
your
database
infrastructure
to
be
expressed
as
scored
again
configuration
things
of
that
nature
and
put
it
in
the
same
source,
control
repository.
A
And
finally,
you
want
to
build
us
not
just
into
your
CI
CD
pipeline,
but
as
early
as
possible.
So
you
automate
all
your
database
changes
testing
as
your
application
development
happens
and,
finally,
in
case
you
missed
the
subtle
icon
in
the
middle
of
this
atom.
We
really
believe
that
you
should
use
kubernetes
to
tie
it
all
together.
Now,
at
this
point,
you
might
ask
me
hey
Neeraj
wait.
All
of
this
is
great.
A
This
is
fine,
but
wouldn't
this
also
equally
apply
to
any
legacy
environment,
whether
I'd
be
running
on
VMs
or
bare
metal,
and
the
short
answer
to
that
is
yes.
This
applies
across
a
variety
of
environments,
but
given
the
advantages
that
kubernetes
provides
us,
I
think
will
be
foolish
for
all
of
us
do
not
explore
how
we
can
leverage
that,
in
particular,
because
of
the
reasons
you
see
on
screen,
kubernetes
by
itself
will
enforce
good
DevOps
hygiene.
The
mobility
aspect
of
things
ensures
you
will
move
away
from
snow
flexing
infrastructure.
A
The
factory
expressing
configuration
and
infrastructure
is
code
will
make
automation
and
testing
simple,
reliable
and
easy
for
you
in
large-scale
testing
environments.
It's
also
much
better
from
the
cost
and
efficiency
perspective.
If
you're
giving
developers
their
own
individual
copy
of
databases
in
these
environments,
it's
much
better
in
terms
of
bin
packing
that
you
can
do
compared
to
spinning
up
say,
for
example,
manage
services
individually
for
a
developer
there
and
finally,
as
a
number
of
customers
have
told
us,
it
is
also
about
making
sure
you
have
a
universal
control
plane
in
the
long
term.
A
A
So
a
faster
change
hydration
process
will
over
time,
lead
to
better
and
faster
deployment
to
production
versus.
Sometimes
you
know
bunching
changes
together
to
do
it
on
only
on
a
periodic
basis,
doing
database
infrastructure,
and
you
will
catch
issues
all
the
there
say,
for
example,
if
you're
testing,
with
synthetic
data
which
was
might
failover
in
production
or
in
any
situation
where
code
that
you
know
works
well
with
small
data
sets,
might
not
work
with
large
ones.
A
No
sequel
systems,
relational
databases-
all
of
those
under
that
umbrella
here,
but
we
haven't
spoken
about
data
so
far,
which
is
an
essential
component
just
of
the
term
itself,
and
how
do
we
handle
that
as
something
people
have
struggled
with
traditionally
and
our
belief
is
that
again
get
the
full
advantage
that
we
believe
you
should
have
you
want
to
test
with
production
data?
But
you
do
not
want
to
do
this
in
production.
I
think
you
know
people
that
say:
hey
go
test
in
production
or
crazy
people.
A
You
definitely
don't
want
to
do
this
with
precious
data
that
your
application
or
your
enterprise,
it
depends
on
which
leads
because
of
issues
such
as
sensitive
data.
You
don't
want
exposed
developers,
you
don't
want
to
accidentally
corrupt
data
running
in
your
production
environment,
but
then
how
do
you
do
this
safely
and
efficiently
becomes
a
question
for
all
of
us
and
in
particular
to
get
to
an
ideal
state
of
the
world
you're
going
to
need
to
integrate
against
different
layers
in
the
stack
for
efficiency
reasons.
A
First
of
all,
you
probably
will
need
some
storage
integration
when
you
talking
about
integrating
with
volume
level
snapshots
on
copy
and
rate
file
systems.
It's
great
when
you
have
large
datasets.
Better
efficiency
will
result
from
that
to
say:
hey,
take
a
snapshot,
move
that
and
do
my
desktop
environment,
but
you
can't
just
stop
there.
You
really
need
to
think
broader
than
that,
because
you
want
your
hooks
into
your
no
sequel
system
or
your
a
relational
database
for
consistency,
reasons
when
you're
capturing
data.
A
So
those
are
some
of
the
issues
that
come
up
at
the
database
level
and
then,
finally,
when
you
go
look
at
application
integration
right
there,
we
see
a
wide
spectrum
of
applications
being
deployed,
whether
you
talk
about
kubernetes,
open,
share,
foundry
things
of
that
nature,
but
polyglot
persistence
shows
up
over
and
over
again
in
some
of
the
micro
service
based
applications.
We
see
so,
if
you're
not
familiar
with
that
term.
A
B
You
so
near
is
just
described
a
set
of
challenges
associated
with
something
we
refer
to
as
data
mobility
when
using
data
mobility
to
improve
your
CSU
pipeline,
it's
important
to
consider
the
data
at
different
layers
in
your
application
stack.
In
fact.
In
many
instances,
it's
important
to
perform
operations
on
multiple
layers
of
your
application
at
once,
as
well
as
interact
with
communities
itself,
so
we've
released
a
project
called
canister
telepods.
B
Just
these
challenges
in
a
fully
open
source
framework
whose
goal
is
to
address
data
mobility
challenges
inside
kubernetes,
it's
easy
to
integrate
with
your
CI
CV
pipeline
because
it
uses
kubernetes
api
extensions,
called
custom
resources,
and
so
you
can
invoke
it.
For
example,
by
using
coop
cuddle,
you
can
easily
extend
it
to
work
with
your
custom
application,
but
we
also
have
integrations
for
several
of
the
common
databases
that
we've
seen
in
a
cloud
native
context.
B
Cancer
supports
a
lot
of
features,
so
it
supports
file,
block
and
object
storage.
You
can
push
and
pull
to
object,
storage.
If
your
application
is
built
using
PVCs
and
that's
how
it
consumes
its
file
and
block
storage,
then
it
can
integrate
with
cannister.
We
look
forward
to
more
providers
employing
CSI
or
the
container
storage
interface
right
now
we
have.
We
invoke
library,
calls
of
different
providers
directly
for
things
like
snapshotting
and
copying
volumes.
B
However,
as
more
providers
implement
the
CSI
we'll
move
over
to
those
invitations,
many
features
and
data
mobility
have
to
happen
at
the
database
level.
So
things
around
security
and
performance
are
a
part
of
our
enable
through
cannister.
The
specific
features
that
we're
talking
about
are
things
like
filtering
masking
and
incremental
backups
many
data
mobility
features
require
coordination
with
your
applications.
Lifecycle
who
canister
supports
this
by
providing
library
functions
for
things
like
coop
exec,
as
well
as
modifying
the
spec
of
your
resources,
so
that
you
can
take
can't
Carini's
actions.
B
Excuse
me
to
just
give
a
brief
walkthrough
of
how
canister
works.
Let's
say
that
you
have
a
careers
cluster
and
in
that
cluster
you
have
your
stateful
application.
You've
installed,
you've
installed,
canister
the
canister
controller,
and
you
have
a
blueprint.
The
blueprint
is
specific
to
your
application
and
we
have
example
ones,
but
you
can
also
build
your
own.
So
in
order
to
initiate
a
canister
action,
you
first
create
an
action
set,
for
example,
bike
by
using
coop
cuddle
the
actions
that
will
get
picked
up
by
the
controller.
B
B
Finally,
we
will
update
the
status
of
the
action
set,
so
you
can
see
what
happened
with
your
action
to
dig
a
little
deeper
to
see
what
an
action
set
in
a
blueprint
look
like
so
an
action
set
itself
is
a
is
a
korean
object
if
it's
a
community
the
custom
resource.
So
it's
an
extension,
meaning
it
has
all
the
same
metadata
as
the
careers
api
objects
you're
used
to.
So
it
has
an
api
version
and
kind,
for
example,
this
this
action
set.
B
B
So
if
we
look
at
the
corresponding
blueprint
itself
is
also
a
custom
resource,
meaning
it
has
the
same
metadata
as
an
action
set
as
well
as
the
other
kubernetes
api
objects,
in
this
case
we're
implementing
the
action,
backup,
backup
works
on
a
stateful
set
and
we'll
be
doing
the
coop
exec
to
run
inside
a
container.
The
arguments
to
blueprints
are
templated,
so
you
can
you
can
pass
in
you.
B
But
what's
really
interesting
for
the
context
of
this
talk
is
that
we'll
pull
data
from
production
for
use
inside
our
integration
tests
so
we'll
be
using
the
most
recent
data
that
we
have
and
hopefully
we'll
we'll
discover
any
issues
that
we
would
otherwise
hit
in
production,
to
figure
out
to
digging
even
deeper
in
to
see
what
this
data
movement
looks
like.
We
have
the
set
up,
so
there
are
two
clusters:
the
cluster
on
the
left
is
our
production
cluster.
It
includes
our
application
and
database.
The
cluster
on
the
right
is
just
our
integration
task
cluster.
B
In
both
clusters
we
installed
k10
and
cannister.
K10
is
a
commercial
product
whose
goal
is
to
help
manage
stateful
applications,
often
by
invoking
canister
for
its
data
mobility,
we've
created
a
policy
which
will
tell
tell
k10
to
snapshot
and
then
export
this
application,
a
snapshot
create
sort
of
something
we
call
a
restore
point,
which
is
a
collection
of
configuration
and
data,
and
then
we
export
that
restore
point
into
some
public
cloud
or
some
objects,
or
rather
the
other
store,
will
contain
both
data,
but
also
references
to
data.
B
In
the
case
of
things
like
cloud
provider
of
long
snapshots
in
the
integration
cluster
will
periodically
import
this.
These
restore
points
so
that
they
can
be
made
available
for
to
restore
when
it
test
runs
as
part
of
CI.
We
will
create
a
new
namespace
with
a
new
app
and
a
new
database
instance,
and
then
we'll
trigger
a
restore
action
to
populate
that
database.
B
A
B
Thank
you.
So
here
we
have
an
application
that
we've
developed
just
for
this
demo.
Our
applications
goal
is
to
just
simply
record
orders
and
store
the
zip
code
for
each
of
those
orders.
It's
relatively
simple.
We
can
look
at
what
is
contained
in
the
application,
so
there's
a
web
server
which
which
is
hosting
our
web
content
and
then
a
database
for
storing
our
data.
The
database
has
a
PVC.
We
can
actually
log
into
this
and
we'll
work
forward
there
and
we
can
see
what
the
content
of
our
production
cluster
is.
B
B
So
now
I'll
show
the
the
setup
we
have
for
k10
and
specifically
our
policies
that
we've
set
up
to
export
and
import
data.
We
create
a
policy
which
will
periodically
every
our
snapshot,
our
our
full
application
and
then
we'll
export
that
snapshot.
We've
configured
this
to
use
s3
and
to
use
this
bucket.
B
B
B
B
B
So
there
many
things
we
didn't
get
to
cover
in
this
talk
and
they're
listed
here.
Schema
changes
are
always
very
dangerous,
and
so
it's
important
to
build
proper
test
infrastructure
around
them
to
make
sure
that
they're
safe
in
this
talk.
We
talked
a
lot
about
using
databases
inside
kubernetes,
but
managed
services
can
also
benefit
from
the
same
testing
techniques
that
we
showed.
We
barely
scratch
the
surface
with
masking
and
sampling,
but
it's
also
a
great
way
to
make
sure
that
your
your
insensitive
data
is
safe
and
also
that
you
can
perform
efficient
data
mobility
actions.
B
B
And
so
to
conclude,
we
hope
that
you
will
build
test
on
infrastructure
around
your
databases.
We
hope
that
you'll
leverage
kubernetes
for
your
tests
of
infrastructure.
We
hope
that
you'll
use
real
data
to
help
expose
real
issues
that
will
only
arise
in
production
otherwise,
and
finally,
we
hope
that
you'll
combine
all
these
things
to
make
sure
that
your
database
engineering
is
agile.
B
B
That's
a
great
question,
so
the
question
was
what,
if
you're
using
something
like
RDS
for
your
production
data,
is
there
a
way
that
you
can
still
use
the
same
techniques?
And
the
answer
is
absolutely
yes
for
something
like
RDS,
there's
really
two
ways
that
you
can.
You
can
do
copies
of
your
data
for
testing.
So
one
way
is
by
using
the
cloud
ap
is
you
can
do
a
arias
copy
of
your
data
and
then
use
that
the
other
way
is
is
doing
a
my
sequel
level
copy.
C
B
Great
question
so
you're
asking
all
the
advanced
topics,
which
is
awesome
so
for
the
question
was
for
gdpr.
Can
you
scrub
PII
from
your
data?
Essentially
right
and
the
answer
is
yes,
you
have
to
do
it,
but
you
have
to
get
the
database
level
right.
You
have
to
respect
to
see.
You
know
fields
that
you
cannot
use
for
your
test,
you're
testing
the
pipeline,
and
so
we
essentially
provide
the
framework
so
you're
able
to
do
that-
and
you
know
we'd
love
to
hear
about
your
specific
use
case
kind
of
offline.
So.
A
Just
to
clarify
in
the
blueprint
Dom
it's
shown
earlier
as
a
part
of
the
data
capture
step
there,
you
could
have
there
multiple
phases
possible.
That
can
you
know
you
can
have
in
an
order.
So
once
you've
captured
data,
you
can
either
encrypt
an
entire
column
out
completely
mask
it,
fill
it
with
random
data.
Whatever
you
choose
to
do,
and
so
there
is
actually
deeper
functionality
in
canister
that
allows
you
to
spin
up
with
sandbox
covers
the
database
master
data
and
then
push
it
out
into
an
object
store
as
an
example.
B
Great
question,
so
the
question
was:
if
you
have
a
really
large
data
set,
and
you
want
to
use
that
in
your
test
infrastructure.
So
one
of
things
again,
events
topics
we
didn't
mention
was
sampling,
and
so
you
could
reduce
your
data
set
on
import
so
that
you
would
still
be
able
to
test
with
it
very
quickly.
B
A
I
think
you
know
Tom
covered
it
that
ultimately
radiative
will
have
gravity
and
the
fundamental
limits
to
what
someone
can
do
if
you
can't
down
sample
as
an
example.
The
best
thing
is
to
evaluate
the
store
system
of
choice,
so
you
can
have
copy
and
write
snapshots
underneath.
So
it's
easy
to
do
copy
data
management
for
your
test
of
clusters.
A
Potentially,
maybe
not
even
freshness,
but
if
there's
no
masking
step
involved,
you
might
not
freshness
will
not
get
impacted
because
you
can
do
this
on
a
very
regular
basis.
Hopefully
it'll
be
cheap.
Almost
you
know
zero
cost
papi
Android
snapshots,
but
Redwood.
We
can
dig
into
this
offline
as
to
what
your
environment
might
look
like
and
yeah.
A
Okay,
so
all
right,
so
it's
a
great
question.
So
a
couple
of
things
kitten
was
used
here
just
for
ease
of
illustration.
For
demo,
you
actually
don't
need
kitten
to
do
any
of
the
things
that
we
showed
today.
You
can
do
it
all
within
kubernetes
natively
itself.
Now,
when
you
compare
what
aww
state
a
pipeline,
it's
not
so
much!