►
From YouTube: Declarative & Imperative Kubernetes Operations with Temporal & M3 - Matt Schallert & Dominik Tornow
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Declarative and Imperative Kubernetes Operations with Temporal and M3 - Matt Schallert, Chronosphere & Dominik Tornow, Temporal
Operators allow users to orchestrate complex software, such as databases, running on Kubernetes. However, due to their declarative nature, expressing and coordinating imperative actions (such as “create a backup”) is awkward and cumbersome. In this talk, Matt and Dominik will explore how declarative solutions, such as Operators, along with imperative solutions, such as Temporal, an OSS Workflow Orchestration Platform, can be used to automate the operations of complex software systems on Kubernetes. They will demonstrate how Temporal workflows can leverage the same Go libraries users are familiar with. Finally, using the OSS M3DB Operator as an example, they will demonstrate how Operators and Workflows work in tandem and mitigate each others' limitations. You will walk away with an accurate, actionable, and dependable understanding of Kubernetes Operators and Temporal Workflows and how to craft scalable and reliable operation automation for your Kubernetes applications and clusters.
A
A
We
have
two
lovely
tour
guides,
rabbit
number,
one
that
is
matt
senior
engineer
at
chronosphere
and
then
rabbit
number
two,
that
is
this
guy
dominic
principal
engineer
at
tempura.
So
I
would
argue
the
answer
to
the
question:
what
is
kubernetes?
We
have
at
least
as
many
answers
as
there
are
people
in
the
room
and
probably
also
online,
but
let
me
give
it
a
shot,
so
I
would
argue:
kubernetes
may
be
characterized
as
a
container
orchestration
platform.
A
That
is
a
very
agreeable
definition.
However,
it
should
give
us
some
pause,
because
kubernetes
does
do
more
than
just
orchestrate
containers.
There
is
at
least
networking
and
storage
involved
as
well.
So
let's
broaden
this
definition
a
little
bit
and,
let's
say
kubernetes
may
be
characterized
as
an
infrastructure,
orchestration
platform,
but
again
the
defining
characteristic
of
infrastructure.
Orchestration
is
operation
automation,
so
I
would
say
we
may
safely
arrive
at
the
conclusion
that
kubernetes
can
be
characterized
as
an
operation,
automation,
platform.
A
A
First,
detection
of
state
drift
detection
of
state
drift
is
the
determination
that
the
current
state
differs
from
the
desired
state
and
second
mitigation
mitigation
of
state
drift.
That
is
the
determination
and
the
execution
of
a
sequence
of
steps,
a
sequence
of
actions
that
transition
the
system
from
its
current
state
to
its
desired
state.
A
A
Let's
look
at
an
example,
because
this
works
very
well
very
well
for
straightforward
reconciliation.
For
example,
the
replica
set
controller
replica
set
controller
manages
a
set
of
port
objects.
The
replica
set
controller
performs
two
actions
with
instantaneous
effects,
create
a
part
if
there
are
too
few
and
delete
a
part.
If
there
are
too
many
now
reading
the
replica
set
object.
The
replica
set
controller
makes
sure
that
there
is
just
the
right
amount
of
part
objects
and
reading
the
part
objects.
A
The
cubelet
itself
makes
sure
that
there
is
just
the
right
amount
of
containers
having
multiple
controllers
with
multiple
control.
Loops
involved
in
the
reconciliation
is
called
cascading
reconciliation,
each
control
loop,
managing
the
part
of
the
world
that
it
knows
about.
There
is
no
central,
no
all-knowing
controller.
A
A
So
imagine
we
are
performing
a
database
upgrade
our
business
requirements
require
that
we
upgrade
the
database
one
short
at
a
time
without
ever
falling
under
the
replication
threshold
per
shard.
How
could
you
implement
that?
Well,
you
may
very
well
attempt
to
express
that
sequence
of
interrelated
actions
as
a
number
of
control
loops
and
a
number
of
cascading
reconciliations.
A
A
A
A
However,
a
workflow
is
not
inherently
reliable.
A
failure
like
a
crash
is
a
major
at
least
a
minor
catastrophe.
If
step
1
has
been
performed,
but
step
2
and
step
3
have
not.
Your
system
is
in
an
inconsistent
state
and
the
next
run
of
the
control
loop
may
actually
not
be
able
to
detect
that
we
all
know
the
consequences,
dangling
resources,
difficult
garbage
collection
and
lots
of
lots
of
messages
on
slack.
If
anybody
knows
what
machines
they
are
and
who
they
belong
to
well
meet
temporal
temporal
is
an
open
source
platform
for
reliable
workflow
execution.
A
You
write
your
workflow
in
plain
old
java,
plain
old,
go
plain
old,
javascript,
plain
old
php.
What
have
you
and
temporal
guarantees
that
your
workflow
execution
cannot
fail
if
a
crash
happens,
temporal
resumes
your
workflow
execution
in
the
exact
same
state
at
the
exact
same
location
it
was
in
when
it
crashed.
A
So
from
the
point
of
view
of
the
workflow
execution.
That
crash
never
happened
now,
that's
quite
a
promise
and
temporal
as
a
technology
can
back
that
up
and
to
prove
that
temporal
can
back
that
up
the
hard
part
of
this
presentation.
I
will
faithfully
hand
over
to
matt
for
the
rest
of
this
presentation.
B
B
M3Db
itself
is
an
open
source
time
series
database
that
was
particularly
built
for
large
scale
and
high
throughput
use
cases
and
part
of
what
makes
it
really
well
suited
for
this
is
that
m3db
is
highly
fault
tolerant,
and
it
accomplishes
this
by
sharding
data,
as
you
write
it
to
the
database
replicating
those
shards
across
different
failure
domains.
So
you
can
think
of
this
as
like
zones
in
a
cloud
region
and
ensuring
that
those
shards
remain
available.
B
B
So,
as
I
said,
m3db
you
know
provides
fault,
tolerance
and
it
ensures
or
provides
this
by
doing
all
reads
and
writes
based
on
a
quorum
response.
So
in
this
case
you
can
see
that
we
have
three
replica
sets
con
or
three
m3db
replicas
contained
in
three
stateful
sets
each
pinned
to
a
zone
and
it's
okay
to
have
an
entire
one
of
these
replicas
be
down.
So
you
can
think
if
you
know
you
lost
an
availability
zone
or
had
some
other
sort
of
catastrophe.
B
B
You
can
imagine
that,
then
you
would
have
to
deploy
the
database
in
order
to
deploy
or
in
order
to
change
the
way
that
it
checks
its
own
health
and
also
each
node
would
have
to
have
a
much
higher
level
view
of
the
cluster
than
it
typically
has
on
a
onenote
basis
and
then,
in
terms
of
how
practical
deploys
are.
We
also
need
them
to
be
fast.
So
when
a
database
node
comes
up,
it
has
to
build
up
a
lot
of
in
in-memory
state
from
disk
before
it
can
serve.
B
Requests
is
part
of
a
process
that
we
call
bootstrapping
and
if
you
only
deployed
one
pod
at
a
time
which
would
be
the
default
with
the
stateful
set,
it
could
take
potentially
days
to
deploy
larger
clusters
around
the
time
that
we
were
trying
to
improve
the
speed
and
safety
of
deploys.
We
had
already
started
using
temporal
in
other
parts
of
our
stack
at
chronosphere,
so
temporal
drives
our
internal
deployment
platform,
a
bunch
of
end-to-end
tests,
as
well
as
a
variety
of
kubernetes
cluster
level,
tooling,
and
throughout
this
throughout
introducing
it.
B
B
Deployer
is
also
run
inside
of
the
kubernetes
cluster
and
it's
a
single
binary.
That
is
both
that
serves
both
grpc
service
for
interacting
with
deploys,
as
well
as
a
temporal
worker
and
then
upon
receiving
a
new
manifest
to
deploy.
Deployer
first
performs
some
validation
and
then
submits
it
to
kubernetes
that
the
operator
is
aware
of
it.
B
Instead
of
the
operator
immediately
acting
on
each
stateful
set,
it
actually
waits
for
the
temporal
workflow
to
kind
of
unblock
processing
of
that
given
stateful
set.
So
this
is
all
driven
via
annotations
and
when
you
first
go
to
deploy
a
new
manifest,
the
operator
actually
won't
do
anything
on
a
given
stateful
set
until
it
notices
this
annotation
present.
B
Once
the
annotation
has
been
set,
then
the
responsibility
for
the
next
part
of
the
deploy
is
handed
off
to
the
operator,
so
the
stateful
sets
are
all
configured
with
an
on
delete,
update
strategy,
meaning
that
we
actually
control
when
pods
get
restarted
and
brought
up
on
the
new
image.
So
the
operator
will
patch
the
stateful
set
and
then
proceed
to
delete
pods
in
batches
until
the
entire
stateful
set
is
processed
and
then
upon
finishing
processing.
B
B
B
The
processing
steps
are
pretty
much
what
I
mentioned
earlier,
so
the
workflow
will
set
an
annotation
on
the
stateful
set,
wait
until
the
operator
has
deleted.
All
the
batches
of
pods
and
the
stateful
set
is
healthy
and
it'll
also
actually
wait
for
that
staple
set
to
remain
healthy
for
a
given
settle
period
before
moving
on
to
the
next
batch
and
then,
finally,
when
that's
when
it's
time
for
that,
it
prompts
the
user
via
slack
interaction,
to
unblock
the
deploy,
and,
what's
nice,
too,
is
that
all
of
this
is
configurable
as
inputs
to
the
workflow.
B
What's
great,
is
that
all
these
workflows
run
while
another
workflow
is
in
parallel
checking
for
errors
and
will
actually
stop
the
deploy
if
anything
in
the
system
becomes
unhealthy
and
again,
this
is
based
on
more
than
just
any
one
pods
given
health
checks.
This
is
based
on
a
view
of
the
entire
system,
so
if
any
alerts
begin
firing
for
the
cluster,
then
we
will
stop
and
then
we'll
halt
execution
of
the
workflow.
B
B
The
slack
prompts
that
I
mentioned
are
provided
as
part
of
these
generic
helper
activities,
so
workflow
authors
can
configure
which
actions
at
any
point
in
a
workflow
would
correspond
to
kind
of
the
next
steps
in
the
code,
and
then
users
will
get
slack
notifications
and
interact
with
the
deployment
system.
This
way,
so
you
can
see
in
this
example,
deployment
failed.
The
workflow
that
runs
health
checks
realized
that
there
were
some
failing
pods
in
this
case
and
asked
the
user
what
to
do.