►
Description
Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18)
GitOps for Istio - Manage Istio Config like Code - Varun Talwar, Stealth Startup & Alexis Richardson, WeaveWorks (Intermediate Skill Level)
Services Meshes like Istio enable devops to drive service behavior via configurations. The model with Istio is to enable changing service behavior and their interactions including things like timeouts, circuit breakers, traffic routing, load balancing, A/B testing, canary releases via configuration and such critical configurations need lifecycle, audit, rollout, versioning, rollback etc like code does. This makes developers lives easier because config can be changed and managed like code code using familiar CI/CD pipelines. This can make changing configuration for resiliency controls and their subsequent testing with feedback metrics very easy. Overall, In this talk we talk about how istio config can be managed like code through git based workflows.
About Alexis
Alexis is the co-founder and CEO of Weaveworks. He is also the chairman of the TOC for CNCF, and the co-founder of the Coed:Code meet ups. Previously he was at Pivotal, as head of products for Spring, RabbitMQ, Redis, Apache Tomcat and vFabric. Alexis was responsible for resetting the product direction of Spring and transitioning the vFabric business from VMware. Alexis co-founded RabbitMQ, and was CEO of the Rabbit company acquired by VMware in 2010, where he worked on numerous cloud platforms. Rumours persist that he co-founded several other software companies including Cohesive Networks, after a career as a prop trader in fixed income derivatives, and a misspent youth studying and teaching mathematical logic.
About Varun
Product Management lead for gRPC and Istio at Google
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
So
we're
going
to
talk
today
about
gear
tops
for
ISTE,
oh
and
I'll
start
by
introducing
it
up
so
that
we
know
why
it's
important,
who
does
it,
what
the
patterns
are
and
hopefully
give
you
some
information
to
go
and
think
about
and
use
and
then
we'll
go
into
the
sto
section,
and
you
know
guests
get
ups.
Is
it's
not
it's
not
just
the
buzzword,
there's
some
some
real
technology
here
underneath
some
very
specific
practices.
A
At
the
same
time,
we
are
absolutely
building
on
top
of
years
and
years
and
years
of
people
doing
DevOps
and
best
practice
around
platform-as-a-service
and
many
other
cool
things
so
you'll
see
patterns
that
you
recognize
and
see.
Iii.
Do
that?
That's
great!
That's
a
good
thing,
but
there
is
some
also
some
detail
here
to
pay
attention
to
I'll.
Try
the
highlighter
as
we
go
and
give
ops
is
the
practice
that
we
just
started,
calling
it
that,
because
it
we've
works,
we
were
deploying
our
entire
systems
from
gear.
A
This
is
important
because
once
one
time
somebody
I
heard
I
was
sitting
in
the
office
one
afternoon
and
I
heard
one
of
the
engineers
say
I'm
about
to
make
a
change
which
you
know
could
wipe
out
our
entire
systems.
Ok,
so
I
still
like
this
and
then
I
had
to
click
followed
by.
Oh
I,
have
wiped
out
our
entire
systems,
global
clusters
and
everything
Thanks
anyway.
A
What
was
really
impressive
to
me
as
a
business
person
watching
this,
is
that
the
team
immediately
rallied
and
took
about
45
minutes
to
rebuild
our
entire
global
multi
zone
multistage
systems
from
scratch,
not
including
the
data
which
stays
on
this
okay
from
scratch,
and
we're
back
up
and
running
serving
customers
Wow.
Okay.
How
do
we
do
that?
I
asked
them
what's
going
on
here?
What
have
you
done?
I
need
to
know.
A
A
So
the
world
state
of
our
system
is
in
git,
which
means
that
we
have
a
atomic
sequence
of
transactional,
commits
representing
that
history
of
that
state
and
go
backwards
and
forwards
in
time,
and
we
can
recover
from
any
snapshot
we're
down
to
about
5
minutes
for
some
of
these
recovery
scenarios
now,
which
is
pretty
pretty
cool.
But
you
know
this
is
a
very
fundamental
change.
A
The
idea
is
the
desired
state
of
your
entire
system
is
in
get,
and
that
means
the
cluster
and
it's
observed
state
can
be
compared
with
that
at
all
times
we
can
fire
diff
alerts
if
they're
different
for
any
layer
of
the
stack
and
then,
of
course,
on
top
of
all
of
this,
you
get
the
benefit
that,
if
you're
a
developer,
all
you
need
to
think
about
is
writing
code.
Do
a
git
push
merge
and
it
could
go
straight
into
production,
so
we
don't
use
coop
cuttle
to
interfere
with
our
operational
cluster
anymore.
Pretty
amazing.
A
The
challenge
for
SC
OV
o
is
wait
a
minute.
This
thing's
happening
inside
sto
like
traffic
management,
where
we
actually
might
be
sliding
you
know
the
percentage
weight
on
a
canary
in
real
time.
We
don't
necessarily
want
to
commit
that
to
get
as
we're
trying
out
different
scenarios,
so
we're
wrestling
with
how
to
do
this
best
and
we're
not
completely
at
a
solution.
We
will
get
there
and
these
guys
will
show
you
a
solution.
The
tip
theta
the
ass
on
this
today,
that
will
be
the
demo.
A
Can
I
go
through
this?
Why
is
it
not
moving?
Oh
there
we
go
so
I'm,
just
gonna
run
through
the
get
up
story,
one
more
time.
It's
a
practice
around
cloud
native
and
server
less
and
things
like
that.
It's
all
about
pushing
code,
not
containers
through
get,
and
it
gives
developers
a
sense
of
operational
control
through
pull
requests.
A
A
What
can
be
described
can
be
automated,
because
orchestration
can
use
the
desired
state,
the
descriptions,
the
annals
and
use
that
to
enforce
conformance,
I
guess
in
the
running
state.
So
you
wanted
therefore
described
as
much
as
you
can
and
get
not
only
other
Cuban
Eddy
stuff,
but
everything
else
too,
you
can
do
differ
Lots.
We
have
three
kinds:
we've
works
tariff,
ansible,
diff
and
cube
diff,
which
tell
you,
if
you're
getting
divergence
respectively
for
terraform,
ansible
and
kubernetes
and
we'd
love
to
have
more.
A
A
A
Quick
example,
this
is
one
of
our
customers.
I
should
explain,
we'll
have
a
demo
users,
our
product
we've
cloud,
but
many
of
the
things
we're
talking
about
you
can
do
with
other
technologies.
Okay,
the
Google
guys
asked
me
to
say
that
these
guys
are
a
twenty
five
person
team
in
SF.
They
do
optimize
local
websites
in
different
languages,
for
big
brands
like
Nike
and
so
on.
A
Keeping
those
websites
up
to
date
with
compelling
content
for
the
customer
is
a
hard
problem
to
solve
at
scale,
because
you've
got
lots
of
different
languages,
lots
of
different
sites
and
you're
trying
to
measure
which
pieces
of
content
people
engage
in
so
believe
it
or
not.
There's
a
whole
class
of
new
products
based
on
machine
learning
and
AI,
and
data
crunching
to
figure
out
how
to
do
that.
That's
what
these
guys
do,
but
they
were
doing
one
deployment
a
week
two
deployments
a
week.
A
Sometimes
they
would
break,
they
would
use
things
like
helm
or
Jenkins
to
push
changes
and
you'd
have
a
group
of
changes
and
then
half
of
them
would
succeed
in
half
of
that
would
not
succeed
and
they
wouldn't
know
what
state
they
were
in
and
they
would
have
to
go
back
to
the
beginning
and
reinstall
whole
cluster.
This
is
not
cool
when
they
started
using
the
approaches
we're
talking
about
today
and
some
open
source
technology
and
then
our
commercial
products.
A
They
found
that
they
could
go
to
four
different
teams,
each
one
of
which
does
between
30
and
50
updates
per
day.
Each
update
is
instantaneous
and
either
fails
cleanly
or
succeeds
cleanly,
and
they
can
also
roll
back.
So
what
that's
meant
is
their
velocity
has
increased
and
they
have
a
lot
more
confidence
in
what
they're
doing
they
can
try
out
business
experiments
like
a
bee
testing
just
on
the
fly.
A
A
So
how
does
it
work?
Who
knows
what
an
operator
is,
or
the
operator
pattern
in
kubernetes
good,
an
operator
basically
lives
within
the
security
and
domain
of
control
of
kubernetes
itself
and
provides
essentially
additional
services
and
capabilities,
taking
advantage
of
the
orchestrator
to
start
and
stop
things.
So
lots
of
people
are
writing
operators
to
do
things
like
manage.
You
know:
Redis
inside
kubernetes,
Nats
inside
communities,
Prometheus
inside
Cuba
Nettie's.
We
wrote
one
to
do
deployments
from
inside
kubernetes.
A
A
A
Why
does
it
do
that?
It's
because
it
can
continue
to
synchronize
if
it
fails,
and
it
has
a
description
of
the
correct
state
if
you
use
we've
written
something
to
integrate
this
with
helm,
so
you
can
do
this
with
helm
as
well,
but
it
was
quite
difficult
to
do
because
how
am
is
not
atomic
non
compositional
and
non-deterministic
makes
it
hard.
A
A
To
summarise
core
principles
very
simple,
and
this
is
why
this
get-ups
thing
is
really
kubernetes
centric
and
cloud
native
centric.
It's
about
declarative
configuration.
Okay.
If
everything
is
imperative,
it's
very
hard
to
get
this
right.
It
certainly
has
to
be
at
least
idempotent.
If
you
want
to
get
it
right,
imperatively,
but
declarative,
you
can
enforce
convergence
by
essentially
reapplying
the
changes
if
they're
not
succeeding,
and
you
can
also
observe
and
compare
with
the
desired
state.
All
changes
should
go
through
get
including
review.
No
one
should
use
q
couple
directly.
A
I
should
explain
by
the
way
you
are
allowed
to
break
these
rules.
Okay,
this
is
not
a
cult.
It's
just
a
set
of
principles,
but
very
important.
It's
not
a
good
idea
to
push
directly
from
CI
to
production.
Many
people,
let
their
CI
tool,
drive
deployment
you're,
giving
a
notoriously
hackable
thing
access
to
production.
If
you
do
that,
I
use
the
operator
in
the
cluster
we've
written
something
called
weave
flux
which
is
open
source
and
underneath
the
demo
that
Stephan
will
show
you
which
you
can
do
to
drive
convergence
in
your
cluster.
A
It's
very
easy
to
use.
You
load
it
into
kubernetes
and
it
sinks
or
the
get
some
benefits.
Everything
is
atomic
and
transactional.
So
you've
gotten
get
a
complete,
sequential
audit
log,
which
you
can
roll
back
and
forward
to
everything
succeeds
or
fails
cleanly,
and
it's
very
very
easy.
So
you
have
that
magic,
Heroku
style
git
push
your
code
centric,
which
is
very
important.
If
you
want
developers
to
care.
I
mentioned
the
transaction
log.
This
is
a
benefit
for
teams,
because
you
can
have
metadata
like.
Why
was
this
commit
made?
When
is
it
made?
A
Who
made
it?
We
have
customers
who've
actually
gone
through
cybox
compliance
because
of
this.
Normally
people
pay
20
million
dollars
for
a
very
expensive
compliance
database
and
the
auditor
will
come
along
and
look
at
the
compliance
database
for
days
and
you'll
pay
the
money
with
this.
You
can
just
show
them
on
the
record
and
get
of
who
made
changes
to
the
system
and
when
and
their
whether
they
were
permitted
to
and
they'll
just
fall
on
the
floor,
because
it's
free
nice
and
easy
and
then
this
is
important.
A
You
know
if
you
think
about
how
this
works,
when
you
do
a
day
of
test
cycle,
you're
loading,
our
images,
image,
changes,
label
changes,
config
changes
and
potentially
code
changes
into
a
set
of
repos.
You
can
think
of
those
as
a
firewall,
because
you've
got
dev
on
one
side
of
the
firewall
and
prod
on
the
other.
The
production
update
from
the
repos
is
happening
by
things
being
pulled
into
kubernetes
by
the
operator.
So
at
no
point
does
somebody
push
out
from
Devon
to
prod
through
the
firewall,
which
is
really
cool.
A
Oops,
okay,
I
did
have
a
slide
saying
who
is
talking
about
get-ups,
but
I'll
run
through
them
verbally,
because
Google
seems
to
not
have
saved
it.
There's
a
whole
bunch
of
companies.
Looking
at
this
there's
anyone
here
from
haserror,
no,
they
have
a
booth.
They've
got
something
called
git
cube,
which
is
really
neat
sort
of
dev
centric
geared
ops
tool.
Cloudbees
Jenkins
axe
is
built
on
get-ups
principles
and
is
a
CD
tool
for
kubernetes
and
Jenkins
James
strikin
who's.
The
author
is,
is
around
on
the
cloud
B's
booth
as
well.
A
Do
we
have
an
open,
faz
project
lead
here
there?
He
is
get
ups
for
open
files,
give
ups
or
server.
Let's
think
about
that.
That's
really
cool
Financial
Times
Sarah!
This
morning
she
described
how
the
Financial
Times
do
all
of
their
work
with
this
sort
of
observable
pipeline.
That's
a
good
ops
pipeline.
Okay,
though
the
supermarket
the
same
thing
and
many
others,
so
you
should
join
them.
Okay!
A
B
Now
the
key
word,
as
we
were
sort
of
getting
sto
out
is
all
of
the
actions
or
configuration
driven,
and
this
is
where
key
tops
and
sto
story
is
going
to
get
connected.
So
if
we
have
to
think
about
this
problem,
how
do
you
make
in
a
world
where
sto
is
taking
a
lot
of
the
service
behavior
things
like
what
is
the
coda
that
should
be
applied
to
a
given
service
things
like
whether
the
service
should
call
securely
to
another
service
things
like
how
should
I
route
traffic
and
load
balance
between
service
to
service?
B
Now
all
these
service,
behavioral
things
are
moving
from
code
to
config,
that's
kind
of
the
promise
of
service
mesh-like
architectures.
Now
you
have
cool
code
management,
workflows
and
tools,
you
know
how
to
iterate
make
code
changes,
push
get
a
new
version
out
and
what
about
config
tools
now,
if
in
a
world
where
all
of
this
is
moving
to
configuration,
you
need
a
proper
config
workflow
to
make
all
those
operations
happen
happen
via
different
teams.
B
Now
imagine
there's
like
hundred
teams
managing
hundred
services,
each
of
them
making
configuration
changes
for
each
of
their
own
services,
and
this
is
affecting
real
traffic.
So
this
is
like,
if
you,
if
you
screw
this
up,
you're
like
basically
your
systems
down
so
sensitive
in
terms
of
like
criticality,
and
you
want
to
have
live
in
a
world
where
you
can
operate
them
cleanly
separately,
each
service
owner
and
observe
the
behavior,
and
if
go
back,
things
go
bad.
You
can
roll
back.
So
this
is
a
pretty
challenging
sort
of
problem.
B
So,
for
those
of
you
who
are
familiar
with
this,
do
probably
you'd
relate
to
this,
but
Easter
is
a
bunch
of
capabilities
across
a
few
areas
right,
and
it
has
almost
everything
here-
is
configuration
driven.
It
deals
with
four
broad
areas
if
you
are
not
familiar,
it's
traffic
management,
which
is
everything
to
do,
managing
all
your
ingress,
as
well
as
service
to
service
traffic.
It's
observability,
which
is
monitoring,
latency,
QPS
error
rates
of
all
of
that
traffic.
B
It's
security,
which
is
authenticating
all
of
the
service
to
service
calls,
as
well
as
authorization
and
it's
performance
in
the
sense
in
a
world
of
micro
services.
If
you
want
to
do
failure,
scenarios,
retries,
timeouts
fault,
injection
circuit,
braking,
etc,
those
are
all
configuration
driven
things
within
sto.
Now,
there's
two
things
to
note
here:
each
of
those
each
of
these
things
has
different
owners.
Typically
in
a
in
an
enterprise
in
a
workflow,
they
are
owned
by
network
team
gets
involved.
Sre
and
DevOps
team
is
involved
as
either
a
horizontal.
A
B
Or
per
service
team,
the
security
and
InfoSec
and
sec
ops
gets
involved
and,
of
course,
developers,
building
services
and
there's
some
sort
of
common
themes
across
these.
These
are
not
one-time
things
so
imagine,
for
example,
let's
take
retries
you're
trying
to
tune
the
performance,
you're
saying:
okay,
let
me
retry
every
five
seconds
from
this
call.
When
it's
service
service,
a
tow
service,
B
right,
you
try
it
out
you
test
it,
you
see
what
the
performance
is.
You
observe
it
if
it
works,
you
do
you
get
it
into
production.
Otherwise,
you
go
back,
make
changes.
B
These
are
all
iterative.
So
this
is
not
a
one-time
thing.
This
is
a
living
breathing
you're.
Making
these
configuration
changes
in
and
out
right,
like
it's
same
thing
for
the
other
key
sort
of
piece
for
these
configurations
is
they're
critical,
so,
like
I
said,
if
you
in,
if
you
inject
something
which
is
going
to
take
over
all
the
authentication
and
going
to
encrypt
all
of
the
traffic
in
transit,
one
breakage
there,
if
one
of
the
services
you're
not
able
to
handle
TLS,
it
can
screw
up
the
whole
system.
B
A
lot
of
these
configuration
driven,
things
is
cross-team,
so
one
team
decides
okay,
I
want
to
use
Zipkin
for
tracing
or,
but
the
other
team
says
okay,
I
want
to
use
Jaeger
for
tracing.
These
are
things
that
these
are
also
even
backends
of
where
things
are
written
for
all
of
the
service
to
service
traffic
is
configuration
in
sto,
so
somebody
writes
it.
Somebody
should
have
prove
it
and
that's
how
it
should
happen
right
and
it
could
be
for
one
service.
B
It
could
be
a
cross
services
and
when
it's
across
the
approval,
workflows
could
look
pretty
complex.
So
these
are
cross
team
in
terms
of
what
these
configurations
are
and
how
they
should
be
rolled
out
and
they're,
not
atomic.
So
things
like
canary
releases
right,
it's
not
like
a
one-time
atomic
thing,
and
this
is
what
we
will
try
and
show
in
the
demo.
It's
a
complicated
workflow.
Where
you
make
the
change
you
get
it
out
through
your
pipeline,
you
send
some
percentage,
some
select
subset
of
traffic
to
it.
You
observe
it,
it
works.
B
If
it
doesn't
work,
you
keep
making
changes.
Take
the
next
version
tag
it
get
it
to
the
same
pipeline.
So
it's
it's
iterative.
They
are
not
single
atomic
operations.
So
this
is
what
makes
like
config
management
overall
for
sto
sort
of
a
complex
problem.
It's
across
stakeholders
and
it's
a
continuously
changing
thing.
B
So
just
I'm
going
to
just
browse
through
a
quick
set
of
config
examples.
Just
to
give
you
a
flavor
for
those
of
you
who
are
using
or
have
seen
a
steer.
This
is
very,
very
familiar,
but
for
those
of
you
who
haven't
this
will
at
least
tell
you
what
is
the
complexity
of
configuration?
We
are
talking
about
here
right,
and
this
is
probably
a
good
complex
example.
B
B
So
that's
what
you're
configuring
here
right
so
and
then
in
terms
of
actually
how
your
route
traffic
is
like
a
set
of
load,
balancing
policies
that
you
can
have
so
it's
you
can
see
how
these
values
around
these
configurations
is
critical
iterative
and
something
that
is,
you
know
core
to
your
performance
and
your
system
take
another
example.
Egress
you
just
want
to
say:
ok
from
this
service,
you
can
go
outgoing.
You
can
talk
to
a
given
domain,
HTTP
or
HTTPS
simple
egress.
Config,
take
a
classic
a
be
testing
kind
of
thing.
You
release
something
new.
B
There
was
a
user
who
was
working
on
it.
He
wants
only
his
requests
to
be
routed
to
the
v2
of
a
service.
So
what
is
this
thing
saying?
This
thing
is
saying:
if
it's
coming
from
ratings
v2
and
the
URL
path,
is
that
and
if
cookie
has
this
user
value
then
go
to
v2
right,
so
classic
staged
rollout,
a/b
testing
scenario
looking
at
things
like
performance,
so
there's
also
simple
things
like
redirect
or
fault
injection.
B
So
you
often
come
across
these
scenarios
when
you
are
in
testing
mode
performance
tuning
mode-
and
you
want
to
get
all
these
configurations
out,
make
sure
these
are
approved
by
the
right
set
of
people
there's
a
log
for
it
of
who
did
what
change
and
what
they
committed,
and
this
is
where
sort
of
get
ups
and
sto
configuration
starts
to
come
together.
So
just
in
terms
of
what
these
are
saying,
so
the
last
one
is
interesting.
B
B
B
C
So
I
have
a
git
repo
inside
this
git
repo
is
my
cluster
definition.
I
have
a
simple
application.
My
application,
the
application
name,
is
pod
in
fall
and
I
have
I.
Have
my
namespaces
here:
I
have
a
single
dead
space
test
and
aside.
Here
are
all
the
files
that
are
needed
for
my
application
to
be
deployed.
The
GE
deploy
the
cannery
deployment,
the
gateway,
the
service,
the
virtual
service.
So
what
I'm
describing
here
is
how
I
have
one
to
deploy
my
application.
C
I'm
using
the
new
the
new
version
of
alpha
3
version,
where
he
still
has
little
services
and
gateways.
So
this
is
the
initial
state
where
all
the
traffic
goes
to
the
MiG
release,
so
the
weight
is
100
the
canary
zero.
So
this
is
now
deployed
on
my
cluster
and
I
can
see
using
with
cloud,
for
example,
scope
that
from
my
load
test,
all
the
traffic
goes
to
my
G
release
and
the
load
tests
I
card
talks
to
the
east
EO
pilot.
C
C
And
using
primitives
I
can
getting
the
primitives
metrics
out
of
out
of
my
cluster
I
can
see
that
all
the
traffic
right
now
goes
to
the
the
GA
commit.
Now
the
our
CD
operator
has
detected.
There
is
a
change
in
get
a
plat
applied
it
to
to
our
cluster,
and
now
we
should
see
that
some
part
of
the
traffic
gets
routed.
C
Down
here,
I
am
measuring
the
request,
latency
and
you
have
all
sorts
of
other
other
things
that
you
can
measure
to
make
sure
that
your
cannery
list
is
not
failing.
So
this
is
the
most
important
part
right.
You
are
not
doing
Kennedy.
If
you
cannot
measure
it
so
using
Prometheus,
you
can
actually
detect
errors
and
stuff
like
that.
So.
B
Yeah,
so
this
is
the
setup
here
you
have,
you
have
GA
release
and
a
canary
release,
and
you
made
some
changes
and
you
started
sending
X
percent
traffic
notice.
How
all
of
these
two
files,
the
in
word
in
the
in
the
repo,
the
virtual
service
gateway
config
files,
are
all
in
the
git
repo
and
he's
making
good
changes.
C
C
C
Deployment
definition
for
my
canary
lists
using
annotations
I
am
instructing
my
the
operator
to
automate
this
release
and
deploy
any
new
version
that
ends
up
in
in
the
docker
registry.
That
is
a
minor
change,
minor
version
to
my
to
my
release.
So
what
I'm
saying
here
any
patch
that
it's
available
on
0.3?
Please
deploy
that
automatically.
So,
instead
of
me
going
here
and
do
changes
every
time,
I
iterate
through
the
code
and
increment,
this
version
I
instruct
the
operator
to
do
that
using
using
this
annotation.
C
B
Notice
what
happened
here?
He
has
not
done
any
cube,
cuddle
or
history
of
control
flux
detected.
He
set
up
flux
to
say,
okay,
if
there's
any
minor
version
of
zero
3x,
then
sort
of
auto
release
and
deploy,
and
that
pushed
as
soon
as
in
the
registry
there's
a
new
image
with
that
zero
3x
tag
that
was
pushed-
and
he
has
the
log
of
the
fact
that
this
was
pushed
right
and
infamy
in
our
monitoring.
As
we
observe
you
can
see
that.
C
Its
own
zero,
three
one,
so
the
next
logical
step
will
be
to
keep
increasing
the
the
traffic
route.
Okay,
maybe
you
could
find
it
annoying
that
you
have
to
always
commit
when
you,
when
you
increase
the
the
weight,
maybe
in
the
future,
somebody
will
do
an
interface
that
will
talk
to
the
operator.
The
operator
will
do
those
commits
for
doing
it,
so
you
don't
have
to
manually
increment
all
the
time
great
implementation
will
be
to
rollback
if
from
some
Prometheus
metric
you
can
actually
detect,
can
set
like
this.
C
B
Set
up
a
threshold
like
if
you
can
set
up
a
threshold
of
like
if
it
goes
below
this
threshold,
then
roll
back
like
this
lot
of
magical
sort
of
things
you
can
do
once
you
have
this.
You
know
your
workflow
connected
to
sort
of
the
desired
state,
so
this
goes
back
to
sort
of
the
get-ups
sort
of
the
main
principle
which
is
here,
is
what
I
want
and
desired
state
I'm
observing
and
it's
based
on
how
far
it
is
from
desired
state.
A
A
So,
while
we're
watching
this
I
just
want
to
make
the
point
that
this
is
a
good
illustration
of
how
observability
is
integral
to
the
pipeline
deployment
process.
Okay,
if
you
can't
see
what
you're
doing
as
you
do
it,
you
can't
get
the
benefits
of
seeing
if
you're,
correct
or
wrong
I
need
to
make
changes
on
the
fly
so
to
go
fast.
You
need
both
I
think.
We
really
are
out
of
time,
because
someone
else
is
going
to
talk
in
a
minute.
Okay,
is
it
restarting
yeah
its
restarted?
Thank
you
very
much
Stefan.
Thank
you.