►
Description
Automated Verification of Deployments for Risk Assessment - Gopinath Rebala , OpsMx
Safe and reliable deployments are critical for continuous delivery of application services. Time taken to detect an issue has direct impact on bottom line. Web scale companies like Netflix practiced deployment strategies like canary deployments and blue/green deployments and use statistical techniques to determine problems before they cause an outage and improve the MTTR in case of an outage. In this talk we will present an automated verification model combined with canary and blue/green deployments in Spinnaker pipelines. We will present the usefulness of the models in verification of performance and functionality of deployments using metrics and logs generated in distributed microservice environment for identifying problems in deployments.
A
A
Today
we
will
discuss
what
is
automated
verification
and
how
can
one
use
it
in
the
current
delivery
pipelines,
we
will
go
a
little
bit
of
background
on
how
it
works
and
then
do
a
demo
of
automated
verification
of
using
a
pipeline
based
on
the
analysis,
promotes
the
production
and
then
we'll
leave
some
time
for
q
a
so
who
are
we?
We
are
opsimix.
We
are
leading
continuous
delivery,
delivery
solution
provider,
so
our
vision
is
to
deliver
software
without
human
intervention.
A
A
A
The
failure
significantly
increased
so
as
some
of
these
organizations
reach
hourly
delivery,
most
all
of
their
deliveries
were
failures,
so
the
cost
of
the
failed
delivery
is
really
high.
Essentially,
if
you
find
problems
at
tail
end
in
the
production,
the
cost
is
much
higher
than
you
would
find
in
the
testing.
A
The
magic
essentially
lies
in
this
ability
to
do
small
changes
on
a
continuous
basis
to
deploy
for
the
applications,
so
this
essentially
decreases
the
cost
and
makes
the
mean
time
to
recovery
a
lot
faster
for
recovering
from
small
changes,
but
it
still
requires
deployment
strategies
and
continuous
verification.
So
why?
A
If
you
look
at
current
delivery
mechanisms,
they're
generally
organized
as
a
continuous
integration
and
continuous
delivery
in
the
continuous
integration,
you
have
the
build
and
unit
tests
and
maybe
static
code
analysis
that
you
do,
but
then
you
hand
it
over
to
the
delivery
mechanism,
which
has
certain
number
of
steps
in
it.
Where
you're
deploying
tests
on
the
tests
maybe
run
the
dynamic
code
analysis
and
go
through
some
approval
process
and
then
deploy
to
production
using
some
deployment
strategies.
A
So
today,
the
manual
approval
and
verification
gates
that
occur
at
different
stages
are
based
on
some
judgment.
Calls
one
makes
either
in
the
devops
model
the
developer
after
the
test
moving
to
production,
it
could
make
it
a
different
decision,
based
on
the
results,
whether
it
occurs
on
a
wednesday
morning
versus
friday
evening
right.
These
are
not
a
consistent
results
because
of
decisions
they
are
making,
and
it
also
has
the
manual
stages
where
someone
has
to
approve,
let's
say,
for
example,
doing
a
sarbanes-oxley
kind
of
systems
for
them.
A
So
for
all
of
these,
it
makes
the
current
manual
verification
process
error
prone
and
also
starts
introducing
delays,
because
there
is
no
confidence
in
it.
So
there
is
a
certain
amount
of
delay
that
gets
introduced
for
every
decision
that
is
going
to
be
made
so
to
counter
that
you
actually
need
this.
Automated
verification.
A
In
a
multi-service
environment
as
you're
promoting
your
services,
it's
not
just
the
service
that
you
are
promoting,
that
you
look
need
to
look
at
the
logs
or
the
metrics
that
are
being
generated
by
that
service.
That
you
need
to
verify.
Everything
is
good,
but
you
also
want
to
see
the
entire
ecosystem
of
the
product
and
all
the
dependencies
and
make
a
decision
on
whether
the
service
can
move
forward,
and,
typically
you
you
have
these
slos
that
you
look
at
once
you
promote
the
service
and
say
everything
is
good
and
there
is
a
secondary
thing.
A
You
also
look
at
logs
of
the
system
and
now
looking
at
these
logs
in
these
services
with
the
spring
boot
kind
of
systems,
they
generate
tremendous
amount
of
logs
and
looking
at
all
of
these,
to
find
anything
new
that
could
be
causing
a
problem
either
in
terms
of
architectural
regression
or
compliance
issues
or
anything
related
to
scale
scalability
very
hard
to
find.
So
this
is
essentially
the
core
drive
towards
going
to
the
automated
verification.
A
Now,
where
can
we
plug
in
this
automated
verification?
What
does
it
need
to
support
this
in
a
typical
delivery?
Workflow,
you
have
your
artifacts
built
and
go
through
the
integration
test
load
test,
and
then
there
are
gates
there
you,
depending
on
the
p1
issues
that
you
see
or
what
the
functionality
is
you
decide
to
promote
the
staging,
and
once
you
promote
the
staging,
there
are
additional
gates
there,
based
on
the
environment
and
configuration
that
you
have
set
up.
A
So
at
each
of
these
gates
is
where
you
can
look
at
the
digital
exhaust
and
determine
you
can
use
the
automated
verification
here
so
the
digital
exhaust.
Here,
essentially,
it
refers
to
the
logs
and
metrics
that
you
are
providing
for
each
of
the
services
that
you
are
promoting
through
your
delivery
front
line
and
make
a
decision
based
on
that
data,
because
this
is
automated,
it's
consistent.
A
A
This
automated
verification
is
a
service
that
you
can
call
from
a
single
script
or
integrate
into
any
of
your
delivery
pipelines,
whether
it's
a
spinnaker
or,
if
you're,
using
jenkins
today
or
any
other
delivery
pipeline
that
you
use.
You
can
integrate
that
as
a
test
service
to
trigger
off
your
analysis.
As
you
do
the
deployment
either
in
testing
staging
or
production
and
based
on
the
score,
you
decide,
you
can
decide
whether
you
go
forward
and
the
score
you
can
retrieve
based
on
the
api.
A
You
observe
the
data
that
is
coming
out
of
your
services,
this
the
data
that
is
coming
out
or
the
log
files
from
your
software,
multiple
services
or
the
metrics
that
you're
measuring
in
terms
of
your
slos
or
even
the
system
metrics.
All
of
them
are
part
of
the
data
that
we
observed
and
based
on
the
data
over
a
period
of
time.
A
You
can
build
a
profile
for
that
service
and
that's
aligned
with
that
service,
and
then
this
model
is
continuously
updated
as
we
deploy
new
services
and
based
on
this
model,
when
a
new
version
is
deployed,
we
analyze,
if
there
is
any
deviation
that
we
find
or
any
classification
of
the
logs,
that
we
can
do
as
an
error
and
based
on
the
errors
and
the
metric
deviations.
There
is
a
score
that
gets
generated.
That
score
is
then
used
for
automated
decision.
A
So,
given
that
I
just
want
to
give
a
quick
demo
with
this
spinnaker
integrated,
automated
verification
for
some
of
you.
That
may
not
know
what
spinnaker
is
spinnaker
is
a
multi-cloud,
continuous
delivery
solution
that
enables
faster
deliveries
of
your
software.
It
provides
basically
three
primary
functions
in
my
mind.
A
One
is
the
ability
to
construct
the
delivery
pipeline
at
different
stages,
stitch
together,
a
bunch
of
different
steps
that
you
need
to
deliver
in.
In
this
case,
you
can
see
there
is
a
build
stage
and
test
deployment,
and
then
we
do
an
automated
analysis
based
on
the
test
suite
that
runs
against
it
and
and
based
on
the
score
that
we
get
from
automated
analysis.
A
A
It
helps
you
look
at
the
current
state
of
your
application
that
is
currently
deployed
and
even
act
on
it
based
on
the
rpac
configured
with
spinach,
for
example.
In
this
case,
you
can
go
back
and
say
hey.
I
want
to
undo
the
rollout
to
his
previous
version
or
the
version
before
this
is
based
on
the
history
amount
of
history
you
keep.
A
This
can
also
be
done
as
part
of
your
pipeline,
say
as
you
do
a
canary
deploy
and
then
decide
that
canada
is
not
performing.
You
can
do
a
role
roll
back
from
there.
It
also
helps
you
with
diagnosing
any
errors
with
services
that
you
deployed
in
this
particular
case.
Let's
say
we
have
a
part.
We
can
go.
Look
at
the
logs
that
gets
generated
all
in
one
place.
A
But
the
steps
that
we
want
to
take
are
build.
That
code
from
the
master
branch
deployed
to
the
test,
run
automated
analysis
against
the
test
cases
that
it
stuns
and
based
on
the
result
that
come
up,
we
want
to
promote
the
production.
So
let
me
redo
the
same
thing
here.
What
I'm
saying
is
I
want
to
use
this
one
as
my
baseline
and
also
don't
want
any
manual
judgment.
It
is
possible
for
for
us
to
specify,
even
after
doing
automated
analysis,
to
look
at
the
analysis
report
and
make
a
manual
judgment
to
promote
the
production.
A
A
We
don't
see
any
specific
problems
with
the
confidence
in
promoting
to
production,
so
zero
is
a
low
confidence
hundred
is
a
high
confidence,
then,
in
the
case
where
it
fails
and
if
it,
if
you
want
to
do
diagnostics
most
of
the
times,
if
it
succeeds,
you
don't
need
to
do
anything
more.
You
basically
promote
the
production
because
there's
no
difference,
but
in
the
cases
where
it
fails-
and
we
want
to
go,
do
the
diagnostics
we
can
go
to
the
report
and
see
what
the
report
details
and
find
responses
fairly
quickly.
A
So
let
me
show
you
a
report
that
has
a
failed
score.
These
reports
can
be
used
to
quickly
identify
what
the
issues
are.
It
does
the
analysis
both
for
logs
and
the
matrix
here.
This
gives
us
scores
independent
independently
for
both
logs
and
metrics
in
the
logs.
It's
basically
found
one
error
that
seems
serious
enough
for
for
us
to
warrant
a
stoppage
of
the
promotion
to
production,
and
one
is
to
take
a
look.
So
here
the
the
analysis
is
done
over
100
kilobytes
of
data
for
the
short
period
that
the
logs
are
generated.
A
This
view
logs
is
essentially
takes
us
to
an
external
system
like
the
elastic
that
you
are
already
using,
let's
say
for
collecting
the
log
information
once
blank
gets
the
data
from
there.
So
in
the
logs
in
the
details,
if
we
go
of
the
100
kb
of
data,
it
was
able
to
find
these
lines
of.
Data
that
are
together
is
forms
a
clusters
of
the
logs,
essentially
using
the
natural
language
processing.
A
It
is
able
to
identify
a
log
lines
that
form
clusters
both
in
terms
of
similarity
as
well
as
closeness
to
each
other
and
then,
and
then
shows
classifies
them
as
an
error
or
informational
and
based
on
that
classification,
it
promotes
this.
It
computes
the
score
and
basically
allows
us
to
use
their
automated
score
for
analysis
decision.
A
So
if
on
this
error,
you
can
see,
this
is
a
having
trouble
connecting
the
sql
server
and
we
can
also
see
that
the
timeline
of
the
test
and
the
errors
occurred
or
in
the
beginning
part
of
the
first
five
minutes
of
this
and
of
the
test
where
it
hasn't
occurred.
After
that
and
in
the
first
five
minutes
it
did
multiple
retries.
So
it
seems
like
there
is
an
issue
where
there
was
initial
network
configuration
error
and
it
had
problem,
but
it
recovered
after
that.
A
So
maybe
this
is
not
an
issue
that
we
should
really
worry
about.
Unless
it
occurs
again,
it
may
not
be
the
issue
with
the
software.
However,
if
you
go
and
look
at
the
log
metrics
data,
you
see
that
we
are
collecting
matrix
for
those
services,
so
these
can
be
the
application
level
metrics
or
the
system
level
metrics,
where
we
find
the
data
coming
from
operating
system
or
pod
level
metrics
like
memory
cpu
and
those
kind
of
things,
as
well
as
the
latency.
A
Here
it
found
that
the
latency
is
high,
so
clearly
based
on
the
previous
error
and
how
it
recovered
and
the
latency
here
there
is
a
network
configuration
issue
or
database
latency
issue
here,
so
we
need
to
look
at
and
so
in
this
particular
case,
what
we
did
was.
It
was
a
transient
issue.
At
that
point
there
was
a
network
configuration
with
the
vpc.
There
was
an
issue
with
routing
was
a
problem
and
by
correcting
that
we
were
able
to
identify
and
fix
the
issue
for
the
service.
A
The
other
thing
I
wanted
to
also
show
is
this:
the
ability
to
do
multiple
services
simultaneously,
so
there
in
the
analysis
in
the
template,
we
said,
run
this
analysis
for
one
service.
So
here
it
is
possible
to
run
this
analysis
as
we
deploy
one
service
on
multiple
services,
and
you
could
provide
a
number
of
the
services
that
you
know
as
a
dependent
and
run
the
analysis.
A
To
going
back
to
it,
it
probably
takes
another
a
few
minutes
to
complete,
but,
as
you
can
see
now
we
are
able
to
deploy
to
test
environment,
run
automated
analysis.
That's
beyond
the
test
cases
that
you
do.
You
may
have
incomplete
test
cases
in
some
of
these
functional
systems.
Sometimes
we
cannot
completely
automate
all
the
test
cases
for
100
coverage
for
particularly
integration
environments.
So
this
is
an
added
benefit
where
you
could
automatically
test
verify
your
deployments
without
having
to
do
any
additional
coding.