►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Custom Deployment Strategies for Kubernetes - Nail Islamov, Atlassian
Many tech companies are using continuous deployments (CD) to deliver changes to their users faster and more frequently. One of the challenges with automated deployments is making them safe by detecting and quickly rolling back in the event of a bad release. Standard CD practices include using canary and blue-green deployments; unfortunately, Kubernetes only supports the "rolling update" deployment strategy out of the box, which can only prevent trivial failures. Thanks to extensibility of Kubernetes, it is possible to build custom advanced deployment strategies while reusing Kubernetes core concepts. Nail Islamov will give an overview of how Deployment, ReplicaSet and Pod objects work together along with Service and Ingress, and will provide examples of implementing blue-green and canary deployments reusing these concepts by introducing extra CRD resources.
To learn more: https://sched.co/GrUI
A
Hi
everyone
thanks
thanks
for
coming
today,
I'll
run
a
session
about
custom,
diplomat,
searches
for
communities,
I
work
at
Atlassian.
We
are
building
into
a
internal
platform
as-a-service
built
on
top
of
communities.
We
use
lots
of
theories
and
like
effectively
the
talk
is
inspired
by
my
actual
work
at
Atlassian.
Hopefully,
you'll
find
it
useful,
so
just
some
brief
overview
for
the
context.
I
suppose
that
most
people
here
are
familiar
with
continuous
delivery
concept
and
continually
is
an
approach
where
teams
release
products,
frequently
and
predictably,
from
source
code
repository
to
production
in
an
automated
version.
A
So
for
this
particular
kind,
I
will
be
focusing
on
the
automation
side
of
things
and
the
typical
CI
CD
pipeline
have
multiple
stages,
which,
with
it
reject
transitions,
which
are
either
automatic
or
manual
triggered,
and
normally
we
have.
The
Glitter
stage
is
building
artifact
running
some
unit
tests.
Then
we
can
deploy
artifact
to
the
staging
environment,
and
then
we
run
more
tests
against
the
staging
environment.
A
So,
first,
let's
talk
about
community
deployments.
So
the
way
to
deploy
deploy
applications
in
communities
is
using
unities
deployment
resource
that
has
to
update
strategies
or
create
unrolling
update.
So
they
recreate
strategy
is
killing
all
existing
pods
before
creating
new
ones.
It
may
be
useful,
mostly
for
working
notes.
It's
not
as
much
useful
for
user
facing
applications
which
have
some
end
points,
so
the
rolling
update
strategy
is
actually
the
default
one.
So
if
you
don't
specify
the
rollout
strategy
in
your
deployments
pack,
this
is
the
strategy.
You'll
get
and
rolling
update
scales
creates.
A
A
new
replica
set
scales
is
up
while
the
old
replica
set
is
being
scaled
down.
So
it
was
just
quick,
a
brief
look
at
and
they're
rolling
updates
through
surgery,
so
deployment
manages
replica
sets
and
replica
sets
manage
pods
and
when
you
first
create
your
deployment,
you
get
one
replica
sets
with
all
version
one
creating
multiple
as
many
pots
as
you
want
of
this
particular
version.
A
So
next,
when
you
update
the
diplom
spec,
you
get
a
new
replica
set
created
and
this
new
replica
set
is
scaled
out,
while
the
old
is
being
scaled
up
and
down,
and
in
the
end
you
again
have
just
one
replica
set
that
is
active
and
serving
all
creating
all
the
pods
and
making
sure
that
the
certain
number
of
replicas
are
running.
The
old
replica
cells
are
normally
not
deleted
straight
away.
It
depends
on
the
revision
history
limit
parameter
in
the
des
plaines
back,
but
for
the
purpose
of
this
talk,
it's
not
really
important.
A
So
next
continuous
deployment.
So
with
the
continuous
deployment
the
the
thing
is
that
we
don't
just
focus
on
the
happy
path
where
and
we
our
rolling
update,
for
example,
works
as
except
expected,
and
we
don't
have
any
bugs.
The
more
important
part
is
the
automation
of
the
are
allowed
and
rollback
and
for
the
rolling
update
the
traffic
timeline
looks
like
this.
A
We
eventually
replace
all
the
pods
of
version
1
with
the
plaza
version
2
and
once
we
are
done,
we,
although
the
pods,
are
running
a
new
version
of
the
application
and
in
case
of
a
bad
release,
what
happens
that
as
soon
as
we
detected
T
some
some
issue,
we
roll
back
the
deployment
and
we
back
effectively.
We
do
the
reverse.
We
create
new
pods
of
the
old
version
which
replace
the
the
version
2.
A
So
next
question
is:
how
do
we
detect
issues
in
productions,
so
we
use
metrics.
For
example,
we
can
monitor
a
number
of
failed
requests
over
the
last
five
minutes
and
based
on
the
of
this
number,
we
can
make
a
decision,
whether
there's
in
some
regression
in
the
functionality
and
the
next
question
is,
as
you
have
seen
you
they.
A
If
there
is
a
new
bed
release,
it
affects
all
users
100%
of
the
users,
and
it
would
be
nice
to
reduce
the
impact
in
case
of
bad
release
and
to
achieve
that,
we
can
use
some
custom
diplomat
search
this
rather
than
just
a
single
different
object
and
the
way
we
can
implement
custom
diplomat
strategies
is
just
by
creating
multiple
different
objects,
which
gives
us
more
flexibility.
You
have
seen
you'll
see
it
on
some
of
the
next
slides,
so
the
first
strategy
is
blueberry
in
deployment
where
we
create
to
deployment
objects.
One
is
labeled
as
blue.
A
A
So
when
we
need
to
roll
out
a
new
version,
we
update
the
the
inactive
deployment
which
is
green
on
this
picture,
and
once
the
deployment
is
ready,
we
can
run
some
tests
against
it
to
make
sure
that
some
trivial
happy
scenario,
for
example,
passes
and
once
we're
confident
that
it
looks
fine.
We
update
this
service
port
selector
so
that
now
all
the
requests
from
the
users
are
being
served
by
the
green
deployment
and
blue
deployment
is
inactive.
So
next
time
we
will
need
to
update
the
our
application
to
version
3.
A
Let's
say
we
will
be
using
blue
deployment,
so
we
will
update
blue
deployment
with
a
spec
of
version
3
and
eventually
again
will
update
service
pod
selector
and
now
the
glute
deployment
is
and
is
active.
So
we
repeat
this
operation
every
time,
and
this
is
how
we
can
implement
Google
deployment
in
communities.
A
Traffic
timeline
for
blue
green
deployment
looks
like
this:
the
cut
over
his
instantaneous
because
we
only
update
our
pod
selector
once
the
deployment
is
ready,
so
it
has
been
scaled
up
with
it,
with
the
new
pods
and
in
case
of
red
release.
The
rollback
is
also
instantaneous,
since
we
keep
the
old
deployment
running
either
for
error
or
for
certain
period
of
time.
We
can
quickly
rollback
once
we
detect
that
the
issue
next
two
days
can
I
release
and.
A
A
A
A
Now,
let's
talk
a
bit
a
little
bit
about
the
automation
for
this
custom
deployment
surgery.
How
do
they
automate
the
deployment
rollout?
So
we
I
think
the
most
common
pattern
being
widely
used
is
just
implementing
this
logic
on
this
side
of
si.
Si
do
si.
Do.
For
example,
you
can
write
a
groovy
script
in
Jenkins
or
bamboo
which
will
manage
this
deployment
objects.
So
let's
first
have
a
look
at
how
this
script
would
look
like.
It
would
be
perfectly.
A
We
will
be
able
to
just
use,
keep
city
reply,
which
is
a
fact
or
keep
control
our
code
to
use
a
declarative
way
of
updating
the
deployment,
spec
and
other
objects
as
well.
It
means
that
we
have
minimal
logic
in
the
in
our
CI
CD,
but
the
problem
here
is
that
we
cannot
actually
update
both
deployments.
A
At
the
same
time,
we
need
some
extra
logic,
so,
for
example,
after
updating
deployment,
a
canary
deployment
object,
we
sleep
for
some
period
of
time,
wait
until
all
the
my
it
processes,
a
certain
number
of
requests
that
the
metrics
are
now
up
to
date
with
check
the
metrics,
and
then
we
make
a
decision
whether
to
proceed
or
rollback.
So
are
there
any
other
options?
First
of
all,
I'd
just
like
to
ask
audience
if
anyone
has
brought
some
code
against
given
a
tcpi
using
climb
goal
library,
can
you
please
raise
your
hands?
A
So,
instead
of
writing
skills
creep
on
CI
CD
tool,
we
can
actually
implement
the
automation
on
the
community
side
and
to
do
that,
we
can
implement
the
CDP
Rd
controller,
also
known
as
a
curator
I
think
it
was
a
very
popular
target
at
this
conference,
so
most
people
are
probably
already
familiar
with
it.
So
why
CRE?
Just
to
name
a
few
benefits
of
series
declarative,
we
describe
the
desired
state
rather
than
the
steps
to
reach
it,
self-healing,
meaning
that
communities
control
of
project
will
take
care
of
making
sure
that
we
have.
A
We
reach
the
final
state,
it
will
retry
if
needed
and
finally
reusability.
So
if
you
build
this
year
as
early
as
a
building
block,
you
can
use
it
along
with
other
series
or
communities
native
resources
and
in
the
same
way
as
you
would
do
with
any
other
communities
resources.
So
let's
just
go
back
to
the
Canario
deployment
picture
and
see
what
we
actually
need
to
automate
to
implement
this
theory.
A
So,
first
of
all,
let's
forget
for
a
moment
about
replica
sets
and
posts
because
they
are
managed
by
deployments
a
controller,
and
now
these
are
the
objects
we
need
to
manage
for
our
Canary
deployment
controller
next
services,
service
and
linguist
objects
are
actually
very
effective.
Aesthetic
we
just
apply
them
at
once.
We
don't
need
to
update
them
every
time
we
ever
have
a
new
version
of
the
application,
so
we
only
need
to
manage
deployment
objects,
and
this
is
where
the
new
Sierra
comes
in.
A
We
can
implement
a
new
canary
diplomacy
ID
with
a
controller
that
will
take
care
of
the
deployment
objects.
And
finally,
we
have
this
type
of
picture
where
all
of
these
objects
can
be
managed.
Declarative,
declaratory
and
particular
controller
will
take
care
of
them.
So
in
the
end,
this
is
how
your
CI,
a
CDA
script,
would
look
like,
and
this
is
exactly
what
we
wanted.
It
has
all
the
declarative
apply
without
extra
any
extra
logic
and
specifics
of
which
monitoring
tool
you
use,
for
example,
or
anything
like
that.
So
next
I'll
be
talking
about
metrics.
A
So
when
not
all
controller
needs
to
integrate
with
something
metrics
API,
the
question
is
which
one
to
choose,
so
the
most
common
is
probably
in
open
source
is
Prometheus
for
communities,
but
the
rels
stick
driver
for
Google
cloud,
for
example,
or
cloud
watch
in
AWS.
There
are
data,
Dirk
and
other
tools,
and
all
of
them
have
their
own
api's
and
like.
If
we
just
integrate
with
one
of
these
tools,
it
doesn't
scale
we'll
have
to
add
more
and
more
every
time
we
need
a
new
metrics
API.
A
What
a
parenting
community
offers
a
set
of
standard
metrics
api's,
which
we
can
use
and
if
you
are
familiar
with
horizontal
proton,
a
scalar
it
be
a
resource.
This
is
the
one
which
is
currently
using
cuny's
metrics
api's,
and
this
is
the
resource
which
you
configure.
You
provide
a
reference
to
the
deployment
and
let
the
controller
of
HPA
scale
up
or
down
your
deployment
based
on
the
metrics,
normally
CPU
and
memory,
but
there
are
other
ways
as
well.
So
this
is
how
HPA
controller
works.
It
talks
to
the
three
standard
communities.
A
Metrics
ap
is
and
manage
the
deployment
object.
Similarly,
we
can
do
exactly
the
same
for
our
controller.
We
can
also
talk
to
the
same
api's
about
instead
of
scaling
up
and
down,
we
make
a
decision
whether
to
proceed
or
roll
back
with
a
different,
so
first
metric
CPI
is
a
resource,
metrics
API
that
is
specific
to
pods
and
notes,
and
it
just
provides
CPU
and
memory
utilization
metrics.
A
Next,
there
is
a
custom
metrics
API
that
provides
arbitrary
matrix
for
any
community
resource,
for
example,
pod
service
and
the
ingress,
and
this
is
actually
the
most
interesting
metric
CPI
for
us,
because
this
is
effectively
an
abstraction
layer
where
you
can
implement
an
adapter
for
your
particular
metric
solution.
For
example,
there
is
already
an
open-source
parameters
adapter
which
you
can
configure
as
a
implementation
for
Prometheus.
A
So
you
can
use
that
as
well.
So,
finally,
let's
talk
about
Canary
deployments,
Eid
format,
and
this
is
how
the
standard
deployment
looks.
Like
probably
most
people
here
are
familiar
with
it.
It
has
a
number
of
replicas,
it
has
a
selector
for
the
ports
to
manage
by
this
deployment
and
it
has
a
bottom
plate
for
the
pods
aspect
to
be
created.
So
we
can
reuse
the
same
structure
for
our
Canary
deployment
resource.
It
also
has
self
selector
in
template
and
it
has
some
extra
fields
specific
to
Canary
and
stable
tracks.
A
So
let's
have
a
look
at
the
specifics
of
they
Canary
deployment
configuration.
These
are
the
two
sections.
I
will
take
a
look
now
and
these
two
sections
have
separate
numbers
of
replicas
to
manage.
For
example,
we
want
to
run
just
one
port
or
canary
deployment.
Even
we
want
to
run
five
calls
for
stable
deployment.
There
is
also
configurable
labels
to
make
sure
that
service
can
be
can
route,
the
request
for
tea
for
Basel
particular
deployment.
And
finally,
there
is
a
section
for
metrics
and
first
we
can
figure
how.
A
How
long
do
we
want
to
wait
until
we
check
the
metric
values
once
they
we
updated
the
canary
deployment,
and
then
we
provide
a
list
of
metrics
to
check
against.
So
here
is
a
service
called
foo
canary
and
the
metric.
We
want
to
check
is
the
request
failure
rate,
so
the
target
value
here
says
that
if
we,
if
the
number
of
failed
requests
over
the
last
minute,
has
exceeded
10%,
we
will
mark
this
deployment
as
failing
and
we
will
need
to
roll
back
so
time
for
the
demo.
A
Just
a
little
bit
more
details
on
the
demo
I
have
implemented
so
for
my
demo.
I
am
running
our
community
fast
on
my
local
machine
and
it's
using
Prometheus
as
with
Prometheus
adapter
is
an
implementation
of
custom.
Metrics,
API
and
also
I
have
installed
Gahanna
just
for
a
better
visualization
for
the
demo.
A
The
demo
is
available
on
github.
You
can
check
it
out.
It's
both
the
the
demo
in
control
itself,
so
everything
is
on
github.
Please
check
it
out.
Leave
a
feedback,
radius,
npr's
and
so
on
at
this
moment
is
mostly
a
proof
of
concept,
but
I
would
be
happy
to
make
it
production
radius.
All
so,
let's
see
yeah
I'm,
afraid
I
don't
have
as
good
internet
as
Kelsey
this
morning,
but
hopefully
the
Wi-Fi
is
not
about.
We
stop
downloading
your
movies.
If
you
do
so.
A
With
extra
companies
installed
so
here,
I'm
writing
a
communities
in
docker
and
I
have
installed
a
bunch
of
companies,
including
contour,
for
the
ingress
with
weighted
load
balancing.
There
is
a
graph
on
Oh
ramita
separator
for
running
your
Prometheus
on
inside
communities.
Cluster.
There
is
a
Prometheus
adapter,
which
runs
as
custom,
API
server
inside
communities
and
finally,
I
have
installed
an
application,
a
time
application
to
show
to
showcase.
How
does
it
all
work?
So
the
application
I'm
using
here
is
very
simple.
It
just
shows
the
version
number
of
the
application.
It's
every
single.
A
So
here
we
can
see
that
all
of
our
requests
for
version
1
here
are
succeeding.
So
it's
a
healthy
version
of
the
application.
The
next
version
I
will
be
using
for
showcasing
the
canard
opponent
is
a
version
2,
which
is
a
bad
release
effectively
and
it
needs
the
failing
metrics.
So,
let's
check,
if
it
does
so
here,
you
can
see
that
most
of
the
requests
are
failing
and
finally,
I
have
a
version.
3
of
the
vacation
that
just
generates
different
kinds
of
emergency,
but
it's
a
happy
version
of
the
application.
A
So
this
is
the
application
that
is
configured
with
ingress
currently
and
if
we
check
the
dashboard,
all
of
the
requests
are
processed
by
version
1.
There
are
no
errors,
and
here
is
a
breakdown
on
the
left.
You
can
see
the
canary
pods
and
metrics
and
on
the
right
there
is
a
stable,
Basel
metrics.
So
there's
one
canary
pod,
three
stable
ports.
All
of
them
are
succeeding
and
both
running
version
1
of
the
application.
So,
let's
see.
A
What
do
we
have
here
for
the
demo
for
our
application?
So
this
is
just
a
watch
for
the
ports.
All
of
my
application
here
are
just
for
pause
once
canary,
through
stable
or
running
version
1,
and
to
configure
the
ingress,
for
example.
What
I
do
here
is
I
just
specify
the
wait
for
canary
service
wait
for
this
table
service,
so
20%
of
the
requests
are
served
by
the
canary
a
deployment
and
the
rest.
80%
are
served
by
the
stable
deployment.
A
Next
I
have
also
configured
parameters
here,
with
an
endpoint
to
watch
against
and
the
labels
for
to
find
the
services
that
we
need
to
collect,
metrics
from
and
finally
we'll
also
here,
I
configured
custom
metrics
that
to
derive
from
the
metrics
that
I
exposed
by
exposed
of
the
dime
application.
So
here,
I'm,
just
collecting
the
failure
rate
over
the
last
minute
and
based
on
this
metric
I
will
make
a
decision
whether
to
proceed
or
rollback.
And
finally,
this
is
the
instance
of
the
canary
deployment
we
have
here.
A
So
we
have
selector
template
as
usually
we
have
in
this
pack.
We
have
version
1
of
the
application,
and
here
we
have
configured
the
metrics
to
before
to
wait
for
one
minute
before
we
check
the
metrics.
Here
is
the
name
of
the
metric
which
I
have
just
showed
you
in
the
Prometheus
row,
and
the
value
is
10%.
A
So
now,
let's
try
updating
our
canary
deployment
to
version
2,
which
is
I,
showed
you
before
is
a
bad
version,
so
I
just
updated
the
docker
image,
I
applying
the
change
and,
let's
see
what
happens
in
our
watch
immediately.
We
see
that
version
2
of
the
placate,
a
pod
of
version
2,
has
been
created,
it's
not
ready
yet,
which
means
that
they
all
body
still
running,
while
once
it
becomes
ready.
The
pod
of
Russian
top
2
will
be
sorry.
The
old
port
of
version
1
will
be
terminating.
A
So
let's
go
back
to
our
your
phone
and
the
report.
First,
let's
just
check
what
happened
has
happened
to
our
application,
so
20%
of
the
requests
are
served
by
the
Canary
deployment
and
we
see
that
it
actually
arrives
version
2
of
the
application,
so
some
of
the
requests
are
processed
by
the
version
2
of
the
application.
A
Here
we
see
that
the
portal
version
1
has
been
replaced
with
the
provision
2,
and
here
we
see
that
actually
we
see
more
and
more
failures
over
the
last
minute,
while
the
stable
port
is
still
running
version
1,
so
everything
is
fine
with
a
stable
one,
but
the
cannery
is
running
unhealthy
version.
Here
you
can
see
the
breakdown
of
the
total
requests,
so
20%
and
requests
are
now
processed
by
the
version
2
of
the
application
and
the
rest
requests
sto
process
by
the
version
1.
A
A
We
need
to
rollback,
and
what
we'll
see
here
is
that
version
two
of
the
application
of
the
port
is
now
terminating
while
the
version
1
has
been
created,
just
30
seconds,
60
seconds
old,
so
we
have
rolled
back
to
the
previous
state
of
the
system
and,
let's
see
what
happens
here
so
as
I
showed
you
I
just
did
keeps
it
up.
I
declared
way.
A
So
now,
let's
see
if
we
will
be
able
to
update
it
to
the
version
3,
which
is
healthy
one,
so
I
just
update
the
docker
image
to
version
3
and
because
it
keeps
it
apply
and
wash
with.
What
should
we
see
here?
Is
the
Russian
three,
what
has
been
created
exactly
the
same
ways
as
we
seen
before
with
version
2
once
it's
ready?
A
It
will
stop
processing
requests
and
the
old
port
of
version
1
will
be
terminated,
yep.
That's
what
we
see
here
again
in
our
event
block
we
see
that
controller
is
waiting
for
one
minute
and
you're
making
decision,
whether
to
proceed
to
the
stable
deployment
or
rollback,
and
if
we
go
back
to
our
application,
which
we
should
see
that
20%
of
the
requests
should
be
processed
by
the
version
3.
A
Are
we
unlucky?
Okay,
yeah?
Here
we
go
so
20%
of
the
requests
are
processed
by
the
canary
deployment,
which
runs
for
some
three
now
and
if
we
go
here,
you'll
see
this
picture
exactly
so
we
we
were
running
version
1,
then
version
2
and
we
rolled
back
and
now
we
are
writing
version
3
and
in
the
arrow
breakdown
we
see
that
actually
everything
seems
fine.
We
don't
see
any
new
errors.
So
after
a
minute
we
should
be
able
to
proceed
with
a
rollout
if
I
go
back.
A
I
should
see
that
yep,
the
matter
check
was
successful.
Now
the
controller
has
updated
the
stable
deployment,
and
now
we
should
see
all
of
the
ports
of
version
1
of
the
stable
deployment
replaced
by
version
3,
so
eventually
we'll
see
that
we'll
see
3
or
3
ports
of
version
3
for
the
stable
deployment
and
all
the
positive
version
1
being
terminated
once
this
is
done,
we'll
see
that
actually
all
of
the
requests
are
being
processed
by
the
new
version
of
the
application.
A
A
So
on
the
traffic
timeline,
what
we'll
see
what
we
will
see
is
that
eventually,
all
of
their
requests,
all
of
the
user
requests-
will
be
processed
by
the
version
3
and
if
you
go
to
our
application,
this
is
what
we
see.
100%
of
the
requests
are
being
served
by
the
version
3,
and
this
is
the
success
path
for
our
Canary
deployment.
Controller.
Ok,
I
think
we
are
done
with
the
demo
and
now,
let's
switch
back
to
the
slides
for
a
moment.
A
So
these
are
some
links
from
my
demo.
First,
this
is
the
canary
deployment
controller
that
I
have
implemented.
Please
check
it
out,
and
there
are
some
other
companies
I
have
used
for
this
demo.
The
interesting
part
here
is
that
parameters:
adapter
is
a
custom.
Metrics
API
server
and
from
it
is
a
curator
is
a
CRT
controller,
as
well
as
help
to
control,
which
is
also
CID
controller,
so
I
have
used
CID
controllers
created
by
other
from
either
other
companies
of
other
people
and
I
have
just
integrated
with
them,
reuse
them.
A
So,
first
of
all,
we
reuse
existing
resources,
no
need
to
reinvent
the
wheel,
meaning
that
in
Canary
deployment,
controller
I
haven't
been
creating
replicas,
as
opposed
directly
I
have
just
reused
the
normal
deployment
which
runs
rolling
update,
and
it
worked
perfectly
next.
Keep
it
simple
so,
rather
than
trying
to
encapsulate
the
entire
logic
of
your
canary
are
allowed
in
your
theory,
it
might
be
a
better
idea
to
just
implement
this
subset.
That
needs
to
be
automated
like
in
this
example,
I
don't
manage
service
and
ingress
objects,
I
only
own
manage
deployment
objects.
Next
use
abstractions.
A
In
this
example,
I
have
used
given
its
metrics
API,
which
allows
for
a
lot
of
extensibility.
You
can
integrate
with
any
other
any
metrics
provider
without
a
single
line
of
code
in
CID
controller
and
finally
use
the
power
of
open
source.
I
have
read
lots
of
in
communities,
mana
rapport,
primarily
primarily
a
deployment
controller
code.
You
reuse,
some
patterns
and
I
have
uploaded
my
code
on
github
as
well,
so
you
can
use
it
as
well,
so
yeah
just
follow
the
best
practices
and
you'll
be
good.
A
B
A
So
so
the
question
the
question
is:
once
the
canary
deployment
has
passed,
the
initial
matter
check
whether
the
controller
keeps
monitoring
it
so
for
this
proof
of
concept,
I
didn't
implement
that,
but
obviously
you
can
be
extended
and
make
it
I
think
it
can
be
made
optional
because,
probably
not
in
all
cases,
you'd
want
to
keep
monitoring
by
the
controller
it
could
be
ad
like
normally
once
you
have
finished,
you're
allowed.
I
think
the
better
way
is
to
set
up
some
alerts,
rather
than
full
controller,
to
keep
monitoring
the
the
object
so
yeah
I.
C
A
Yes,
yes,
the
great
question,
so
the
question
is
whether,
instead
of
using
ingress
capture
control
controller,
you
can
use
East
Joe
yeah,
actually
both
East
you
and
contour
eye
using
envoy
underneath
so
it's
like
internal.
It
will
be
exactly
the
same
solution
so
actually,
for
this
is
the
perfect
example
of
this
Yeti
control
that
it
doesn't
manage
ingress,
so
you
can
just
implement
ingress
with
SEO
or
any
other
way.
You
are
out
your
request
to
the
final
to
the
deployment
and
it
will
just
work
fine
without
any
code
change,
yep.
A
Right
yeah,
so
the
question
is
why
the
standard
deployment
only
supports
could
recreate
enrolling,
updated.
It
doesn't
support
blue,
green
or
canary
or
any
other
advanced
researcher.
I
think
the
answer
is
that
it's
probably
quite
hard,
like
everybody,
understand
the
concepts,
the
generic
concept
of
Bluegreen
deployment
and
canary
deployment,
but
most
people
have
different
implementation,
details
and
I
think
it's
pretty
hard
to
get
the
abstraction
right.
What
actually
and
at
this
point
they
were,
for
example,
metrics,
IP,
I
I,
believe
didn't
even
exist.
A
A
A
Yep,
so
in
terms
of
ppi,
it's
it's
very
easy
to
do.
This
was
mostly
a
proof-of-concept
I
have
implemented
like
within
the
last
month,
but
yes,
we
can
easily
extend
it.
So
the
questions
are,
it
was
whether
we
can
check
the
metrics
not
just
for
canary
deployment
but
profit
stable
deployment
as
well.
Yes,
it's
very
easy
to
implement
it.
We
can
just
reuse
exactly
the
same
code
and
we
just
provide
the
name
of
the
department
which
is
a
stable
one
rather
than
canary.
A
With
HTTP
sure
Oh
with
HPA
yeah,
also
a
great
question
so
currently
I
think
the
my
controller
will
be
actually
conflicting
with
the
HPA
because
they
will
be
fighting
against
the
replicas
value,
but
I
yeah,
actually
I
have
to
thought
about
that.
But
I
haven't
implemented
yet
yet
yeah
I
think
we
need
to
have
some
kind
of
flag
saying
that
if
we
replicas
like
don't
touch
the
replicas
field
or
something
like
that
yeah,
it's
very
easy
to
implement.
A
So
the
question
is,
we
do
apply
the
declarative
state
in
our
script,
but
now
our
desired,
like
our
desired
spec,
doesn't
match
the
reality.
But
this
is
kind
of
part
of
the
design
of
communities
in
community.
You
have
spec,
which
is
affected
by
a
desired
state
of
the
status
which
is
actual
state
and,
in
my
case
I,
don't
roll
back
this
back
of
the
canary
deployment
object,
which
manages
the
deployment
underneath
right.
A
So
it
still
says
that,
yes,
the
desired
state
is
still
version,
even
if
it's
failing,
but
the
status
will
say
that
hey
I
cannot
make
it
happen
because
you
have
a
better
image.
So
I,
don't
roll
back
the
canary
deployments
back
itself.
I,
don't
touch
it
at
all,
just
to
be
clear
that
this
is
I,
think
it's
consistent
with
community
CPI,
where
you,
rather
than
touching
this
back.
You
just
report
on
this
status
that
something
is
wrong.
Please
go
and
fix
your
speck.
A
The
question
is
whether
I
considered
integration
with
ham
releases.
So
the
answer-
the
short
answer
is
no,
but
I,
think
that
it
is
possible,
for
me,
is
more
of
that
ham
is
hell,
is
probably
the
wrong
level
of
traction
for
this
type
of
theory,
because
theory
is
managing
like
unisys
objects
directly
rather
than
using
so
it
you
can
use
hell
on
top
of
this
theory,
for
example,
in
your
help
chart
rather
than
having
a
deployment
Hammel,
you
can
declare
a
canary
deployment
URL,
and
this
is
also
should
work,
but
in
between
canary
deployment
and
deployment.
A
Yeah,
so
the
question
is
a
deployment
there
are
more,
probably
less
known
configuration
files,
for
example,
progress
deadlines,
I
think
for
now,
I
just
copy
the
progress
there
live
from
canal
deployment
back
up
to
both
deployments
and
like
well,
communities
take
care
of
these
things,
but
yeah.
We
can
probably
improve
that
as
well.
A
Right
so
the
question
is
HP
is
like
also
using
metrics
and
make
it
makes
the
decision
whether
to
scale
up
or
scale
down
right
and
during
the
rollout.
It
could
happen
that,
for
example,
HP
a
will
just
like.
Let's
say
that
your
new
application
is
running
out
of
memory
constantly
and
HP
a
controller
keeps
up
just
it's
going
up
and
up
and
up,
and
it
means
that
metric
can
can
be
still
showing
that
the
applications
healthy
when
it's
not
and
I,
think
for
that
we
can
just
like.
A
First
of
all
in
the
HP,
you
can
specify
the
max
number
of
replicas
you're
happy
to
run,
and
next
here,
I
have
a
delay
to
check
so
I,
like
I,
think
you
can
increase
the
delay,
for
example.
So
if
HP
a
scales
up
for
like
5
minutes
and
10
mins
later,
is
you
see
that
even
with
hundreds
of
replicas,
they
still
running
out
of
memory?
This
is
definitely
a
reason
to
roll
back.
So
yeah,
it's
a
bit
tricky,
but
I
think
you
can
get
it
right
with
the
product
numbers.
A
Yeah
yeah,
so
this
is
actually
a
different
way
of
implementing
canary
deployment
and
it's
definitely
possible
because
a
CRT
controller
only
manages
deployment
objects,
it
doesn't
manage
the
percentage
of
traffic,
so
you
can
just
imagine
updating
your
ingress
route,
which
has
weighted
weights
configured
right.
You
can
have
some
other
logic,
control
or
whatever,
which
will
update
this
percentage
dynamically
yeah,
it's
not
only
possible.
It's
like
it's
not
specific
to
this
controller.
Actually.