►
From YouTube: Toward Full Adoption of GitOps and Best Practices at RingCentral - Ivan Anisimov & Tamao Nakahara
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Toward Full Adoption of GitOps and Best Practices at RingCentral - Ivan Anisimov, RingCentral & Tamao Nakahara, Weaveworks
This talk will cover RingCentral’s current journey and commitment to adopt GitOps and best practices in the AI department. Ivan will share their needs for managing infrastructure at scale, serving the internal developers, implementing progressive delivery such as canary deployments, and managing hybrid clouds. All of these choices benefit the team and RingCentral as a company: money savings, greater security, reliability, faster troubleshooting, and more. This session will cover how the team does this by using a range of tools such as Kubernetes, Flux, Kustomize, Terraform, Istio, Prometheus, and more, as well as some decisions they are making to ensure a Kubernetes-first architecture. The session will include considerations for how the tools are designed, how that helps to drive best practices, and pitfalls to avoid. By enabling both technical and business benefits, the GitOps journey at RingCentral is helping them to focus on innovation, increased velocity, and the company’s success.
A
Hi,
my
name
is
Ivan
and
I'm,
director
of
engineering,
kindering
Central,
so
I
work
in
different
roles
in
central
I
work
as
a
software
developer,
software
architect
team
lead
product
manager
like
everything,
is
manageable
and
right
now,
I'm,
working
internal
and
advocating
can
implementing
github's
processes
across
the
whole
company.
So
it
will
be
my
pleasure
to
share
with
you
our
experience
how
we
approach
githubs
in
a
large
Enterprise
company,
thousands
upon
thousands
of
users,
huge
workloads
and
what
challenges
we
faced.
B
B
It
links
to
a
site
where
we
have
all
the
flux
things
for
this
week,
whether
it's
our
flux,
booth
and
all
these
talks
and
activities
that
we
have
and
for
those
of
you
watching,
the
recording
we'll
still
have
the
site
up
so
that
when
the
recordings
are
up,
hopefully
you
can
see
it.
It
also
has
ways
for
you
to
follow
up
with
questions.
So
please
do
chat
with
us.
I'll
show
this
at
the
end
as
well.
B
So,
first
of
all,
security
and
compliance,
so
RingCentral
is
a
company
that
absolutely
needs
security
and
compliance
and
git
Ops
helps
with
that.
First
of
all,
if
you
don't
know
they
have
multiple
products,
there's
telephone
and
messaging
and
video
I'm
sure
many
of
you
have
worked
at
companies
where
you
actually
have
those
physical
phones
as
well
as
other
capabilities.
So
imagine
it's
a
company
that
has
a
chief
compliance
officer.
It
has
Auditors
both
inside
the
company
and
outside
and
so
auditability
and
observability
are
absolutely
required,
needs
that
they
have
they
both.
B
They
also
have
Legacy
and
modern
systems
which
brings
with
it
dependencies
and
requirements
and
configs
and
secrets
management
stuff,
which
brings
certain
levels
of
complexity
that
they
need
to
take
care
of,
and
similarly
because
of
that
they
have
multiple
Solutions
and
products
that
bring
those
same
challenges.
So
that
means
they
absolutely
need
to
have
a
reliable,
get
UPS
system
and
tools
like
flex
help
with
that
and
to
be
able
to
give
them
the
security,
compliance,
auditability
and
policy
management
that
they
need.
That
will
cover
in
this
talk,
especially
when
you
have
human
beings
involved.
B
It's
just
not
gonna.
Everything
that
you
see
on
the
left
here,
the
things
that
they
need
to
do
is
not
going
to
happen
if
they're
chasing
after
human
beings
of
like
who
made
what
change
under
what
context.
When
you
know
all
that
is
too
manual,
and
that
would
not
be
able
to
be
possible,
so
flux
helps
with
that
so
quickly.
B
If
has
anybody
not
used
flux,
yet
right
so
great
if
you're
new
to
flux,
we'll
kind
of
segue
and
share
little
bits
about
it,
so
flux
is
a
project
in
the
cncf,
we're
incubating
and
actually
right
after
kubecon,
we'll
be
finishing.
The
final
steps
we
hope
to
get
to
graduation.
So
it's
been
a
wonderful
Journey
here
in
the
cncf
and
flux
like
I
said
it
comes
from
weworks
and
the
company
that
coined
git
Ops.
B
So
it's
been
providing
git
Ops
both
for
apps
and
infrastructure,
and
one
thing
that
it
does
is
it
listens
to
the
repo
and,
if
there's
a
change
in
that
their
repo
as
a
single
source
of
Truth,
it
talks
with
kubernetes
and
implements
that
change.
So
it
kind
of
created
that
that
idea
of
having
that
single
source
of
Truth
in
git
and
because
you
already
have
a
version
controlling
system.
That
means
you
already
have
an
audit
Trail,
that's
part
of
that.
B
So
there
we
go
about
the
need
for
auditability
and
also
provides
added
security
because
you're,
not
exposing
your
API
to
the
outside
world
and
you're,
also
not
putting
your
secrets
in
CI,
and
if
people
have
been
following
the
the
news
you
know
more
and
more.
Hacking
is
starting
to
move
in
the
CI
space,
because
they're,
seeing
vulnerabilities
there
and
being
able
to
access
a
lot
of
data.
So
flux
is
already
you
know,
steps
ahead
of
this
and
always
making
sure
that
we're
evolving
so
that
we
meet
those
security
needs.
A
Okay,
yeah
and,
like
tomorrow,
said
we
will
start
firstly
with
security
in
mind,
because
if
you
work
in
a
large
Enterprise
and
if
you're
a
large
Enterprise
when
you're
trying
to
implement
some
delivery
tools
or
development
tools
which
tightly
coupled
with
your
production
system,
I
think
the
first
question.
Everyone
answers
is
what
you
will
do
with
compliance
and
security
and
I
would
not
talk
only
about
just
the
compliance.
I
will
talk
about
actual
security
because
ring
central
is
a
as
you
can
see
from
this
map.
A
Every
country
marked
in
Orange
we
actively
operating
in
you,
can
buy
all
services
and
most
likely
every
one
of
you
in
your
life,
encountered
or
dialed
to
the
phone
line
or
contact
center
line
which
operated
or
hosted
by
the
ring
central.
So
we
like
biggest
Telephone
Company
in
the
United
States,
at
least
on
unified
communication
as
a
market
space.
A
So
when
we
yeah
so
when
we're
trying
to
tackle
such
a
DVRs
geography
and
Tackle
different
types
of
compliance,
so
we
are
creating
government
and
health
care.
So
it's
a
HIPAA
compliance
different
sub-compliance.
We
have
public
company,
so
you
can
imagine
it's
like
audit
upon
audit
and
we
have
it
constantly
during
our
years.
We
also
always
try
to
Target
it
by
the
hackers.
We
have
different
types
of
attempted
breaks
in
our
system
and
we
want
to
make
sure
all
our
clients
are
safe
and
every
data
is
protected.
A
So
the
first
first
and
most
important
question
is
the
security,
not
the
scale,
not
any
other
ones.
But
if
you
fail
at
security,
you
basically
do
not
need
to
use
this
tool
or
if
you
cannot
answer
this
question
so
when
we
talk
before
we
talk
about
the
security
itself
and
how
Flagger
and
flags
approach
it
I
want
just
to
segue
into
one
particular
important
part
we
use
and
we
use
it
I
think
it's
very
tight
to
all
success
of
using
githubs
in
the
company
and
we
use
it
everywhere.
A
It's
a
grammatic
build
so
we're
using.
If
anyone
is
views
familiar
with
build
systems
such
as
Basel
or
nixos,
or
just
a
term
of
hermetic
build,
it
means,
regardless
of
the
environment,
regardless
of
the
computer.
You
run
you
build
on
or
build
your
contain,
your
own
daytime,
different
libraries.
You
get
bit
to
bit
exact
copy
of
the
container
if
you
build
from
the
same
source.
So
what
does
it
means?
The
same
source
source
is
a
source
of
truth,
it's
all
source
code.
A
So
if
we
have
everything
which
needs
to
be
so,
it's
no
Dynamic
variables
Dynamic
dependencies,
everything
is
pinned
down
and
if,
if
we
have
everything
pinned
down
and
inside
the
repository,
we
can
build
a
geometry
container
and
we
can
argue
it's
basically,
we
can
build
on
10
different
machines.
It
should
be
bit
to
bit
exact
replica
if
something
goes
wrong
and
the
replica
is
not
exact,
it
means
the
machine
could
be
compromised
to
the
process
is
compromised
and,
as
you
can,
and
it's
a
development
practice
Yeah.
A
It's
not
have
nothing
to
do
with
githubs
yet,
but,
as
you
can
see,
we
how
we
go
from
here
to
githubs,
because
in
our
development
life
git
repository
is
a
source
of
truth.
Yes,
it's
a
source
of
all
source
code
to
know
the
dependencies,
so
we
wanted
to
do
the
same
in
our
infrastructure,
so
how
we
can
approach
it
in
our
infrastructure
and
here
is
devops
or
githubs,
helps
us
and
answered
questions.
It's
a
little
bit.
Complicated
image.
I
think
with
slides
will
share.
A
A
So
going
from
here,
we
can
definitely
see
from
the
security
point
of
view,
we're
also,
while
we're
using
Flags
we're
implementing
different
security
practices
such
as
static
scan
for
the
Civil,
vulnerabilities
and
civilian
lists.
So
if
we
have
every
dependency
pin
down,
we
can
easily
do
the
reverse
scan,
which
helps
the
security
a
lot.
So
what
does
it
mean?
It
means
if
you
have
zero
days
as
vulnerability
in
your
in
your
system.
You
can,
and
you
know,
every
desire.
A
It
is
no
desired
state
of
our
cluster
and
if
desired,
State
diverges
it's
a
incident,
so
we
we
manage
and
with
basically
always
monitor
the
version
of
the
state.
But
if
we
have
the
desired
State
and
the
repository
we
can
go
and
see
on
the
release
of
zero
day
vulnerability
and
know
exactly
where
this
vulnerabilities
existing
or
thousands
of
clusters.
So
we
do
not
need
to
time
to
scan,
because
if
we
scan
every
system
for
vulnerability
After,
we
receive
the
Civil
vulnerability.
A
It
can
take
weeks
in
a
large
company
such
as
ours,
so
it
definitely
helps
us
to
with
detecting
the
Civil
vulnerabilities
abilities.
We
also
employ
flux.
Integration
with
Mozilla
stops.
So
if
you
don't
know,
it's
really
great
tool
to
keep
your
secrets
inside
the
git
repository
sounds
counterintuitive,
but
it's
very
nice
thing
very
nice
solution
to
keep
things
Atomic,
so
it
changes
I'm
not
distributed
among
different
repository
like
a
secret
world
and,
for
example,
your
git
repository,
it's
all
in
one
place,
an
atomic
State
and
a
git
car
made.
A
So
we
do
not
wait
with
solely
adhere
to
the
practice
of
storing
every
fan
everything
can
get,
and
it's
helped
us
and
help
security,
compliance
and
security
audit
on
our
company.
It
made
it
a
lot
easier.
All
the
trails,
all
the
changes
and
git
merge
requests
is
a
change
request.
So,
if
you,
for
example
Enterprise,
you
know
this
change
management
system,
Etc
so
merge
request
is
completely
a
Maps
one
to
one
and
we
managed
to
convince
all
our
Auditors,
such
as
KPMG,
pricewaterhousecoopers
and
different
other
Auditors
and
our
internal
security
teams.
A
B
All
right
so,
hopefully
company,
it's
like
yours,
that
have
regulations
and
such
that's
something.
That's
relatable,
I'm
sure
many
of
you
have
issues
with
scale
so
ring
central
is
company
as
well
as
we
talked
about
their
global
company,
but
they
so
they
have
millions
of
users,
hundreds
of
thousands
of
businesses,
200
kubernetes
clusters
and
growing,
and
thousands
of
deployments
and
workloads
I.
Think
tens
of
thousands,
perhaps,
and
as
we
showed
on
the
map,
multiple
regions
around
the
world,
they've
got
hybrid
clouds.
B
They've
got
their
own
data
centers
as
well
as
GK
and
eks,
and
again,
as
we
mentioned,
multiple
Solutions,
as
well
as
different
development
practices
within
house.
So
there's
a
lot
that
needs
to
get
managed
and
internal
customers
that
need
to
be
happy
at
scale.
So
how
does
flux
help
with
scale?
So
the
flux
project
is
actually
a
microservices
architecture
and
a
set.
It's
a
set
of
controllers
that
do
things
like
creation,
deletion
and
updates
to
both
your
app
deployments
and
to
changes
that
you
make
for
your
infrastructure.
B
B
So
that
means
like,
if
you've
seen
the
story
in
the
cncf,
the
Department
of
Defense
absolutely
wanted
to
use
flux
because
they
needed
to
use
their
entire
Helm
ecosystem
and
flux
allows
that
to
happen
because
it
has
native
support
and
you
have
many
git
Ops
options,
as
you
hopefully
heard
in
the
Keynotes.
You
know.
B
There's
the
gitups
principles
by
the
get
Ops
working
group
flux
completely
complies
with
them
and
within
those
options
you
have
the
version
that
we
talked
about
out
where
you
have
a
git
repo
that
has
a
single
source
of
truth,
but
also
with
oci.
Now
you
have
the
availability
to
have
flux,
consume
artifacts,
oci
artifacts
and
to
implement
get
Ops
in
that
way.
So
some
of
our
companies
Enterprise
companies
who
I
can't
name
who
have
major
major
scale
issues.
You
know
they're
at
the
Forefront
and
their
major
uses
of
flux.
B
So
we've
made
sure
that
we've
met
those
needs,
so
they
can
not
have
scale
issues
that
come
with
Git
itself,
so
git
is
still
there.
It's
just
moved
to
the
left
so
that
we
can
meet
their
scaling
needs.
So
wherever
you
are
in
your
journey,
just
starting
or
wherever
you
know
that
you
can
start
with
a
project,
that's
already
built
for
scale.
B
A
Yeah,
so
the
second
need
is
the
scale.
So
if
you
figure
it
out
your
compliance
within
the
security
need,
you
may
need
to
make
sure
you
can
run
it
at
a
large
scale,
so
meaning
meaning.
Yes,
you
can
definitely
try
and
many
project
Styles
like
this
as
a
toy
project
inside
the
company,
but
usually
it
does
not
make
much
sense
to
have
many
different
processes,
many
different
ways
to
deploy
and
manage
your
infrastructure
and
large
Enterprise.
You
want
basically
to
make
sure
the
things
you
build.
A
Also,
we
present
on
the
cloud
servers
and
in
Google
and
Amazon.
So
it's
it's
a
huge
infrastructure,
and
if
we
cannot
answer
how
do
we
scale
that
infrastructure
efficiently,
we
will
most
likely
fail,
so
it
will
not
be
adopted
widely
by
the
company.
It
will
be
adopted
by
the
some
departments,
but
might
need
it
will
not
work
will
not
work.
So
how
do
we
approach
it?
It's
already
on
the
slide,
so
we
approach
it
on
scale,
income
mono
repository.
So
what
does
it
mean?
A
A
We
also
have
huge
lab
environments
which
have
sponsor
bound
like
about
200
workloads,
kubernetes
workloads,
there's
already
old
number,
around
500
kubernetes
clusters
up
and
running
right
now
and
half
of
them
on
the
lab
and
the
staging
environment
and
yeah.
So
you
have
multiple
clusters.
So
how
do
you
manage
these
multiple
clusters
efficiently
and
here's
a
mono,
reappa
I
think
is
a
great
answer
to
that,
because
you
could
have
in
the
single
repository
information
and
configuration
of
all
your
clusters
in
the
same
place.
A
You
can
have
multiple
model
repositories
if
you
can
distinct
them
by
some
common
I.
Think
like,
for
example,
in
central,
we
have
multiple
product
lines
which
are
more
or
less
separate
like
video
and
phone,
a
messaging
contact
center
and
voice
OIP
in
telephony
Solutions.
So
we
have
mono
repositories
for
issues
of
slides
but
I,
but
it's
still
a
huge
product
line.
So
what
we
do?
We
basically
do
very
simple
thing,
and
it's
worked
for
us
great.
A
So,
first
of
all
one
thing
I
want
to
mention:
try
to
keep
it
simple,
especially
in
a
large
Enterprise.
So
what
we
do
we
just
using
customize
with
flux-
and
we
have
our
main
branch-
acted
as
a
as
a
basis,
so
we
keep
our
bases
all
our
basis
and
common
things
in
the
main
branch,
and
we
have
multiple
branches
and
folders
inside
this
repository
for
different
environments,
which
top
Builds
on
the
stop
main
branch.
A
But
what
it
gives
us
it
gives
us
ability
to
infer
the
state
of
all
our
infrastructure
and
will
it
will
come
a
lot
more
important
later,
when
I
will
talk
about
how
we
keep
it
reliable,
but
essentially,
as
it's
very
simple
distinction
and
flex,
helps
you
to
basically
as
long
as
you
get
repositories,
performant
enough
and
you
can
pitch
changes
and
you
can
fetch
a
state
of
the
git
repository
fast
enough
from
the
multiple
clusters.
You're
good,
we're
using
custom,
gitlab
deployment.
A
I'll
sell,
hosted
one
Enterprise
solution,
so
we
have
multiple
caches
and
set
up
and
making
sure
our
git
repository
is
always
available
and
performance.
But
yes,
this
is
important
part
of
it.
But
if
you
again
a
large
company,
you
really
want
to
have
your
git
repository
and
your
workplace,
where
every
developer
is
relying
on
to
be
always
available,
performant
and
well
behaved.
So
if
you
already
have
it
I
think
it's
a
great
way
to
scale
flux
and
any
github's
repository
simply
without
any
complicated
solution
to
to
basically
multiple
installation
of
multiple
clusters.
B
A
B
Excellent,
all
right,
third
Point
third
need
velocity,
I'm.
Sure
that's
important
for
all
of
you.
So
imagine
at
ring
central
right.
They've
got
various
teams.
They
got
to
move
quickly.
They
can't
have
slow
downs
from
miscommunications
or
people,
not
knowing
what
the
desired
state
is,
and
human
error
has
come
up.
B
A
lot
in
our
topics
of
conversation,
so
get
Ops
with
flocks
provides
this
ability
to
have
a
single
pane
of
glass
for
the
desired
State,
and
that
really
helps
with
these
issues
of
miscommunication
and
that
helps
them
really
reach
a
state
of
being
able
to
implement
devops
in
their
company,
which
means
a
shared
ownership
and
end-to-end
ownership.
So
their
teams
even
talk
about
their
own
design,
creation,
maintenance,
testing
to
even
support
tickets,
and
that
means
there's
no
separate
delivery.
Team
they're
not
detached
from
the
production
process.
B
So
how
does
flux
help
with
that
flux?
Has
a
lot
of
capabilities
with
observability
and
notifications
through
its
own
use
and
Integrations
that
our
community
has
worked
on
I
love,
the
quote
that
Yvonne
was
talking
about
that
flux
makes
kubernetes
easier.
So
if
you're,
you
know
having
certain
challenges,
we'll
talk
about
how
flux
can
help
with
that.
B
So
one
of
the
basic
things
that
flux
can
do
is
it
has
events,
and
so
you
can
get
notifications
through
the
platforms
that
you
use,
such
as
slack
or
Discord
or
others
that
we
support
and
we're
also
increasingly
working
across
various
Integrations.
So
flux
is
moving
into
your
IDE.
So
if
your
app
developers
are
doing
deployments,
they
can
do
less
context
switching
and
they
can
just
be
within
one
place.
So
we've
started
with
Visual
Studio
code,
how
many
people
use
Visual
Studio
code
or
people
in
there
excellent?
B
So
you
should
definitely
check
out
it's
our
git
Ops
extension
kingdin's
here
who's
working
on
it,
it's
really
fantastic,
so
you
can
literally
manage
flux
and
do
your
deployments
from
whether
you're
an
IDE
and
not
have
to
contact
switch.
If
you
like
uis
there
are
options.
There's
a
free
and
open
source
week
get
Ops
UI
that
we
have
out
there.
B
You
can
just
check
it
out
and
I
think
it's
really
important
to
mention
here
that
you
can
also
use
tools
like
Arc,
kubernetes,
eks,
anywhere
and
d2iq's
platform,
and
so
Microsoft,
Amazon
and
dqiq
have
chosen
flux
and
they
trust
flux
as
the
tool
that
delivers
git
Ops
to
their
end
customers.
So
if
you're
already
using
those
clouds,
then
you
have
those
options
as
well,
so
the
QR
code
will
have
links
for
you
to
check
them
out
so
I
think
that's
it.
A
Okay,
so
I'll
try
to
keep
it
very
short,
because
we
have
not
much
time
and
I
would
want
to
leave
it
for
the
end.
So
very
important
like
side
again
Segway
into
the
velocity,
is
a
devops,
so
as
a
devops,
so
I'm
very
heavy,
advocated
devops
and
devops,
not
I,
think
it's
misunderstood,
German
many
terms
of
special
in
many
ways,
especially
an
Enterprise,
because
right
now,
but
not
right
now,
but
before
we
had
so-called
so-called,
we
still
have
devops
Engineers.
A
So
the
people
who
try
to
build
infrastructure
and
try
to
support
the
infrastructure
for
for
our
development
teams-
and
we
have
around
three
thousand
developers
since
3
000
Engineers
under
Accenture
and
about
half
of
them,
was
on
the
support
Pro.
So
it
was
not
not.
They
were
not
software
Developers
and
like
big
bunch
of
them
was
developed.
Devops
Engineers.
A
It
outlines
the
big
problem,
because
if
you
need
so
many
Engineers
specific
Engineers
to
maintain
your
infrastructure,
something
is
wrong
and
for
me,
devops
was
always
been
a
shared
responsibility
between
the
team
and
Charity's
responsibility
for
the
product.
So
the
devops
team
is
a
team
who
owns
the
product.
Who
makes
the
changes
to
this
product
from
maintain
it
and
deploy
it?
So
when
we're
talking
about
githubs
and
flags
and
all
other
Solutions,
we
bringing
this
infrastructure
closer
to
the
development,
so
we
were
able
to
simplify
infrastructure
management.
A
We
were
able
to
make
it
in
in
a
way
it's
easy
for
our
developers
to
work
it
because
it's
a
source
called
it
gitreprit
model
repository
is
always
familiar
with
and
it
makes
so
basically
I
think
it
makes
the
devops
with
like
githubs
and
flags.
The
main
develops
what
it's
meant
to
be
and
what's
it
evolved
right
now,
which
is
like
kind
of
some
kind
of
abomination.
A
From
my
point
of
view
so
coming
and
it's
of
course
it
increased
sorry
about
it,
increased
our
velocity
and
sometimes
10
volts
when
we
used
like
other
practices
when
you
have
this
waterfall
process,
so
different,
agile
processes
and
pipelines.
So
we
do
not
have
any
pipelines
in
the
company
right
now
when
we're
using
githubs
and
it
may
now
deliver
a
like
10
times
or
even
20
times
faster.
So
it's
it's
a
great
tool
and
devops
is
a
key
practice
to
utilize
when
you're
working
with
GitHub,
so
you
always
should
think
about
it.
B
Awesome
all
right
and
the
fourth
need
reliability,
which
I'm
sure
is
important
for
all
of
you
as
well.
So
here
again
we'll
talk
a
little
bit
about
Flagger,
which
is
a
sub
project
of
flux,
and
this
is
really
important
at
ring
central
because,
as
we
talked
about
millions
of
views,
there's
hundreds
of
thousands
of
businesses-
these
are
business
critical
needs,
I
mean
their
telephones
have
to
work
right.
So
these
are
things
where
they
need
reliability
built
into
the
infrastructure.
B
They
also
need
deployment
processes
that
are
verifiable,
testable
and
safe,
and
we'll
talk
about
how
exciting
it
is
that
we
can
create
policy-led
environments
for
your
developers
to
be
able
to
do
the
things
that
they
need
and
know
that
they
have
the
guardrails
up,
and
so
the
human
error
won't
come
into
play
so
Flagger
the
sub-project
that
provides
Progressive
delivery,
which
I
really
love
about
this
story.
It
helps
users,
if
all
of
you
are
kind
of
on
your
git
Ops
Journey.
B
We're
aware
that
there's
a
lot
of
changes
that
you
have
to
make
or
your
teams
have
to
make,
even
in
the
way
that
you're
thinking
that
things
can
feel
new
and
unsafe,
so
Flagger
really
helps
with
that,
and
it's
really
great
to
hear
people
out
in
the
wild
feeling
that
safety
net
for
errors,
because
Flagger
provides
so
this
term
is
fairly
new
Progressive
delivery.
It's
an
umbrella
term
for
things
like
Canary
deployments,
blue
green,
a
b
testing
Etc
and
what
it
does
is
it
decouples
the
app
deployment
process
from
the
release
process?
B
So,
basically,
if
you
have
a
service,
mesh
or
Ingress
controller,
Flagger
can
use
that
to
decide
to
Route
traffic
from
you
know
your
first
source
and
then
slowly
Dole
it
out
to
the
new
one
based
on
metrics
of
success
or
failure
and
roll
back.
If
it
doesn't
meet
that
threshold,
so
basically
flux
will
notice
the
change
and
start
the
process.
B
The
get
Ops
process
but
before
kubernetes
can
take
over
Flagger
will
come
in
and
say:
okay,
the
user
set
thresholds
for
me,
I'm,
going
to
take
metrics,
for
example,
from
Prometheus
or
datadog
or
whatever,
and
based
on
those
metrics
I'll
keep
checking.
You
know
every
10
seconds.
You
know
I'll
route,
a
little
bit
of
traffic
are
there,
errors
am
I
within
the
threshold.
B
Okay
I'll
route,
a
little
bit
more
and
a
little
bit
more
until
you've
completely
moved
over
or
if
it
says
no
I
haven't
met
the
threshold,
it
will
roll
back,
and
so
what's
really
great
about
this,
as
Stefan
has
said,
is
like
you
can
deploy
on
a
Friday
and
know
that
if
anything
goes
wrong,
you
know
it's
not
going
to.
Have
you
be
on
your
pagers
on
a
Saturday,
so
you'll
talk
about
how
that
helps
with
you.
A
Yeah
so
very
important
thing
about
the
flux,
so
if
you're
just
using
flex,
adjust
using
git
Ops,
it's
very
important
to
understand
it's
a
very
powerful
tool
and
if
you
approach
it
blindly,
it
can
make
things
worse,
because
you
can
make
a
change.
We
can
make
changes
which
bring
the
whole
production
down,
especially
if
you
use
monorails,
which
is
counterintuitive,
but
still
so.
A
What
you
want
to
do
is
to
build
this
like
set
on
enforcement
of
rules
which
you
enforce
upon
the
developers
and
and
enforce
upon
the
process
in
a
way
we
cannot
avoid.
So
what
I'm
talking
about
so
I
have
I
have
not
much
time
but
I
will
bring.
One
simple
example
of
enforcement
is
a
high
availability
and
calc
engineering.
A
So
when
you
build
your
system,
you
have
this
non-functional
requirement
of
variable
ability.
Every
like
everyone
needs
to
be
high
available
cool,
but
if
you
just
write
it
a
non-functional
requirement
and
do
not
test
it
and
do
not
run
it
an
environment
which
is
inherently
unstable,
you
will
miss
it
you'll
most
likely
it
will
be
just
High
availability
on
paper
which
will
not
work
when
something
hits.
A
The
fan,
so
what
you
do
you
implement
chaos
engineering,
so
we
use
litmus
for
that
purpose
and
you
bring
you
you
make
sure
you
all
your
environments,
even
production
ones,
you
run
house,
engineering
and
production
sounds
crazy.
Sometimes,
but
it
is
you
make
it
inherently
unstable
and
developers
have
no
way
to
avoid
those
non-functional
requirements.
The
same
way
goes
with
everything
else,
and
when
we
build
our
systems
and
when
we're
trying
to
like
basically
make
it
Universal
across
a
large
Enterprise,
we
want
to
have
this
enforcement
rules.
A
It's
best
practices
to
be
shared
and
to
be
enforced
upon
every.
So
this
is
I.
Think
it's
the
most
important
part
of
success.
If
you
do
not
do
it,
you
most
likely
will
either
break
your
management,
will
stop
it.
Something
goes
wrong.
People
will
not
understand
that
we'll
have
too
many
ways
to
do
stuff
and
it
will.
It
will
break
so
again.
We,
as
I,
said
we
have
run
time
and
configuration
domain
for
configuration
domain.
A
We
run
gitlab
State
like
it's
not
pipeline,
it's
just
a
single
verification
with
GitHub
Runner,
but
we
have
custom
because
we
have
one
repository
like
I
said
before
we
can
run
static
analysis
on
this
model
repository.
We
can
detect
drifts
in
configuration,
for
example,
if
you
have
different
fqdns
with
different
Services,
which
aims
each
other,
you
can
statically
check
and
analyze
what
the
fqdn
is
the
correct
one
or
it
leads
to
some
another
resource
which
will
will
not
exist
on
the
class.
You
can
run
policy
management
and
policy
enforcement
using
career
net.
A
Specific
resources
exist,
and
we
also
have
run
time.
So
configuration
is
not
enough,
because
during
runtime
things
can
have
go
wrong.
People
make
mistakes,
yes,
developers
especially
makes
a
lot
of
mistakes,
so
we
use
flag
gear
for
the
purposeful,
like
with
questionna
asked
about
red,
green
blue.
We
have
Canary
deployment,
we
have
Progressive
deployment,
different
types
of
deployments
by
using
flagger
which
flag
your
support
and
we
tied
it
with
test
Cube.
So
it's
run
test.
A
Cube
checks
in
all
our
environments
in
lab
and
stage
and
production
is
very
important
and
it
it
makes
sure
what
when
we
deploy
stuff
when
the
developers
deploy
stuff,
we
are
not
afraid.
So
we
build
trust
into
the
system
and
when
you
build
trust
into
the
system,
when
we
started
basically
trusting
what
we
can
really
Deploy
on
a
Friday,
we
start
to
deploy
stuff
on
a
Friday
because
we're
not
afraid
to
break
stuff,
and
last
fingering
Central
have
five
nines
SLA
for
availability.
A
B
Again,
if
you
didn't
take
this
QR
code,
we've
got
plenty
of
talks
throughout
the
week,
so
we
also
have
the
slack
channel
so
we'll
be
outside.
If
anybody
has
follow-up
questions
or
if
you
want
to
follow
up
questions
in
slack,
please
do
that.
We
also
have
our
email
address
and
you
can
email
us
if
you
have
questions
so.
Thank
you.
So
much
and
I
hope
you
have
success
with
flux
and
flagger.