►
From YouTube: Panel: GitOps User Stories
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Panel: GitOps User Stories - Tamao Nakahara & Stefan Prodan, Weaveworks; Javeria Khan, Palo Alto Networks; Hubert Chen, Branch; & Edward Lee, Intuit
The expression, GitOps, has taken off and resonated in the Kubernetes community since its launch by Weaveworks in 2017. GitOps is a way to do Continuous Delivery by using git as a single source of truth for declarative infrastructure and applications. Meet actual GitOps practitioners in this panel, hear their use cases, challenges, constraints, and which tools they use to execute GitOps. If you’ve been wanting to get started, learn from these actual solutions: *Automated Helm-based deployments for code and infrastructure changes through Jenkins and GitLab at Palo Alto Networks *Simplified access control for Kubernetes clusters using Weave Flux at Branch *Flux, Terraform and Vault, oh my! Unique ways Under Armour leverages GitOps *GitOps at Scale: Patterns and processes enabling Intuit to manage thousands of applications and repositories, across 100+ clusters using Jenkins and Argo CD
https://sched.co/UaYh
A
Thanks
so
much
so
we
will
advance
the
site.
So,
yes,
we've
got
a
tight
schedule
here.
My
name
is
Tom.
O
nakahara,
like
you
said,
I,
run
the
developer.
Experience
team
at
a
company
called
we
works.
I
have
lost
my
voice,
so
if
I
run
away
from
the
stage,
do
not
panic
it's
because
I'm,
coughing
and
I
want
to
just
not
distract
the
speakers.
So
just
warning
you
without
that.
So
we'll
do
a
quick
overview.
Hopefully
you've
heard
the
term
get-ups.
A
If
you
have
not
welcome,
you
will
be
able
to
hear
a
lot
of
great
user
stories,
and
that
was
the
goal
of
my
wanting
to
organize
this
panel
because
we
feel
like
the
history
is
that
we
saw
certain
use
cases
happening
out
in
the
field.
We
were
doing
our
own
things
within
we've
works
and
our
esteemed
CEO
Alexis
Richardson
that
you
might
have
seen
in
the
previous
talk
put
out
the
blog
post
in
the
word.
What
about
the
term
get-ups
and
I
kind
of
took?
A
You
know,
kind
of,
took
off
around
I
guess
it
was
2017
and
a
lot
of
people
even
said.
Oh
we're
kind
of
doing
this
thing,
but
Damian
I
wish
I
was
the
one
who
created
up
that
word.
You
know
that's
a
pretty
good
one,
and
so
as
part
of
that
we've
kind
of
seen
it
find
momentum
so
just
high
level
definition.
If
this
is
your
first
time
you
know,
hopefully
people
know
git
and
it's
sort
of
operations
by
pull
request.
So
this
can't
this
concept
that
you
know
we
feel
like
git
has
really
been
pervasive.
A
We
kind
of
have
understanding
that
most
stems
and
a
lot
of
people
in
operations
how
that
is
a
common
language.
So
why
don't?
We
use
the
benefits
of
git
as
a
way
to
have
an
audit
log
to
do
approvals
and
to
do
roll
backs
if
things
are
necessary.
So
as
part
of
that,
we
are
excited
that
we
started
it,
but
we
do
not
own
that
term.
We
really
want
to
build
like
a
community
around
this
concept
of
kid
ops.
A
We
want
to
hear
all
these
different
use
cases
and
different
tools
that
people
use
and
we're
really
excited
about
where
this
concept
of
get-ups
will
take
us
in
the
next
year
or
two.
So
that
was
really
the
motivation
for
my
putting
together
this
panel
to
sort
of
see
already
today
what
some
may
be
the
forefront
users
of
get-ups
are
doing,
and
they
can
share
it
with
you.
If
here's
the
about
what
this
might
mean
for
you-
and
you
know
what
are
the
pros
and
cons,
what
are
the
right
use
cases?
A
What
are
the
stories
that
they
have
in
terms
of
decisions
they
made,
and
we
really
hope
that
it
will
be
useful
for
everybody
here.
So
that
is
the
goal.
So,
with
that
I'll,
do
you
introductions
so
we're
going
in
order
or
the
slides?
So
we
have
stuff
on
pradhan
who's
on
our
team,
he's
a
developer
experience
engineer
and
has
been
deeply
involved
in
our
open
source
project
called
flux,
which
is
what
really
created
this
concept
of
get-ups
as
well
as
flagger.
A
You
might
have
heard
of
we're
very
lucky
to
have
Lee
here
also
from
Intuit
who's.
A
distinguished
engineer
and
I'll.
Do
a
quick
shout
out
if
you
haven't
heard
yet
we
put
out
an
an
announcement
the
other
day
before
cube
con,
that
the
argos
CD
project
that's
been
that
was
started
at
Intuit,
is
merging
or
becoming
a
family
together
with
flux
CD,
because
we
really
want
to
create
the
best
use
case.
A
Possibility
get
up
so
actually
I'll
shout
out
so
at
4:30
after
this
we're
all
gonna
meet
over
at
the
Starbucks
in
near
exhibitor
a
if
anybody
wants
to
hear
more
about
that,
then
we
also
have
Hubert
Chen
from
France.
You're
really
excited
to
me
as
infrastructure
manager
and
I
give
him
a
talk
about
get
ops
with
terraria.
When
you
guys
were
both
there
branch,
the
last
coupon
right
in
the
US.
Is
that
correct
me
getting
right
so
we
reached
out
and
we
wanted
to
hear
and
then
we'd
heard
that
Javari
had
moved
on
to
Palo.
A
Two
networks
has
also
brought
this
get
ops
and
methodology
and
she's
a
senior
sree
over
there.
So
it
was
really
excited
that
we
got
these
people
together
again,
so
you
are
in
good
hands
and
hopefully
we
will
go
through
now
about
three
to
four
minutes
of
what
their
particular
use
cases
are
and
they
each
have
like
a
slight
or
two,
and
then
we
will
get
to
questions
because
I'm
sure
lining
we
have
them
so
you've
caught
a
couple
mics
there
and
we
will
get
to
that.
B
Hey
so
our
Kitab
story
started
very
small.
We
had.
We
started,
of
course,
with
Tara,
Furman
and
small.
Like
everybody
else,
we
are.
We
are
a
long
time,
kubernetes
users,
so
we
had
to
provision
everything
on
our
own.
There
wasn't
no
installer
back
then
afterwards
we
we've
thought
how
it
can.
We
apply
same
things
that
were
doing
to
infrastructure
to
applications
that
are
running
on
kubernetes,
so
a
natural
way
of
doing
it
was
to
store
everything
in
a
git
repository.
We
had
some
kind
of
outside
the
cluster
automation
at
first
and
afterwards.
We
realized
that.
B
Why,
wouldn't
you
put
the
automation
inside
kubernetes
and
have
back
then
there
wasn't.
The
operator
concept
was
the
controller
concept,
so
we
build
a
controller
that
was
just
looking
at
git
and
continuously
applies.
The
state
on
our
cluster
we've
taught
that
the
idea
of
what
continuously
appliance
tape
was
important.
So
if
someone
made
a
mistake,
someone
access
our
cluster
and
made
some
manual
modification
we'll
be
able
to
undo
any
any
of
that.
So
that
was
step
one
step
two
for
us
was
okay.
We
have
this
controller.
That
applies
the
state.
How
can
we?
B
How
can
we
build
automation
in
such
a
way,
then,
when
we
release
a
new
container
image,
that
image
is
deployed
on
a
cluster
without
us
having
to
go
into
the
git
repository
and
modify
the
image
tag
every
time,
so
we
will
build
into
flux
a
component
that
scans
the
image
registry
detects.
That
is
a
new
image
there,
an
image
tag
and
based
on
some
policies
that
we
we've
defined.
B
For
example,
we
have
a
policy
based
on
semantic
version,
semantic
versioning,
so
we
can
say
if
I'm
doing
a
patch
release,
I
trust
that
actually
this
is
backwards
compatible.
So
why
not
just
automate
the
whole
thing
so
flux
now
is
able
to
scan
container
registries
and
push
that
modification
back
to
get
and
afterwards
the
other
part
of
flux.
The
operator
will
just
apply
the
new
version
we
got
to
that
stage
and
afterwards,
almost
a
big
thing.
B
Everybody
loved
him
and
we
decided
to
create
a
sub
project
in
flux,
which
is
today
called
the
flux,
helm
operator,
and
we
applied
the
same
principles
to
help
we
created
custom
resource
or
alham
release
that
has
everything
can
wise
inside.
You
can
set
up
your
values
there.
You
can
configure
your
helm
installation
through
a
declarative
way
and
we
we
apply
the
same
principles
to
our
home
operator.
Every
time
you
do
a
change
in
your
hearing
star,
we
do
an
upgrade
and
so
on.
B
Also,
if
someone
runs
home
comments
on
our
cluster,
the
operator
will
undo
it
afterwards.
The
last
step
was
okay.
Now
we
can
automate
this
new
version
in
production,
but
if,
if
there
is
a
is
a
bar,
there
is
a
problem:
how
can
we
detect
it
and
not
affect
all
our
users?
This
is
how
we
came
up
with
Pfleger
project
that
I've
been
working
on
in
the
last
year.
B
C
D
B
E
B
E
Your
slide,
so
let
me
know
I,
guess
a
good
way
to
kind
of
understand
into
us.
Get
off.
Your
knee
is
to
kind
of
understand
what
constraints
and
kind
of
goes
we
were
trying
to
solve
for
so
it
into
it.
We
have
kind
of
a
large
number
of
kubernetes
clusters
and
we
use
AWS
a
lot
of
AWS
accounts
as
well.
A
lot
of
that
is
for
isolation
and
security
purposes
in
a
performance.
Isolation
excel
bra,
but
also
for
kind
of
administrative
domain
purposes.
E
So,
for
example,
and
when
we
create
a
kubernetes
cluster,
will
create
a
separate
account
for
all
of
the
resources,
like
databases
and
so
on
that
are
not
directly
managed
by
kubernetes,
and
then
we
give
the
applications
teams
that
actually
use
those
databases,
direct
management
over
their
database,
but
the
clusters
themselves
run
in
a
separate
account
so
that
applications
teams
cannot.
You
know
we
don't
step
on
each
other,
so
they
cannot
modify
the
management
of
the
cluster,
but
they
have
full
access
to
their
resources.
E
So
so,
basically
we're
looking
at
a
kind
of
a
multi
account
multi
cluster
environment.
Many
teams
also
deploy
their
application
into
multiple
clusters.
It
could
be
different
environments
like
pre,
prod
versus
prod
versus
performance.
It
could
also
be
for
dr
purposes.
You
may
be
running
a
separate
instance
of
your
application
in
a
two
different
geographical
reasons
of
regions.
So
so
we
needed
a
system
that
could
manage
even
for
a
single
application.
Multiple
hydrogen
counts,
as
well
as
over
multiple
clusters.
E
Each
cluster
also
hosts
can
host
multiple
applications,
so
each
cluster
can
be
multi
tenant,
so
we
needed
appropriate
access
control
and
you
know
for
in
order
to
lock
down
each
each
of
the
namespaces
that
a
particular
applications
team
that's
needs.
We
obviously
wanted
to
provide
a
very
simple
low
friction
developer
experience
which
we
found.
Yet
ops
is
a
really
great
way
to
do
that.
E
It's
very
easy
to
understand
what
should
be
running
in
production
on
each
particular
cluster
and
if
there's
anything
ever
running
on
a
cluster
shouldn't,
be
it's
very
easy
to
detect
that
we
also
wanted
to
completely
separate
the
CI
and
CD
pipelines.
We
didn't
want
the
CI
system
to
have
any
access
to
production
resources,
obviously,
for
security
purposes.
There's
no
item,
that's
not
on
this
slide,
but
we
also
wanted
to
be
able
to
declaratively
specify
tempora
behavior
on
a
cluster.
E
So,
for
example,
if
you
want
to
do
a
Bluegreen
deployment
in
order
to
do
the
Bluegreen
deployment,
your
application
has
to
go
through
a
sequence
of
steps,
but
instead
of
doing
a
github
commit
for
each
of
those
sequences
of
steps
we
wanted
to
provide.
You
know
one
description
which
you
could
commit
once
and
it
would
perform
a
Bluegreen
deep
and
then
there's.
Of
course,
all
of
the
standard
enterprise
features
that
people
want
like
our
back
and
auditability
and
so
on,
and
then
there's
one
quick
slide,
which
will
kind
of
summarize
a
lot
of
this
stuff.
E
So
this
is
kind
of
a
picture
of
our
environment.
You
can
have
multiplied.
Those
accounts,
an
application,
could
be
running
in
multiple
accounts.
Multiple
clusters
there's
also
a
separate
kind
of
resource
account,
which
shows
resources
like
databases
which
are
not
directly
managed
through
kubernetes.
We
also
use
a
CRS
for
managing
some
AWS
resources
through
the
kubernetes
cluster,
but
there
are
some
resources
whose
life
cycle
is
not
closely
tied
to
that
of
a
particular
cluster.
So
we
find
that
we
always
end
up
with
some
resources
that
are
managed
outside
of
kubernetes.
E
So
what
we
do
is
we
use
Argos
CD
to
manage
anything
that
gets
deployed
into
the
cluster
itself,
but
for
anything
that
gets
deployed
into
like
a
AWS
account,
we
basically
have
created
our
own
tour
called
Aggie
AGI.
It's
kind
of
the
chemical
notation
for
silver
iodide
in
case
you're,
curious
to
manage
anything
that
gets
deployed.
You
know
in
today
that
account
itself
so
so
I'll
go.
Cd+
Aggie
provides
a
complete
set
of
get-ups
tools
for
controlling
everything
in
the
cluster,
as
well
as
everything
running
outside
the
cluster.
F
So
we
found
that
when
we
were
doing
hot
fixes,
we
would
do
things
like
not
mono
not
and
go
back
and
could
fix
configs
and
then
maybe
the
CIA
and
the
CD
system
would
deploy
redeploy
old
fixes
or
have
out-of-date
configurations,
and
we
would
essentially
replay
the
same
errors
that
we
had
already
solved,
which
is
very
frustrating.
So
we
did
things.
So
we
looked
at
some
tools
and
that's-
and
we
said,
oh
really.
What
we
want
is
just
to
deploy.
F
We
have.
We
want
to
have
one
place
to
make
changes.
Gay
and
modifying
production
is
already
bad,
but
the
problem
was
we
had
sort
of
tightly
coupled
our
CI
and
CD
systems
such
that.
If
I
wanted
to
do
a
quick
fix,
a
rollback
or
something
like
that,
that
I
would
need
to
do
go
through
the
CI
system,
which
takes
twenty
plus
minutes,
and
if
you
want
to
run
full
regression
to
us.
F
It'd
take
hours
so
I
never
wanted
to
have
a
critical
situation
where
I
would
have
to
respond
through
CI
to
go
through
CD
and
go
through
get
to
get
to
the
deploy.
So
we
use
flux
to
essentially
deploy
directly
from
the
the
get
of
the
CD
part.
So
it's
very
well
designed
to
fix
the
CD
part
and
that
allows
us
to
decouple
the
CI
part
from
our
CD.
So
now
I
can
do
a
hotfix
for
five.
F
In
five
minutes
we
had
also
looked
at
things
like
home
to
essentially
abstract
various
parts
of
the
build
and
deploy
process.
So,
but
when
I
looked
at
it,
I
would
I
was
a
little
confused
because
now
I
have
the
helm.
Chart
and
I
have
five
different
configurations
for
five
different
clusters
and
mentally
I
have
to
reconstruct.
What's
going
to
happen
when
I
just
want
to
deploy
one
change
to
one
cluster
for
a
hotfix,
so
I
felt
like
that
solution
was
not
right.
F
For
me,
what
I
wanted
was
to
be
able
to
deploy
a
hotfix
to
one
cluster
where
I
wanted
it
right
very
quickly
without
having
to
reverse
engineer
this
problem
of.
What's
helm,
plus
the
CI
system,
gonna
do
to
get
to
the
final
configuration
and
then,
when
I
thought
about
it,
I
generalized
that
to
a
problem
we
actually
have
in
other
configuration
management
systems.
So
we
have
salt
for
systems,
management
and
provisioning,
and
we
actually
have
a
similar
problem
there.
Where
we
don't.
F
We
have
to
reverse
engineer
what
the
salt
is
going
to
do
when
we
want
to
manage
and
deploy
systems,
so
it
gets.
It
gets
very
scary
when
you're,
possibly
touching
hundreds
or
thousands
of
instances,
all
at
once
and
saying
this
is
a
very
scary.
Change.
I
really
want
it
very
targeted
so
by
using
a
git
ops
tool
to
essentially
abstract,
to
split
out
the
CI
NCD
and
obstruct
and
give
me
a
essentially
very
readable
configuration
I
can
now
have
much
more
confidence
to
say.
F
This
is
what
we
want
to
do
right
now
to
fix
this
problem
in
a
targeted
way.
So
we're
moving
forward
with
this
and
we're
trying
to
expand
on
it
and
moving
with
for
other
parts
of
our
infrastructure.
Prometheus
is
one
example
where
now
it
gives
us
the
ability
to
target
for
developers
to
just
make
it
commits
to
make
changes.
I
really
want
to
get
to
a
system
where
the
developers
can
make
a
lot
of
changes
directly,
and
they
don't
have
to
make
changes
in
clusters.
They
don't
touch
anything
except
to
get
good.
F
Ops
is
about
the
up.
Part
of
it
is
really
about
control
and
processing
consistency
and
getting
to
a
point
where
we
have
one
consistent,
shared
view
between
developers
and
operators
like
myself,
is
really
really
important.
So
now,
if
somebody
somebody
doesn't
have
to
look
in
the
cluster
to
say,
okay,
this
is
what
the
configuration
should
be
now
I.
Just
share
a
git
repo
get
it
commit.
This
is
what
it
should
be.
F
If
there's
a
weird
problem,
then
maybe
I'll
take
a
look
at
it
and
maybe
they
would,
but
in
general
those
configuration
should
always
match
and
also
gives
us
a
very
good
auditability,
rollback
and
isolation.
So
now
they
don't
need
cluster
access
to
see
what
should
be
in
the
cluster.
There
are
certain
situations
where
that's
valuable,
but
but
we're
moving
forward
and
trying
to
build
more
and
more
infrastructure
in
sort
of
this
methodology,
where
everybody
commits
to
gate
it's
a
shared
view
and
it
automatically
gets
deployed
things
like
will
be
it.
F
A
lot
of
these
changes
are
all
about
preflight
checks
to
make
sure
that
what
you
think
is
gonna
go
out
will
will
God
being
able
to
roll
back
being
able
to
handle
partial
failures
and
being
able
to
rectify
all
of
the
changes
that
you
think
that
that
the
system
wants
to
get
and
that's
why
the
controller
paradigm
is
really
really
important
in
here.
Thank.
G
Basically,
apply
good
offset
three
humane
levels
for
the
first
one,
which
is
a
kubernetes
infrastructure.
We
have
about
40
clusters,
on-prem
running
on
bare
metal
nodes
across
various
data
centers
across
the
world.
We
manage
all
of
these
through
CI
CD
pipelines
and
get
labs
in
Jenkins
and
the
cluster
states
are
maintained
in
version
control,
danceable
pipelines
out
playbooks,
so
the
newer
platforms
were
deploying
to
gke
those
are
managed
with
separate
law
pipelines
running
in
GCP
for
all
the
infrastructure
codes,
such
as
cluster
components
at
CDE
or
the
CNI
or
the
cubelet.
G
Everything
goes
through
PR
processes,
so
we
have
a
review
process
and
everything
gets
deployed
to
production
that
way,
even
for
pure
clusters
as
well.
Our
CD
problem
for
kubernetes.
They
use
vault,
helm
and
internal
fact,
management
system
to
fill
in
secrets
and
server
information
before
they
deploy
to
production
and
then
coming
to
the
second
layer,
which
is
applications
for
all
kubernetes
application
manifests
which
our
deployments
and
services
and
secrets
we
kept.
We
keep
them
and
get
repose
as
helm
charts
and
we
build
new
app
images
and
we
release
them
as
new
releases
to
clusters.
G
That
way,
this
also
goes
through
PR
processes.
At
this
application
layer.
We
are
now
also
looking
into
incorporating
of
a
convergence
tools
such
as
flux,
which
can
automatically
keep
the
cluster
stay
in
sync,
with
the
version
that
we
have
in
source
control
outside
of
releases
as
well,
and
then
at
the
very
last
stage,
which
is
outside
of
kubernetes.
We
also
use
git
ops
practices
for
building
all
of
our
new
GCV
infrastructure.
G
So
this
is
important
because
all
of
our
infrared
eV
and
sre
teams,
they
can
now
build
their
projects
on
demand
and
PISA
determined
ways
through
PR
processes,
and
they
can
basically
just
open
a
PR
for
the
changes
that
they
need
and
the
review
and
approval
process
goes
through
that
PR
person
and
they
get
their
changes
applied
and
they
get
the
things
that
they
need
in
GCP.
So
this
is
managed
through
version
control,
terraform
scripts,
which
is
applied
as
Atlantis
pr's.
G
Besides
that
everything
which
is
on
Prem,
which
is
for
provisioning,
orchestration
config
management,
is
done
through
salt
as
scheduled
States.
We
also
have
a
bit
of
terraform
over
there
as
well.
Prometheus
alert
manager
rules
live
with
their
play,
books
also
and
get
repose,
and
we
have
sync
jobs
running
under
prometheus
global
servers
that
pull
these
in
and
apply
them.
Cluster
logging
is
done
through
fluency
and
we
manage
that's
also
kept.
Announcer
will
play
book
and
template
Annette's
with
these.
Likewise,
through
the
other
cluster
configurations.
G
A
You
thank
you
so
much
uhm,
so
we
have
two
mics.
If
anybody
wants
to
come
up
to
have
questions
while
you're
thinking
about
your
questions,
we
do
have
a
few
kind--
questions
that
we
prepared
so
yeah.
Please
take
advantage
of
the
speakers
that
we
have
here,
while
you're
waiting,
I'll
just
do
a
quick
thing
that
stuff
on
it
made.
We
have
a
git
ops,
hands-on.
A
You
can
take
a
picture
of
this
if
you'd
like
to
try
it
on
your
own
laptop
and
if
you
want
to
get
this
shirt
we're
giving
away
by
the
end
of
tomorrow,
as
I
mentioned,
we
have
these
Fargo
Fluxx
office
hours
right
after
this
we'll
be
heading
toward
the
lobby
de
areas
actually
because
they're
closed
off,
and
if
you
have
more
questions
about
that,
please
join
so
it's
excellent.
We
have
questions
here,
we'll
move
on
to
yes,
hi.
H
My
question
is
centered
around
sort
of
like
a
organizational
changes.
We
I
work
for
Salesforce.
You
know
huge
company
and
we
have
a
lot
of
release
teams
that,
essentially,
you
know,
take
care
of
making
sure
that
things
go
where
they're
supposed
to
go
when
they're
supposed
to
go
and
all
of
that
stuff,
it
seems
like
get-ups
kind
of
takes
care
of
a
lot
of
those
things
that
we're
doing
manually.
Have
you
guys
had
to
go
through
a
transition
of
that
sort
where
you're
kind
of
making
certain
teams
or
certain
jobs
irrelevant.
A
E
Well,
yes,
well,
he
definitely
went
through
a
big
transition.
Add
into
it
I
wouldn't
use
the
word,
absolutely
sin
or
you
know
unnecessarily
unnecessary,
but
for
application
teams.
What
what's
happened
is
that
a
lot
of
the
ops
responsibility
has
kind
of
shifted
left
to
the
application
seen
so
many
of
our
big
application
seems
used
to
have
big
SRA
organizations
associated
with
them,
and
now
they
don't
because
they
can
do
all
of
that
works
through
get
out.
E
But
what
has
happened
is
that
a
lot
of
that
burden
has
also
shifted
onto
the
platform
teams,
the
teams,
maintaining
the
kubernetes
clusters
and
all
of
the
trolling
that
these
applications
seems
use,
and
so
those
sarees
are
still
needed
for
those
teams
in
order
to
provide
infrastructure
and
platforms
that
the
applications
infuse.
But
definitely
the
application
seems
today
at
Intuit.
They
do
all
their
own
operations.
E
D
B
Begin
there
are
several
solution
that
works
in
a
negative
spy
plane.
One
of
it
is
a
open
source
project
built
by
benami,
which
is
now
maintained
by
vmware,
is
called
the
banana
secret
controller,
which
allows
you
to
generate
a
public
and
a
private
key.
The
private
key
never
leaves
the
cluster.
So
since
next
flux
is
there
in
the
cluster,
the
public
key
can
be
shared
with
your
team.
Anyone
can
create
encrypted
secrets
in
get
the
also
the
public
key
can
be
also
in
get
public
is
no
problem.
Everyone
can
create
those
equals.
B
No
one
from
your
team
can
decrypt
them
only
the
COC
press
control
can
and
how
it
works.
Basically,
when
you,
when
you
commit
a
secret
so
synchronize
it
the
Silla
secret
control,
retransform
that
secret
into
a
kubernetes
secret
and
that
kubernetes
secret
gets
mounted
as
a
volume.
Our
is
a
environment
variable.
There
are
other
solutions
out
there
get
plugins
like
get
secret
and
PGP,
and
we
are
working
now
with
the
Mozilla
team.
B
A
I
Hi
thanks
thanks
for
talking
here.
My
question
was
actually
about
secrets
as
well,
but
I
have
another
one.
So
what's
a
good
way
to
start
with
kind
of
introducing
get
ops
to
an
organization
like,
obviously
you
can
take
the
output
of
every
resource
in
your
cluster
and
put
it
into
a
git
repo
and
start
consuming
it.
I
But
do
you
recommend
kind
of
scoping
changes
to
maybe
a
few
objects
that
you
care
about
deployments,
config,
Maps
things
like
that
and
then
Duke's
do
you
is,
is
the
end
of
get
ups
to
be
like
to
manage
every
single
resource
you
know
and
and
the
entire
state
is
represented
there?
Is
that
something
that
and
also
another
bonus
questions?
How
do
you
organize
your
repos
so
that
your
files
don't
conflict
with
each
other,
so
someone
wouldn't
clobber
a
deployment
in
another
namespace
or
something
like
that.
So
how
do
you?
F
I
can
take
the
firt
to
the
first
question.
We
don't
do
everything.
Definitely
the
highest
velocity
I
would
pick
first
and
the
mode
things
you're
most
likely
to
change.
We
do
deployments,
we
do
HPA's,
we
do
services,
we
do
config
Maps,
but
our
back
is
a
little
trickier,
but
we'll
probably
eventually
get
to
that
for
certain
things.
F
Part
of
it
is
the
control
of
whether
you're,
giving
it
to
the
application
teams,
which
you
definitely
want
to
do
on
the
deployment
side
versus
whether
you're
doing
more
on
the
operations
side
like
this
to
keep
secrets
and
security
and
those
kinds
of
things,
you
probably
probably
want
to
get
a
little
careful
more
careful
about
that.
But
I
think
you
definitely
want
to
have
a
staged.
Rollout
have
a
face.
I'm.
Actually
you
know
I
would
advocate
that
you
should
also
not
try
and
do
everything
at
once
like
it's
getting.
F
The
UPS
process
is
about
control
and
coordination
and
audio
dabeli,
and
it's
fine
to
throw
in,
like
a
emanuel
audit,
a
manual
step
to
say,
okay
for
things
that
are
very
risky.
You
want
to
have
a
second
person.
Look
at
it.
You
want
to
have
a
manual
audit
trail
that
and
a
manual
approval
step.
That's
fine!
That's
just
on
the
way
to
a
process
right.
F
So
if
you
get
to
do
this,
this
step,
we're
literally
the
only
person
thing
that,
if
that
person
is
click
a
button
you
might
as
well
remove,
but
until
you
get
the
assurance
that
you've
done
it
fifty
done
a
hundred
times
that
you
get,
you
can
do
it
without
with
a
with
full
automation.
Nothing's
gonna
go
wrong.
You
should
definitely
stage
robot.
F
E
G
I
would
kind
of
pick
them
out
with,
by
impacts
to
like
pick
out
the
most
impactful
things.
First,
for
example,
the
things
that
people
are
more
prone
to
hotfix
and
then
those
things
caused
me
major
outages
and
there
are
more
important
so
obviously
start
with
staging
interior
first,
but
then
move
on
to
production.
So
maybe
like
deployments,
people
have
a
habit
of
going
and
changing
resources
or
painting
the
image.
J
So
sorry,
I
kind
of
get
kid
ops
for
like
pull
requests
for
CI
for
two,
like
that.
But
I
don't
really
hear
anybody
talking
about
like
the
local
development
process,
because,
like
a
lot
of
people
will
say
like
look
I
don't
really
want
to
commit
my
changes.
Push
them
up,
wait
for
something
to
happen.
Right,
like
I,
want
to
be
able
to
iterate
and
kind
of
innovate
quickly
to
know
whether
this
is
the
right
thing.
J
So
how
does
a
lot
of
these
practices
kind
of
enable
those
patterns
without
like
completely
reproducing
the
process
right,
so
you
have
like
scaffold
so
you'd
be
able
to
like
trigger
from
like
Visual,
Studio
or
IntelliJ
and
say,
like
hey,
look
like
I've
made.
Some
changes
like
just
go.
Roll
them
out
just
go
update
it
like
I,
don't
really
care
but
like
how
do
you
like
not
reinvent
the
world
I.
B
Can
take
this
one
so,
for
example,
we
have
customized
support
in
both
are
going
clocks
and
you
can
use
Kapil
that
does
a
customized
build
and
deploys
on.
Let's
say
kubernetes
kind,
on
your
own
laptop
when
in
the
development
and
that
customized
build
the
same
customized
build.
Our
tuning
will
be
running
the
same
stuff
on
your
clusters.
So
if
you,
if
you
have,
is
this
tool
customized,
you
want
to
do
that.
You
can
use
it.
B
You
can
also
use
help
and
scaffold
will
be
able
to
do
a
upgrade
every
time
you
change
something
it
can
work
different
with
different
types
of
templating
systems,
on
the
on
the
client
side,
on
a
laptop
for
dev,
so
I
think
it's
not
it's
not
an
issue
to
replicate
everything
locally
and
in
the
cluster.
As
long
as
you
have
this
consistent
view
of
your
of
your
the
classic
desire
state
somewhere
stored
like.
J
One
of
the
things
I
typically
hear
is
like
hey.
Look
like
one
of
the
best
practices
is
separating
the
the
operational
process
from
the
application
source
code
such
like
you're,
not
kind
of
having
to
deal
with,
like
you
know,
merge
conflicts
and
rebasing
and
stuff
like
that
when
the
application
changes.
So
that's
kind
of
living
in
this
kind
of
like
separate
repository
yeah
like
how
does
that
so?
Where
does
like
your
customize
or
your
helm?
Charts
live
if
they're
in
the
application
repository
right
like
how
do
you
so.
B
That's
how
my
support
thing
called
remote
base
remote
base,
so
we
can
say
on
customizing,
have
this
small
customization
in
your
app
repository
that
we'll
pull
from?
Let's
say
the
the
cluster
repository
the
common
thing
and
you
can
do
small
customization
while
you
are
developing,
for
example,
adding
a
new
environment
variable
or
what
you
find
a
flag
right,
so
you
don't
have
to
clone
the
whole
whole
thing.
Estimize
allows
you
do
that!
Also
we
can
you
can
just
too
easily.
B
E
Could
actually
I
mean
gold
couple
different
ways
like
you
could
put
the
deployment
manifests.
You
know
all
of
those
things
in
the
same
repository
as
your
code,
if
you
like,
or
you
could
put
it
in
separate
repositories,
particularly
that
particularly
works.
Well,
is
you
want
the
people
who
you
want
to
give
control
over
deployment
to
be
a
different
set
of
people
or
a
smaller
set
of
people
than
the
ones
who
actually
develop
the
application?
Also
in
terms
of
testing
at
Intuit,
first
developers
will
always
want
to
run
unit
tests
and
those
run
under
laptop.
E
As
you
get
into
kind
of
more
micro
service
style
environments,
it
becomes
harder
and
harder
to
run
a
meaningful
test
on
your
laptop
because
it
just
needs
to
talk
to
so
many
different
services.
So
unless
you
spend
a
lot
of
time
studying
everything,
you
know
it's
very
hard
to
run.
Something
like
that
tough,
so
another
approaches
work
we've
taken
is
to
make
the
deployment
process
itself
super
fast
and
responsive.
So
people
can
just
do
something
and
then
deploy
it
within
like
10
seconds
say,
and
so
that's
also
a
really
good
way
to
test
I.
C
When
comes
to
step
over
preventing
the
delivery
pipeline
from
accessing
production,
kubernetes
cluster
I'm,
guessing
we'd
have
to
separate
out
the
continuous
integration
pipeline
from
the
delivery
pipeline.
In
that
case,
how
do
you
verify
that
the
deployment
took
place
and
was
successful,
and
how
do
you
provide
that
feedback
to
the
developer
and
then
I
have
a
second
question?
If
you
have
time
unrelated
a
little
bit,
but
I
noticed
that
some
companies
have,
you
said
you
have
like
200
clusters
and
or
40
clusters.
What
what
boundaries
you
use
define?
E
Good
question,
of
course,
what
you
want
to
do
is
you
don't
want
the
CI
pipeline
to
have
total
control
over
all
aspects
of
deployment?
However,
you
can
have
the
like.
The
a
CI
pipeline
kick
off
deployment,
but
you
can't
say
what
repo
to
deploy
where
you
know
or
daca,
or
modify
what
is
being
deployed.
So
that's
one
way
to
kind
of
still
have
some
control
or
linkage
and
from
your
CI
pipeline,
but
it
can't
control
anything.