►
From YouTube: Tales from the Branches: GitOps in the Real World - Moderated by Dan Garfield, Codefresh
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Tales from the Branches: GitOps in the Real World - Moderated by Dan Garfield, Codefresh; Cornelia Davis, Amazon; Christopher Lane, Chick-fil-A; Mae Large, State Farm; Mike Bowen, Blackrock & David Lewis, Starbucks
Join a panel of GitOps end users from major companies in different industries – Blackrock, Chick-fil-a, Lyft, Starbucks, State Farm – as they tell stories from their GitOps journey. Attendees will hear about problems they sought to solve, challenges faced along the way, and how GitOps strategies and tools were used to make it work.
A
All
right,
we
are
ready
to
get
started.
We
have
an
incredible
panel
of
amazing
experience
a
whole
a
whole
lot
of
of
people
from
really
interesting
companies
with
really
interesting
use
cases,
some
of
them
very
famous
and
well-known,
some
of
them
less
famous
and
less
well-known.
So
I'm
hoping
we
can
expose
those
and
get
them
clear
for
those
we
are
going
to
do
some
q
a
during
this.
We
do
want
it
to
be
somewhat
of
a
live
session.
Now
there
is
one
trick
with
that.
A
C
Sure
I
will
start
I'm
cornelia
davis.
I
have
spent
the
last
decade
or
so
working
on
developer
platforms.
I
was
at
pivotal
working
on
cloud
foundry
and
then
I
was
at
weave
works,
which
was
really
just
an
extension
of
that
agenda,
which
is
to
continue
the
devops
agenda
into
the
space
of
git,
ops
and
I've.
Just
recently
moved-
and
I
am
at
amazon,
doing
alexa
artificial
intelligence
stuff
these
days.
So.
B
A
A
So
one
of
the
first
things
that
I
wanted
to
talk
about
is
the
announcement
of
the
get
ops
1.0
standard
and
this
idea
of
having
a
standard.
Now
none
of
you
had
the
benefit
of
a
standard,
but
there
were
a
lot
of
guide
posts
put
out
there
by
people
like
alexis,
richardson
and
others
who
were
writing
about
it.
So
maybe
does
anybody
have
like
a
strong
feeling
about
something
they
ran
into
when
they
were
adopting
git
ups?
That
really
surprised
them.
A
A
B
So
one
of
them
is
the
perception
switch
from
more
of
a
ci
ops,
driven
and
manual
processes
and
as
a
in
a
highly
regulated,
firm
moving
that
to
things
that
are
more
autonomous,
a
little
bit
higher
velocity
and
where
you're,
trusting
more
declarative
code
and
describing
policies
and
regulation
and
such
in
a
way
to
facilitate
the
get
ops
journey.
I
wouldn't
say
it
was
a
challenge,
but
it
was
a.
It
was
an
opportunity
that
was
was
one
that
took
a
little
bit
longer
to
realize.
A
B
It
did
it
did
in
pockets,
it's
a
big
firm,
so
it
came
up
in
pockets
but
typically
showing
seeing
is
believing
a
lot.
So
the
more
we
sat
down
and
worked
with
people
had
conversations
showed
them.
The
real
value
showed
them
where
we
were
getting
the
gains
where
we
still
had
some
rough
edges,
where
maybe
some
higher
order
tools
or
integrating
some
of
our
existing
tools
needed
to
be
done,
but
it
was
really
about
having
that
conversation
make
sure
we
were
collaborating
and
showing
that
it
it.
B
D
It's
actually
very
exciting
because,
prior
to
the
principles
right
you
it's
almost
like
you're
handed
a
clean
sandbox
and
it's
up
to
you
to
be
creative
about
it,
but
we're
also
very
intentional
to
share
out
whatever
it
is
that
we're
learning,
because,
yes,
we
have
a
large
community
within
state
farm.
But
we
all
understand
it's
a
larger
git
ops
community
that
we're
all
a
part
of.
G
C
There
there
is
something
that
I
would
add,
because
I
think
that
there's
two
principles
that
are
are-
and
one
of
them
is
not
expressed
quite
this
directly
in
the
principles,
but
the
two
of
them
them
that
come
together.
That
are
very
interesting
are
the
notion
that
the
automation
is
convergent,
and
that
is
back
to
the
point
that
you
just
made,
which
is
something
that,
from
a
principal's
perspective,
we
want
to
express
the
principles.
C
But
if
you
take
the
principles,
just
one
step
further
and
you
say
declarative
configuration,
what
does
that
imply?
That
implies
that
all
of
the
automation
that
is
tied
to
a
git
action
is
convergent
and
that's
very,
very
different,
and
that
is
in
fact
what
allows
the
other
thing,
which
I
think,
still
surprises
a
lot
of
people,
which
is
that,
when
you're
doing
get
ops
get
is
the
interface
for
operations
and
what
I
still
see
a
lot
of
people
doing,
and
you
mentioned
it
as
well
as
infrastructure
is
code.
C
Well,
I
have
infrastructures
code,
of
course
I'm
putting
that
in
a
source
code
control
system.
But
if
then
after
you've
checked
it
in
you
go
and
you
execute
a
script
somewhere
else
or
you
go
to
some
user
interface,
which
now
draws
things
from
that
git
repository,
then
you're
not
doing
git
ups
all
the
way.
It's
that
git
is
the
interface
for
operations.
It
is
now
you
have
to
get
away.
You
have
to
shift
that
mindset
from.
Oh
I've
still
got
my
consoles,
be
it
an
aws
console
or
a
vsphere
console
or
whatever.
No,
no.
No.
C
E
A
Yeah
this
is,
I
mean
something
that
famously
aws
and
cloudflare
both
had
very
well
known,
outages
that
happened
because
of
people
making
manual
configuration
changes,
one
of
the
other
kind
of
interesting
things.
That's
had
been
happening
with
git
ops.
That,
I
think
was
unexpected
for
a
lot
of
people
is
the
rise
of
edge
gitops
and
this
idea
of
having
sites
and
satellites
and
using
githubs
in
those
locations.
A
Chick-Fil-A
very
famously
came
out
two
years
ago
with
with
the
talk
about
how
they
had
a
kubernetes
cluster
in
every
store
and
it
kind
of
blew
everybody's
minds.
I
think
starbucks
is
in
a
followed
a
similar
kind
of
pattern,
and-
and
I
don't
know
if
state
farm
is
doing
something
similar,
but
do
you
want
to
maybe
to
the
edge
use
case?
Was
there
something
unique
about
git
ops
that
really
brought
you
guys
to
that.
F
Yeah,
so
I
mean
you
know:
managing
2600,
plus
clusters,
kind
of
drove
us
towards
that.
There's
really,
I
don't
think
an
effective
way
to
do
it
other
than
a
git
ops
process,
especially
when
you
have
this
fleet
of
clusters,
you're
managing
so
you're.
You
know
it's
not
just
one
cluster
and
those
components.
F
It's
every
single
one
and
rolling
it
out
has
to
be
happen
in
a
controlled
fashion
and
so
using
git
as
decentralized
tool
to
do
that
and
well-known
processes
that
all
developers
can
immediately
come
into
because
it's
scary
to
run
things
at
the
edge
of
the
restaurant.
You
can't
replace
them
easily
and
quickly,
like
you
can
in
the
cloud,
and
so
you
have
to
get
it
right,
and
that
involves
quite
a
few
gates.
F
F
A
E
So
there's
there's
this
thing
that
we
refer
to
as
starbucks,
and
I
know
chick-fil-a
deals
with
the
same
thing
when
you
have
all
these
stores
any
kind
of
configuration
change.
A
lot
of
times
requires
what
we've
referred
to
as
a
truck
roll.
I
don't
know
if
they
call
them
that
a
chick-fil-a
as
well
but
truck
bowls
are
expensive,
because
it
requires
a
technician
to
go
out
there
and
install
a
piece
of
hardware
or
software
or
make
a
a
change,
and
we
have
10
000
stores,
just
in
in
the
united
states
alone.
That's
anytime!
E
You
want
to
deploy
like
a
new,
a
new
application
that
requires
a
new
piece
of
hardware
that
is
really
expensive,
and
that
takes
a
long
time
and
to
roll
out,
and
so
the
idea
is
to
kind
of
bring
it
bring
it
together
into
you
know:
kubernetes
really
makes
sense
in
those
situations,
because
then,
if
you
can
run
all
your
applications
in
just
a
small
subset
of
hardware-
and
it
doesn't
require
a
charcoal
to
do
it,
you
can
just
deploy
it
out
to
that
location.
Then
that's
the
ideal
scenario.
F
We've
had
that
problem,
the
restaurants
are
have
limited
bandwidth,
so
even
like
pulling
down
images
at
the
restaurant
can
be
challenging
and
we've
had
to
really
strengthen
our
widen.
Our
bandwidth
constraints
still
allow
us
to
do
some
of
the
more
faster
things
we
like,
but
but
yeah.
We
run
to
that
even
like
pulling
logs
and
metrics
and
how
you
stream
that
you
have
to
think
very
very
carefully
about
how
you
pull
all
of
that
from
the
restaurants,
because
you
can
take.
You
can
just
swamp
the
network
and
take
down
sales
so,
but.
C
But
I
I
wonder,
but
I
wanna
I
wanna
knit
on
something
here,
because
you
said
all
at
once
and
yes,
I
think,
if
they're
all
going
at
once,
you
can
have
this
problem,
but
the
whole
point
I
think
of
this
is
that
you
achieve
a
certain
amount
of
autonomy
where
not
all
of
the
the
stores
need
to
go
all
at
once.
That's
the
beauty
of
distributed
systems
and
the
magic
of
distributed
systems
when
you've
done.
That
right
is
that
you
don't
have
these
bottlenecks,
and
so
that
you
know
I'm
sitting
here.
C
Thinking
of
course,
I'm
thinking
about
well,
how
do
we
build
that
into
the
tooling
that
supports
git
ops
so
that
we
can
take
it
advantage
of
this
inherent
advantage
of
autonomy
that
comes
with
distributed
systems
done
the
right
way?
So
how
can
we
do
that
so
that
it's
not
just?
Oh
we've
got
to
up
the
bandwidth,
but
maybe
there's
ways
that
we
can
navigate
through
that
hierarchy
of
distributed
systems
to
kind
of
balance.
The
load
interesting.
A
D
For
sure
opposite
to
the
many
clusters
that
you
guys
have
to
maintain,
we
introduce
get
ops
in
a
well
large
multi-tenant
cluster
super
large
with
production
workloads
so
having
to
figure
out.
There's
obviously,
relationships
that
had
to
be
figured
out
and
part
of
the
excitement
like
I
talked
about
earlier,
is
there's.
A
H
H
A
F
A
A
B
Reinforce
the
training
with
more
training,
because
it's
varying
degrees
right,
especially
when
you
get
into
the
kubernetes,
is
wrought
with
complexity.
It
doesn't
make
hard
things.
Easy
makes
hard
things
more
efficient,
more
distributed
more.
It
facilitates
a
a
better
path,
using
git
ops
as
the
delivery
mechanism
too.
So
not
just
people
that
knew
how
to
use
git,
but
good
git
practices
and
understanding
all
the
git
semantics
understanding.
B
Everything
there
is
to
know
about
shads,
one
of
the
big
values
that
we've
gotten
out
of
it
is
the
ability
to
roll
back
and
roll
back
very,
very
quickly
over
a
thousand
apps
across
150
clusters.
Two
short
years
ago,
we
had
five
clusters
a
handful
right
wow,
and
so
we
we've
scaled
up
very,
very
quick,
but
that
configuration
nested
configuration
remote
bases
with
customized
helm
dependencies,
whether
it's
an
add-ins
platform
service
level
thing
versus
a
tenant
application
thing.
We
also
support
single
tenant
versus
multi-tenant.
B
We've
had
both
clusters
given
privacy
right,
certain
apps
cannot
run
with
others
very,
very
specifically
and
intentionally.
So
in
that
model
we
do
have
to
kind
of.
We
don't
have
a
a
series
of
snowflakes.
We
have
two
snowflakes
right:
one
single
tenon
one's
multi-tenant,
but
the
way
we
reason
about
around
that
is
through
those
add-in
layers
right
through
those
segments.
We
call
them
that
go
into
every
cluster
based
on
its
composition
right,
but
to
get
training.
B
E
Better,
I
would
say
enforcing
the
pattern,
is
really
really
important
to
back
up
the
training,
because
if,
given
the
opportunity,
an
engineer
has
the
opportunity
for
a
back
door,
they
might
take
it
so
make
sure
that
you've
got
your
safeties
in
place.
You
know:
go
to
read
only
permissions
for
your
portals.
Do
things
like
that,
so
that
the
only
path
for
them
to
take
is
through
get.
E
F
B
I
Hello
how's
it
going
so
this
is
a
follow
on
from
actually
the
previous
question
developers
love
git,
but
what
about
so?
I'm
finding
myself
in
a
situation
now
where
someone
wants
to
use
hui
or
another
tool,
and
I
go
well-
you
know
in
read-only
mode
sure
I
don't
want
it
to
write
to
the
cluster
in
a
non-declarative
way.
Presumably,
new
tools
are
coming
online
that
are
kind
of
adopting
git
ops
principles,
new
gui
like
base
tools,
but
is
that
always
the
right?
I
I
guess
I'm
wondering
about
kind
of
performance
introducing
kind
of
interposing
get
in
between
rather
than
writing
directly
to
the
database
or
writing
directly.
To
whatever
is
that
is
that
the
model
is
that,
where
we're
going,
where
all
these
tools
start
to
use,
get
on
the
kind
of
on
the
back
end
so
that
we
can
get
out
the
flow
or
the
area
there.
So
I
can.
E
I
E
And
whatnot,
and
so
they're
asking
these
guys
to
receive
information
from
the
from
our
customer
and
then
it's
writing
to
git
and
then
that
get
code
is
then
driving
configuration
for
it.
So
and
what
I
would
like
to
see
for
a
lot
of
products
and
vendors
to
do
is
when
they
are
developing,
like
a
nice
portal
to,
instead
of
just
making
the
configuration
change.
E
That
applies
directly
to
the
infrastructure
or
cluster
that
it
writes
to
git,
which
then
can
flow
through
your
own
set
of
tools
and
and
workflows
like
ci
and
policy
validation,
compliance
and
that
sort
of
thing.
And
then
it
applies
to
the
infrastructure.
At
that
point.
So
I
really
do
like
the
idea
of
a
ui
writing
to
get
as
a
means
to
use
get
ops.
G
So
kind
of
going
off
that
as
well,
we
have
a
lot
of
operations
where
we
want
to
make
bulk
changes
and
then
query
and
git
is
not
always
the
best
language
or
mechanism
to
query
for
things
like
we
want
to
see.
You
know
how
many
of
our
applications
are
using
version
xyz
of
their
image
and
instead
of
going
directly
to
the
clusters,
because
we
might
not
have
access
to
them.
You
know
because
get
is
the
the
gitops
engine
is
the
only
thing
with
access?
E
So
I'll
answer
the
the
first
question
somebody
answered
can
answer
the
second
first
question
I
would
say
in
regards
to
like
having
some
visibility
into
what
versions
and
things
that
are
out
there.
We
use
monitoring
for
that.
So
there's
lots
of
tools
we
use
datadog
new
relic
and
things
like
that.
That's
what
we
use
to
gather
the
metrics
of
what
versions
things
are
on
what
state
they're
in
and
things
like
that.
We
don't
actually
query
get
for
that
kind
of
thing.
A
A
D
For
question
one
we
rolled
out
our
own
framework,
it's
a
it's
the
don't
mind
us
we're
just
listening
type
of
deal
because
a
code
change
as
soon
as
you
push
it
remotely
till
the
time.
It's
realized
to
prod
there's
lots
of
steps
that
happen
there
and
we
do
watch
that
we
capture
all
those
milestones.
Those
events
from
your
testing
scans
chaos,
tests,
performance,
all
the
levels
of
testing
right
until
ultimately,
it's
going
to
get
applied
to
production.
D
So
we
we,
we
set
up
all
that
infrastructure
in
aws,
again
proprietary
homegrown
solution,
but
we
harvest
that
vast
amount
of
data
right
to
increase
our
comfort
for
this.
This
is
this
is
good
to
go.
This
can
go
to
prod
your
second
question
about
large
changes.
We
just
take
advantage
of
windows
to
do
that.
We
primarily
use
terraform,
we've
gotten
really
good
at
writing,
sweeping
changes
and
I'll
we'll
tell
more
at
our
session.
I
don't
want
to
steal
priyanka
ravi's
thunder
here.
C
Where
there's
some
differences
and
of
course
it
leverages
things
like
customize
and
so
on
for
those
differences.
But
there
is
emerging
this
hierarchical
model
so
that
you
can
go
and
say
the
things
that
are
common
across
my
clusters.
So,
instead
of
putting
everything
in
a
single
get
repository,
it
allows
you
and
by
the
way
these
profiles
are
get
repository,
get
repositories
themselves.
C
So
it
essentially
allows
you
to
have
kind
of
base,
get
repositories
and
then
specialized
get
repositories
so
that
the
things
that
you
do
want
to
roll
out
at
scale
you
can
from
you,
don't
have
to
go
and
update
it
in
a
bunch
of
different
repositories.
You
can
update
it
in
a
root,
git
repository
and
it'll
filter
down
for
you.
So
that's
something
that
works
has
in
the
works.
A
So
another
topic:
when
we
actually
started
the
get-offs
working
group,
we
had
discussions
with
aws
and
with
azure
who
were
both
founding
members,
and
I
remember
asking
them
like:
why
are
you
interested
in
pushing
to
get
off
standard?
Like
we've
worked
in
code
fresh?
I
know
why
we're
interested
we
both
make
and
sell
tools
that
are
new
get
ups.
You
know
and
the
answer
that
I
heard
was
well.
A
We
want
to
have
a
standard
that
we
can
hold
up
against
our
tools
and
say:
are
we
building
cloud
components
that
are
compliant
that
can
meet
this
standard
that
are
going
to
work
with
this?
So
now
that
you
work
at
aws,
I
was
hoping
that
you
could
give
a
very
hard
and
public
commitment
about
everything
in
aws.
C
Well,
my
big
cop-out
is
that
I
don't
actually
work
for
aws.
I
work
for
alexa,
so
I'm
not
at
liberty
to
make
any
statement
like
that.
Even
if
I
was
at
aws,
I
wouldn't,
but
the
thing
that
I
will
tell
you
that
is
not
unique
to
aws.
It
is
something
that
amazon
has
across
the
board
is
we
are
customer
obsessed,
and
I
can
tell
you
that
the
reason
that
amazon
that
aws
participated
in
the
getups
working
group
right
from
the
beginning
was
because
we
this
serves
customers.
C
G
D
H
G
H
We
had
a
question
asking
github's
practitioners
identified
configuration
management
at
scale
as
an
unsolved
problem
in
the
community.
How
do
you
handle
this,
and
what
more
can
the
industry
do
to
solve
this
problem
and
she
was
referring
to
things
like
customize,
helm,
etc?
That
manageml
configuration
files.
F
A
Awesome
well,
thank
you.
Everyone
we're
out
of
time
on
this
panel,
but
please
chase
these
people
down
as
they
leave
the
room.
I'm
sure
they
would
love
to
have
more
questions
from
you.
Thank
you.
So
much
I
mean
there's
an
incredible
amount
of
experience.
You
are
the
pioneers
of
git
ops.
You've
been
doing
it
for
a
few
years,
massive
scale
massive
success,
and
it
really
shines
away
for
all
of
the
rest
of
us.
So
thank
you
for
that.
Thank
you.