►
From YouTube: GitOps in the Real World: Opportunities for Developer Experience Im... Christopher Lane & Alex Crane
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
GitOps in the Real World: Opportunities for Developer Experience Improvement - Christopher Lane & Alex Crane, Chick-fil-A
GitOps is a pattern for managing the state of Kubernetes clusters using git as the source of truth. The entire state of the cluster is declared in manifests stored in git repositories and any changes to the manifests follow well-known git processes. Once the manifests are version controlled in git, then there's a number of state reconcilers (Flux, ArgoCD and the like) that can automatically apply changes from the repository. However, this leaves a significant gap in the process: How do we build and get the manifests *into the git repositories* in the first place? This talk will walk through Chick-fil-A's experiences with GitOps to manage the state of our production clusters at scale and offer what we see as opportunities to improve the frontend of the process.
A
Well
good
afternoon,
my
name
is
christopher
lane
and
my
co-presenter
today
is
alex
crane.
We
are
both
members
of
the
enterprise
architecture
team
with
chick-fil-a
and
we're
going
to
talk
about
get-offs
in
the
real
world,
improving
the
developer
experience-
and
this
is
based
on
our
experiences-
managing
chick-fil-a's,
customer-facing,
digital
properties.
A
So,
first
a
little
bit
about
who
chick-fil-a
is.
We
are
privately
owned.
Quick
service,
restaurant
based
in
atlanta,
georgia
chick-fil-a
began
with
a
simple
idea
from
our
founder
truett
kathy
to
fill
the
needs
of
his
customers.
The
chick-fil-a
chicken
sandwich
very
famously
we've
grown
from
that
idea
to
more
than
2
400
restaurants,
mostly
in
the
u.s,
but
we've
opened
locations
in
canada
and
puerto
rico.
In
the
last
few
years.
A
Our
per
restaurant
scale
is
significant.
We
have
by
far
the
highest
per
restaurant
sales
in
the
qsr
industry.
Currently
more
than
20
percent
of
revenue
is
flowing
through
our
digital
channels.
Our
mobile
app
chick-fil-a
one
and
chick-fil-a.com
drives
the
vast
majority
of
these
sales
and
we've
seen
significant
growth
in
these
channels.
Since
march
2020,
when
the
impacts
of
covet
started
in
north
america,.
A
So
I
want
to
talk
a
little
bit
about
what
our
back
end
looks
like
for
our
digital
properties.
We
we
call
that
back
in
dxe
for
the
digital
experience,
and
this
is
the
number
of
requests
through
dxe
on
a
given
day.
There's
nothing
special
about
this
day.
I
just
picked
a
recent
one,
but
all
the
times
are
us
eastern,
but
the
problem.
The
pattern-
probably
isn't
terribly
surprising.
A
Together,
this
team
pushes
thousands
of
commits
and
opens
hundreds
of
prs
every
single
day.
I
really
like
this
github
insight
graph
because
it
looks
like
we
have
no
issues
which,
of
course
we
do,
but
we
just
track
them
elsewhere,
but
our
application
platform
is
entirely
based
on
git
ops,
so
all
of
our
developers
need
to
transverse
traverse
this
process
to
get
their
code
deployed.
A
We
have
no
blackout
times
where
we
prevent
deployments,
so
we
need
this
process
to
flow
smoothly.
At
all
times,
alice
is
going
to
explore
a
little
bit
more
about
what
we
discovered
using
git
apps
at
our
scale,
but
I
think
it's
helpful
to
set
a
little
context.
So
let's
talk
a
little
bit
about
what
the
dxe
architecture
looks
like
in
the
get
out
process
behind
it.
A
So
this
is
a
high
level,
dt
architecture,
it's
relatively
straightforward
requests
from
cfa1,
chick-fil-a.com
and
other
digital
properties
are
routed
to
our
services
running
in
eks.
We
have
clusters
in
both
u.s
east
1
and
u.s
west
2,
though
the
ladder
is
mostly
for
failover.
The
setup
is
the
same
in
in
both
regions
and
we
regularly
route
all
of
our
traffic
to
us
west
2
to
make
sure
there
aren't
any
hidden
dependencies
in
there.
Our
services
are
mostly
java
spring
movement
applications
with
some
mixture
of
go
and
python
apps.
A
A
The
app
data
from
both
dynamo
and
the
analytics
databases
are
stream
to
rs3
back
data
lake
and
the
digital
born
order.
So
digital.
The
orders
that
start
with
chick-fil-a.com
and
cpa1
are
combined
with
the
traditional
orders
so
that
our
analytics
teams
have
a
complete
picture
of.
What's
going
on
from
any
given
day.
A
A
Cap
has
opinionated
defaults
and
you
know
you'll
get
the
you'll
get
those
opinionated
defaults,
but
it
does
allow
flexibility
for
teams
to
switch
things
out,
say
the
get
ops
operator
for
a
custom
one
or
different
one
things
in
that
layer,
but
the
base
layer
is,
is
eks
on
top
of
oops.
Sorry
that
when
I
go
back
there,
we
go
the
base
layers
eks.
On
top
of
that
is
the
the
syslayer
which
includes
things
like
iron
rolls
or
service
accounts,
external
dns
for
managing
around
53
entries,
the
aws
load
balancer.
A
A
The
prometheus
and
grafana
stack,
which
we're
using
prometheus
operator
for
we're
using
thanos
operator
for
managing
long-term
storage
of
those
metrics
and
then
finally,
a
tool,
we're
very
excited
about
speed
scale,
which
we
use
for
capturing
traffic
in
one
environment
and
replaying
in
another
speed.
Sales
folks
are
here
today
if
you'd
like
to
talk
to
them
a
little
bit,
but
we're
excited
about
that
and
the
the
top
layer
is
our
app
layer
and
we
draw
a
pretty
distinct
line
between
what
cap
manages
and
what
our
dev
teams
manage.
A
A
A
We've
moved
over
to
github
actions
last
year
from
jenkins,
the
app
containers
built
all
tests
are
run
and
we'll
assume
a
happy
path
here
and
that
they
all
pass
and
the
image
is
pushed
to
artifactory
next,
the
workflow
pulls
the
base
manifests
from
for
the
app
from
our
customized
repos
and
our
customized
referral
our
set
of
repos
that
we've
developed
that
contain
our
standard
manifests
for
various
applications.
So
you
know
job
api
go
api
python
api
react
app.
A
All
of
those
are
stored
within
the
set
of
customized
repos
and
the
base
manifests
are
merged
with
any
app
specific
overlays
found
in
the
app
repo.
At
this
point,
developers
are
free
to
patch
anything
in
the
standard
manifest,
but
we
offer
sane
defaults
that
work
for
most
use
cases,
and
this
provides
kind
of
an
easy
on-ramp
for
developers
to
get
started
quickly
and
increase
their
k,
knowledge
incrementally
and
change
things
as
they
need
to.
Without
having
to
wade
through
a
gigantic
pile
of
yaml,
the
the
complete
merge
manifest
are
then
committed
to
a
special
recall.
B
Thanks
christopher
and
I
will
probably
hop
skip
through
the
slides
a
little
bit
since
we
have,
I
think,
we're
still
gonna
run
up
on
a
similar
time,
stop
that
we
had
before
the
kind
of
the
interlude
there,
but
so
kind
of
one
quick
slide
here.
You
know
this
is
kind
of
the
simple
version
right.
You've
got
users
contributing
stuff
to
the
git
repo
you
have.
B
So
christopher
kind
of
showed
the
the
real
version
right.
You
know
we
we
sell
this
to
directors,
etc.
This
is
what
it's
gonna
look
like
and
then
we
know
it's
gonna,
look
like
what
christopher
showed
so
I'll
jump
right
into
some
pain
points
and
then
go
to
a
kind
of
a
couple
examples
that
kind
of
show
these
things
but
yeah.
So
so
it's
get
ups
right
and
actually
it
was
great.
This
panel
covered
like
50,
of
what
we
wanted
to
cover
in
the
slide
deck.
B
B
So
you
know
how
do
you
do
you
know?
Do
you
do
the
pull
request,
validation
and
testing
that
catches
some
of
it,
but
that's
not
terribly
proactive
to
the
user
right,
they
do
it,
they
submit
it.
Then
they
find
out
it's
almost
like
the
mainframe
days,
but
I
mean
better
and
faster,
hopefully,
but
still
a
little
bit
delayed.
B
You
know.
So
you
know
what
are
some
solutions
you
know
for
some
of
that.
You
know
ide
plug-ins
as
well
right.
You
can
do
all
that
lending
front
load,
all
that
and
that's
good,
but
some
of
the
users
still
aren't
kind
of
ide
people
right,
I
mean
think
about
today.
Some
people
are
command
line
people
some
people
like
web
uis.
Some
are
love
doing
it
in
the
ide
and
kind
of
the
other
piece.
There
is
and
again,
as
was
mentioned
in
the
panel
aws
azure,
google,
argo
cd.
B
B
B
You
lose
audit
history
of
what's
here
unless
argo
separately
fully
implements
that
you
lose
re-creatability
right,
think
about
anyone,
who's
create
had
a
team
that
turned
out
they
created
a
database
or
similar
through
a
web
ui
and
a
cloud
provider
and
later
on,
you
need
to
go
recreate
it.
You
got
no
idea
what
that
state
was.
B
You
know
an
example.
You
know
with
git
ops,
we've
fixed
those
two
problems
by
doing
this
in
git,
we
know
the
audit
trail.
We
can
recreate
it,
we
can
re-pull
it.
But
what
are
the
field
values
that
I
can
put
here,
which
ones
are
not
listed
in
some
basic
template
where
you
have
things
that
would
be
auto,
complete
clusters
to
be
targeted,
etc.
B
Usually
you
know
you
go
ahead,
you
get
it
as
close,
as
you
know,
you
think
you
submit
it
and
then
you
wait
for
things
to
explode,
hopefully
in
the
pull
request,
but
sometimes
it
gets
all
over
the
cluster
before
you
find
out.
That
thing
is
exploding,
and
hopefully
that's
also
not
scary.
Hopefully
your
automated
tooling
for
rollbacks
etc
makes
that
a
clean
and
transparent
process,
but
as
a
dev
I
hate
waiting
five
or
ten
minutes.
B
Sometimes
to
find
out
that
the
thing
I
deployed
actually
exploded
and
they're
waiting
for
a
production
fix
so
getting
to
some.
You
know
proposed
solution,
so
one
I
went
ahead
and
crossed
out
entirely
was
the
abandoned,
get
ops
in
favor
of
some
new
model.
I
just
went
to
get
ups
and
I
really
don't
want
to
change.
No,
I
think,
there's
a
lot
of
great
value
found
in
get
ops
and
the
auditability
what's
there
and
for
those
of
us
that
are
deep
in
the
weeds
of
some
of
the
stuff.
B
You
know
it's
really
nice
being
able
to
be
in
the
guts
of
it
and
not
pulled
out
into
some
menu,
but
you
know
I.
I
just
think
how
great
it
would
be
when
we
look
at
ui
editing
with
git
as
the
data
store.
If
I
jump
back
a
few
slides
real
quick,
we
see
in
this
top
right
hand
corner.
It
might
be
too
small
to
see
on
the
deck,
but
it
says
edit
as
yaml
right.
The
number
of
cloud
providers
number
other
pieces
in
various
places.
B
B
Those
who
do
need
to
go
edit,
something
more
directly
in
the
repo
and
then
one
other
kind
of
interesting
option.
There
is
ide,
plugins
and
also
command
line
right
and
those
both
both
would
and
could
function
similarly
to
the
ui
piece
there.
But
the
core
part
of
that
being
that,
after
you
made
your
changes,
done
your
validation,
some
of
it,
it
gets
committed
to
get
and
is
able
to
follow
all
the
rest
of
those
deployment,
practices
and
validations.
B
A
Yeah,
so
we
always
like
to
finish
with
a
quote
from
our
founder
truett
kathy,
and
we
pick
the
no
goal
it's
too
high
to
climb
or
sorry
no
go
so
high.
If
you
climb
with
care
and
confidence,
this
is
a
quote:
that's
particularly
meaningful
to
the
the
dxe
team.
We
actually
named
one
of
our
conferences
climb
with
claire,
but
you
know
please
check
us
out
at
our
tech,
blogs,
tech,
post,
techpost.medium.comtechblog.