►
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
A Day in the Life of the GitOps Platform Team - Mae Large & Priyanka Ravi, State Farm
Mae Large and Pinky Ravi have been a part of the GitOps team at State Farm since its inception in February 2019. GitOps at State Farm is adopted in all our strategic cloud platforms - AWS, Multi-tenant Kubernetes (on-prem), and Cloud Foundry (on-prem). This talk will walkthrough how we scaled GitOps in our large organization. The GitOps team is 5-person team. We have successfully rolled out GitOps, supported our customers, and continue to advance the offering towards observability and resiliency of the applications or products using GitOps. We will give an insight on how our typical day is like and also highlight the different bumps and challenges (including outages) we’ve faced and how these all became opportunities for us to continuously improve.
A
B
A
A
As
soon
as
we
understood
the
gravity
and
the
benefits
that
we're
going
to
get
out
of
get
ops,
we
knew
we
needed
to
go
to
work
and
by
we
I'm
talking
a
three-person
git
ops
platform
team
that
that
started
this
all
myself,
russ
and
russ
who's,
our
engineer,
who's,
probably
watching
and
and
pinky.
Here
we
were
og
of
the
get
off
platform
team.
So
way
before
get
ops
was
generally
available
at
state
farm.
We
spent
cycles
in
making
sure
that
everybody
gets
excited
about
get
off.
A
Yes,
we
would
still
talk
about
the
current
deployment
release
process
that
we
have,
but
we
really
would
spend
a
significant
amount
to
get
everyone
excited
about
get
offs,
and
this
is
to
complete
that
get
ups
pitch
to
get
this
organizational
change
to
happen.
So
out
of
the
pitch,
we're
really
different
reactions,
one
from
our
compliance
folks,
it's
about
hey,
prove
to
me
that
in
the
case
of
an
audit,
we
are
able
to
pass
or
fulfill
requirements
one
through
five.
A
The
second
reaction
is
really
from
our
first
line
leaders
the
one
signing
off
on
changes
affecting
their
products,
and
so
it's
a
mix
of
hey,
I'm
not
very
familiar
with
git
workflows,
which
was
brought
up
earlier
in
the
panel
discussion
and,
on
the
other
hand,
managers
who
are
really
excited
about
the
transparency
like.
Oh,
I
can
actually
see
the
files
change.
The
actual
lines
of
code
changed
to
realize
this
feature
that
empowers
them
and
enables
them
to
better
review
the
evidence.
A
The
testing
that
go
along
with
it,
giving
them
the
better
confidence
that
this
is
good
to
go
to
production
last
and
certainly
not
the
least
I'm
talking
about
the
7
000
developers
who
were
really
excited
about
get
ops
because
they're,
the
ones
who
were
catering
to-
and
I
remember
this
one
feedback
from
an
engineer
in
in
atlanta-
and
he
said,
if
you
all
can
realize
this.
If
you
can
make
get
off
the
thing
at
state
farm,
you
all
will
be
our
heroes,
and
so
we
knew
we
needed
to
go
to
work.
B
Yeah,
so
obviously,
in
order
to
make
get
off
the
thing
at
state
farm,
there
was
a
lot
of
enablement
that
we
would
do.
Starting
with.
How
do
we
even
set
up
the
config
repos?
How
do
we
make
sure
that
they
meet
the
compliance
standards
that
are
set
forth
to
us?
We
actually
landed
on
using
the
terraform
gitlab
provider,
and
that
way
we
could
actually
take
advantage
of
like
infrastructures
code
benefits.
You
know
reusability,
manageability
and
lots
of
other
things
right.
B
B
Another
thing
that
we
have
are
a
handful
of
scripts
and
some
run
on
one-off
situations,
and
then
we
have
a
handful
of
other
ones
that
actually
run
on
a
scheduled
pipeline.
One
of
those
is
one
that
we
call
affectionately
the
enforcer
and,
basically
that
one
runs
nightly.
It
uses
it
utilizes,
terraform
enterprise,
and
so
that
runs
nightly
to
make
sure
that
they're
all
still
meeting
those
compliance
standards
that
are
set
forth.
B
It's
a
place
for
our
consumers
to
go
get
announcements
to
have
answers
to
frequently
asked
questions
to
like
see
our
some
of
our
other
enabled
api
and
cli,
and
then
also
the
way
that
our
consumers
interface
with
us
directly,
is
through
our
rocket
chat
channel
and
that's
where
they
come
and
ask
us
questions
and,
and
other
people
can
kind
of
see
the
questions
that
they
ask
as
well
and
get
theirs
answered
as
well.
And
then
one
thing
that
mae
is
going
to
touch
on
in
a
in
a
bit
is
our
monitoring
and
dashboards?
B
B
For
us
it
could
be
a
pipeline,
it
could
be
flux
for
kubernetes,
and
so
that
will
do
the
deployment
and
then
the
other
thing
that
happens
is
another
thing
that
we
created
internally.
It's
our
git
ops
api
and
that
actually
kicks
off
using
a
web
hook
from
git
lab.
That
creates
a
a
change
record
in
our
asset
management,
repo
and
then
so.
B
But
basically
our
last
one
and
the
most
recently
enabled
one
thanks
to
nimi
over.
There
is
cloud
foundry
and
so
that
one,
that
one
is,
we
actually
ended
up
utilizing
pipeline
templates
to
make
that
process
more
friendly
for
our
consumers.
And
then
we
also
so
that
actually
utilizes
the
cloud
foundry
cli.
We
had
to
adjust
some
of
the
config
repo
modules
and
then
we
also
use
short-lived
access
just
to
fold
to
fulfill
deployments
through
uaa,
okay,
so
continuing
with
kind
of
like
the
the
cloud
foundry
side.
B
Obviously,
as
the
get
ups
platform
team,
a
huge
part
of
what
we
do
is
migration.
All
the
time,
so
one
thing
that
we're
still
working
on
is
migrating
from
jenkins
to
git,
lab
ci,
and
that
is
actually
still
a
work
in
progress.
Our
cloud
foundry
users
are
mostly
on
jenkins
and
we
actually
have
this
dashboard.
That
shows
us
where,
like
that
status
is,
and
we
can
monitor
what
the
progress
is
on
that
migration.
A
All
right,
another
huge
part
of
being
a
get
ups
platform
team
is
education.
Earlier
we
said
training,
training,
training
right,
it's
it's
the
best
way
to
realize
a
large
organizational
change
like
adopting
git
ops.
So
over
time
we
have
gotten.
It's
now
become
an
expectation
that,
as
soon
as
we
enable
git
ops
into
one
of
our
strategic
platforms,
it's
a
given
that
we're
going
to
do
a
road
show.
A
We
focus
obviously
on
the
the
core.
The
foundational
pieces
as
pinky
walked
us
through,
but
then
focus
on.
How
do
you
really
apply
your
changes
to
this
particular
platform
versus
this
particular
platform,
and
as
pinky
mentioned,
we
recently
enabled
it
for
cloud
foundry
and
we
just
did
a
road
show
for
that.
All
right
beyond
the
road
shows
I
touched
on
this
earlier.
A
We
realized
we're,
obviously
part
of
a
larger
community,
so
we've
really
been
intentional
in
sharing
our
progress
beyond
outside
a
state
farm,
and
it's
through
that
that
we
really
got
to
connect
with
wonderful
people.
A
lot
of
them
are
here
to
help
get
that
help
that
dialogue
to
start
going
and
exchange
ideas
on
how
to
solve
those
problems.
B
All
right,
so
basically,
I'm
gonna
walk
through
like
what
our
day-to-day
looks
like
with
support
and
all
that.
But
before
I
do,
I
want
to
mention
that
we're
actually
a
five-person
team
right
now,
so
we
went
up
we
upgraded
from
the
three-person
team,
we
started
with
and
it's
nice
because
there's
may
obviously
she's
our
manager
and
she
does
a
lot
of
technical
stuff
too
and
then
there's
nimi
who's
more
like.
B
She
knows
more
about
cloud,
foundry
she's,
our
expert
on
that
and
then
there's
russ,
our
engineer,
who
is
our
aws
expert,
then
I
focus
more
on
kubernetes
and
then
we
actually
our
new
member
adam.
He
does
a
lot
of
our
data-driven
stuff.
So
that's
really
cool.
It's
like
nice
to
have
that
balance.
Okay,
so
our
day-to-day
we!
Actually
we
were
getting
really
swamped.
It
was
really
hard
for
us
to
manage,
like
keeping
up
with
new
tasks
and
then
also
like
trying
to
answer
support
questions
all
the
time.
B
So
we
actually
came
up
with
a
system
that
we've
been
recently
doing
where
we
have
a
weekly
support
rotation.
So
one
of
us
will
be
like
on
call
and
we'll
be
the
ones
that
will
be
monitoring
that
get
ops
channel
the
yeah
the
get
off
channel
and
rocket
chat
that
I
mentioned
earlier
and
that's
actually
been
working
out
really
well,
because
we
don't
have
to
constantly
context
switch
as
much
as
we
used
to
another
thing
that
we
do
daily.
B
We
have
frequent
touch
points
with
our
platform,
admins,
just
to
make
sure
that
we're
all
on
the
same
page
and
kind
of
talk
about
if
they've
heard
any
new
tech
and
we've
heard
any
so
we're
all
just
like
you
know,
kind
of
hashing
it
out
it's
kind
of
fun
and
then
outside
of
state
farm.
We
are
pretty
active
in
like
github
discussions
about
flux,
terraform
gitlab,
there's
a
bunch
of
other
things,
and
then
we
were
really
blessed
to
have
weave
works.
B
A
Next
is
the
fun
part
outages
right,
there's
no
support,
that's
complete
without
having
to
deal
with
outages,
and
so
as
pinky
and
I
were
preparing
for
this.
We
kind
of
reflected
on
what
were
at
least
the
top
two
top
two
things
that
we
had
to
rally
around
drop
everything
we're
doing
right,
recover
our
customers,
so
the
top
has
to
do
with
secrets.
A
Thankfully
those
exposures
that
happen
were
internally
initiated
by
our
penetration
test
team,
but
the
key
takeaway
is
over
time,
we've
gotten
better
at
limiting
the
number
of
secrets
that
get
passed
around
into
the
the
components
or
the
solution
stack
behind
github,
the
less
that
we
require
expect
our
customers
to
provide
secret
a
secret
b
just
so
it
can
do
such
and
such
deploy,
keys
and
etc.
The
better
we
still
follow
the
convention.
A
The
second-
and
this
is
funny
because
it
happened.
While
I
was
on
vacation-
and
I
came
back
like
oh
crap
type
of
news,
so
flux
get
deployed,
those
secrets
are
all
still
there,
but
if
you
look
into
any
one
of
them,
the
flux
get
deployed,
the
actual
data,
the
actual
secret,
is
gone
and
so
again
following
a
standard
operating
procedure,
dropped
everything
we're
doing
and
we
fixed
it
by
rolling
out
just
a
simple
bash
script
that
looped
through
all
the
name,
spaces
that
were
impacted
exact
into
the
flux.
A
The
running
flux,
pod,
grab
the
identity
that
we
needed
and
patch
that
secret
again
to
recover
our
customers.
But
that's
just
step
one
right.
The
real
benefit
is
how
do
we?
How
do
we
mature?
How
do
we
learn
from
this?
And
so
we
started
with.
We
know
the
time
frame
on
when
the
situation
occurred
right.
So
it's
a
lot
of
combing
through
logs.
A
So
I
say
I
say
it's
a
lot
of
splunking
and
then
out
of
that
right
that,
in
addition
to
obviously
understanding
the
logs,
it's
also
working
with
our
kubernetes
operators,
and
so
it
oil
it
all
boiled
down
to
the
fact
that
there
is
a
worker
node
that
had
gone
bad.
It
was
cubelet
that
would
report
itself
as
hey,
I'm
still
part
of
this
cluster,
but
other
times,
I'm
I'm
not,
and
so.
A
Okay,
that's
a
contributing
factor,
but
what
rendered
cubelet
in
that
bad
state
and
really
it
came
down
to
again
lots
of
working
sessions,
but
it's
the
cni
stuff
that
was
getting
flaky.
So
it's
unable
that
this
particular
worker
know
was
unable
to
report
itself,
and
so
it
got
into
a
taint
manager,
execution
error
that
just
dropped
the
the
flux
pods,
and
so
the
takeaway,
though,
is
yes.
Now
we
understand.
Why
did
this
happen
right?
A
and
over
time,
we've
gotten
better
at
the
metrics
and
observability.
So
it's
an
expectation.
It's
a
given
that
a
lot
of
the
pieces
that
make
up
the
getup
solution
stack
come
with
prometheus
and
grafana
the
alerts,
and
we
take
action
and
that's
for
the
onboarding
app
and
the
ui
that
pinky
walked
us
through.
A
A
And
that's
really
what's
getting
into
a
couple
of
dashboards
that
we
took
a
screenshot
of
one
is
three
three
metrics
that
I'm
really
excited
about.
One
is
deployment
events,
how
many
people
are
actually
using
git
ops
and
by
which
platform
aws
being
the
top
consumer
of
git
ops
at
state
farm
next
is
change,
lead
time
and
welcome
to
further
dialogue
on
this,
but
how
we're
measuring
change
lead
time
right
now
is
how,
from
the
moment,
a
code
change
is
pushed
remotely
in
a
source
repo.
A
We
captured
that
time
frame
from
that
point
until
that
same
commit
sha
is
applied
and
merged
in
your
target,
config
repo,
which
means
it's
realizing
to
your
target
environment,
your
production,
environment,
another
side
effect
or
another
metric
that
got
added
as
a
result
of
change.
Lead.
Time
is
change,
size
which
it's
super
exciting
to
us,
because
github
is
about
small
miniature
changes.
A
Next
is
hardening.
I
already
touched
on
this
earlier
right
about
lessening
the
number
of
secrets
that
we
pass
around,
but
it
also
goes
without
saying
that
we
we
embrace
policy
as
code
for
our
terraform
related
resources
we
use
sentinel
and
for
our
kubernetes
cluster.
We
started
using
kyberno
to
enforce
tenant
isolation
last
and
certainly
not
the
least,
is
governance
and
a
lot
of
this
leading
up
to
governance.
A
We
were
good
at
gathering
the
data
making
meaning
out
of
the
rich
deployment
githubs
deployments,
and
now
this
is
a
recent
enhancement
that
we
just
added.
We
are
able
to
in
real
time
alert
the
necessary
stakeholders
when
there
is
a
merge
request
that
gets
applied
in
the
config
repo
without
any
prior
approval,
without
any
first
line
leader
approval,
because
at
the
end
of
the
day,
we
still
have
that
compliance
requirement
that
we
have
to
abide
by.
B
A
A
That's
a
great
question
and
we
can
we
can
tag
team
on
answering
the
ones
that
we
remember
we're
heavy
on
measuring
adoption.
First
right
with
cloud
foundry
recently
enabled
another
bit.
Is
here's
a
product,
we're
hybrid
cloud,
we're
multi-cloud
right,
so
we
have
the
ability
to
capture
here's
product
a
and
where
does
it
have
footprint
in
our
many
strategic
platforms,
trying
to
think
if
there's
anything
else,
that's
like
kind
of
worthwhile,
even
our
even
our
cookbook,
we
measure
how
many
folks
really
get
value
out
of
the
different
sections.
It's
funny.
A
All
data-driven
decisions
even
earlier
when
there's
a
question
about
deployment
events
and
when
you
have
massive
changes.
Well,
we
have
visibility
into
how
often
what
time
of
the
day
and
what
time,
what
time
in
the
week,
what
day
in
the
week
do
these
deployments
happen?
So
we
have
an
informed
decision
on
when
to
do
sweeping
changes
like
this.
Does
that
help
okay.
E
B
Oh
yeah,
I
guess
I
should
okay,
so
that
was
actually
like.
Primarily,
I
think
I
took
the
lead
on
that
last
year.
Yeah,
there
was
it's
a
little
hard.
I
think
too,
when
there's
an
established
platform
team,
you
kind
of
have
to
get
a
working
balance
like
you
have
to
make
sure
that
you're
not
stepping
on
toes,
but
you
want
to
get
them
on
board
as
well.
B
We
were
lucky
they're,
they're,
very
willing
to
hear
us
out
and
you
want
to
describe
who's
they,
oh
they
as
in
the
platform
team,
are
on
prem,
kubernetes
platform
team
they're.
Already
there
right,
they
were,
they
they've
been
there
for
years,
our
operators
so,
and
basically
it
was-
I
mean
it
was.
It
was
really
nice
because
there's
a
lot
of
documentation
out
there
on
how
to
do
it.
I
don't
think
we
found
too
much
hardship
on
that.
B
It's
nice
also
having
a
dedicated,
get
ops
team,
because
that's
our
jobs
right.
We
we
were
able
to
mob
on
it
and
make
it
happen
and
like
actually
sit
there,
there
were
countless
meetings.
I
mean
that's
what
we're
going
through
even
now.
Every
morning
we
have
a
recurring
meeting
on
setting
up
plugs
too.
So
basically,
it
was
just
a
lot
of
trial
and
error.
B
A
E
A
C
Okay,
so
you
represent
a
large
commercial
organization
bravely
and
you
know
traveling
down
the
github's
row.
So
I
have
a
multi-part
question.
So
since
you
operate
in
a
large
organization,
two
guys
have
a
semblance
of
a
sort
of
unified,
enterprise-wide
get-up
strategy,
both
on
the
methodology
and
tool
chain
and
infrastructure
side.
That's
sort
of
the
first
part
of
the
question,
and
second,
do
you
guys
incorporate
any
runtime
policy
or
what
other
way
image
security
scanning
and
do
you
have
any
policy
around
that
how
to
incorporate
it
and
get
apps
process?
C
C
B
B
A
If
I
may
add,
okay
we're
not
to
the
state
that
we
want
to
be
yet
we
realize
we
have
kubernetes
customers,
not
just
on-prem.
Yes,
it's
a
large
multi-tenant
cluster,
but
we
have
chris
here
for
also
from
state
farm
who
has
this
little
eks
cluster,
so
our
desired
state
is
to
continue
to
use
terraform
and
even
how
we
stand
up
and
standardize
and
how
we
stand
up
these
clusters.
B
A
B
D
D
B
So
I
think
we
got
this
question
online
too
for
me
earlier
and
the
so.
I
have
mixed
feelings
on
this.
I
mean
I
know
that,
like
we
obviously
have
created
a
heavy
reliance
on
our
gitlab
instance,
and
we
have
seen
it
be
down
before
my
opinion
is
that,
like
if
git
lab
is
down,
you
can't
run
the
test,
you
can't
run
the
pipelines
right,
so
your
code
really
shouldn't
be
going
to
pride
either
way,
but
and
we're
I
mean
there
was
something
you
had
earlier
too.
I
think
to
answer
that.
A
Reconciles
the
concept
of
reconciling,
but
that's
really
a
break
glass
scenario
and
not
may
may
or
may
not
cater
into
dr
stuff
right.
So
it's
on
git
we
can
roll
back
a
commit
sha.
I
also
understand
that
we're
talking
lots
of
components,
so
good
luck,
rolling
back
the
in
the
right
order
right
so
that
that's
where
we're
at
we're
not
to
that
state.
A
Yet
where
we
can
accurately
report
okay,
it's
this
entire
stack
of
components
that
make
up
a
product,
but
I
will
counter
that,
though,
right
with,
like
I
said
earlier,
small
miniature
changes
going
straight
to
production,
less
context.
Switching.
So
if
what
I
change
breaks,
I
know
exactly
how
to
fix
it
before
I
compound
that
with
other
problems.