►
From YouTube: Eating Your Vegetables: How to Manage 2.5 Million Lines of YAML - Daniel Thomson & Jesse Suen
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2021 Virtual from May 4–7, 2021. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Eating Your Vegetables: How to Manage 2.5 Million Lines of YAML - Daniel Thomson & Jesse Suen, Intuit
Configuration management, while overlooked and underappreciated, is a necessary component of a healthy Kubernetes diet. With over 200 clusters, 2500 git repositories, and 9000 namespaces, Intuit manages and deploys 2.5 million lines of Kubernetes configuration using GitOps. To handle this scale, over the years Intuit has experimented with different config management approaches including templating (Helm), overlays (Kustomize), codifying and abstracting (jsonnet, cdk8s), ultimately settling on Kustomize. Kustomize’s remote base capabilities allow our central team to define and distribute standard patterns and templates, while still allowing developers to overlay last-mile customizations on these templates. This session will explain our journey, hard lessons faced for managing YAML at scale, and where Intuit thinks the future of Kubernetes configuration management needs to head.
https://sched.co/ekAg
A
Hi
everyone
thank
you
for
attending
our
talk
eating
your
vegetables.
How
to
manage
2.5
million
lines
of
yaml.
My
name
is
daniel
thompson.
This
is
my
colleague
jesse
suen,
and
we're
really
excited
to
get
this
talk
before
we
dive
into
it.
Just
a
brief
intro.
So,
like
I
said
earlier,
I'm
danny
thompson,
I'm
a
software
engineer
at
stitch.
If
you
haven't
heard
of
stitch,
stitch
is
a
platform
for
simplifying
user
authentication
through
passwordless
lock-ins.
A
B
A
Great
thanks
jessie,
so
why
don't
you
dive
into
it?
So,
let's
start
with
just
a
single
kubernetes
font
before
we
get
to
25
or
sorry
2.5
million.
So
what
we
have
here
is
a
deployment,
and
this
deployment
has
a
anti-affinity
rule
that
guarantees
that
the
pods
within
this
deployment
won't
be
scheduled
on
the
same
nodes,
and
so
what's
really
great
about
this.
Is
that
there's
no
code?
You
need
to
write
all
of
it
is
configured
through
yaml.
A
This
is
one
concern
just
taken
off
my
plate.
However.
That
being
said,
this
yaml
is
also
very
verbose.
It'd,
be
very
challenging
to
write
this
without
googling
the
spec
and
the
other
part
is
it's
pretty
long.
This
yaml
isn't
even
complete.
You
might
notice
that
we
left
out
parts
of
the
selector
because
we
couldn't
fit
the
entire
yellow
on
this
slide,
and
so,
while
it
is
challenging
that
kubernetes
yemel
is
this
verbose?
A
It's
almost
a
bit
necessary
for
kubernetes
to
kind
of
solve
all
the
different
use
cases
it's
trying
to
address,
and
so,
as
a
result,
us,
as
developers
have
to
have
a
challenging
problem
of
managing
kubernetes
files,
and
this
is
just
one
kubernetes
file.
It's
when
you
run
a
cluster
you're
going
to
need
more
than
just
one
deployment
you're
going
to
install
a
bunch
of
add-ons,
just
as
an
example
of
that.
A
Here's
a
slide
of
a
bunch
of
really
popular
open
source
projects
that
are
installed
across
a
wide
variety
of
clusters,
and
these
add-ons
are
adding
new
functionality
to
kubernetes.
That's
not
already
there
and
helps
you
run
a
production
grid
cluster,
and
this
list
isn't
even
exhaustive,
either
and
so,
which,
with
each
of
these
add-ons,
they
generally
require
modifications
to
get
them
to
fit
into
your
cluster,
and
so
that
is
in
addition
just
to
the
ammo
for
these
add-ons.
A
Configuration
management
is
very
much
a
very
much
a
problem,
that's
kind
of
unique
to
each
organization,
so
each
approach
that
you
take
doesn't
necessarily
mean
it
will
work
in
every
organization
and
what
you
need
to
think
about
are
the
things
that
are
kind
of
unique
to
your
org.
What
is
your
culture?
What
is
the
responsibility
of
your
teams,
and
this
will
help
inform?
What
is
the
appropriate
approach
to
take?
So
a
great
place
to
start
would
be
with
the
different
personas.
A
A
A
They
will
deploy
as
needed
and
don't
necessarily
need
semantic
versioning
to
version
their
applications,
and
so
with
each
of
these
personas,
they
actually
have
very
different
needs,
and
you
might
decide
that
actually,
it's
better
to
take
two
different
use.
Two
different
approaches
with
configuration
management
to
solve
their
computer,
their
needs,
and
for
this
talk
we're
going
to
focus
more
on
the
developer
experience.
A
A
For
example,
if
you
extract
everything
away,
you
solve
the
zero
interest
use
case,
but
you
might
alienate
some
power
users
who
might
have
a
special
use
case
they're
trying
to
solve
on
the
other
end
of
the
spectrum.
If
you
just
give
developers
raw
emo,
the
zero
interest,
developers
or
zero
interest
in
kubernetes
developers
will
want
will
be
confused
by
the
ammo
and
won't
necessarily
understand,
but
the
highly
the
power
users
will
run
with
it,
and
so.
A
A
You
also
want
the
centralized
control
team
or
to
be
able
to
deploy
changes
out
and
not
have
to
go
bother,
hundreds
or
thousands
of
developers
to
kind
of
go
margin
of
pr,
and
so,
with
all
of
these
considerations
in
mind,
this
is
where
you
need
to
kind
of
evaluate,
what's
important
to
your
organization,
and
so
at
this
point
I
think
we're
ready
to
kind
of
evaluate
what
are
the
different
approaches
and
how
the
pros
and
cons
of
each
of
them.
So,
let's
start
with
an
easy
one:
raw
yellow,
rye
ammo.
This
is
pretty
easy.
A
A
So
templating
is
where
you
take
a
list
of
parameters
that
you
inject
into
the
predefined
template
and
that
output
is
your
configuration.
An
example
of
this
would
be
helm.
The
n
and
jsonnet,
and
the
image
we
have
here
is
a
helm
chart
that
we
have
listed
and
what
you'll
notice
is
that
it
allows
you
to
list
kind
of
different
values
that
you
can
fill
in
in
another
file
which
the
developer
would
provide.
A
So
the
advantages
here
is
that
it's
a
simpler
configuration
and
it's
very
flexible,
like
you
might
notice.
We
have
like
an
if
statement
with
within
this
deployment
template,
but
there's
some
disadvantages
here.
Where,
generally,
these
templates
start
to
grow
in
complexity
as
time
goes
on
and
you
start
to
parameterize
everything,
and
so
maintaining
these
templates
and
understanding
what's
going
on,
becomes
really
challenging.
A
You
can
think
of
overlays
as
defining
a
common
base
that
you
share
across
environments
and
then
for
each
environment.
You
define
a
file
that
has
the
specific
changes
that
you
want
to
make
to
that
environment.
So
then,
when
you
want
to
deploy
you
take
your
base
and
apply
those
environment
specific
changes
on
top
of
that
base.
An
example
of
this
would
be
customize
the
image
we
have
here.
The
image
in
the
background
would
be
a
defined
base,
a
define
or
which
is
a
deployment.
A
A
That
being
said,
some
disadvantages
to
the
overlay
approach
is
that
it's
not
always
immediately
intuitive
to
developers
what's
kind
of
going
on.
It
takes
a
bit
for
it
to
click
sometimes,
and
the
lack
of
parameterization
makes
things
that
should
be
easy
hard,
and
so
that
is
overlay
in
a
nutshell,
and
the
next
approach
I'm
going
to
talk
about
is
kind
of
a
little
bit
more
of
a
pivot,
and
that
is
abstractions.
A
A
So
here
we
have-
or
at
least
the
image
we
have
here,
is
like
an
example
of
an
abstraction
where
the
developer
would
only
kind
of
provide
the
things
that
they
care
about.
Like
okay,
I'm
running
a
web
service.
This
is
the
dns
name
I
want.
This
is
how
I
want
to
upgrade
it,
and
I
want
it
to
be
on
the
mesh,
and
so
the
user
only
has
to
list
these
fields,
and
then
that
is
translated
into
the
all.
A
However,
there
are
some
disadvantages:
where
you're
giving
up
flexibility,
the
developer
is
giving
up
flexibility,
and
usually
most
of
these
abstractions
tend
to
kind
of
start
to
leak,
the
underlying
details
and
so
kind
of.
As
a
result,
we
we
found
that
there
really
hasn't
been
the
right
obstruction
found
for
kubernetes
yet
and
that's
kind
of
just
abstraction.
In
a
nutshell,
the
last
approach
we're
going
to
talk
about
is
codifying
your
configuration
management
and
this
one's
pretty
simple.
B
A
That
being
said,
you
get
all
the
disadvantages
of
a
code
base,
so
this
is
another
code
base
you
have
to
manage
and
you
have
to
so
like
you
might
have
bugs,
and
you
have
to
kind
of
figure
out
how
to
debug
those,
and
it
can
be
a
challenge
for
your
developers
to
figure
out.
How
does
this
chart
that
we
have
listed
here
translate
to
our
final
result?
B
These
developers
are
managing
multiple
environments
and,
for
us,
environment
equates
to
a
kubernetes
namespace,
and
these
environments
are
mostly
identical
and
they
have
slight
variations
in
the
in
their
config.
So
maybe
the
differences
are
that
they
have
different
dns
names,
they
use
different
advanced
resources
and
they
have
maybe
different
im
roles
and
privileges.
B
B
And
so
with
that
we
came
up
with
a
set
of
requirements
and
as
a
platform
team,
we
wanted
to
provide
a
standard
set
of
patterns
and
best
practices.
What
we
refer
to
as
the
paved
road,
but
at
the
same
time
we
also
want
to
empower
developers
and
provide
them.
The
flexibility
so
that
they
would
be
unblocked
in
the
event
they
need
more
capabilities
that
aren't
provided
by
a
default.
B
Even
if
this
came
at
the
cost
of
simplicity,
and
so
we
made
the
conscious
decision
to
actually
expose
our
developers
to
kubernetes
ymo
and,
and
so
they
can
leverage
all
the
power
and
benefits
of
kubernetes
and
finally,
any
approach
that
we
took
it
needed
to
be
git.
Ops,
friendly,
git
ops
is
extremely
important
to
intuit
because
of
a
lot
of
compliance
and
security
requirements,
and
so
whatever
solution
we
took
had
to
fit
into
our
strategy.
B
And
so
the
solution
we
came
up
with
actually
after
years
was
actually
to
use
customize
and
with
customize
since
you're,
exposing
your
users
to
actually
kubernetes
resource
and
manifest
you're,
preserving
all
the
full
power
and
capabilities
of
of
kubernetes
and
because
customize
is
kubernetes
native.
It's
well
supported
and
documented.
B
Our
users
or
developers
are
often
able
to
get
help
from
open
source
documentation
and
and
not
always
rely
on
the
the
platform
team,
because
it's
it's
just
kubernetes
yamaha.
At
the
end
of
the
day,
readability
is
very
clear.
There's
you
just
again
looking
at
manifest,
and
this
is
actually
both
important
for
the
developers
but
the
platform
team
who
is
supporting
that
service.
So,
if
say,
there's
some
problem
with
the
bug
in
the
yaml.
B
And
finally,
what
I
think
is
probably
customizes
best
feature
is
its
ability
to
reference
a
central
or
remote
base
of
git
repository,
and
that
allows
us
to
do
that.
Distribution
of
standard
patterns
across
our
organization,
so
yeah,
it's
a
central
remote
base,
is
how
we
leverage
customize.
Basically,
as
a
platform
team,
we
provide
what
I
call
a
a
catalog
of
generic
starter,
yaml
and
so
in
a
simple
example.
The
developer
will
get
something
like
a
web
service
which
is
comprised
of
a
deployment,
a
service
and
an
ingress
and
later.
B
If
the
developer
wants
some
to
leverage
some
of
the
more
advanced
capabilities
of
kubernetes,
they
can
then
include
say
like
a
hpa
base
or
a
canary
analysis
base
as
part
of
their
service,
and
we
cementally
version
this
remote
base,
so
that
developers
they
can
upgrade
to
newer
future
at
their
own
pace.
And
so
all
of
this
is
to
kind
of
gives
us
this
standard
distribution
of
patterns,
but
best
practices
and
things
like
you
know,
setting
pod
readiness,
gates
and
resource
limits
and
ingress
annotations.
B
So
what
we
have
here
is
is
actually
a
ingress
that
we
as
a
platform
team
came
up
with
that
suits
our
use
case.
As
you
can
see,
we
use
the
lb
ingress
controller
and,
if
you
use
alb
english
controller
you'll
know
that
it
just
has
a
ton
of
different
annotations
to
control
the
underlying
load.
Bouncer
and
most
of
these
details
are
things
that
you
don't
want
to
expose
to
your
developers.
B
And
so
what
the
developer
gets.
Is
this
git
repository
and
it's
structured
like
this
at
the
top
level,
they
have
the
application
based
directly
that
it
serves
as
the
common
definitions
that
they
want
to
apply
to
all
of
their
environments,
and
then
they
get
a
list
of
environment
directories
and
each
of
these
environment
directories
correspond
to
a
kubernetes
namespace,
and
it
only
includes
the
changes
specific
for
that
environment.
B
If
you
look
at
that
top
level,
customization
you'll
see
that
here
it's
referencing
that
centrally
managed
a
base
that
I
referred
to
this
one's
referencing,
our
4.0
version,
and
then,
if
you
look
at
the
customization
and
environment
you'll,
see
that
it
then
derives
from
the
top-level
app
base
of
their
own
git
repository
and
so
with
this
approach.
Developer
can
leverage
that
same
configuration
reuse
that
they
get
by
also
inheriting
from
that
central
repository.
B
B
It's
counted
a
little
bit
differently,
but
you
have
a
base
of
90
lines
of
yaml
and
four
overlays
one
per
environment,
and
each
of
those
overlays
are
about,
let's
say,
45
lines
that
equates
to
270
lines
of
ammo
per
service.
You
multiply
that
by
2
500
services
and
you
get
675
000
lines
of
yaml.
This
is
roughly
about
a
little
over
25
of
the
deployed
ammo.
B
So
this
has
been
working
well
for
us,
but
it
hasn't
been
without
its
challenges,
and
so
these
are
some
of
the
challenges
that
you'll
come
across.
Should
you
take
a
approach
like
this
and
the
number
one
is
actually
user
support
and,
as
I
mentioned,
we
chose
to
expose
our
users
to
kubernetes
yaml,
and
so
by
making
that
decision,
we
also
gave
our
users
a
lot
of
loaded
foot
guns.
You
know,
people
who
may
not
be
as
familiar
with
kubernetes
can
make
lots
of
mistakes.
B
The
second
challenge
that
we
had
was
just
the
automation
necessary
to
support
this.
You
know:
2500
services
equates
to
2500
git
repositories,
and
so,
in
order
to
support
that,
we
had
to
build
a
lot
of
automation
to
to
do
things
like
automatically
send
pull
requests
in
the
event
say
we
want
to
deprecate
like
a
certain
api
and
then.
Finally,
the
last
challenge
is
with
the
tool
customize
itself.
B
You
know
when
we
you
depend
so
much
on
this
tool,
which
is
responsible
for
rendering
out
the
yaml
that
you
end
up,
deploying
any
change
to
that
tool
can
impact
a
lot
of
your
services,
so
if
we
were
to
say,
upgrade
customize
and
it
had
it,
it
rendered
dml
a
bit
differently
which
it
has
then
we
have
potentially
are
breaking
all
at
once.
A
thousand
different
applications-
and
another
thing
you
should
know
about
the
customize
is
its
support
for
crds
is
a
bit
limited.
B
It
has
great
support
for
native
kubernetes
kinds,
but
when
it
comes
to
crds,
you
aren't
able
to
leverage
some
of
the
convenient
features
like
strategically
merge
patching
your
resources.
We
actually
ended
up
forking
customize
in
order
to
run
a
version
of
customize
that
natively
understands
the
crds
that
we
we
use
at
intuit.
B
A
Yeah
so
hopefully,
as
you
could
tell
that,
we
really
think
there's
no
perfect
solution
with
configuration
management,
no
matter
what
you
do.
You're
gonna
have
kind
of
certain
edge
cases
or
road
bumps
that
are
gonna,
be
a
challenge
and
it's
kind
of
the
nature
of
the
problem,
and
even
within
that
also
like
with
whatever
solution
you
choose.
It's
going
to
be
highly
dependent
on
kind
of
your
organization
and
how
it
divvies
out
different
responsibilities
and
just
the
general
culture,
and
so
what
we've
kind
of
taken
away
is
that
at
a
certain
scale,
just
act.
A
A
This
is
through
a
ui
assisted
configuration
management
tool
that
can
allow
our
developers
to
kind
of
easily
edit
their
yaml
in
a
way
that
kind
of
gives
them
more
insight
into
what
changes
they're
actually
making,
but
also
kind
of
enforce
those
organizational
standards
that
you
want
to
see,
and
so
with
that
a
good
example
of
that
would
be
spinnaker
as
a
project.
One
thing
they
do
really
well
is
they
make
it
really
easy
to
create
a
ec2
instance,
so
they
walk
you
through
all
the
different
fields.
A
You
need
to
provide
and
kind
of
show
you
an
opinionated
way
of
bringing
up
any
c2
instance
and
it's
easy
gui
to
follow,
and
so
we're
hoping
we
could
find
something
similar
with
kubernetes
with
that.
Thank
you
for
attending
our
talk
here
is
a
list
of
resources
that
we
have
and
please
feel
free
to
reach
out.
We're
excited
to
see
what
we
can
kind
of
do
with
the
configuration
management
portion
of
kubernetes
have
a
great
day
and
enjoy
the
rest
of
the
conference.