►
From YouTube: Lightning Talk: How Intuit Enables GitOps at Scale For All Its Developers - Omer Azmon, Intuit
Description
Lightning Talk: How Intuit Enables GitOps at Scale For All Its Developers - Omer Azmon, Intuit
This is the story of how we at Intuit learned what it really takes to enable our front-end, back-end, and AI developers – all our developers – to rapidly create, update, and dispose of applications. Our learning from building such a GitOps on-boarding/asset-management system include: * What are the independent personas whose needs must be addressed by any such system, and how to resolve their disparate needs. * What are and how to handle the overlapping needs of the assortment of application types: service, serverless, UI, ML, etc.. * Why is updating and cleanup harder than onboarding * Why workflows, orchestration, and even traditional choreography can't handle the variability of such a system, and what we did about it. * How to avoid flurries of PRs. * What happens when a developer can rapidly standup an asset and abandon when not needed like cattle, including build, deploy, persistence, everything * Why is it impossible for one team to develop this alone, and how to make it possible to deliver such a system. We hope that you will find the needs common and join us in building an open source community based on our proven tools.
A
This
is
the
cliff
notes
version
for
those
who
know
what
cliff
notes
are.
So
our
learning
involved
that
we
had
to
do
deployment
we
had
to
do
configuration
we
had
to
cover
monitoring.
We
had
to
cover
cloud
interactions,
cloud
resources,
alerting
and
more
and
more
and
more,
there's
really
a
lot
that
goes
into
this,
and
so
what
we
found
out,
like
most
people,
know
in
most
software
that
get
working
with
githubs
create
different
software
assets.
You
need
to
create
a
repo.
A
You
have
to
populate
it
with
sample
working
code
that
demonstrates
and
propagates
the
standards
we'd
like
to
follow
in
the
company
for
consistency,
security
and
compliance,
and
so
it
wasn't
intuit,
and
we
were
very
proud
of
ourselves
about
three
years
ago.
When
we
got
to
this
point
and
then
we
realized
that
there's
add-ons
that
there
are
things
that
people
want
to
add
afterwards
and
we
have
to
give
them
documentation.
A
Instead
of
automation,
database
access,
messaging
access
and
so
on,
and
they
used
to
file
a
ticket,
they
used
to
create
the
new
resources
or
get
access
permissions
and
then
follow
some
wiki
to
update
their
code,
but
not
anymore,
and
then
there
were
different
updates
that
had
to
be
happening.
For
example,
the
platform
change.
We
went
to
kubernetes
116,
and
suddenly
we
had
to
send
3000
prs
and
trace
all
those
teams
and
work
with
them
and
everything
else.
A
And
while
there
are
tools
like
the
pandabot
and
snick,
and
so
on
that
help
with
the
simple
dependencies
when
you
need
to
handle
the
more
complicated
ones.
There
is
a
lot
of
work
that
goes
into
keeping
that
system
update
from
a
central
team's
point
of
view.
Well
beyond
what
needs
to
happen
and
finally,
there's
the
whole
issue
of
cleaning
up,
because
there
were
tons
of
resources
that
were
being
created
with
all
this
wonderful
automation
and
they
cost
money,
as
the
aws
guy
knows
very
well,
and
so
in
order
to
understand
how
to
solve
the
problem.
A
The
one
person
may
be
running
around
spinning
plates,
but
it's
still
a
capability
team
and
that
was
they
had
their
own
needs.
The
next
piece
that
had
to
be
taken
into
account
is
a
paved
road
team.
This
is
an
intuit
term.
I
hope
it
will
become
more
general.
These
are
people
that
handle
a
specific
type
of
resource,
such
as
a
service,
web
application,
etc,
etc,
etc,
and
want
to
propagate
everything
in
their
paved
road
off
to
a
specific
team,
and
then,
of
course,
there's
the
product
development
team.
A
Idiom
potent
retain
what
should
be
give
very
good
opposite
know.
What
is
so?
We
could
do
the
diff,
we
wanted
it
to
be
data
driven.
We
didn't
want
to
have
to
build
a
workflow
for
every
single
pattern
that
was
getting
unmanageable.
So
quick.
You
can't
believe
it.
We
wanted
that
the
data
for
each
one
would
be
the
true
source
of
truth
for
jenkins
would
be
jenkins.
A
Forget
would
be
git,
etc,
etc,
and
yet,
from
the
point
of
view
of
somebody
accessing
the
system,
it
had
to
all
operate
as
one
it
had
to
support
making
changes
after
the
fact,
and
it
had
to
be
low,
no
code
for
people
that
want
to
create
patterns
or
for
developers
to
consume,
and
it
needed
that
all
these
different
teams
can
work
independently,
instead
of
having
tons
of
meetings
to
coordinate
with
each
other
on
every
piece
of
work
that
had
to
be
done.
So
what
we
did,
we
did
not.
A
We
built
this
on
kubernetes,
but
not
using
the
kubernetes
infrastructure.
We
used
graphql
as
federation
as
the
technique
to
assemble
all
these
microservices
each
microservice
acts
as
a
controller.
We
used
a
bus
to
shuttle
messages
between
them
to
allow
all
this
occur,
eventual
consistency
and
what
the
users
ended
up
doing.
A
Is
they
go
to
a
ui
which
is
set
up
by
one
of
these
paverold
builders?
That
paved
road
builder
then
fills
in
this
huge
yaml
document.
I
need
two
repos.
I
need
this
code.
I
need
this.
I
need
this
everything
splunk
indexes
certificates,
anything
cloud
resources,
anything
they
build
it.
That
was
the
majority
of
it.
One
of
the
squares
you
look
there
and
we
have
currently
25
different
integrations
that
go
into
this
to
all
the
different
resources
is
content.
A
Our
content
system
is
allows
both
this
system
to
send
either
new
code
or
send
updates
as
prs.
It
also
allows
through
the
ui,
to
get
the
list
of
prs
available
for
prs
on
demand,
for
example,
upgrade
me
from
java
8
to
11,
which
requires
changes
to
the
code,
changes
to
the
palm
or
to
gradle,
build
on
to
the
dockerfile,
etc,
etc.
So,
all
these
automations
and
they
all
use
tools
like
combi
and
gradle
and
other
tools,
but
they
put
it
all
together.
So
it's
really
simple
to
operate
all
of
this.
This
is
the
ui.
A
That's
used
to
see
people
accepting
the
pr
and
nudge
them
when
that's
needed
and
some
metrics
about
what
happened.
So
currently,
we
have
in
excess
of
17
000
durable
assets
which
are
managed
through
this
system,
and
it
grows
on
a
regular
basis.
We
have
15
active
paved
roads,
of
which
12
were
built
through
contribution
in
the
company.
A
We
had
nothing
to
do
with
them.
We
had
we're
managing
20.
Well,
it's
now
up
to
25
micro
services
that
12
were
them
and
now
it's
even
more
contributed.
This
is
our
growth
pattern
and
65
engineers
internally
contributed
to
this,
but
we
over
the
last
three
years
only
had
15
engineers
ever
working
for
us.
A
So
now
what
happened
as
a
result
of
making
it
so
easy
that,
with
a
click
of
a
button,
you
can
get
this
git
ops,
everything
a
through
z,
and
if
you
need
to
add
the
database,
you
go
to
the
ui
and
you
say:
add
a
database,
you
select
and
you
connect
or
whatever
it
is,
is
two
things
one,
because
it
was
easy.
50
of
new
resources
being
created,
live
between
one
hour
and
one
month,
and
then
they
abandon
it.
A
The
experimentation
level
has
skyrocketed
and
innovation,
as
measured
by
features
pushed
to
production,
has
gone
up
well
over
a
hundred
percent.
It's
actually
gone
up.
I
didn't
want
to
put
it
down
here,
but
I'll
say
it.
It's
gone
up
over
400
percent
and
I
checked
that
number
several
times
with
several
people
because
it
didn't
seem
likely,
but
that's
what
happened
so
that
that's
a
10-minute,
quick
story
about
there
is
no
question
and
answer
to
a
lightning
talk,
I'm
at
kubecon
all
week
and
can
be
reached
at
this
address.
Thank
you
very
much.