►
From YouTube: Progressive Delivery at Intuit, the Next Chapter in a GitOps Journey. - Henrik Blixt & Jesse Suen
Description
Progressive Delivery at Intuit, the Next Chapter in a GitOps Journey. - Henrik Blixt & Jesse Suen, Intuit
After successfully implementing a GitOps based platform, now spanning over 350 clusters and thousands of services, Intuit is embarking on a new journey of progressive delivery. Accelerating developer velocity and innovation while ensuring smooth service introductions and upgrades, are some of the key goals of this effort. Come and hear Henrik and Jesse dive into their vision of a progressive delivery platform, what they have already done to realize this and how it’s being rolled out at scale at Intuit. The talk will draw experiences and examples from their work at Intuit and the use of Argo CD and Argo Rollouts, but the concepts, ideas and results are equally applicable to other GitOps/progressive delivery tools and for companies in similar situations.
For more Continuous Delivery Foundation content, check out our blog: https://cd.foundation/blog/
A
A
And
just
quickly
before
we
get
started
just
a
brief
note
on
on
who
we
are
so
intuit
is
a
us-based
financial
technology.
Software
company,
founded
in
1983
here
in
california
and
already
from
the
start,
was
focused
around
a
software
called
quick
in
a
personal
finance
management
application
and
we've
continued
to
build
financial
software
for
the
last
few
decades,
and
we
now
have
a
a
number
of
occasions
for
those
in
the
u.s
you're,
probably
most
familiar
with
turbotax
and
the
financial
tax
preparation
software.
A
So
we
had
a
lot
of
data
centers
and
we
spent
a
lot
of
time
managing
our
data
centers,
managing
the
hardware,
setting
things
up,
configuring
things
and
being
a
financial
technology
company.
That's
definitely
not
part
of
our
core
business
they're,
not
solving
any
business
case
other
than
delivering
compute.
A
We
didn't
really
know
what
was
being
used
where
if
there
was
a
security
vulnerabilities
and
if
there
were
some,
it
took
a
long
time
to
remediate-
and
there
was
just
a
lot
of
various
things
being
built
and
used
across
the
company.
One
of
our
business
unit
had
29
different
tools
to
manage
their
their
life
cycle
for,
for
example,
and
there
are
a
lot
of
complicated,
integrations,
scripting
and
just
a
lot
of
snowflake
and
things
going
on
to
get
things
to
run.
A
And
this
also
led
to
a
lot
of
unreliable
and
complex
and
unrepeatable
deployment
processes.
Engineers
just
spent
way
too
much
time
doing
things
other
than
writing
code.
So,
for
example,
it
could
take.
You
know,
15
weeks,
to
create
a
new
application
because
they
had
to
provision
environments,
make
sure
ports
were
open
in
the
firewalls
and
just
worry
about
a
lot
of
other
things
that
were
not
innovating.
A
So
this
just
led
to
our
engineers
solving
the
wrong
problems,
not
focusing
on
the
core
business
value,
delivering
the
core
business
value
a
lot
of
frustration
because
they
want
to
innovate.
They
want
it,
they
want
to
move
fast,
but
they
were
bogged
down
by
some
of
these.
These
legacy
things
and
they,
how
did
it
take?
We
had
meant
they
didn't
have
to
work
with
old
versions,
old
software
and
everyone
wants
to
work
on
the
latest.
A
A
So
that
kind
of
spurred
this,
this
transition
and
modernization
and
some
of
the
key
errors
we
focused
on
was
operational
excellence,
making
sure
that
we
can
have
the
observability.
We
can
see
what's
going
on,
understand,
what's
going
on
and
introduce
the
right
amount
of
processes
that
doesn't
burden
engineering
with
bureaucracy,
but
still
have
enough
rigor
to
to
keep
things
in
line
and
know
what
we're
doing
and
make
sure
that
we
can
move
faster
while
still
lowering
our
mean
time
to
to
detect
the
mean
time
to
to
resolve
issues
that
that
occur.
A
All
these
various
tools
we
had
before
also
means
there
was
a
lot
of
various
ways
of
solving
things
and
we
realized
we.
We
didn't
have
a
platform.
We
had
many
platforms,
so
we
needed
to
standardized
api
standardized
tools,
leverage
standard
capabilities
as
as
much
as
humanly
possible
and
then
introduce
self-service
and
automation.
A
A
We
have
for
some
of
the
guided
paths
for
engineers
to
do
things
like
deploying
service,
mesh,
lambda
and
especially
grouping
of
best
practices,
automation
and
script,
to
help
them
standardize
things
and
get
things
deployed
in
a
standardized
way,
and
this
journey
started
back
in
late,
2017,
2018
and
then
his
latest
in
the
last
three
a
little
bit
over
three
years
now
and
has
also
led
us
to
introduce
applications
like
argo,
cd
and
argo
rollas.
That
jesse
will
talk
about
later.
A
There
were
things
that
we
discovered
during
this
process
that
there
wasn't
really
anything
that
fulfilled
in
the
tool,
tooling,
that
we
could
find.
So
you
know
we
started
this
this
journey
of
building
some
of
the
tools
we
needed
ourselves
as
as
well
and-
and
this
journey
has
led
us
down
a
path
of
of
automating,
all
the
things
using
git
ups
using
devops.
A
If,
if
you
build
it,
you
run
it
making
sure
that
engineering
takes
responsibility
for
everything
from
writing
the
code
testing
until
it
actually
gets
deployed
in
production,
operationalizing
everything
we
have
an
operational
data
lake,
where
we,
where
we
dump
all
our
operational
data,
making
sure
that
we
can
track
things
trace.
Things
run
a
lot
of
queries
and
dashboard
against
this.
A
Releasing
applications
that
that
help
our
customers
solve
their
business
needs
and
also
enabling
them
to
move
faster.
Don't
have
to
worry
about
all
the
my
new
details
on
on
how
the
pipeline
gets
created.
They
can
just
get
have
it
done
for
them
that
lets
them
focus
their
energy
on
the
things
that
they
want
to
focus
on,
and,
lastly,
modernizing
our
platform,
shifting
everything
to
using
cloud-native
technologies
using
kubernetes
and
lately
using
things
like
service
mesh
and
lambda
and
transitioning.
A
All
our
engineering
teams
to
to
adopting
that
and
being
comfortable
that
was
was
also
a
big
change
and
and
using
things
like
githubs
and
devops
in
this
whole
process,
you
know
to
make
things
more
automated
and
move
smoother
through
the
software
development
life
cycle,
and
I
will
hand
it
over
to
jesse
to
give
some
more
details
on
the
progressive
delivery
part
of
this.
This
platform.
B
B
One
is
their
application
source
code
repo
which
is
by
default.
Is
this
hello
world
of
some
spring
boot
application
they'll
get
a
git
repository
containing
application
config
and
we
use
spring
cloud
config
to
hold
application
config
and
most
interesting
for
this
talk
is
their
deployment
repository,
which
contains
kubernetes
manifest,
and
we
currently
use
customize
for
managing
configuration.
B
In
addition
to
that,
they
will
get
a
container
repository
for
their
service.
They'll
get
a
jenkins
ci
pipeline
and
the
pipeline
is
actually
what
promotes
that
application
between
environments,
they'll
get
a
set
of
kubernetes
namespaces
and
the
namespace
corresponds
to
an
environment.
Typically
they'll
get
a
you
know:
a
qa,
a
stage:
a
prod
west,
a
prod
east
and
finally,
they'll
get
an
argo
cd
project
and
a
set
of
applications
tied
to
this
service.
B
So
right
away
after
onboarding,
a
service
owner
will
be
able
to
start
developing
their
code
and
with
their
next
commit,
they
can
actually
start
deploying
to
multiple
environments
and
exposing
it
to
their
clients.
And
when
I
talk
about
environment
promotion,
what
I
mean
by
that
is
that
the
developers
have
these
handful
of
environments.
You
know
the
qa
ede
stage
prod
and
they
need
to
first
deploy
their
changes
to
qa
run
some
tests
that
that
goes
well.
B
B
So
now,
let's
talk
about
progressive
delivery
or
what
actually
happens
inside
a
single
environment
and
how
we
do
safe
deploys
during
an
update.
So
if
you
don't
know
what
progressive
delivery
is,
progressive
delivery
is
a
practice
where
you
are
incrementally
exposing
new
versions
of
your
application
to
initially
small
subset
of
your
users
and
then
gradually
larger
and
larger
subsets,
and
the
reason
you
do
this
is
you
want
to
limit
the
number
of
users
who
might
have
been
negatively
impacted
by
a
problematic
version
of
your
software
and
chances?
B
Are
you
might
already
be
doing
some
form
of
progressive
delivery?
Already
you
might
not
just
not
have
known.
There
was
a
term
for
it,
and
examples
of
this
is
canarying,
where
you're
doing
a
traffic
split
between
the
canary
and
the
old
version
feature
flags
where
you
build
dormant
features
into
the
application,
and
you
just
turn
on
those
features
on
and
off
to
various
customers
and
ring
deployments,
which
is
a
fancy
way
of
just
organizing
your
deployments
into
larger
and
larger
rings
of
users.
B
So
the
way
we
do
progressive
delivery
at
intuit
is
through
a
project
that
we
started
called
argo
rollouts
and
argo.
Rollouts
is
a
progressive
delivery
operator.
It's
a
controller
that
runs
in
a
cluster
and
it's
designed
to
be
a
drop-in
replacement
for
the
deployment
object,
but
with
these
added
functionalities.
So
let's
walk
through
what
robots
offers.
B
So
first,
as
you
are
probably
aware,
the
deployment
strategies
in
the
basic
kubernetes
deployment
are
are
really
basic.
You
have
rolling
update,
but
rolling
update
is
limited
in
that
one.
It
can't
do
an
atomic
cut
over
to
that
new
version
like
like
you
can
with
blue
green.
So
your
service
better
not
require
something
like
api
consistency
to
its
clients
on
two.
It
can't
stop
and
pause
in
the
middle
of
the
update,
since
a
rolling
update
is
trying
to
get
you
from
version
one
to
version
two
as
as
fast
as
possible.
B
B
B
So
as
soon
as
a
user
makes
a
change
to
their
their
rollout
pod
spec
argo
wallace
will
create
a
new
replica
set
using
that
new
pod
spec
and
then
it
starts
scaling
that
up
gradually
and
the
rollout
will
also
then
modify
the
ingress
and
intel
to
start
splitting
traffic
between
the
canary
and
stable
versions.
Here
it
just
bumped
up
the
percentage
to
five
percent.
B
B
If
that
measurement
is
successful,
the
roll-up
continues
on
with
the
update
by
then
scaling
up
the
canary
stack
and
then
increasing
its
the
weight
to
the
canoe,
and
it
just
continues
to
repeat
this
process
or
according
to
whatever
steps
you
define
in
the
rollout
until
it
reaches
100,
and
then
once
we
reach
the
end
of
the
steps
the
update
is
considered
complete
and
the
revis.
The
the
new
revision
is
promoted
as
the
stable
and
and
we
start
sending
we,
we
market
a
stable
and
send
100
traffic
to
stable.
B
So,
as
I
mentioned,
cargo
wallets
integrates
with
a
number
of
different
service
meshes
and
ingress
controllers,
including
all
of
the
ones
you
see
here,
and
that's
really
made
it
very
flexible
for
our
developers,
even
at
intuit,
which,
where
we
we
actually
run.
You
know
they
have
several
options
of
of
ingress
two
services
and
so
now
I'll
pass
it
back
to
henrik
who
can
describe
how
this
has
improved
our
release,
velocity.
B
A
What
what
are
the
the
other
values
that
we've
seen
with
this
git
ups
and
progressive
delivery?
We've
seen
how
the
within
empowered
engineering
things
are
moving
a
lot
faster
things
that
used
to
take
weeks
now
takes
minutes
and
to
create
an
application.
It's
done
through
the
self-service
platform
in
in
minutes
sets
everything
up
automatically.
A
A
We've
also
seen
our
release,
laws
that
are
talked
about
and
improved
three
to
four
times
now.
Our
cycle
and
deployment
times
also
see
big
improvements
in
5x
improvements
here
and
on
the
operational
side,
with
our
data
lake
and
all
the
the
new
data
standardized
data
we
get
from
all
these
applications,
we've
seen
an
improvement
of
5
to
10
times
in
our
time
to
defined
issues
and
to
resolve
them.
A
And
lastly,
but
not
least
given
all
the
time
that
was
spent
by
engineering
before
in
things
around
deploying
an
application
or
releasing
an
application,
rather
than
writing
code,
we're
now
saving
around
1
million
developer
hours
annually,
with
these
new
processes
and
through
this
journey.
But
we're
not
we're
not
done
yet
in
the
upcoming
year.
Here.
A
B
Hello
thanks
for
coming
to
the
talk
so
now
we'll
start
answering
questions.
So
first
question
I
see
is
what
did
we
use
to
build
our
developer
portal
and
that
one
is
actually
that
was
completely
homegrown,
so
I
think
it
has
a
like
a
java
tech
stack
and
databases
and
stuff,
but
I
did
want
to
mention
that
spotify
has
open
source
the
open
source
developer
portal
called
backstage.
I
personally
haven't
checked
it
out,
but
it's
actually
quite
similar
to
what
it's
trying
to
accomplish.
A
So
I'm
going
to
do
so.
The
next
one
I
think
here
is:
how
did
you
all
convince
your
product
organization
to
allocate
budget
for
increasing
debt
velocity?
I
think
the
the
the
key
question
and
the
key
point
is-
is
convincing
your
product
organization
that
developer
velocity
is
important
once
once,
you've
reached
that
goal
and
and
your
management
and
the
executives
actually
understand
why
developer
velocity
is
important,
then
getting
to
the
next
point
of
of
getting
the
budget
to
to
to
increasing.
That
is
that
discussion
is
a
lot
easier.
B
All
right
and
then
the
next
question
is
how
does
intuit
utilize
argo
cd
or
a
paper
for
git
ops
for
lambdas.
So
actually
we
actually
have
a
separate
tool
that
we
use
for
aws
cloud
resources,
which
is
actually
based
on
cloud
formation
templates
and
argo.
Cd
is
purely
used
for
kubernetes
deployments.
There.
A
B
Open
source
project
that
amazon
runs
called
ack,
which
stands
for
aws
controllers
for
kubernetes
and
it
basically
lets
you
manage
aws
resources
using
kubernetes
style,
manifest
and
that's
an
option
for
people
who
like
to
use
kubernetes
as
their
control
plane
all
right.
Next,
I
see
how
does
rollouts
compare
with
spinnaker.
B
Actually
migrated
from
spinnaker
to
argo
cd
because
both
of
those
are
deployment
tools,
one
is
obviously
just
focused
on
the
git
ops
deployment
tool,
specifically
for
kubernetes
and
spinnaker's,
more
general
purpose
and
rollouts
actually
can
integrate
with
a
spinnaker.
In
fact,
one
of
our
user
sales
force.
I
think
they
contributed
they
some
kind
of
spinnaker
plug-in
or
something
that
lets
you
use
roll-outs
with
spinnaker.
A
B
Okay,
so
how
does
hardware
world
support
db
migrations
that
are
sometimes
required
so
as
a
best
practice?
This
is
true
for
developer
services.
Intuit
recommends
that
our
developer
teams
perform
database
migrations
out
of
band
separately
from
the
deployment
process
and
also
in
the
backwards
compatible
way.
So
you,
for
example,
you
should
add
columns,
but
you
should
never
remove
the
column
until
certain
that
it's
no
longer
used
and-
and
this
should
also
happen
out
of
band
from
your
regular
deployment
so
that
you
can
roll
back
your.
A
B
This
is
a
good
question.
So
what?
What
is
the
rationale
behind
numerous
repos
per
application?
Does
that
generate
repo
sprawl?
Has
it
been
intentional
to
not
use
a
yuno
repo
approach
so
keep
in
mind
that
intuit
we
have
like
4
000
developers?
I
think
maybe
it
might
be
more
than
that,
and
so
it
was
important
for
us
to
have
a
lot
of
self-service
capability.
B
B
B
B
Okay,
so
if
there
was
a
follow-up
on
the
repo
question,
so
stan
did
not
mean
to
imply
a
global
uni
repo,
but
per
service.
Why
not
merge
runtime
development?
Oh
I
see
okay.
So
why
did
we
get
three
repo?
So
there
was
an
application
source
code,
repo,
a
kubernetes
deployment,
repo
and
a
spring
cloud
config
repo.
I
think
one
of
the
answers
is
a
technical
limitation.
I
believe
spring
cloud.
B
Config
I've
been
told
that
it
kind
of
wants
the
whole
repo
so
automatically
you
need
to
one
for
your
source
and
one
for
your
spring
cloud.
Then
the
deployment
one
is
necessary
because
if
it
lives
with
your
app
code,
then
you
have
this
kind
of
chicken
and
egg
problem
where,
like
our
pipeline,
wants
to
commit
back
to
the
deployment
repo,
but
also
that
could
trigger
a
jenkins
build,
and
so
you
either
have
to
account
for
that
in
your
jenkins
pipeline
or
you
separate
out
the
repos.
B
But
the
other
reason
you
want
to
separate
your
repos
is
that
oftentimes,
the
people
who
can
affect
the
environment
should
not
be
the
same
developers
who,
who
can
make
source
code
changes
and
that's
not
true
if
you're
into
it.
Like
the
same,
we
we
practice
devops,
but
that's
one
reason
why
you
might
want
to
separate
out
the
repos
just
because
of
privilege.
A
Yeah
so
a
few
seconds:
let's
maybe
how
many
developers
that
introduce
the
platform
framework.
I
think
it's
between
four
and
somewhere
between
four
and
five
thousand,
probably
close
to
five
thousand
by
now,
and
I
think
with
that,
I
think
we're
out
of
out
of
time.
So
thanks
everyone
for
for
tuning
in
and
if
you
have
any
questions
you
have
our
twitter
handle
so
feel
free
to
reach
out
and
we'll
do
our
best
to
help
you
out
thanks.
Everyone.