►
Description
With a pivot to a DevOps mindset at Northwestern Mutual, we wanted to chart and demonstrate our progress on that journey. We developed a process to leverage the robust project and pipeline data from GitLab and aggregated it into dashboards to provide visibility on current state and trending over time. In this session, we'll show what types of metrics we captured and how this data drives improvements.
Speakers:
Bobbi Wenzler - Senior Product Manager, Northwestern Mutual
Nicole Schultz - Assistant Director of Engineering, Northwestern Mutual
Get in touch with Sales: http://bit.ly/2IygR7z
A
B
Out
hi,
everyone
welcome
to
get
lab
commit.
My
name
is
nicole
schultz,
and
I
have
my
product
manager
bobby
wenzer
with
me
today,
and
we
are
going
to
be
presenting
on
how
we
are
leading
devops
transformation
with
gitlab
data
at
northwestern
mutual.
So
for
those
of
you
who
haven't
heard
of
northwestern
mutual,
we
are
a
financial
services
and
planning
company.
So
everything
you
need
from
life
insurance
to
investments
to
financial
planning.
Nm
can
help
you
with
it.
We
have
two
office
locations,
one
in
milwaukee,
wisconsin
and
another
one
in
greenwich
village,
new
york.
B
We
have
a
fairly
large
tech
organization
with
a
wide
variety
of
technologies.
So
if
that
sounds
interesting
to
you,
check
us
out
online
find
us
on
linkedin
we'd
love
to
connect
with
you.
So
let's
do
some
quick
introductions
so,
like
I
said,
I'm
nicole
schultz
and
I'm
really
passionate
about
helping
application
teams
move
the
needle
in
terms
of
ci,
cd
and
devops.
C
My
name
is
bobby
wenzler.
I
am
the
product
manager
for
nicole's
team,
I'm
always
up
for
a
challenge
and
I
really
enjoy
partnering
with
the
engineering
team
and
the
clients
that
we
have
to
bring
solutions
to
life.
So
I'm
going
to
talk
a
little
bit
about
our
agenda
today.
First
we're
going
to
cover
our
problem,
then
we're
going
to
cover
the
industry
research
we
did
on
that
problem.
Then
we'll
talk
about
the
industry
findings
that
we
applied
at
northwestern
mutual
talk
about
our
solution,
the
cic
pipeline
certification
and
last
talk
about
our
outcomes
and
learnings.
C
So
first
I
wanted
to
talk
about
our
problem
and
that
is
there's
so
much
data,
but
there's
not
a
lot
of
visibility
in
action.
So,
as
you
can
see
here,
this
is
the
standard
pipeline
that
we
have
at
northwestern
mutual.
It's
got
a
lot
of
stages
and
jobs
included,
which
is
great
you're,
going
to
get
a
lot
of
really
fast
feedback
from
your
pipeline,
but
the
challenge
becomes,
as
you
look
at
this
screen.
C
C
C
So,
as
we
looked
at
this,
we
knew
we
wanted
to
learn
from
what
else
was
going
on
out
in
the
industry.
So
we
looked
at
this
book
called
accelerate
and
we
had
a
lot
of
really
great
learnings
and
I
wanted
to
share
a
few
with
you.
The
first
was
there
are
a
lot
of
gaps
with
maturity
models.
So
when
you
look
at
maturity
models,
it's
often
a
one-size-fits-all,
very
static,
never-changing
set
of
criteria
that
apply
across
the
board.
C
The
other
thing
we
learned
is
that
quality
automation
gives
you
something
great.
You
get
speed
and
stability
you're
not
having
to
sacrifice
one
or
the
other
to
go
faster,
you're,
not
giving
up
stability
to
get
stability
you're,
not
going
slower.
It's
really
the
power
of
having
great
automation
in
your
pipelines.
C
Next,
the
icd
and
sem
are
supporting
capabilities
for
the
door
metrics.
So
this
means,
if
you
want
to
really
get
the
sustained
dora
metrics
in
your
organization.
You've
got
to
have
some
good
source
code
management
and
ci
cd
best
practices,
if
you're
not
sure
what
the
door
metrics
are.
There's
another
presentation
at
getlab
commit
actually
being
done
by
google
would
highly
recommend
you
go.
Listen
into
that.
C
The
last
point
was
there's
light.
Mate
change
management
process
is
really
important
and
what
this
means
is
they
found.
Studies
showed
that
if
you
had
an
external
team
approving
your
changes,
you
actually
didn't
get
better
stability
at
all.
So
we
found
this
very
intriguing
and
wanted
to
learn
more
about
that
as
well.
C
C
The
next
is
automate
and
integrate
again
by
having
that
robust
pipeline.
You
know
we
saw
that
the
automation
for
the
high
performers
was
way
above
the
categories
of
all
other
performance
groups.
So
it's
really
important
to
actually
automate
and
build
out
things,
so
you
don't
have
to
keep
doing
the
same
processes
manually
over
and
over
communities
of
practice.
So,
like
many
big
organizations,
things
are
going
to
change
over
time.
C
So
I'll
go
back
to
automate
and
integrate,
so
it's
important
that
you
continue
to
automate
and
integrate
and
why
this
is
the
book
showed
that
basically,
the
teams
that
are
the
highest
performers
have
the
most
integration
and
automation
built
into
their
pipelines.
The
second
is
communities
of
practice.
C
We
want
to
make
sure
that
we've
got
a
good
community
of
practice,
so
as
things
reorg
and
change
in
one
company,
there's
still
our
champions
out
there
for
ci,
cd
and
devops
that
doesn't
it
doesn't
go
away
if
the
people
that
were
tied
to
it
are
there
start
with
foundations.
So
this
is
really
important.
There's
a
lot
of
great
practices
and
learnings
in
in
both
of
these
industry
researches,
but
you
don't
want
to
do
it
all
at
once.
C
You
want
to
kind
of
break
it
off,
just
like
you
would
in
agile
and
break
it
into
small
pieces
and
make
progress
over
time.
So
then,
the
next
piece
I'll
talk
about
is
how
we
apply
the
industry
findings
at
northwestern
mutual
we're,
not
ready
for
the
dora
metrics.
Yet
so
what
I
mean
by
that
is,
we
want
to
really
start
with
those
sem
and
cicd
best
practices
and
get
those
off
the
ground,
so
teams
have
somewhere
to
start.
If
we
were
to
tell
teams,
you
need
to
have
more
deployments
right
now.
C
C
We
wanted
to
address
the
maturity
model
gaps,
so
our
criteria
doesn't
say
static.
It
definitely
also
doesn't
fit.
One
size
fits
all
models.
We
wanted
to
really
make
sure
that
it
was
flexible
to
support
the
necessary
tools
and
technologies
that
were
in
our
company,
and
the
last
piece
is
increased
voice
of
customer.
So
it's
really
important
that
people
understand
why
you're
doing
this
and
what
they
stand
to
benefit
from
it
and
they
feel
like
they're
part
of
the
process.
C
We
really
felt
we
wanted
to
make
sure
people
were
involved,
so
it
didn't
feel
like
a
mandate
where
you
must
do
this.
It
felt
more
of
like
a
partnership
where
everyone
understood
what
they
were
going
to
get
out
of
this
tool,
and
next
I'm
going
to
turn
it
over
to
nicole
to
talk
about
our
solution.
Thanks
bobby.
B
So,
to
address
the
gaps
that
bobby
just
covered,
we
developed
something.
We
call
the
cicd
pipeline
certification,
which
is
essentially
an
automated
process
that
evaluates
a
repository
in
gitlab
and
gives
it
a
mature
rating
of
how
well
it's
doing
in
terms
of
predefined
predefined
cicd
criteria.
So
we
actually
started
with
a
version.
One
of
this
last
year
called
the
cic
assessment
and
we
got
really
positive
voice
of
the
customer
feedback
on
it.
B
Application
teams
really
liked
it
and
at
the
time
the
assessment
was
running
on
all
of
the
repositories
we
have
in
gitlab,
which
is
a
lot
and
something
that
we
learned
is
that
in
order
to
really
support
this
and
have
this
work,
we
need
to
create
project
types.
So,
as
we
were
looking
and
doing
our
industry
research
and
developing
kind
of
a
new
version
of
this
assessment,
we
had
we
decided.
You
know
what
we
really
need:
project
types,
that
we
can
have
specific
criteria
for
specific
project
types,
so
projects
in
gitlab.
B
You
know
they
have
different
languages
which
are
going
to
have
specific
needs
so,
for
example,
a
docker
repository
one
of
the
criteria
we
have
for
that
is
that
you
need
to
have
security
scanning
in
your
pipeline.
However,
for
a
database
repository
that
container
scanning,
you
know
that
is
not
going
to
be
there
and
there's
going
to
be
a
different
tools
that
that
are
used
for
the
database
repositories.
B
We
decided
to
start
with
just
evaluating
aws
docker
projects.
That's
really
because
for
our
cloud
platform
that
we
have,
we
just
have
more
standardizations
in
place
and
set
up.
So
it
would
be
a
lot
easier
for
us
to
automatically
grab
all
the
data
we
needed
to
check
each
criteria
using
using
just
projects
that
have
docker
so
our
maturity.
Our
csc
pipeline
certification
has
three
maturity:
levels,
bronze,
silver
and
gold.
B
So
a
little
bit
of
gamification
there,
each
one
has
a
theme,
so
the
theme
for
bronze
is
project
and
pipeline
basics,
as
you
can
see
on
the
screen
here.
With
all
the
criteria
listed,
you
know
we
have
your
default
branch
protected.
You
don't
want
to
have
people
directly
pushing
to
master.
We
want
our
source
code
to
be
tied
to
an
asset
for
us,
that's
really
how
we
do
ownership.
B
So
when
we're
tying
source
codes
to
assets
source
code
assets,
then
we
can
correctly
know
who,
which
team
owns
that
repository
in
that
code
and
should
be
the
contact
for
any
questions.
We
have
a
lot
of
the
things
on
the
screen
here
might
seem
really
basic
to
you.
But,
as
bobby
pointed
out
and
as
I
said
in
the
intro,
we
have
a
really
large
tech
organization
and
everyone's
on
in
a
different
place
in
their
devops
journey,
some
teams
that
are
more
advanced.
B
Yes,
these
criteria
are
going
to
be
done
on
day,
one
they're
going
to
have
them
already
when
we
roll
this
out.
We
knew
that,
but
there
are
also
teams
that
really
have
are
just
beginning
their
cic
and
devops
journey
and
might
not
know
where
to
start-
and
you
know
might
not
have
a
lot
of
time
or
bandwidth
to
really
think
about
what
what
they
should
be
doing
in
terms
of
that.
So
we
wanted
to
make
sure
that
our
certification
and
the
bronze
criteria
were
project
basics,
things
that
could
apply
to
everyone
across
the
board.
B
We
also
have
unit
test
coverage
added
in
here.
Unit
test
coverage
as
a
whole
is
still
something
that's
being
adopted
across
our
organization,
so
you
can
see
for
silver
right
now.
We
just
have
the
percentage
set
at
20
or
more.
We
definitely
plan
on
looking
at
increasing
that
in
the
future
and
adding
in
more
types
of
testing
in
silver.
We
also
have
the
fact
that
you're
pushing
artifact
to
registry
so
that
you
can
use
it
later
for
deployment
and
for
production
deployments.
You
are
utilizing
our
change
management
integration.
B
The
theme
for
gold
is
addressing
fast
feedback
and
having
automated
builds
and
deployments
for
your
default
branch,
so
addressing
fast
feedback.
That
really
means
that
your
pipeline
is
going
to
stop
when
previous
jobs
fail.
So
the
point
is,
if
you
have
security
scanning
in
your
pipeline,
that
should
stop
right.
If
it
fails,
it
should
be.
It
should
not
be
allowed
to
fail
and
have
your
pipeline
keep
progressing.
That
should
stop
so
that
you
address
the
fast
feedback.
You
fix
any
vulnerabilities
that
are
found.
B
B
You
obviously
definitely
want
to
have
automated
builds
and
automated
deployments,
and
that's
you
know
both
signs
that
a
team
is
fairly
advanced
in
the
cisd
space.
You
know
you're
continuously,
building
your
change,
you're
continuously,
deploying
that
most
likely
means
you're,
hopefully
deploying
small
iterative
changes.
You
know
in
into
production
to
minimize
risk.
B
Another
thing
we
have
here
is
the
unit
test
coverage
being
at
50.
Again,
we
needed
to
just
start
somewhere
and
that's
something
that
we'll
be
talking
about
how
we
are
we're
going
to
be
looking
to
increase
that
in
the
future,
so
future
changes.
I've
talked
about
a
couple
of
them,
one
of
them
at
the
bottom,
increasing
unit
test
coverage
right.
So
as
teams
progress
and
hopefully
add
more
and
more
unit
tests,
we
want
to
definitely
increase
those
percentages
to
be
higher
in
silver
and
gold
to
challenge
people
again
to
to
get
better.
B
B
B
We
also
want
to
add
additional
testing
types
into
the
pipeline,
so
right
now
we
just
have
unit
tests
because,
again,
with
all
these
criteria,
we
need
to
be
able
to
get
the
data
automatically
and
right
now.
Unit
test
coverage
was
the
easiest
piece
to
get
in
an
automated
fashion.
We
definitely
are
working
with
our
testing
organization
and
partnering
with
them
to
understand
what
other
types
of
testing
can
people
use
in
the
pipeline,
and
how
can
we
get
that
data
automatically
and
even
beyond
that?
B
Lastly,
we
would
like
to
create
an
exception
process
that
allows
teams
you
know
to
to
override
things
and
and
when
they're
doing
a
hot
fix.
So
we
all
know
that
you
know
hot
fix
is
coming
in
production.
Things
maybe
have
to
go
in
right
away.
Maybe
you
don't
have
the
luxury
of
waiting
for
your
entire
pipeline
to
execute,
or
you
need
to
allow
a
certain
step
to
fail
just
so
that
you
can
get
the
fix
in
in
a
timely
manner.
B
So
we
definitely
want
to
you
know,
figure
out
what
that
looks,
like
obviously
have
some
sort
of
audit
trail
etc,
but
something
that
customers
have
asked
for
that
we'll
be
looking
into.
So
I'm
going
to
turn
it
back
over
to
bobby
now
and
she's
going
to
be
walking
you
through
the
ui
of
our
application.
C
Yeah,
so
this
is
it.
This
is
the
pipeline
certification,
and
this
is
actually
the
report
that
will
display
for
an
individual
project.
So,
as
you
can
see
here,
get
lab
project
is
what
is
what
the
project
that
we're
looking
at?
You
can
see
bronze.
We
met
all
the
criteria.
That
nicole
talked
us
through
silver
is
where
it's
going
to
start
us
automatically
expanded,
because
we've
got
you
know
some
criteria
left
to
do
here.
One
thing
that
we
talked
about
was
the
need
for
upcoming
feedback.
So
here
you
can
see
this
last
feedback
line.
C
It's
going
to
give
you
a
lead
time
to
make
sure
that
teams
have
some
runway
to
actually
get
out
there
and
do
these
changes.
So
this
is
a
feature
that
our
customers
ask
for.
So
you
know
one
day:
if
we
ask
for
some
improvement,
they
don't
necessarily
drop.
So
this
gets
to
the
the
point
I
made
about
our
criteria
aren't
static.
C
You
can
also
see
here
clearly
where
things
aren't
met,
so
they
still
have
some
work
to
do
on
this
particular
criteria.
So
that's
what
you
know
they
need
to
do
before
their
pipeline
will
go
to
silver.
We
also
have
gold,
so
it's
not
expanded
again
because
right
away,
we
want
teams
to
focus
on
silver
and
kind
of
iterate
through
those
steps.
C
On
the
left
hand
side,
we
give
you
a
lot
of
information
about
the
project
where
we
certify
it.
As
you
know,
what
the
project
is
some
of
those
details,
nicole,
was
telling
you
about
how
to
tie
it
back
to
people,
and
then
we
also
added
in
a
few
other,
really
awesome
features
for
our
customers.
The
first
is
this
project
out
button,
so
there
could
be
teams
who
are
sun
setting
something
in
six
months.
C
Do
they
want
to
spend
time
on
that
project
or
when
they're
touching
every
day,
that's
going
to
be
around
for
a
while
they
might
not.
So
they
can
then
opt
that
project
out
and
say
this
project
is
going
to
be
sunset.
I
don't
want
to
focus
on
it
right
now.
We
really
wanted
to
empower
teams
to
focus
on
what
made
sense
for
their
organization
and
not
necessarily
dictate
you
know
everything
has
to
be
done
and
let
the
teams
focus
on
you
know
what
metrics
they
wanted
to
set.
C
The
next
button
is
recertify.
This
is
really
important
in
bronze.
Well,
you
can't
see
all
the
settings.
A
lot
of
those
settings
are
in
the
get
lab
project
themselves,
teams
really
loved
this
pipeline
assessment,
but
like
their
pipeline,
they
wanted
fast
feedback.
They
did
not
want
to
wait
overnight
for
things
to
run
so
this
recertify
project
button
will
go
out
and
get
the
latest
and
greatest
get
lab
settings
and
then
update
them
to
tell
them
where
they
are
now.
C
The
next
part
of
the
pipeline
certification
is
our
dashboard.
So
out
here
we
give
the
organization
a
rundown
of
how
we're
doing
percentage-wise
and
project
count
wise
for
the
types
of
projects
we're
certifying
right
now.
So,
if
you're
an
executive,
you
could
come
out
here
and
say
for
product
a
I'm
at
discount
gold
discount
over
bronze.
My
team's
opted
out
this
many
there's
also
another
piece
of
this
dashboard
that
shows
you
by
assets
as
well,
so
for
a
particular
asset.
C
You
know,
where
is
that
asset
health
at
from
a
cicd
pipeline
perspective,
and
then
we
also
show
trending
so
over
time,
do
we
see
gold
going
up
like
we
expect?
Do
we
see
silver
going
up
those
types
of
things
as
well,
and
now
I'm
going
to
turn
it
over
to
nicole
you're,
probably
really
excited
to
hear
more
about
how
we
actually
did
this
from
a
technical
perspective,
thanks
bobby
so
this.
B
Is
our
high
level
architectural
diagram
on
how
we
collect
all
the
necessary
data
for
our
cicd
pipeline
certification?
As
you
can
see,
on
the
left
hand
side,
it
all
starts
with
gitlab,
so
shout
out
to
gitlab
for
having
such
a
great
api
and
tool.
That'll.
Allow
us
to
take
everything
we
need
from
the
application
and
all
the
repositories
in
it.
B
So
we
utilize
a
system
hook
in
get
lab
when
a
commit
is
triggered
that
triggers
a
system
hook
for
us
which
puts
the
name
of
the
project
into
an
sqs
queue
that
we
have,
that
triggers
or
will
get
read
by
a
lambda,
and
we
basically
have
a
repo
metadata
lambda.
We
call
it
which
collects
metadata
about
the
repository,
so
lots
of
general
information.
B
You
know
it
collects
the
default
branch,
the
asset,
some
asset
information
contributors,
all
kinds
of
different
stuff,
a
lot,
a
lot
of
metadata
information
about
the
repository
in
general
that
we
use
once
that
lambda
is
complete
that
publishes
to
an
sns
topic.
We
then
have
our
certification
lambda,
which
actually
reads
from
that
sns
topic
and
gets
the
name
of
the
project.
So
you
can
see
the
first
language
that
runs
the
repo
data,
one
that
stores
the
data
in
a
dynamo
database
which
is
accessible
by
a
devops
api
that
we
have.
B
So
when
our
certification
lambda
is
running,
we
actually
pull
some
of
the
metadata
from
the
devops
api
to
grab
the
asset
information.
You
know,
like
I
said,
the
default
project.
We
pull
some
of
the
settings
like
do
you
have
your
default
branch
protected?
You
have
merge,
requests
pulled
on.
We
pull
some
of
the
criteria,
information
from
the
devops
api
and
then
the
certification
limit
does
a
lot
of
the
calculation
and
logic
around.
You
know
for
all
the
criteria
we
have
for
bronze
silver
gold.
Is
this
met?
True
false?
Is
the
criteria
upcoming
or
not?
B
Is
it
active
and
then
it
also
will
calculate
the
percentages?
That's
that
bobby
should
on
the
ui
of
like
six
of
seven.
What
percentage
is
that
et
cetera
for
each
maturity
level?
So
after
the
lambda
is
complete,
it
stores
the
results
in
another
database.
We
have
just
for
the
certification
and
then
that
database
is
read
by
our
certification
api,
which
is
our
middle
tier,
and
then
that
eventually,
is
what
feeds
the
ui,
which
bobby
showed,
which
is
our
front
end
for
the
certification.
B
So
let's
talk
about
some
outcomes
and
lessons
learned
so
one
thing
we
learned,
as
I
mentioned
before,
we
kind
of
had
this
version,
one
called
the
assessment,
and
that
was
collecting
data
on
a
nightly
basis.
So
one
thing
that
we
that
we
learned
was
people
don't
want
to
wait
overnight
to
see
if
the
changes
they
made
on
in
their
repository
got
them
to
bronze
silver
or
gold.
So
we
definitely
with
the
new
architecture.
B
I
just
showed
you
we're
able
to
collect
it
in
near
real
time
for
some
of
the
data
that
we
don't
have
set
up
real
time
right
now,
such
as
on
this
gitlab
settings
page.
So
when
you
go
into
gitlab
your
project
and
you
go
to
the
settings
page
and
make
some
changes
right
now,
we
don't
have
a
way
to
automatically
trigger.
Have
that
have
those
changes
trigger
our
data
collection
process.
So
that's
where
the
recertify
button
comes
in,
that
bobby
talked
about
so
for
bronze.
B
As
I
said,
if
people
are
making
changes
in
their
gitlab
ci
file
and
their
pipeline,
that's
going
to
trigger
a
commit
which
will
automatically
trigger
our
process,
so
the
data
collection
from
our
site
is
much
much
faster.
Now,
and
teams
can
get
on
near
real
time
data
collection
and
feedback
when
they're
making
changes
to
meet
a
maturity
level
zombie
talked
about.
We
people
also
really
wanted
the
opt
out,
as
teams
started,
setting
more
okrs
or
team
goals
around
this.
You
know
they
wanted
to
say
hey.
B
We
want
to
get
100
of
our
projects
to
bronze
okay.
Well,
for
these
10
projects,
you
know
we're
sunsetting
them
in
six
months.
It
doesn't
make
sense.
So
that's
really
where
people
wanted
the
app
doll
feature,
because
they
wanted
to
be
able
to
hit
team
goal
numbers
without
having
to
actually
work
on
repositories
where
it
doesn't
make
sense.
B
The
other
thing
we
heard
was
automation.
So
a
lot
of
teams
have
a
lot
of
repos.
In
some
cases,
teams
might
have
a
hundred
or
more
so
they
don't
really
necessarily
want
to
go
through
all
100,
repos
and
click.
You
know
four
or
five
check
boxes
to
meet
the
bronze
criteria
right
to
protect
their
branch.
B
Turn
on
mrs
all,
that
kind
of
stuff,
so
something
we're
actually
working
on
right
now
is
figuring
out
a
way
to
how
to
automate
that
so
on
our
ui,
adding
in
a
button
or
some
options
to
say
hey
for
this
group,
this
pro
these
projects
automate
these
criteria,
we're
starting
with
just
a
simple
criteria:
right,
we're
not
going
to
be
starting
by
making
changes
to
get
lab
ci
pipeline
files
and
stuff.
Like
that,
but
a
lot
of
the
bronze
criteria
which
are
just
updating
settings
on
a
gitlab
project.
B
We
can
automate
a
lot
of
that
for
the
application
teams.
The
other
thing
we
heard
was
teams
wanted
time
for
improvements
so,
and
that
really
gets
to
the
the
point
that
bobby
was
making.
We
have
the
upcoming
criteria,
so
teams
can
adjust
and
in
general
the
fact
that
we
can
show
all
this
data
helps
teams
give
more
of
a
view
to
their
leadership.
To
say,
hey,
we
want
to
make
changes,
we
want
to
get
to
bronze
or
silver.
C
Bobby
yep,
thanks
nicole,
so
I'll
talk
a
little
bit
about
some
of
the
outcomes
that
we
saw.
So,
as
you
can
see
here,
we
had
three
to
pace
three
to
eight
percent
increases
and
now
projects
that
actually
have
ci
cd
pipelines
in
those
pipelines,
they're
actually
including
static
application
security
scanning
more
often
now,
they're
also
leveraging
artifact
management
tools
that
we
have
more
often
they're
now
requiring
their
pipeline
has
to
run
successfully
before
they
merge
more
often
they're.
Integrating
our
change
management
integration
as
well.
C
They're
reporting,
more
unit
test
coverage
now
as
part
of
their
pipeline
and
they're,
requiring
their
merge
requests
or
they're
requiring
that
merge
requests
are
needed
before
you
can
get
them
changes
into
your
branch.
So,
while
three
to
eight
percent
might
not
sound
big,
we
wanted
to
to
call
out
that
we
have
you
know
somewhere
in
the
neighborhood
of
20
000
projects.
So
it's
not
a
small
drop
in
the
bucket
to
get
this
increase
across
the
organization
and
also
it's
important
to
note
that
all
of
this
was
done
without
a
lot
of
outreach.
C
Nicole
and
I
just
pretty
much-
had
some
dashboards
out
there
and
told
people
they
were
out
there,
there
was
no
mandate.
You
know,
teams
really
were
doing
grassroots
adoption
because
they
were
so
excited.
They
wanted
to
be
bronze.
They
wanted
to
know
when
gold
was
coming.
When
is
silver
coming?
How
do
I?
How
do
I
get
there?
So
it's
really
important.
C
I
think
the
other
point
I
stressed
in
an
earlier
slide
too,
about
voice
of
the
customer
really
getting
out
there
and
understanding
what
our
customers
wanted
and
what
problems
they
were
seeing
also
really
helped.
You
know,
I
think,
seeing
this
type
of
increase
in
six
months
is
awesome,
and
I
can't
wait
to
see
where,
where
the
next
version
of
the
the
pipeline
certification
is
going
to
take
us
now
that
we've
gotten
even
more
feedback
from
our
customers.