►
From YouTube: CI/CD as a Service: Moving Towards Scalable Continuous Delivery - Samed Guner & Abhinav Srivastava
Description
CI/CD as a Service: Moving Towards Scalable Continuous Delivery - Samed Guner & Abhinav Srivastava, SAP
Speakers: Samed Güner, Abhinav Srivastava
At SAP Artificial Intelligence we build and run large number of microservices from different teams across different time zones, different technologies and different hyperscalers. Given this diversity in people, time and technologies everyone starts to build their own CI/CD, infrastructure management, testing strategies and many more. This leads to repetitive work, unfilled security/product standards, high lead time, firefighting and a sleepless operations team. So what is the answer to all of these problems? As the central AI DevOps team we want to share our journey and experience in building a centrally governed CI/CD, one-click onboarding, infrastructure management, test strategy and delivery process. In particular how to use Jenkins pipelines at scale to support over 200 developers and how to streamline deployments using Spinnaker and ArgoCD while managing the infrastructure.
For more Continuous Delivery Foundation content, check out our blog: https://cd.foundation/blog/
A
Good
morning,
good
afternoon,
good
evening,
everyone
from
wherever
you
are
joining
us,
my
name
is
abner
swastif.
I,
along
with
my
colleague,
samad
gunal,
work
with
sap
artificial
intelligence
and
senior
devops
developer.
Our
topic
today
is
cicd
as
a
service
moving
towards
large
scale,
continuous
delivery,
where
we
talk
about
how
we
build
a
centrally
governed
solution
which
work
as
a
service
to
different
microservices
and
how
it
makes
their
life
easy.
We
are
very
obliged,
speaking
here
and
very
thankful
to
cdcon
for
inviting
us.
A
A
But
if
we
try
to
see
behind
the
scenes-
or
often
we
tend
to
forget
some
behind
the
scenes,
aspects
of
these
particular
pillar
is
from
application
site.
It
could
be
from
the
linting
till
deployment
packaging
managing
all
the
dependencies,
versioning
and
observability
side
of
thing
for
each
microservices
building.
These
kind
of
different
angle
is
tough
infrastructure
side.
The
infrastructure
management
as
code,
the
hyperscaler
supports
the
security
angle.
A
The
disaster
recovery,
the
and
access
management-
all
those
concepts-
needs
to
be
streamlined
for
infrastructure
and
for
developers
to
make
their
life
easy
and
do
what
they
do
best.
We
have
to
provide
a
out
of
the
box
automation
in
terms
of
configuration,
feed,
release,
nodes.
The
environment
should
be
automated,
created,
multi-technology
support,
multi-language
support
and
all
in
all,
when
these
things
are
getting
combined,
they
actually
do
the
maximize
business
value.
A
We
had
this
problem
and
we
tried
to
solve
this
by
building
a
centrally
governed,
ci
cd
solution,
where
we
build
a
central
solution
with
all
the
end
to
end
aspect
of
a
cicd
offering,
and
we
just
offer
it
as
a
service
to
our
internal
stakeholders
or
internal
microservices
team.
Let
me
take
a
real
world
example.
So
suppose
I
have
a
50
micro
service
architecture,
application
and
initial
time
taken
per
micro
service
is
around
five
days,
given
the
fact.
A
If
the
developer,
who
is
working,
who
knows
the
things,
what
he's
doing
it
across
40
hours
and
maintenance
per
micro
service
is
two
hours
per
day
means
it
could
be.
Maintenance
could
be
anything
it's
relating
to
ci,
builds
or
ci
failures,
test
failures,
infrastructure,
failures,
etc,
etc,
and
if
you
just
calculate
all
these
numbers
for
a
50
micro
service
application,
it
should
be
around
it
could
come
around
200,
2,
000,
initially
and
22
hours
per
month
as
a
maintenance.
A
A
So
now
we
also
had
this
problem
and
we
do
realize
that
implementing
ci
cd
for
loyal
micro
services
architecture
could
require
more
time
and
cost,
and
in
the
next
slides
we
are
going
to
tell
you
our
experience
and
the
best
practices.
What
we
follow
to
deliver
our
application
with
large
number
of
microservices
very
smoothly,
and
very
often
on
that
note,
I
would
hand
over
to
sam
for
describing
the
architecture
and
different
parts
of
our
solution.
B
Yeah,
thank
you
very
much
avinas.
Now
I
want
to
go
with
you
I'll.
Give
you
an
overview
about
our
architecture.
It's
an
overview
so
about
the
technical
details.
We
will
discuss
them
as
we
progress
through
the
session
in
this
architectural
overview.
We
want
to
give
you
our
cases
and
overview
of
our
key
systems
relation
between
the
systems
and
the
resulting
ci
cd
as
a
service
offering.
So
in
general,
the
main
entry
points
were
offering
are
self-service
jobs
as
well
as
ci
pipeline
templates
available
on
all
jenkins.
B
We
actually
operate
in
large-scale
jenkins
cluster
hosting
multiple
teams
from
for
the
whole
ai
unit,
all
jenkins
hosts
the
ci
templates,
which
represent
sap
compliant
sentry
manage
as
well
as
battle
test
ci
pipelines.
These
pipelines
are
agnostic
to
any
programming
language
and
target
kubernetes,
as
well
as
close
fundraising
environments.
B
Since
we
are
a
cicd
offering,
we
give
the
people
by
the
possibility
to
have
a
one
click
experience,
meaning
with
one
job.
We
can
actually
onboard
a
new
application
to
our
platform
and
this
job,
then,
does
everything
necessary
in
the
background
like
creating
a
github
repository,
creating
a
necessary
credential
to
deploy
the
application
to
or
to
our
infrastructure,
as
well
as
all
of
the
small
dot
dotted
unknown
things
like
white
source
scans
protocol
scans
and
so
on.
B
B
We
have
our
jobs
in
addition
to
that,
we
have
our
jobs
to
manage
our
infrastructure,
especially
apply
changes
through
terraform
for
infrastructure,
where
terraform
is
used
to
deploy
or
kubernetes
classes
or
backing
services
from
sap
business
technology
platform,
as
well
as
from
gardner
or
open
source.
Kubernetes
management
plan
and,
in
addition
to
that
or
terraform,
is
even
able
to
rotate
secrets
and
manage
secrets,
since
we
are
taking
full
responsibility
from
cicd,
also
in
the
infrastructure,
part
and
delivery
part.
B
Even
more
spinnakers
are
capable
of
generating
status,
as
well
as
traceability
reports,
which
are
directly
mapped
to
gyra
or
reported
in
slack.
So
the
teams
program
management
pos
can
review
them
for
monitoring.
We
heavily
use
dynatrace,
which
gives
us
a
pretty
good
overview
about
our
system.
How?
It's
behaving
and
if
there
are
any
issues
we
are
able
to
debug
them
very
easily
trace
requests
from
multiple
layers
of
microservices.
A
Yeah,
so
that's
the
first
job,
what
we
have
where
we
onboard
a
particular
repo
or
a
developer
or
any
requirement
to
our
all
the
configuration
end
to
end
to
see
icd
suppose
in
a
real
world
scenario.
If
I
have
a
developer,
you
have
a
project
type,
say
python
or
javascript
or
whatever,
usually
what
you
do.
There
is
always
some
confusions
or
always
lots
of
tasks.
In
order
to
properly
run
a
ci
cd
pipeline.
You
have
to
integrate
with
lots
of
lots
of
tools
starting
to
collaborate,
build
test,
deploy,
run.
A
B
So,
generally
speaking,
a
developer
just
needs
to
add
one
single
line
to
run
a
centrally
governing
test
and
develop
ci
pipeline
with
that
developers
can
focus
on
creating
business
value
rather
than
fighting
groovy,
skips,
unstable
jenkins
slaves
and
other
problems.
So
with
this
approach,
we
are
able
to
push
fixes
to
all
of
our
components,
as
well
as
streamline
compliance
and
delivery
of
all
of
our
applications
at
once.
B
The
shared
library
itself
consists
of
pipeline
templates,
which
exchange
in
exchange
consume
modularized
stages,
which
are
simple
groovy
classes
and
those
groovy
classes
can
be
easily
used
in
different
pipelines.
So
we
have
a
pick
what
you
need
fashion
to
build
new
pipeline
templates
for
different
use
cases.
We
cycle
various
simple
jenkins
file
developers
can
provide
their
own
docker
files
to
run
certain
stages,
such
as
unit
tests
and
acceptance
tests.
B
By
having
such
a
strong
flexible
system,
we
are
able
to
dramatically
increase
the
developer
experience
by
reducing
the
time
spent
on
ci
activities,
as
well
as
streamlining
ci
cd
for
everyone
by
adding
a
streamlined
automate,
for
example,
adding
a
streamlined
delivery
process
for
all
onboard
applications,
and
with
that
I
want
to
move
to
the
next
big
topic.
It's
about
our
infrastructure
management.
B
So
why
else
the
cicd
toolkit
takes
care
of
the
ci.
Somehow
the
application
needs
to
run
on
some
infrastructure
and,
like
the
csc
toolkit,
infrastructure
management
is
one
of
the
most
important
pillars
of
all
offering
and
before
that,
we
would
sit
on
a
bunch
of
wrong
conflict,
diverse
environments
not
being
able
to
spawn
new
environments
or
even
go
to
have
go,
live
to
new
hyperscalers
within
weeks.
B
So
for
managing
our
infrastructure,
we
have
chosen
terraform,
which
allows
us
to
write
down
our
infrastructure
skilled
and
helps
us
scale
our
infrastructure,
both
in
a
number
of
environments
as
well
as
complexity
of
each
environment.
Since
we
write
our
infrastructure
code
as
installed
in
git,
we
also
leverage
github's
principles
that
have
several
benefits.
For
example,
we
have
free
audit
log
with
the
git
history,
so
we
know
what
is
done
when
by
who.
This
is
important.
If
you
want
to
get
specific
certifications
and
have
audits
for
that,
we
have
a
change
validation.
B
We
have
well-defined
interfaces
in
our
infrastructure
where
applications
after
where
deployment
can
during
runtime
find
out
and
what
they
need.
So
this
could
be,
for
example,
injected
configurations
for
certain
access
to
certain
databases
or
logging
services
or
any
other
backing
services,
and
with
that
infrastructure
management
is
responsible
for
creating
this
environment,
for
application,
which
are
agnostic
to
the
infrastructure.
B
B
What
you
need
a
lego
fashion,
building
our
own
infrastructure
and
with
that
we
actually
support
aws,
azure
sap,
banking
services
and
even
dyna
trace
dashboards
can
be
deployed
by
terraform,
and
with
that
strong
isolation
and
integration
could
we
could
do
actually
a
go
live
from
for
some
of
the
products
to
azure
in
seven
to
ten
days
with
just
one
developer.
So
this
helped
us
a
lot,
and
I
just
want
to
move
to
the
next
big
pillar.
It's
the
continuous
delivery
part
of
our
offering
and
we
had
continuous
integration.
B
You
know
how
to
bake
the
application.
We
know
where
to
run
the
application,
but
we
don't
know
how
to
deploy
it
there.
So
let
me
switch.
We
also
talked
about
get
ops
and
actually
in
offering
we
give
the
people
and
the
developers
the
freedom
to
choose
between
spinnaker
and
argo
cd,
so
we
have
infrastructure
managed
by
spinnaker,
and
we
also
have
infrastructure
managed
by
argo
cd
depending
on
the
use
case.
B
We
have
added
a
manual
judgment
step
where
we
take
a
small
step
back,
take
a
breath
and
see
okay.
What
did
we
do
in
the
last
previous
environments
before
hitting
production
with
this
change
and
we're
also
currently
working
to
smooth
out
this
or
to
get
rid
of
this
manual
judgment
stage
by
integrating
data
collected
by
dynatrace
to
spin
a
car
to
do
a
a
very
nice
canary
analysis
offered
by
spinnaker,
and
with
that
I
want
to
hand
over
to
the
monitoring
part
have
enough
to
you.
A
Yeah,
so,
as
you
can
see,
there
are
lots
of
offerings
in
our
solution.
It's
very
important
to
have
very
vigilant
about
what
is
happening
on
the
application
level,
as
well
as
on
infra
level.
So
we
rely
heavily
on
the
monitoring
thing
where
we
monitor
in
two
ways.
The
first
one
is
the
custom
devops
dashboard,
which
we
have
used
our
internal
technologies
to
build
it.
It's
kind
of
a
ui5
dashboard
where
we
track
each
microservices
with
their
version
about
their
quality
kpis
for
each
version.
What
is
the
build?
A
Pass
ratio,
the
security
things,
the
other
quality
measures
as
well
as
the
interesting
part.
We
also
do
the
devops
kpis
means.
We
all
know
that
kpis
the
market
standards
we
all
capture,
all
those
four
kpis,
mainly
the
lead
time.
Where
are
we
struggling
to
make
ourselves
better
and
do
the
internal
audit
in
our
offering?
We
also
wanted
to
have
environment
based
test
results
means
we
wanted
to
make
sure
that
our
all
the
environments
have
same
parity
and
all
the
test
cases
which
are
running
in
the
development
should
pass
in
the
production
environments.
A
Also,
after
the
deployments,
we
do
store
all
the
results
to
further
analyze
analyze
in
the
future.
On
the
dynamics
side,
we
do
the
apm,
which
is
application
performance
monitoring
for
for
each
micro
services,
having
all
the
kpis
mentioned.
Example:
400,
500,
the
failed
requests,
etc,
etc.
Apart
from
that,
we
also
monitor
all
our
infra,
mainly
the
kubernetes
till
the
pod
level.
If
anything
goes
wrong
and
proactively
fix
it,
we
rely
on
the
tracing
part,
which
connects
all
the
micros
micro
services
at
one
go
for
the
debugging
purpose.
A
So
if
any
changes
are
any
request
is
happening
in
one
micro
service,
we
travel
via
all
the
micro
services
and
press
the
issues,
and
we
also
do
the
analysis
on
archiving
the
results
on
this
dynatrace
thing.
So
all
in
all,
this
was
our
offering
and
end
to
end
and
how
we
are
centrally
governing
ci
cd
as
a
service
by
providing
200
plus
microservices
the
same
set
of
solution.
A
B
B
A
A
B
B
A
Yes,
any
other
question,
probably
I
can
ask
you
one
question
which
could
be
relevant
for
lots
of
people
since
we
are
talking
about
as
a
service.
There
is
always
a
you
know,
some
resistance
or
tough
part
from
some
of
the
organization
to
unify
the
things
how
they
do
it.
So
could
you
just
let
probably
answer
this-
that
what
was
the
most
challenging
part
we
have
done
in
this
journey
to
you
know,
make
sure
that
all
the
microservices
are
running
on
our
provided
services.
B
B
Yes,
so
the
killer
feature
is
the
scalability
of
spinach,
it's
way
easier
for
us
to
onboard
multiple
teams
and
projects
to
spinnaker
than
using
githubs
and
maintain
also
large
scale
of
kubernetes
cluster.
This
was
one
of
the
major
points,
but
also
the
inbuilt
capabilities
which
we
are
going
to
adapt
soon
with
canary
analysis
and
the
valve
integration
to
all
of
the
apm
services
like
dynatrace.
These
are
the
key
key
features
which
we
took
over,
but,
of
course,
for
specific
use
cases.
We
still
offer
the
possibility
that
you
use
argo
to
deploy
your
application.
A
Yeah,
so
there
is
one
more
question:
how
do
you
share
the
jenkins
file
templates?
I
can
take
it
so
we
have
made
jenkins
file.
As
I
said,
the
name
was
cicd
toolkit
in
the
presentation.
It
is
basically
a
jenkins,
shared
library
resides
in
a
get
and
we
just
shared
that
library
within
the
git.
As
a
you
know,
git
library,
or
something
like
that.
A
So
once
you
have
it,
you
can
just
easily
import
it
via
get
by
at
the
rate,
and
then
you,
you
know
just
import
the
particular
shared
library
in
terms
of
jenkins,
how
it
communicates
you
can
go
to.
You
know,
jenkins,
set
up
and
add
this
library
in
the
setup
to
make
sure
that
your
jenkins
basically
recognize
this
setup.
B
What
did
you
get
so
from
sean?
How
did
you
get
teams
to
engage
and
adopt
this
service
yeah?
So
this
is
a
very
good
question.
We
actually
got
the
mandate
from
the
management
and
support
from
the
management
to
actually
change
how
we
deliver
and
to
improve
there,
because
we
as
developer
race
topics
like
that
all
delivery
is
unstable,
that
testing
no
tests
are
working
fine,
and
then
we
got
the
mandate
as
a
team
from
management
that
we
actually
create
such
a
solution.
A
So
if
a
product
was
gets
say
supposed
getting
delivered
in,
say
seven
days
or
ten
days
for
a
single
deployment
and
with
our
solution,
just
it's
a
plug
and
play
and
everything
gets
delivered,
say
every
day,
or
at
least
once
in
a
day
and
developer
doesn't
have
to
do
it
worry
about
anything
I
mean
they
can
just
do
the
development
and
all
the
things
and
plug
and
play
with
all
the
sap
policies.
As
samus
said,
it's
a
lot
in
order
to
deliver
it
and
everything
is
automated
yeah.
B
Question
the
queer
section:
is
there
a
way
to
have
multiple
versions
of
a
pipeline
in?
So
this
is
a
very
technical
question.
What
we
do
in
spinnaker
is
we
actually
manage
in
spinnaker
or
spinnaker
pipelines
in
in
good
there's
a
good
way
by
armory
in
doing
that
and
supported
there
using
dinghy
and
but,
as
far
as
I
know,
you
can
only
have
one
state
of
the
pipeline
saved
and
get
you
can
use.
A
B
I
can
pick
this
one
so
when
you
build
especially
when
you
build
applications
at
sap,
cp
has
a
lot
of
product
standards
to
deliver
the
highest
quality
to
our
customers,
and
this
requires
speaking
to
many
systems
which
are
also
third
party
systems
which
do
specific
things
for
you
like
ip
scans,
like
malware
scans
and
so
on.
And
unfortunately,
this
system
landscape
is
not
well
integrated,
yet
that
you
can
have
a
one-click
one-click
experience
in
doing
that.
B
B
B
A
B
A
B
B
A
Yeah
so
yeah
I
mean
we
started
this
journey
around,
you
know
one
year,
one
year
or
half
or
so
so
we
started
with
this
very
small
poc
that
I
mean
when
we
took
this
project.
It
was
a
very
honestly
at
the
mess
and
that's
where
we
took
it
and
we
put
our
brains
that
how
we
can
you
know
do
it
in
terms
of
having
long-term
vision
that
going
forward.
We
knew
that
this
problem
is
gonna,
raise
more
and
more
within
the
company
and
within
our
organization,
so
yeah.
So
we
had
a.
A
We
are
very
basically,
we
believe
in
devops
principle.
It's
a
small
team.
You
probably
heard
of
a
concept
of
a
one
pizza
team,
so
we
are
people
of
four
and
five
people
very
diverse
kind
of
people,
and
then
we
took
around
one
year
or
so
to
you
know,
build
these
things
all
these
things
four
and
five.
Obviously
there
is
some
open
contribution,
also
from
the
developers,
so
we
we
work
in
a
very
open
source
kind
of
fashion.
A
A
A
Okay,
I
think
our
times
are
up,
but
at
the
same
time,
if
you
want
to,
you
know,
discuss
us
more,
we
can
very
happy
to
be
connected
on
linkedin
and
you
know
also
in
the
in
this
session.
Also
I
mean
in
the
cd
con
next
sessions
or
anywhere
the
meeting
rooms
or
something
like
that.
Meetups,
we
are
break
rooms.
Whatever
is
there
I
mean
we
are
happy
to
discuss
it
more.
Thank
you.
So
much
for
joining
us
have
a
nice
session
ahead.