►
Description
Continuous Integration and Delivery/Deployment helps speed up development and review workflows. Developers now can focus on code reliably tested in different environments. Once in a while the operations team gets paged on broken pipelines and jobs being stuck. Then the security audit has unveiled plain text secret exposure and dependency exploits.
The next horror story: The software cannot be deployed anymore since package dependencies are broken in a new distribution.
In this talk Michael Friedrich, Developer Evangelist at GitLab, takes a different look on making CI/CD pipelines more secure.
A
I
want
to
talk
a
little
bit
about
developers,
love
ci,
cd,
the
second
op
sequel,
it's
kind
of
a
snatchy
title,
but
we
will
get
into
it.
I
think
it's
a
little
bit
about
what
I'm
going
to
touch
in
the
next
20
minutes
round
about
a
little
agenda,
a
short
introduction,
then
we're
gonna
dive
a
little
bit
into
ci
cd,
that's
second
ops!
What's
in
the
pipeline,
going
even
more
deeper
down
in
the
pipeline
and
optimizations,
defining
some
team
actions
and
cultural
changes,
maybe
asking
ourselves,
are
we
safe?
A
This
is
kind
of
a
spoiler.
What's
coming
and
later
on,
also
asking
the
question
about
auto
devops
a
little
bit
about
myself.
My
name
is
marty.
I'm
originally
from
austria.
I
moved
to
nuremberg
around
about
eight
years
ago
and
joined
gitlab
is
a
developer
evangelist
in
march
this
year.
So
it's
around
about
seven
and
a
half
months.
Now,
I'm
doing
everything
around
monitoring.
Observability
infrastructure
is
codes,
writing
blog
posts,
social
media
and
things
like
that.
A
If
I'm
not
engaging
on
social,
I
do
love,
build
lego
models
and
have
some
coffee
chats
with
my
friends
talk
about
the
latest
tech.
This
is
about
me
about
gitlab.
Just
just
to
highlight
a
few
numbers
gitlab
was
founded
in
2014,
the
company
exists
even
longer,
and
I
know
christian
you're
miss
missing
russia.
Hexy
just
hit
me
up
with
your
address.
I
might
send
you
some
coming
back
to
gitlab.
The
thing
is,
you
can
use
it
for
the
entire
software
development
lifecycle.
It's
huge
feature
set.
A
You
can
self-host
it
in
the
cloud
offline.
Whatever
you
want
and
speed
up
your
development
lifecycle,
and
today
I
want
to
dive
a
little
more
into
how
ci
cd
evolved
over
time
and
how
development
security
and
ops
engage
with
it.
The
thing
is-
and
I
read
that
in
in
the
forestry
report-
lately
we
kind
of
changed
our
development
model,
so
the
speed
to
market
is
important.
You
need
to
release.
Often
you
need
to
release
fast
and
the
faster.
A
A
So
the
thing
is-
and
this
is
a
screenshot
or
an
experience
of
myself
when
I
started
with
ci
cd,
everything
was
red
and
nothing
did
work.
So
there
were
kind
of
a
few
iterations
necessary
to
adopt
and
say
hey.
The
pipeline
is
working
now
the
tests
are
in
a
good
shape
and
everything
is
fine
and
I'm
probably
never
touching
it
again
because
it
works
thing
is,
others
might
experience
the
same
problem
and
there
is
lots
of
trial
and
error
involved,
and
you
don't
know
about
the
best
practices,
and
sometimes
you
read
a
blog
post.
A
A
Your
ci
cd
pipelines
coming
from
a
developer
view
and
maybe
you're
being
asked
well,
please
fix
the
ci
cd
pipelines
because
well
and
you
you're
gonna
you're
gonna
go
there
and
say:
why
should
I
fix
them
yeah
because
they
consume
so
many
resources,
so
probably
they're
running
in
in
the
cloud
and
the
aws
bill
is
exploding
or
maybe
the
local
storage
is
being
drained
by.
I
o
caused
by
the
ci
cd
pipelines,
and
then
they
say
well,
I'm
just
like
the
developer.
I
just
want
to
run
my
tests.
A
I
have
no
idea
how
this
works
and
I
kind
of
build
it
up
and
how
would
I?
How
would
I
approach
this
problem?
Yeah,
maybe
just
read
the
docs
ask
the
community
because
everyone
has
something
to
share,
and
probably
there
are
best
practices
around,
and
this
leaves
you
we
could
leave
you
with
an
uncertain
feeling.
A
A
A
Another
thing
to
keep
in
mind
is
to
look
at
the
number
of
stages
and
jobs
involved
to
get
an
idea
where
optimizations
can
be
used
and
a
third
thought
would
be
incorporating
dependencies
and
synchrons
because
oftentimes
you
might
build
something
which
runs
in
parallel
and
asynchronous
super
fast.
But
super
hard
to
understand
and
optimize
later
on.
A
Another
thing
to
keep
in
mind
are
patterns
with
fake
jobs.
So
when
you
have
some
flaky
unit
tests
and
then
one
second,
they
work
and
then
five
seconds
they
don't
work
again
and
you
have
no
idea
what's
going
to
happen.
So
this
is
something
to
keep
in
mind.
Sometimes
the
test
coverage
drops
for
some
reason,
also
an
interesting
point
to
monitor
and
around
the
infrastructure.
A
A
So
when
we
start
looking
into
kind
of
optimizing,
cicd
pipelines,
one
idea
could
be
probably
the
tool
of
your
choice
could
be,
gitlab
could
be
github.
Something
else
has
an
analytics
overview
and
you
get
a
crisp
idea
about
the
duration
of
your
jobs
in
the
screenshot.
On
the
right
hand,
side
it
says,
there's
there's
a
job
which
runs
seven
minutes,
which
is
rather
long.
If
I
want
to
have
fast
feedback,
I
might
want
to
be
looking
into
this.
A
The
other
portion
to
immediately
look
into
is
like
failure.
Success
and
gather
get
an
idea
right
there.
Why
are
there
so
many
failed
jobs?
Is
there
like
a
pattern
involved?
How
does
this
evolve
over
time
and
getting
some
some
trending
and
some
reporting
out
of
just
to
get
an
idea
of
what?
Where
to
start
looking?
A
The
other
thing
is-
and
this
is
a
customization
of
of
yourself-
because
each
programming
language
might
need
different
approaches.
You
can
run
unit
tests.
You
can
get
the
feedback
in
your
merge
request,
pull
request
on
every
push
on
every
commit.
Whatever
whatever
is
needed,
super
nice,
it's
super
convenient.
You
can
integrate
it
into
the
merge
request,
you
get
an
output
and
you
can
probably
start
working
rather
rather
easily.
The
thing
is
this
is
something
also
to
watch
and
keep
in
mind
in
terms
of
optimizations,
also
flaky
unit
tests,
analysis
and
so
on.
A
Last
but
not
least,
we
might
have
different
visualization
layers
just
because
the
staging
or
the
stages
and
the
pipelines
are
looking
like
they're
in
a
good
shape.
Everything
is
green,
but
from
a
from
a
quick
glance,
you
cannot
really
say
well.
What
is
the
critical
path
in
there?
Is
this
something
running
asynchronously
but
whatever
comes
to
mind,
and
maybe
the
critical
part
is,
for
example,
in
the
in
the
picture
below
the
screenshot
below
you.
We
want
to
like
format
the
code.
A
A
Now,
when
we
go
a
little
deeper
into
the
pipeline,
and
this
can
get
quite
complex,
so
the
thing
is
as
a
developer,
I
might
need
to
find
my
operator
or
my
system
in
head
right
now
and
say:
hey,
maybe
I'm
looking
at
a
little
more
from
the
monitoring
perspective
and
I
want
to
get
the
global
picture.
I
want
to
check
the
workloads,
bottlenecks,
resources
being
involved.
A
A
In
case,
you
want
to
go
deeper
into
the
monitoring
level
and
like
having
a
visualization
around
ci
pipeline
or
the
icd
pipelines
a
good
way.
A
good
idea
would
be
to
add
it
to
your
existing
monitoring
system.
Probably
you
might
already
have
grafana
and
promises
around
or
like
nagios.
I
think
us
in
zusabics
whatever
comes
to
mind
for
your
monitoring,
and
you
can
immediately
visualize,
for
example,
the
cicd
pipelines
via
rest
api
access.
A
The
screenshot
shows
an
example:
how
to
use
an
existing
gitlab,
ci
pipelines
exporter
and
incorporate
it
into
a
grafana
dashboard.
27
minutes
in
the
screenshot
for
powerplant
duration
is
pretty
pretty
not
so
good.
This
is
probably
something
to
look
at
and
you
can
immediately
see
that,
like
in
the
office
dashboard,
getting
an
idea
and
saying
hey
something,
something
is
weird
and
we
need
to
start
looking
into
that
and
there's
also
a
check
plugin,
which
I
will
be
sharing
the
slides
later
on
another
thing:
to
look
at
the
resources.
A
This
is
something
this
is
like
lots
of
things
to
unpack.
Many
ideas
to
look
at
storage
consumption
can
be
a
thing
in
your
ci
cd
pipelines,
especially
when
your
jobs
generate
binary
assets
store
these
artifacts.
They
might
be
kept
on
the
storage
forever,
so
you
are
kind
of
keeping
terabytes
of
artifacts,
which
you
don't
use
anymore,
and
the
lookup
and
the
index
explodes
and
like
the
normal
runtimes
for
the
cicd
pipelines,
are
getting
slowed
down.
So
looking
at
them
and
like
purging
and
expiring,
the
artifacts
is
a
good
thing.
A
The
other
thing
which
could
influence
your
storage
layer
as
a
whole
are
container
registry
images.
So
if
you're,
using
docker
containers
or
apartment
containers
or
whatever,
whatever
format
you
prefer,
there
might
be
a
certain
change
chance
to
keep
everything
forever
as
well
and
defining
optimizing
the
storage
and
also
optimizing
the
container
image
is
a
good
thing.
A
The
other
point
portion
to
keep
in
mind
is
the
is
the
network
traffic,
like
I
said
before,
when
you're
on
a
low
low,
latency
connection,
or
maybe
you're,
even
paying
for
the
vpn
between
the
us
and
and
europe.
I
know
I
know
someone
who
does
this
might
get
quite
expensive
if
you're
pulling
lots
of
data
all
the
time
and
it's
like
the
ci
jobs
are
installing
their
dependencies.
A
But
when
the
image
is
10
gigabyte
or
20
gigabyte,
this
also
might
incorporate
lots
of
traffic
and
slow
down
your
ci
cd
pipeline
in
in
the
long
run.
Last
but
not
least,
everything
which
comes
to
mind
from
host
monitoring
or
linux
mac,
os
windows,
unix
flavor,
toast
the
cr
runner
might
take
this.
I
o
you
can
monitor
cpu
usage
and
you
might
even
look
into
your
ci
jobs
and
have
some
sort
of
chop
tracing
involved,
collect
the
data
and
integrate
it
into
your
merch
request,
for
example.
A
One
other
point
in
regard
of
merge
requests.
Is
you
want
to
in
want
to
use
them
for
fast
feedback,
so
getting
when
you're
working
on
on
a
new
feature
or
working
on
something?
You
can
also
incorporate
specific
things
like
the
test
coverage.
You
can
add
the
artifacts.
You
get
metric
reports,
the
test
reports.
You
might
even
use
code
quality,
analytics,
accessibility,
testing
and
also
security
reports.
A
So
whenever
there's
some
automated
security
scanners,
around
checking
for
static
application,
security
tests,
dynamic
application,
security
tests,
container
secrets,
dependency
scanning,
so
there's
a
lot
lots
of
things
to
unpack
and
you
can
get
everything
in
integrated
into
your
merch
request.
The
pull
request
might
be
done,
natively
might
be
done
with
bots
and
automating
the
issue
or
the
not
the
issue
the
merge
request.
A
A
Now
this
was
this
was
lots
of
ideas
on
how
you
can
approach
these
things
and
look
into
them.
Maybe
you
want
to
define
some
team
actions
around
this
and
say
hey?
How
can
we
kind
of
getting
getting
things
going?
What
are
the
things
we
want
to
start
with,
and
one
thing
I
I
always
kind
of
see
is
here:
you
probably
have
the
best
structured
stages
in
your
pipeline
and
oftentimes.
A
There
are
jobs
which
are
blocking
each
other
and
one
one
strategy
is
to
say:
okay,
I
want
to
fail
fast
and
when
the
source
code
is
not
compiling
like
the
format
job.
In
the
example
I
wanted,
I
want
to
just
fail
fast
and
say:
hey,
okay,
the
entire
pipeline
fails
just
because
the
suicide
cannot
cannot
be
compiled.
A
Otherwise,
and
this
is
the
screenshot
on
the
left-hand
side
code.
Quality
is
running
for
seven
minutes
like
the
screenshot,
then
at
the
beginning,
this
would
block
the
entire
stage
for
seven
minutes
and
then
ci
cd
says.
Oh
I'm
broke
in
contrast
to
format
which
runs.
I
don't
know
one
second
five
seconds
getting
immediate
feedback
and
also
not
wasting
resources.
A
The
other
thing
is-
and
this
touches
the
cicd
infrastructure,
depending
on
whether
you
are
responsible
as
a
developer,
or
maybe
you
have
your
ops
team
around.
Managing
that
docker
pull
a
docker
image
is
being
pulled,
could
take
quite
a
while,
if
you're
going
to
the
cdn
or
to
the
docker
hub.
This
might
also
be
a
problem.
A
This
is
a
possible
way
to
optimize
that
and
the
other
one
is
speaking
of
the
ci
jobs
which
always
install
the
dependencies.
You
might
want
to
create
pre-built
docker
images
in
where
the
tool
chain
is
already
installed
and
also
think
about.
Caching,
between
the
jobs,
like
optimizing,
the
the
ci
cd
pipeline
runtimes
and
the
example,
the
code
below
just
mentions
that
a
custom
docker
image
has
just
increased
php
dependencies
and
ci
cd
runtimes
after
all.
A
Last
but
not
least,
sometimes
it's
not
about
us
developers
or
like
you're,
seeing
a
stack
trace
in
the
application,
and
you
probably
have
no
idea
how
to
fix
it
as
a
non-technical
person,
and
this
is
something
which
we
also
see
when
we
at
gitlab
deploy
our
blog.
We
might
see
some
ruby
exceptions
and
the
idea
is
to
kind
of
catch
them
and
say
hey.
A
I
want
to
use
colored
error
messages,
maybe
with
an
emoji.
I
want
to
provide
a
clear
cut
message
and
also
link
troubleshooting
documentation,
for
instance
just
to
allow
everyone
fixing
the
pipelines
and
not
just
like
saying
hey.
It's
like
the
responsibility
of
this
developer,
because
git
blame
said
that
you
have
changed
this
line.
Everyone
should
be
able
to
fix
the
ci
ci
cd
pipelines
fast
enough
needs
documentation,
and
it
also
needs
to
create
issues
from
the
from
the
findings
and
encountered
learnings,
especially
because
in
in
times
where
everyone
is
remote.
A
A
You
might
have
forgotten
that
the
ci
configuration
has
some
clear
text
password
inside
so
you're,
giving
access
to
someone
outside
of
your
organization
and
you
figure-
maybe
the
mysql
database
password
has
been
leaked,
and
now
you
revoke
it.
Maybe
you
have
reused
the
password
for
different
applications
to
access
it,
so
monitoring
with
red
everyone
is
like
angry
burnout
and
things
like
that
also
not
so
nice.
So
this
is
the
cycle
where
security
automation
comes
in.
A
So
now
we're
reaching
kind
of
auto
devops
or
automating
everything.
So
we
are
how
to
build
or
to
test
trace,
can
fix
a
chest
like
machine
learning.
We
are
automatically
deployed,
deploying
everything
and
at
a
certain
point
we
might
just
merge
everything
automatically
because
yeah
we
are
automatically
happy.
Maybe
are
we
happy?
I
don't
know
the
thing
is,
and
this
is
an
interesting
thing
I
needed
to
learn
myself.
A
There
are
so
many
changes
coming
into
your
main
branch
and
deployments,
and
things
like
that.
You
totally
need
to
level
up
your
git
game
again,
so
you
need
to
learn
how
to
get
fetched
and
get
rebased
or
rich
in
main,
like
rebasing,
the
local
branch
against
the
remote,
and
maybe
you
also
get
a
lot
of
ai
and
machine
learning,
fixes,
also
suggestions.
A
A
One
thing
I
learned
is
like
to
have
tech,
coffee
chats
and
share
what
we
learn
with
each
other
in
a
kind
of
coffee
chat.
Atmosphere
in
you
know
remote,
but
the
thing
is
there's
more
coming,
so
there
are
some
stories
and
some
like
snarky
comments
inside
well.
If,
for
example,
someone
says
well,
let's
add:
let's
increase
the
test
coverage
to
a
hundred
percent,
because
we
need
a
hundred
percent
and
say:
okay
yeah,
then
I'm
returning
okay
for
every
function,
and
this
is
a
stop
and
everything
is
fine.
A
A
The
thing
is,
you
can
make
things,
but
things
are
complicated
so
when
you're
using
an
object,
oriented
model
and
you're
putting
it
into
into
the
main
object
into
the
constructor,
the
tracing
function,
everything
inherits
from
it
and
the
application
was
slowed
down
by
50.
It's
also
not
not
a
good
thing,
so
we
need
to
go
again
at
the
beginning
of
the
presentation
and
say
hey:
how
can
we
optimize?
How
can
we
learn
things
like
that?
The
other
thing
is
maybe
you've
seen
hashicorp
waypoint
last
week
on
thursday
being
announced
super
convenient.
A
Let's
try
this
we
can.
We
can
do
it,
it's
not
yamalani
moritz
using
htl
yeah,
but
maybe
we
can
wrap
yammer
into
html
now,
because
we
just
have
it
also.
Not
so
not
not
a
good
way
of
doing
things.
Another
snarky
comment:
why
does
our
wrap
our
app
not
running
kubernetes
already
yeah?
Probably
because
it's
a
monolith
and
we
need
to
rewrite
it.
A
We
write
everything
from
scratch,
but
we
cannot
tell
our
manager,
so
maybe
let's
go
ahead
and
put
everything
into
a
single
container
like
application
database
front
and
whatever-
and
this
is
just
fine
and
everyone
is
happy
because
we
run
it
in
kubernetes
as
a
container.
So
last
but
not
least,
the
aforementioned
machine
learning
they
are
suggesting
patches,
and
this
is
also
something
which
is
coming
in
the
in
the
future.
A
So
we
are
going
to
get
started
again
and
learn
and
say:
hey:
how
can
we
optimize
our
ci
cd
pipeline?
We
need
to
identify
that
ai
is
failing,
and
last
but
not
least,
fast
testing
was
coming
or
is
coming
fast
and
it's
exploited
our
production
and
then
you
find
out
or
learn.
Maybe
you
shouldn't
have
changed
the
dns
for
stage
dns
record
from
staging
to
production
for
testing,
because
it's
always
dns.
You
know
that
also
from
a
nickname.