►
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Securing CI/CD Systems Through eBPF - Alex Ilgayev, Cycode
The challenging nature of securing CI/CD pipelines, mainly because of inadequate security tooling and low observability of the underlying system, would make using the eBPF technology inevitable. This technical talk aims to demonstrate how eBPF can be used to secure CI/CD pipelines, whether they run on bare-metal, virtual machines, or ephemeral environments. By combining dedicated research, an innovative approach, and proper tooling, we can inject an eBPF-based implant into every build environment quickly and easily, to inspect, identify, and protect against malicious activity. We will demonstrate the following use-cases: - Visibility over the entire build process - created processes, contacted IP addresses/domains, modified files, traffic inspection, etc. - Ensure code and artifact integrity. - Denying build process tampering. - Maintain a tight network policy to prevent processes from exfiltrating sensitive secrets, such as tokens and environment variables. We will demonstrate how we can stop the deadliest software supply chain attacks while supporting all major CI/CD platforms, such as Github Actions, Jenkins, GitlabCI, and CircleCI.
A
Okay,
so
thank
you,
John
fascinating
talk,
and
this
will
be
yet
another
talk
on
security
and
actually
half
of
the
talk
is
praising
tetragon.
So
maybe
this
talk
will
be
much
shorter
right
now,
so,
okay,
today's
topic
will
be
securing
cicd
through
ebpf
in
the
upcoming
talk.
I
will
showcase
to
you
how
I
take
this
great
technology
of
vbpf
and
try
to
solve
for
it.
Some
of
the
most
some
of
tough
issues
that
we
have
in
securing
cicd
Pipelines.
A
The
community
builds
for
adpf
and
then
we
will
describe
what
is
a
cicd,
what
risks
post
to
CSD
pipelines,
what
the
security
issues
we
have
in
cicd,
then
we
will
merge
this,
both
technology,
BPF
and
cicd,
and
showcase
what
the
possible
solution
could
be
to
solve
these
issues
through
utilizing,
evpf
technology
we'll
tackle
that
solution
through
looking
some
of
the
most
prevalent
attacks,
we
have
over
the
cicd
pipelines
in
the
recent
years.
Some
of
them
were
attacking
the
software
supply
chain.
A
A
Senior
security
researcher
for
a
site
code,
previously
our
team
leader
for
the
malware
research
team
at
checkpoint,
where
I
reverse
engineered
some
complex
pieces
of
malware
that
originated
both
from
cyber
crime
and
apts,
and
nowadays
I'm
a
nobility
researcher
at
cycle,
the
researcher
vulnerabilities
and
also
mitigations
for
software
supply
chain
attacks.
A
cycode
is
a
cyber
security
company
that
provides
a
complete
software,
Supply
Chain
Solutions
for
organizations.
A
A
It
allows
you
all
efficiently
and
safely
to
extend
kernel
capabilities
without
modifying
any
kernel
source
code
or
writing
a
complex
pieces
of
Kernel
modules,
and
so
why
do
we
need
ddpf
I?
Think
we
had
enough
reasons
up
to
this
point,
but
in
short,
we
can
get
a
great
tracing
abilities,
observe
observability
and,
of
course
we
can
create
a
security
mechanism
in
the
kernel
effectively
safely
without
a
lot
of
hard
work.
A
So,
specifically,
in
this
talk,
we
won't
be
talking
a
lot
on
BPF
code
on
how
to
write
BPF
code
or
more
on
the
of
the
concept
of
taking
evpf
technology
and
and
solving
security
issues
with
it.
There's
plenty
of
tutorials
explaining
that
some
of
the
VPN
tooling
that
exists.
The
open
source
nowadays,
like
I,
am
a
bit
some
people
over
here
school
getting
started
with
writing
BPF
code.
So
there's
BCC
and
BPF
trace
a
great
tooling
to
get
it
started
to
get
your
first
BPF
code
compiled
and
running.
A
There's
several
issues
with
these
tools
that
determines
the
kernel,
headers
and
recompiling
for
every
operating
system
is
a
library
that
introduced
a
concept
called
query
compiled
once
run
everywhere.
It
solves
several
issues
like,
as
it
said,
you
don't
need
to
compile
it
for
every
operating
system.
Most
the
latest
Linux
distributions
contained
BTF
headers,
which
is
BPF
type
format
that
allows
This
Record.
Your
applications
run
easily
on
every
modern
Linux
system,
some
of
the
a
popular
open
source
project
that
utilize
this
score:
real,
silium,
petragon
and
more.
A
A
Such
simple,
SCI
CD
pipeline
could
be
taking
your
source
code
check
out
it,
compile
it
either
through
a
Docker,
build
goaling
make
file
or
whatever
your
compilation
method
is
taking
the
artifact
you
created
from
this
compilation,
put
it
in
some
some
registry,
maybe
some
package
manager
and
even
sometimes
take
this
artifact
and
deploy
it
in
your
production,
environment,
staging
testing
or
whatever
such
processes
has
a
few
issues.
For
example,
it
demands
highly
privileged
access
to
some
very
sensitive
assets.
A
For
example,
the
process
I
just
described
demands
sometimes
right
access
to
our
code
repositories,
it
Demands
a
right
access
to
artifact,
Registries
or
package
managers,
and
sometimes
even
to
the
cloud
infrastructure
So.
Lately
we
are
observing
that
the
attacks
on
software
supply
chain
are
arising
because
of
of
these
risks.
A
So
securing
say
CD
is
hard.
Why
it's
hard?
First,
we
don't
have
a
lot
of
security,
tooling,
The,
Standard,
Security
tooling,
like
antiviruses,
ETR,
firewalls,
just
don't
fit
for
these
purposes,
and
the
other
application
security
tooling
focuses
more
on
the
finding
vulnerabilities
and
vulnerable
packages
instead
of
securing
the
the
pipeline
itself.
A
A
Next,
usually
CSD
system
because
of
lack
of
security
tooling
also
have
really
low
visibility.
So
this
is
another
issue.
A
security
sock
team
need
to
handle
it
and,
lastly,
some
of
the
systems,
for
example,
GitHub
actions,
git,
lab
and
more
are
built
on
ephemeral
environments.
This
means
that
every
build
is
creating
a
new
machine
or
a
new
container
environment.
That's
creating
the
build
and
destroyed
afterwards.
So
this
make
it
Security
even
harder.
A
So
I
propose
a
let's
take
this
idea
of
abpf
and
try
to
to
try
to
stop
some
of
the
security
issues
we
have
in
the
cicd.
Why?
Specifically
BPF
it's
safe?
It's
easy
to
build
a
evpf
problem,
relatively
easy.
I,
don't
want
to
say
it's
really
easy
and
it
can
solve
some
of
the
observability
and
security
issues
we
have,
as
I
said
about
Quarry.
A
So,
let's,
let's
tackle
this
concept
for
some
of
the
latest
attacks
we
have
over
the
software
supply
chain
and
cicd
systems
in
the
recent
years.
The
first
one,
maybe
most
famous
one-
is
solarwinds
in
solar
winds.
There
were
a
sunspot
malware
that
was
planted
on
the
build
server
for
a
solarwinds
corporation
and
that
replaced
the
code
file
just
before
it
being
compiled
to
a
malicious
one,
which
resulted
to
a
malicious
artifact
that
was
signed
by
solarwinds
and
deployed
to
its
to
its
many
customers.
A
So
how
this
specific
this
abpf
agent
could
maybe
solve
this
issue.
First,
the
most
straightforward.
The
method
will
be
to
deny
any
right
access
to
the
source
code
file
during
the
CI,
build
it's
plain
simple,
but
also
can
be
very
effective,
I'm,
finding
really
hard
to
find
the
use
cases
where
a
CI,
workflow
or
some
build
process
need
to
have
right
access
to
the
source
code
file
and
another
possible
solution
would
be
to
monitor,
process
execution
file,
system
execution
and
maybe
compare
it
with
the
previous
execution
of
the
same
build.
A
A
A
First,
the
most
naive
and
simple
solution
would
be:
let's
monitor
all
the
network
connections
that
our
builds
are
doing
as
I
said
that
there
are
quite
repetitive
that
doesn't
have
to
like
to
access
hundreds
of
say,
domains
or
IPS.
They
can
see.
The
list
should
be
quite
simple:
a
more
advanced
solution.
Let's
create
an
allow
listing-
let's
say,
let's
say
every
process
that
doesn't
that
connects
tools
or
IP
or
domain?
That's
not
in
the
dial.
It
should
be
terminated
immediately.
A
The
third
use
case
is
more
like
a
that
doesn't
talk
about
specific
incidents,
but
a
group
of
incentives
that
would
see
a
rising
in
the
in
the
recent
year
are
installed
installation
of
malicious
dependencies
through
the
through
the
CI.
Usually,
when
we
are
building
software,
we
are
installing
dependencies.
A
Usually
the
we
don't
know
what
depends
exactly
we're
stalling.
We
don't
know
whether
they
are
malicious
or
not,
and
there
are
many
incidents
where
your
project,
during
the
build
and
compilation
that
install
malicious
packages,
whether
they
have
been
hijacked
by
some
attacker
or
maybe
some
type
of
squatting,
a
mistake
for
the
package
name,
for
example,
you
you
put
in
the
requirements
file
for
python
report
request
instead
of
the
popular
requests
package.
This
request
package
could
be
malicious,
so
similar
to
kodka.
We
could
also.
A
A
So,
first
you
all
heard
John
stock
really
fascinating.
To
talk
about
tetragon
tetragon
is
extremely
powerful
tool
that
allows
you
the
observability
and
run
to
enforcement
for
for
security
purposes,
so
I
will
run
it
quite
quickly
because
you're
already
already
seen
it,
but
why
I
choose
specifically
that
you're
gone
because
it's
highly
generic
it
allows
you
to
build
policies
and
to
load
them
to
tetragon
engine,
which
also
is
a
highly
capable
engine
that
loads
it
into
the
BPF
code.
A
Second,
it
has
limited
powerful
features
to
enforce,
based
on
predefined
filters,
so
we
can
enforce
on
the
Kernel
level
and
that
doesn't
have
to
write
code
in
the
user
mode
to
stop
this
and
third,
unlike
other
powerful
tools
that
exist
on
the
in
the
eppf
ecosystem.
The
dragon
also
works
outside
of
kubernetes
environment.
This
is
important
because
the
cic
system
don't
work.
We
can
work
out
so
outside
combats
environment
and,
for
example,
GitHub
actions
is
not
working
kubernetes,
it's
it.
A
So
the
appeal,
the
architecture
of
the
POC,
is
quite
simple.
It
has
two
parts:
it's
the
agent
which
is
the
most
important
and
it's
some
server
that
in
our
lab
environment
that
will
get
all
the
all
the
results
and
all
the
detection
we
had
on
the
agent
part
we
don't
have
to
the
server
is
not
a
mandatory,
but
it's
only
going
to
to
see
the
results
and
to
ease
the
the
experiment,
the
experiment.
A
So
in
the
agent
part
we
have
two
pieces
of
code,
we
have
tetragon
executable
and
we
have
our
agent
simple
agent,
written
in
golang
that
installs
a
tetragon
on
the
machine
and
communicate
it
with
on
the
it
gives
the
policy
that
it
should
be
loading
to
the
to
the
Linux
kernel
and
extracts
all
the
events
and
sends
it
to
our
to
our
lab
environment
to
understand.
What's
going
on
in
the
in
the
in
the
build
itself.
A
So
some
of
the
functionality
we
implemented
in
this
POC,
the
first
is
observability
for
the
build
system.
Understanding
all
the
executed
processes
build
some
process,
simple
process,
three
out
of
that
to
see
domains
and
IPS.
It's
nothing
fancy
here,
then
we
Implement
also
a
simple
source
code.
Integrity
feature
to
stop
against
attacks
such
as
solarwinds
and
to
experiment.
With
this
feature,
then
the
third,
which
is
the
most
important
to
my
opinion,
is
the
network
protection.
It's
create
some
simple,
allow
listing
for
domains
and
block
any
connection.
That's
not
in
this
specific
list.
A
Okay,
so
I
will
be
running
it
quite
fast
because
I
don't
have
stuff
for
the
demo,
so
the
in
the
installation
is
quite
easy.
It's
through
What's
called
the
custom
GitHub
action.
We
just
add
a
evpf
edge
detection
in
the
CI,
whoever
not
familiar
with
the
GitHub
action
syntax.
It's
not
really
required
here,
in
short,
for
every
installation,
I'm
just
putting
a
Docker
container
privilege
in
a
detached
mode
that
installs
this
tetragon
and
the
agent
we
wrote
to
secure
the
machine.
A
For
the
tracing
part,
it
was
quite
easy
because
the
dragon,
as
I
said,
is
really
powerful
tool.
It's
giving
us
the
entire
process
list
that
is
executed,
including
it
enriches
it
with
additional
information
such
as
the
parent,
the
arguments,
the
the
binary
and
more
when
we
combine
it
with
TCP
connect
tracing,
which
is
interesting
for
all
the
TCP
connection
made
by
the
operating
system.
We
can
also
giving
addition
context
to
the
connection
understand
which
process
did
did
the
connection,
so
we
can
terminate
it
in
case
of
DNS
tracing
there's.
A
A
A
A
So
for
for
integrity,
as
I
said
we
can
do,
we
can
Implement
Integrity
on
several
levels,
specifically
for
this
POC
we
Implement
only
code
Integrity
But,
ultimately
want
also
to
verify
the
Builder,
the
Builder
hash.
That
was
wasn't
altered,
for
example,
if
it's
go,
make
fi
make
or
or
a
docker.
We
want
to
verify
that
no
memory
modifications
were
made
to
the
process.
It
could
be
a
valid
hash,
but
it
was
a
modified
in
the
memory
and
we
want
to
verify
that.
No
right
operation
also
made
for
the
artifact
created
by
the
by
the
Builder.
A
A
B
A
A
A
Let's
start
the
server
right,
so
this
side,
basically
installs
the
ebpf
agent,
see
it
right.
We
also
installed
some
solarwinds
attack
setup.
This
is,
of
course,
artificial.
This
one
have
this
won't
be
in
a
real
CI,
but
it's
only
to
simulate
the
what
the
attacker
can
do
if
he
was
getting
a
hold
of
this
build
server.
A
A
It's
already
over,
we
have
out
of
a
hack
text.
This
means
that
the
attacker
managed
to
replace
the
the
goaling
file
and
and
to
to
create
a
malicious
artifact.
If
we
look
on
our
server,
we
have
a.
We
can
see
the
the
process
tree
that
we
from
that
we're
observing
from
our
agent
and
some
of
the
domains
that
we
were
asking
accessing.
So,
let's
run
it
again,
but
this
time.
A
In
this
case,
we
will
expect
the
will
expect
the
agent
to
understand
that
this
binary,
that
solarwinds,
that
we
have
predefined
in
the
CI
system
is
touching
some
some
go
source
file
and
terminate
the
process
exactly
immediately.
We
also
want
to
see
that
in
our
in
our
lab
environment
that
we
managed
to
to
stop
this
thread.
A
Okay,
so
when
we
run
the
executable
we
have,
this
is
benign
hello
world.
This
is
this
means
that
the
attacker
didn't
manage
to
replace
the
file
and
we
have
like
the
original
source
file
and
when
we
will
look
on
the
on
the
on
their
server,
we
can
see
that
these
two
processes
were
killed
by
the
agent.
We
are
searching
all
the
processes
that
were
received
CQ.
This
is
a
method
of
finding
a
mitigation
made,
so
we
can
see
that
the
solarwinds
is
the
executable
that
we
predefined
at
CI
to
do.
A
First,
we
install
the
the
agent
and
then
we
have
a
very
simple
said:
hello,
world
goaling
problem,
probably
a
pro
program
that
views
The,
Code,
test
it
and
uploads
the
coverage
to
code,
but
through
the
compromised
bash
uploader,
not
the
original
one.
Just
to
simulate
the
the
code
code
attack
from
the
last
year,
so
I
think
it's
over.
No,
not
yet.
A
Let's
give
it
a
few
more
seconds,
yeah
it's
over.
So
when
we're
looking
at
this,
we
can
see
this
is
the
process
tree
executed
and
we
have
an
interesting
line
here.
This
is
under
the
compromise
bash
uploader,
which
is
like
the
quad-core
bash
uploader,
and
it
was
doing
a
curl
to
some
unknown
IPA
or
a
domain.
A
A
Okay,
so
it's
quite
long
and
ugly,
but
this
this
very
long
list,
it's
processor,
skilled
I'm
this.
This
is
the
the
entire
argument
list
of
the
call
comment,
because
it
was
running
at
the
end
of
the
command
inside
the
minus
a
d
parameter
of
curl.
So
that's
what
that's,
how
the
attacker
initially
wanted
to
send
all
the
internal
particles,
so
we
can
see
that
this
call
command
was
terminated
and
it
caused
the
termination
of
entire
malicious
process.
A
A
We
have
to
do
more
smarter
domain
filtering
because
the
user
want
to
give
a
domain
to
to
for
the
allow
list
on
the
content
store,
but
ebpf
receives
IP.
So
we
need
to
understand
this
connection
and
and
implement
it
better,
and
also
we
have
many
additional
features
that
we
want,
ideally
to
implement.
For
example,
this
we
want
to
be
more
service
oriented
with
the
the
so-called
agent
need
to
understand
whether
it's
running
on
GitHub
actions,
on
gitlab,
on
Jenkins
or
Travis,
and,
for
example,
in
GitHub
action.
A
Maybe
you
want
to
allow
certain
steps
inside
the
jobs
to
access
to
certain
domains.
For
example,
we
have
like
a
go
mode
download.
We
want
to
allow
only
this
step
to
access
to
the
go
package
manager
and
in
the
in
the
entire
CI
should
shouldn't.
Do
it,
there's
an
additional
anomaly
agent
that
we
could
implement
the
entire
work
on
on
a
salsa,
whoever
salsa?
A
It's
a
it's
a
standard
that
building
a
some
industry
leaders
into
increasing
the
supply
chain,
software
artifact
Integrity,
so
we
can't
this
is
the
standard
says
that
a
good
practice
for
the
artifact
is
to
contain
their
provenance,
their
their
recipe
for
how
they
were
created.
So
this
recipe
could
also
be
created
using
such
a
tool
that
was
run
during
the
the
build
process.
A
We
could
also
support
additional
CI
systems
and
and
more
and
more
so,
this
project
currently
is
not
open
source
because
I
didn't
make
it
I
didn't
have
time
to
create
it
the
open
source,
but
it
will
be
in
the
in
the
upcoming
weeks,
so
you
can
all
maybe
try
and
experiment
with
it,
and
when
we
look
in
the
future,
we
are
considering
taking
this
concept
and
maybe
creating
a
better
solution
for
many
other
CI
systems.
That
will
work
also
for
a
production
grade
level,
and
maybe
it
could
be
adopted
by
many
other
open
source
projects.
A
So
we
would
love
to
hear
feedbacks
from
the
community
on
this
idea
and
don't
hesitate
to
to
ping
me
or
DM
me
on
Twitter
on
LinkedIn
or
whatever
media
you
want.
So,
let's
wrap
it
up.
What
are
the
takeaways
from
this
from
this
talk?
First,
this
is
an
ebpf
conference
and
I
want
wanted
to
Showcase
how
ebpf
is
on
some
technology
and
with
great
use
cases.
A
The
APF
Community
has
created
really
great
tooling
I
demonstrated
tetragon
how
I
was
able
to
take
it
almost
out
of
the
box
and
use
it
for
for
additional,
maybe
purposes
that
the
maintainers
didn't
think
about
in
the
first
place
and
and
finally,
of
course,
we
welcome
any
individuals
who
want
to
contribute
to
the
idea
and
maybe
cooperate
on
this
and
create
a
better
Security
Solutions
for
a
specifically
for
cicd
system
or
maybe
for
other
purposes
as
well,
and
that's
it.
Thank
you.
C
D
Thank
you
for
the
talk
was
great
in
the
context
of
GitHub
actions
when
you
are
I,
guess
doing
a
Sig
Kill
from
that
UPF
detection.
How
does
actions
handle
that
like
does?
Does
it
just
think
nothing
happened.
It
keeps
going
about
its
life,
or
are
you
actually
able
to
catch
that
and
display
to
the
developer
in
the
GitHub
action
panel,
some
kind
of
meaningful
thing
of
like
what
actually.
D
You,
oh
sure,
when
it
fails
like
from
the
detection
and
say
the
ebpf
detection
is
running.
A
sick
kill,
can
GitHub
action
somehow
capture
that
and
also
capture,
maybe
stand
it
out
or
standard
error
from
that
failure
and
then
display
that
in
the
GitHub
action
context
to
the
developer,
or
do
they
have
to
go
to
the
build
machine
and
find
what
happened?
Yeah.
A
Basically,
GitHub
action
is
just
a
VM.
You
fully
controlling
to
do
whatever
you
want
and
what
I
noticed
that
when
I'm
killing
such
a
process
center
semi-secure,
we
just
sometimes
it
stops.
Sometimes
it
continuous
depends
on
the
on
the
parameters
you
get
for
the
for
the
job
you
can
give.
You
can
give
a
parameter
that
continue,
even
when
it
fails.
So
it
will
just
show
you
like
everything.
A
E
Hey
thanks
for
doing
this.
This
is
great.
It's
just
a
random
thought
about
the
gift
of
action.
We
can
actually
throw
arbitrary
signals
too,
so
maybe
there's
some
way
that
the
GitHub
action
could
catch.
A
specific
signal
that
we
send
it
right,
like
we'd,
have
to
work
with
the
GitHub
folks,
but
it's
possible.
B
E
You
plan
to
take
it
from
a
POC
into
production,
maybe
without
a
mask,
remove
the
mask.
Oh,
will
you
take
this
from
a
POC
and
keep
working
on
it,
because
that
would
be
awesome
and
if
you,
if
you
keep
working
on
it
and
you're
interested
I,
saw
some
of
your
to-do's.
They
looked
like
like
things
that
were
on
my
to-do
list
as
well,
so
we
should
talk.