►
Description
Don't miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from 18 - 21 April, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Verifiable GitHub Actions with eBPF - Jose Donizetti & Itay Shakury, Aqua Security
GitHub actions have been one of the most popular ways to build and release software, with recent developments in supply chain security it became a major target for malicious attacks. A couple of years ago a widespread hack to codecov, a popular service prevalent in build pipelines, caught the industry’s attention. In response, a new solution to protect the build pipeline was created on top of Tracee, OSS Runtime Security solution, and introduced the concept of profiling with eBPF and verifying software builds. In this talk we will present that solution and explore the lessons learned in the past two years since the initial release.
A
All
right,
hi
everyone,
my
name-
is
itaisha
curry
I
lead
the
open
source
team
at
Aqua
security.
I
was
supposed
to
give
this
talk
together
with
my
colleague
Jose,
but
he
got
stuck
in
the
storm.
So
Jose
was
the
inevitable
part
of
this
presentation,
but
unfortunately
he
couldn't
make
it
so
we're
going
to
talk
today
about
something
you
probably
haven't
heard.
A
Moreover,
it
can
help
you
detect
suspicious
behavioral
patterns
in
those
events
and
either
using
a
library
of
building
signatures
that
we
ship
with
with
Tracy,
or
you
can
write
your
own.
We
call
them
behavioral
signatures
based
on
the
event
stream
that
we
generate.
That's
Tracy,
it's
open
source,
go
check
it
out
on
GitHub,
just
to
make
sure
when
I
say
Tracy.
A
You
know
what
I
mean
all
right,
so
I
said
we
want
to
see
if
we
can
take
random
security
Concepts
like
Tracy
and
apply
them
in
the
real
time
to
explain
the
motivation
for
that
I
need
to
take
you
back
a
little
bit
just
a
couple
of
years
ago,
different
times
in
Cloud
native
times.
You
know.
A
lot
have
happened
in
two
years
very
early
days
for
the
supply
chain:
security
category
before
all
the
s-bomb
craze
before
the
executive
order
before
tools
and
standards
that
didn't
exist
so
early
days,
some
meaningful
attacks.
A
That
happened
and
opened
the
our
eyes
and
everyone
else
to
look
into
this
stuff.
Specifically.
For
us
it
was
the
code
Cove,
a
bridge
I,
don't
know
if
anyone
here
remembers
that,
but
basically
he
targeted
GitHub
actions
users
by
so
the
attackers
were
able
to
compromise
a
very
popular
GitHub
action
called
Cove
in
code
coverage
scanning
tool
and
through
the
Upstream
of
that
Upstream
they
were
able
to
actually
compromise
you
or
your
build,
so
classic
supply
chain
attack
and
it
made
us
and
others
things
think
what
we
can
do
to
to
protect
against
that.
A
At
that
time
we
still
had
the
Tracy
open
sourced
for
a
while
Tracy
was
open
source
in
2019.
So
we
thought
why
not?
Why
not?
Just
try
and
use
Tracy
in
the
pipeline
I
was
actually
a
little
bit
skeptical
like
can
we
run
ebpf
in
a
managed
service
like
GitHub,
so
apparently,
yes,
they
don't
care
at
all
and
Tracy
just
ran.
We
wrapped
it
in
a
GitHub
action.
That
was
the
first
incarnation
of
the
Tracy
action
we
just
wrapped
it
in
a
GitHub
action.
You
add
it
to
your
pipeline.
A
It
starts
running
in
the
background
it
look,
it
is
looking
for
any
of
the
suspicious
behavioral
patterns
that
are
indicating
and
it
worked.
It
was
a
very
nice
first
step,
but
first
lesson
that
we
learned
is
that
the
build
time
is
not
the
same
as
production
specifically
Tracy
had
back
then
a
limited
set
of
signatures
that
we
shipped
it
with
and
they
were
very
tailored
to
production.
So,
for
example,
one
of
the
signatures
looked
for,
let's
take
an
example,
so
enforcing
immutable
infrastructure.
A
This
is
something
that
we
wanted
to
do
in
production.
You
know
immutable
infrastructure.
Basically,
you
need
to
pre-bake
your
containers,
ship
them
to
production.
You
don't
need
to
introduce
any
new
software
to
production
directly,
so
that
signature
looked
for
any
new
executables
being
introduced
in
production.
A
A
very
good,
probably
best
practice
to
do
in
in
the
production,
but
very
bad
thing
to
do
in
build
time,
because
the
build
server
pretty
much
has
only
one
job
which
is
to
produce
new
executables,
so
didn't
work
out,
so
we
had
to
fine
tune
the
list,
but
I
think
the
more
important
lesson
here
is
that
we
we
could
make
assumptions
in
the
build
time
that
we
could
not
have
make
in
the
in
the
production.
So
Tracy
was
built
with
production
in
mind.
All
of
the
signatures
had
to
be
very
generic
and
Abstract.
A
We
didn't
want
to
assume
anything
about
what
you
are
running
in
production.
What
tools
you're
using?
What's
your
Tech
stock?
We
don't
care.
We
just
want
to
look
from
the
below
from
the
underneath
and
to
see
if
something
suspicient
looking
for
us,
but
in
the
build
we
actually
can
assume
some
things
like
I
know:
you're
a
go
shop.
You
know
which
tools
you're
using
you
know
your
tool
chain,
you're,
not
going
to
switch
to
be
a
python
shop
every
other
day
right.
A
So
we
know
how
the
pipeline
is
going
to
look
more
or
less
it's
going
to
be
pretty
much
consistent,
so
we
can
leverage
that
to
write
more
specific
signatures,
things
that
we
could
not
have
afforded
to
do
in
production.
We
can
do
here,
for
example,
let's
do
something
specific
to
a
go.
Build
pipeline
like
the
go
mod
file
should
never
change
during
the
build
right.
If
you
want
to
change
it
change
it
before
make
a
full
request.
Have
someone
review
it,
but
it
should
not
change
during
the
build.
A
So
then
we
thought
why
don't
we
do
the
inverse
instead
of
looking
for
the
bad
stuff,
let's
just
Define,
what
is
the
good
normal
behavior
of
your
pipeline
and
just
enforce
it?
So,
in
other
words,
we
started
to
do
profiling
with
Tracy.
This
was
the
second
incarnation
of
this
project
and
we
introduced
a
profiling
feature
to
Tracy.
You
still
introduced
it
the
same
way
to
your
pipeline.
It
builds
the
profile
automatically
for
you,
so
we
ditch
the
signatures.
A
We
build
the
profile
of
every
executable
that
we
encounter
doing
the
the
Run
and
it's
supposed
to
represent
more
or
less
the
composition,
like
not
a
composition,
but
what
your
pipeline
is
made
of.
How
does
it
normally
behaves
like
I,
sometimes
think
about
it,
like
an
s-bomb
for
the
runtime
or
something
like
that?
Maybe
that's
another
talk,
but
we
generate
the
profile,
everything
that
was
executed
during
the
the
the
build,
and
then
you
review
this.
You
accept
it
as
the
Baseline
and
then
next
time
it
runs
again.
A
Another
lesson
learned
here
is
that
building
a
profile
is
not
that
easy,
especially
when
we
consider
executions
and
things
like
that,
because
there's
a
lot
of
very
volatile
information
involved
there
that
that
you
know
that's
going
to
change
like,
for
example,
if
there's
a
process
ID
some
somewhere
in
the
profile
right
we're
dealing
with
executions.
So
probably
the
process
ID
recorded
here
and
there.
You
know
that
it's
going
to
change
the
next
time.
You
run
it.
A
So
it's
a
very
difficult
thing
to
balance
between
collecting
enough
information
to
make
the
profile
meaningful,
but
not
too
much
information
to
make
it
annoying.
So
we
had
to
to
go
through
some
iteration
to
fine-tune
what
we
do
and
what
we
don't
want
to
include
in
the
profile
and
we
removed
a
lot
of
things
from
the
profile
to
make
it
stable,
and
we
thought
that
we
can
compensate
on
the
information
that
we
removed
from
the
profile
with
signatures.
A
That
will
help
balance
that,
and
that
brings
me
to
the
third
incarnation
of
this
project
and
the
current
one,
where
we
basically
take
the
both
The
Best
of
Both
Worlds.
So
you
still
into
those
places
to
pipeline
the
same
way.
Does
the
GitHub
action
you
add
it
to
the
pipeline
and
it
runs
in
the
background
it
will.
It
will
look
for
any
of
the
suspicious
behavioral
patterns
using
the
built-in
signatures.
A
A
We
also
expanded
the
profile
to
include
more
than
executables.
We
also
look
for
file
modifications
that
happen
during
the
pipeline
and
network
activity
that
happened
during
the
pipeline
and
the
profile
lets
you
represent
the
the
normal
behavior.
So
it's
kind
of
like
an
allow
list
and
denialist
approach
the
signatures
lets
you
declare
what
is
the
bad
stuff
that
you're
looking
for
the
profile
lets
you
declare
what
is
the
good
stuff
that
you
want
to
enforce
and
you
can
pick
the
the
best
tool
when
you
want
to
introduce
a
new
security
control.
A
You
see
a
main.go
and
go
mod
and
we
have
a
pipeline
in
GitHub
that
builds
it.
So
you
see
the
the
normal
actions,
you
see,
go
mode,
verify
go
test,
go
build
and
you
can
see
the
Tracy
start
and
stop
actions
that
to
this
Tracy.
You
can
also
see
a
suspicious
action
here,
fake
upload
right
now,
it's
good.
It
will
turn
row
later
on.
So
so
far,
so
good,
let's
push
this
a
project
into
GitHub,
and
this
is
the
first
commit.
A
The
reason
why
it
failed
is
because
Tracy
failed
it
because
it's
the
first
time
you
run
it
there's
an
unknown
profile.
You
need
to
acknowledge
it,
so
we
created
a
new
pull
request.
This
is
the
the
pull
request
that
asks
you
to
review
and
commit
the
profile
you
can
see
here
all
of
the
things
that
we
saw.
That
should
happen
during
the
pro
the
the
pipeline.
A
This
is
the
DNS
profile,
execution
profile
and
files
being
modified.
Three
files
introduced,
so
we
reviewed
everything
looks
good.
We
merge
this.
This
pull
request
again.
This
is
not
a
pull
request
that
we
made.
This
is
the
pull
request
that
Tracy
made,
and
now
we
need
to
go
back
to
our
pull
request
and
update
it,
because
now
there's
a
profile,
so
we
go
back
to
the
code.
We
update
main
Let's
check
out
our
our
branch
and
merge
it
with
main.
A
So,
basically
now
we
update
our
pull
request
with
the
the
newly
built
profile
go
back
to
to
GitHub,
and
this
is
the
sample
request
that
we
had
in
the
beginning
now
the
pipeline
is,
we
ran
and
it
passed
this
time
because
nothing
changed
from
the
from
the
profile
that
it
knows
to
the
current
execution
means
that
we
can
merge
it
safely.
Nothing
looks
wrong
here,
all
right,
so
some
time
passed.
A
A
We
just
want
to
re-trigger
the
the
workflow
it
doesn't
matter
which
change
we
make,
because
what
we
want
to
demonstrate
here
is
that
it
might
be
that
one
of
the
GitHub
actions
that
we
used
has
been
also
updated
during
this
time,
like
happened
in
the
code
bridge
that
we
discussed
earlier
and
let's
say
now,
we're
at
the
maintainer
of
that
action.
Let's
say
we
push
the
new
version
of
that
action
and
and
we
use
the
same
tag
and
it
means
that
our
pipeline
will
now
use
this
new
code
automatically.
A
A
The
the
pipeline
failed.
Also,
we
see
a
comment
here
from
Tracy
saying
that
we
saw
something
suspicious
happening
during
the
pipeline,
specifically
a
minor
domain,
something
that
looks
like
a
minor
in
the
pipeline.
We
have
all
the
raw
data
here
and
also
a
new
pull
request
was
created
to
show
us
what
is
changed.
A
What
exactly
was
changed
so
we
see
that
a
new
process
was
executed.
We
see
the
the
the
shaft
for
it.
We
see
the
arguments
and
we
also
see
that
the
file
was
modified.
Main.Go
was
modified
during
the
pipelines.
It
didn't
happen
before
it
shouldn't
happen,
so
we
use
the
pull
request
here
as
a
kind
of
a
user
interface
to
show
you
what
is
wrong
between
what
is
different
between
the
last
time
and
this
time.
A
A
The
first
challenge
that
we
faced
and
we're
trying
to
recap
like
the
past
couple
of
years
of
this
project,
was
that
running
eppf
at
least
a
couple
of
years
ago
was
not
that
trivial,
especially
in
a
remote
service
or
a
managed
service,
and
in
the
beginning,
Tracy
was
basically
compiling
the
ebpf
code
on
the
Node.
This
is
this
was
the
common
practice.
It
still
is,
in
some
cases,
compiling
the
abpf
code
on
the
Node
when
it
runs.
A
Is
it
in
the
past
that
we
expect
it
to
be
in
will
not
always
show
so
one
of
the
very
significant
changes
that
we
made
was
to
make
the
the
eppf
portion
of
Tracy,
compile
ones
run
everywhere.
This
is
how
it's
called
in
the
ebtf
world
basically
means
to
make
it
portable
in
simple
world,
it's
simple
words,
but
it's
very
hard
to
achieve
in
the
code
and
in
order
to
make
it
happen
again,
there
are
some
dependencies
on
the
on
the
on
the
machine.
A
Not
every
Linux
kernel
supports
this
in
the
same
way
Etc
and
we
had
to
create
like
a
separate
project.
It's
called
BTF
Hub
to
to
solve
the
problem
of
evpf
portability,
but
anyway
long
story
short
it
it
works
now,
and
it's
it's
one
of
the
first
projects
to
do
something
like
that.
So
it's
not
very
easy
to
run
Tracy
everywhere,
including
in
your
pipeline.
A
Another
thing
that
we
had
to
implement
here
and
that
we
learned
the
hard
way
was
that
when
we
are
tracing
stuff,
especially
for
security
purposes,
you
need
to
be
very
certain
that
you
capture
every
single
event
every
single
bit.
You
can't
afford
to
lose
any
information
and
the
way
that
this
orchestration
of
Tracy
in
the
pipeline
worked
in
the
beginning.
We
were
losing
some
events
because
Tracy
was
starting.
The
pipeline
was
starting.
There
was
kind
of
an
opportunity
there
to
lose
events.
A
A
What
else
so
another
thing
we
need
to
consider
here
is
that
there's,
the
GitHub
gives
you
basically
a
VM
to
run
your
code
right
in
this
VM.
There
is
the
workflow
itself
that
that
is
running
and
in
order
to
avoid
noise
from
the
machine
itself
like
this
is
Linux,
a
lot
of
things
are
happening
at
any
given
time.
I,
don't
know
a
system
update
might
start
sometime
and
we
will
see
it
because
Tracy
is
running
in
the
kernel.
A
So
in
order
to
focus
Tracy
to
look
only
on
what's
interesting
for
us,
we
have.
We
have
this
filtering
mechanism
in
Tracy
that
we
can
use
it's
an
old
feature.
You
can
tell
it
exactly
what
what
to
filter
on
so
for.
In
this
example,
we
identified
the
way
that
GitHub
sets
up
the
the
Runner,
and
we
saw
that
there
is
a
one
process
if
we
trace
this
process
tree.
A
Basically,
this
processing
everything
underneath
it
is
basically
means
we're
tracing
only
the
the
pipeline,
so
this
was
not
too
hard
to
do,
but
considering
that
there
are
other
areas
in
the
machine
that
we
needed
to
look
into
as
well
started
to
face
some
problems.
For
example,
some
things
will
run
as
Docker
containers.
A
These
Docker
containers
will
not
will
not
be
under
this
process
tree
that
that
is
basically
the
the
workflow
itself.
The
workflow
would
start
the
container,
but
the
container
process
itself
would
appear
under
the
docker
demon,
or
something
like
that.
So
we
need
visibility
into
that
as
well
right.
But
if
we
filtered
to
see
only
the
process,
three
it
will
conflict.
A
Another
thing
is
that
sometimes
we
need
visibility
into
the
host
itself.
As
generally
because
remember
we
have
the
generic
signatures
that
we
bought
from
Tracy
from
the
production
kind
of
use
case.
Then
why
not
use
them
to
look
into
the
host
as
well?
So
we
have
now
a
challenge
that
we
want
to
trace
different
scopes
for
different
purposes,
for
the
for
the
host
I
want
to
see
this
kind
of
events
for
the
workflow
I
want
to
see.
A
So
this
was
something
that
we
were
dragging
along
for
a
while,
but
recently
we
finally
launched
a
very
significant
feature
in
Tracy
that
we
called
multiscopes,
basically
allowing
us
to
to
score
to
trace
different
to
create
Scopes.
Basically,
a
scope
is
basically
a
set
of
filters
that
are
independent
of
of
one
another,
so
we
have,
for
example,
one
scope.
This
is
the
syntax
how
it
is
in
Tracy,
so
you
tell
Tracy
to
trace
and
we
create
scope
number
one.
A
This
will
be
just
the
list
of
all
the
signatures
that
we
want
to
to
observe
on
the
host.
So
file
is
execution,
is
one
signature.
Hidden
file
created
is
another
signature.
These
are
the
events
that
you
want
Tracy
to
trace
and
we
don't
apply
any
special
filter
here,
except
by
saying
this
is
one
scope
and
then
there's
another
scope.
That
says
that
we
want
to
look
for
file
modification
events.
This
is
a
kind
of
events
that
Tracy
emits,
but,
in
this
scope
number
two
well,
the
file
modification
event
is
in.
A
The
third
scope
is
basically
to
say:
let's
look
at
the
the
guitar
3
and
the
docker
tree,
and
in
these
cases
we
want
to
see
the
executions
and
the
network
activity
in
order
to
build
the
profile.
I
will
show
you
more
about
this
in
a
second,
but
the
the
point
here
is
that
scoping
the
the
trace
into
different
use
cases
and
for
every
use
case
tracing
the
only
the
the
relevant
events
for
that
scope
is.
A
Executions
is
one
category
in
the
executions
category.
There
is
a
bunch
of
signatures
that
are
introduced
just
by
the
nature
of
Tracy.
Is
there
so
looking
for
suspicious
things
that
might
happen
on
the
on
the
hosts
like
suspicious
executions
patterns?
Let's
pick
just
one,
for
example,
code
injection
some
process
is
trying
to
inject
code
to
running
another
process
running
instance
or
a
LD
preload.
Someone
is
messing
with
the
dynamic
Linker
or
something
like
that.
So
this
you
get
for
free,
like
in
quotes
from
just
Tracy
being
there.
A
In
addition,
Tracy
also
Builds
an
execution
profile
of
what
happened
during
the
build.
What
is
in
this
profile?
First
of
all,
binary
paths
like
what's
the
binary
that
was
ran
the
binary
hash
very
important
to
know
if
this
process
that
is
called
LS
is
the
same
as
another
positive.
It's
called
LS
the
user
who
created
it.
A
This
was
this
was
there
for
ages
and
it's
kind
of
goes
without
saying,
but
then
we
found
out
another
lesson
that
we're
missing
by
not
including
the
arguments
for
the
process.
Just
to
give
an
example,
let's
say
you
have
in
your
pipeline
a
curl
into
codecov.com,
and
then
someone
managed
to
change
it
to
curl
into
my
bedminer.com.
It's
the
same
curl,
it's
the
same
hush.
If
you
just
Trace
executables,
nothing
changed
right,
all
good,
but
no
just
by
changing
an
argument.
I
dramatically
changed
the
behavior
of
the
pipeline.
A
A
process
arguments
includes
a
lot
of
volatile
information
like,
for
example,
when,
when
you
do
a
git
clone
or
something
like
that,
GitHub
creates
a
temporary
directory
and
that
directory
name
is
changed
every
time
you
run
the
pipeline
and
that
is
being
passed
as
an
argument.
So
a
lot
of
volatile
information
that
we
that
would
just
pollute
the
profile
so
to
solve
this,
we
have
to
introduce
another
kind
of
feature.
That
is
a
ignore
system
that
lets.
You
basically
say
these.
These
kind
of
things
I
know
that
they
will
happen.
A
So
we
wanted
to
include
that
as
well,
but
that
created
another
problem
of
so
you
know,
environment
variables
contain
Secrets,
usually,
and
if
we
are
including
that
in
the
profile
we're
basically
committing
your
secrets
to
the
source
code,
not
something
that
we
want
to
do
now,
you
have
the
ignore
system,
so
you
could
ignore
this
kind
of
environment
variables
and
say
something
that
is
called
GitHub
token.
Just
don't
include
it
in
the
profile
that
will
solve
the
problem,
but
just
out
of
precaution,
we
decided
to
make
it
an
opt-in
feature,
just
the
environment
variables.
A
If
you
want
you
can
enable
it.
We
would
include
environment
variables
in
your
profile
and
then
you
are
encouraged
to
filter
the
the
secrets
out.
Another
interesting
interesting
thing
is:
how
do
we
even
detect
an
execution?
How
do
we
know
that
something
was
executed,
so
the
the
obvious,
like
the
Nave
thought,
would
be
to
look
into
the
system
call
that
is
invoking
new
executables.
A
First
of
all,
we
need
to
understand
that
the
system
call
is
not
really
like
invoking
function,
it's
more
like
the
user
requesting
the
system
to
do
something.
It's
not
necessarily
what
will
end
up
being
invoked
and
now
that
we
know
that
there
are
cases
where
the
user
might
request
to
do
something,
and
by
the
time
that
the
system
actually
got
to
do
it,
the
user
might
change
the
request,
so
we
traced
X,
but
the
system
invoked
y,
so
that
is
a
kind
of
attack
called
the
time
of
check.
A
A
So
let's
say
I'm
I'm
telling
the
system
to
invoke
this
binary
I'm,
giving
it
a
path
that
path
may
be
relative
to
some
other
directory.
That
path
might
be
a
file
descriptor
that
I
opened
and
obtained
earlier
and
I.
Don't
know
it.
That
path
might
be
a
Sim
link,
for
example,
that
the
scenes
that
the
system
needs
to
resolve.
So,
if
we're
just
tracing
the
exact
ve
calls,
it
might
be
that
we
were
seeing
meaningless
information
for
us.
We
would
see,
for
example,
that
file
descriptor
5
was
executed.
What
does
it
mean?
A
No
one
knows
unless
they
had
access
to
in
the
entire
capture
of
that
Trace,
so
our
solution
was
to
switch
and
use
another
event
that
Tracy
produces.
It's
called
the
sketch
process,
exec
it's
like
an
internal
transport
in
Linux.
It
solved
all
of
those
problems.
It's
not
vulnerable
to
time
of
check
time
of
use
it.
It
gives
us
the
real
path,
how
we
call
it
like
the
the
resolved
path.
That
is
the
absolute
path
to
the
file
on
the
disk.
It
also
gives
us
the
hash
and
many
more
information
that
we
include
on
that
event.
A
A
First
of
all,
there's
a
bunch
of
signatures
same
here
that
look
into
suspicious
file
access
buttons.
For
example,
someone
changed
the
suders
files
on
the
system
shouldn't
happen,
definitely
not
doing
the
build.
This
is
the
kind
of
thing
you
get
again
in
quotes
for
free
just
by
tracing
being
there.
The
profile
also
includes
file
being
modified
and,
and
I
mentioned
this
before
we
want
to
limit
this
to
only
the
the
Gita
workspace
directory.
We
don't
want
any
file
that
was
touched.
We
just
want
the
source
code
files
that
we
touched.
A
How
do
we
know
what
is
source
code?
We
just
say
everything
in
the
GitHub
workspace
directly
again
here
with
the
trigger.
It
was
a
little
bit
tricky.
If
you
want
to
know
when
a
file
has
been
written
to,
then
the
the
intuitive
approach
would
be.
Okay,
there's
a
system
code
for
that.
It's
called
write.
A
If
you
trace
write
system
call
in
your
system,
it's
totally
unmanageable
because
there's
so
many
rights
happening
at
any
given
point,
especially
on
like
on
Unix
Linux.
Everything
is
a
file
so
basically
impossible
to
deal
with
and
it
created
a
little
bit
of
a
challenge.
Our
solution
to
that
was
instead
of
tracing
the
the
actual
rights
to
the
files,
we're
tracing
when
a
file
has
been
opened
for
writing.
So
this
is.
This
is
an
action
that
the
the
program
has
to
take.
A
If,
if
you
want
to
write
to
a
file,
you
need
first
to
open
it
technically,
this
is
how
it
works
and
you
need
to
pass
the
right
flag.
So
this
is
what
we
capture.
Instead,
it's
a
nice
trick
and
we
use
it
later
on
again
like
to
to
trace
the
intent
to
do
something
and
not
necessarily
the
something
itself,
because
it's
a
lot
more
manageable.
A
A
So
this
is
unique
kind
of
in
the
tracing
world
that
you
you
don't
just
Trace
accept
system
core,
for
example,
and
then
you
get
gibberish,
you
actually
like
you
can
trace
HTTP
calls
for
example,
and
know
that
this
process
did
an
HTTP
get
to
this
IP
Etc,
and
what
we
would
like
to
do
here
is
to
know
which,
which
web
services
my
pipeline
interacted
with
right.
This
is
this
is
what
we
want
to
do.
The
problem
is
that
there
isn't
a
concept
of
a
service
in
the
network
world.
A
It's
like
it's,
it's
a
very
high
level
concept,
but
in
the
in
the
low
level
Network
there
isn't
such
a
thing.
There
is
just
TCP
to
an
IP,
but
we
cannot
even
deal
with
IPS,
because
IPS
are
also
very
dynamic.
They
will
change
for
good
reasons
and
and
again
touching
back
on
the
lesson
from
before
that,
instead
of
tracing
the
actual
thing,
sometimes
it's
easier
to
think
about
tracing
the
intent
in
this
case,
if
I
want
to
communicate
with
another
service
before
I,
do
that
I
need
to
do
a
dnsl
solution.
A
This
is
like
the
file
open
from
before.
So
if
my
pipeline
communicated
with
an
external
service
using
a
domain
name,
we
would
catch
that
because
there
would
be
a
domain
resolution
and
we
are
catching
domain
resolutions.
We
give
you
a
list
of
all
the
domains.
If
we
didn't
use
the
domain
name,
then
this
is
something
that
we
consider
a
suspicious
Behavior
like
contacting
a
bear
IP
and
there's
a
signature
for
that.
A
There's
also
like
other
network
related
signatures.
Like
someone
opened
the
reverse
shell,
like
trying
to
create
a
connection
outbound
that
tunnels,
your
shell
to
an
external,
endpoint
and
stuff,
like
that,
the
number
of
signatures
you
can
review
them
later
in
Tracy,
but
network
activity
is
another
section
that
was
added
today.
Github
action
of
Tracy.
A
It
was
another
kind
of
aha
moment
that
I
don't
think
we
have
yet
to
fully
understand
it,
but
just
wanted
to
share
it.
Even
at
this
row
point
that
we
have
a
lot
of
good
information
at
hand
right.
We
have
executions
profile,
we
have
network
activity,
we
have
files
being
modified
during
the
pipeline,
a
lot
of
good
information.
We
use
it
in
Tracy
to
tell
you
if
something
doesn't
look
right
in
your
pipeline,
but
maybe
you
can
use
it
for
other
purposes
as
well.
A
Specifically,
there
is
the
salsa
specification
that
deals
with,
so
they
Define
it
like
they
Define
a
provenance
as
the
verifiable
information
about
software
artifact
describing
where,
when
and
how
something
was
produced.
So
we
have
a
lot
to
add
about
the
how
something
was
produced
right.
We
know
exactly
how
it
was
produced.
So
can
we
use
this
information
in
this
context,
and
then
we
saw
not
too
long
ago
actually
that
there
is
a
proposal
to
create
a
formal
attestation
format.
A
It's
called
a
runtime
Trace
that
should
complement
the
salsa
predicate
and
from
that
spec
I'm,
quoting
the
random
Trace,
can
prove
that
the
build
was
invoked
via
script,
that
the
build
was
executed
in
a
hermetic
environment,
with
no
access
and
so
on.
So
this
is
exactly
the
kind
of
information
that
we
already
collect,
and
definitely
when
this
thing
is
is
a
little
bit
more
matured.
We
would
emit
the
the
the
the
information
in
this
format
as
well,
so
that
you
can
use
it
for
other
purposes
like
complementing
the
salsa
station.
A
A
We
were
able
to
increase
the
coverage
by
using
signatures
as
a
deny
and
profile
as
an
allow
kind
of
controls.
We
looked
at
how
the
blind
spot
of
tracing
tools,
how
we
overcome
them
with
the
specific
features
of
Tracy,
we
talked
about
system
call
tracing
as
opposed
to
other
approaches.
What
are
the
triggers
Etc,
and
this
is
how
you
can
look
at
all
of
this
actually
Tracy
on
GitHub
or
Tracy
action
on
GitHub
and.