►
From YouTube: Keynote: Project Trebuchet: How SolarWinds is Using Open Source to Secure Their Supp... Trevor Rosen
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Keynote: Project Trebuchet: How SolarWinds is Using Open Source to Secure Their Supply Chain in the Wake of the Sunburst Hack - Trevor Rosen, SolarWinds
As you're no doubt aware, SolarWinds was hit in December 2020 with a sophisticated supply chain attack perpetrated by nation state actors. In the months since, they've been working to create an entirely new build system based on a number of CNCF and CDF projects. In this talk, you'll learn about what they're building, why it's necessary, and what it's like to be on the inside when the unthinkable happens.
B
A
A
I
do
a
lot
with
go
and
kubernetes
and
all
lots
of
fun
things
you're
familiar
to
anybody
attending
the
cncf
con.
I've
been
doing
a
lot
of
that
kind
of
stuff
about
the
last
five
or
six
years
before
that
I
spent
a
lot
of
time
building
penetration,
testing
tools,
offensive
security
products,
and
so
when
the
big
bad
stuff
happened
last
december,
I
didn't
know
anything
about
the
product
that
was
compromised
because
I'm
not
on
that
side
of
the
business.
So
they
asked
me
to
start
thinking
about
what
would
come
next.
A
So
what
does
solarwinds
do
we?
I
guess
the
big
headline
is
that
we're
the
market
leader
in
network
management
software
most
of
the
fortune
500,
I
think
about
80
of
it,
uses
our
stuff
pretty
much.
Every
government
agency-
you
can
name
universities,
mom
and
pop
shops,
all
the
way
up
to
tier
1
network
providers.
No
matter
who
you
have
for
an
isp
at
home
chances
are,
they
are
probably
using
the
orion
platform
somewhere
to
monitor
big
cisco,
routers
or
to
help
run
their
knocks,
or
things
like
that.
A
So
this
talk
is
30
minutes
long,
which
is
not
nearly
enough
time
to
give
like
a
comprehensive
introduction
to
supply
chain
stuff
and
outward
to
take
you
through
every
aspect
of
what
we're
doing
I'll,
be
leaving
out
some
things
around
securing
dependencies
around
how
we
leverage
spawn
files
in
the
build
process,
the
merits
and
pitfalls
of
signing
every
git
commit
and
some
of
the
deeper
details
of
how
we're
trying
to
dig
in
and
get
around
some
of
the
surface
level.
Non-Determinism
of
certain
build
technologies.
A
We
do
intend
to
open
source
everything
that
we've
been
building
sometime
over
the
next
couple
quarters.
So
obviously,
in
the
coming
months,
we'll
have
a
lot
more
technical
detail
available,
but
what
I'm
going
to
focus
on
today
is
what
we
call
internally
pipeline
mechanics
and
that
then
I'll
end
the
talk
with
sort
of
a
grab
bag
of
implementation.
Details
and
folks,
of
course,
can
ask
questions
after
that,
which
I
will
answer.
As
best
I
can
fair
warning.
A
A
I
think
it's
important
to
define
what
we
mean
when
we
say
supply
chain
attack.
So
google
has
a
pretty
good,
very
general
definition,
which
is
just
simply
unauthorized
modifications
to
software
packages
right.
But
I'd
like
to
take
that
a
step
further
and
say
that
we
can
divide
that
into
a
couple
different
categories.
A
A
There's
a
first
party
code
compromise
which
or
not
even
code.
Sorry,
first
party
compromise,
which
is
what
happened
to
us
at
solarwinds,
right,
a
system
that
we
own
that
we
maintain
was
compromised
and
some
bad
things
were
done
as
a
result
of
that
access
and
then,
as
kind
of
a
joke.
I've
added
like
a
third
type
here,
which
I'm
not
sure
if
it
even
represents
it,
because
I
don't
know
if
microsoft
ever
has
divulged
exactly
what
took
place
when
they
somehow
were
tricked
into
signing
root
kits
earlier
this
summer.
A
A
So
what
happened
in
sunburst
I'll
first
say
that
there
is
an
amazing
blog
post
from
fireeye
who
was
the
security
vendor
that
originally
discovered
this,
and
it
is
insanely
detailed
and
if
you
want
to
know
absolutely
everything
possible
to
know
about
the
malware
and
about
sort
of
you
know
the
way
that
it
works.
You
can
dig
in
and
if
you
just
google
fireeye
sunburst
blog
post
you'll
find
it.
A
But
basically
what
happened
was
our
build
system.
The
orion
build
system
based
on
team
city
and
not
the
kubernetes
version,
but
the
kind
with
like
long
running,
vmware
based
build
agents
was
compromised
and,
as
with
any
hack
attribution
and
initial
vectors
of
compromise
can
be
extremely
difficult,
if
not
impossible
to
ascertain
for
certain.
A
A
They
actually,
we
determined
they
actually
kind
of
stuck
some
dead
code
into
one
of
our
releases
and
then
when
they
were
decompiled
it.
After
grabbing
that
release,
they
saw
that
that
dead
code
was
there,
and
so
they
knew
that
they
could
then
insert
something
more
interesting
and
malicious.
But
there
was
a
lot
of
sort
of
camouflage.
Dll
had
a
very
innocuous
name.
It
used
a
lot
of
naming
identifiers
that
would
have
looked
normal
to
any
solarwinds
engineer
who
worked
on
orion.
A
This
was
all
directly
with
compiled
code.
There
were
no
compiled
artifacts.
There
was
no
source
code
compromised
at
all.
Once
it
woke
up,
it
would
mimic
call
home
traffic
to
to
our
stats
portal
and,
as
with
a
lot
of
contemporary
malware,
it
used
dns
for
command
and
control
right.
So
it
was
pretty
interesting.
It
would
wake
up
after
some
number
of
days.
It
was
kind
of
like
a
random
number
of
hours.
A
That
would
go
up
to
12
days,
and
then
it
would
emit
a
some
dns
request
and
then,
depending
on
what
it
got
back,
it
would
either
wake
up
and
do
more
stuff
or
it
would
just
stay
completely
dormant.
There
were
certain
ip
blocks
that
it
would,
if
it
was
found
within
there,
it
would
the
command
control
server
would
just
say,
stop,
don't
do
anything,
so
that
was
that
was
pretty
interesting.
A
So
once
we
found
out
about
this
from
fire,
which
happened
december
13th
of
last
year,
we
had
to
start
turning
the
entire
company
upside
down,
and
that
is
not
just
the
orion
platform
which
all
by
itself
is
a
pretty
big
challenge.
At
10
million
lines
of
code
developed
by
you
know
thousands
of
people
over
two
decades,
but
also
everything
in
our
cloud
business
we
had
to
rotate
every
single
password
and
secret
across
you
know.
A
Dozens
and
dozens
of
amazon
accounts
obviously
had
to
decompile
huge
numbers
of
build
artifacts
that
we
had
sitting
around
as
part
of
our
process
and
sort
of
make
sure
that
if
we
recompiled
from
source
that
it
matched
those
dols,
we
wrote
custom
scanners
to
deal
with
intermediate
formats
like
pdb's
and
try
to
use
that
also,
as
a
basis
of
truth,
to
sort
of
find
out
how
about
how.
Far
back
this
went.
A
A
A
First
of
all,
this
was
a
really
good
adversary.
We
obviously
don't
have
any
way
of
understanding
how
it
is
that
people
like
nsa
and
gchq
have
said
that
this
was
russian
foreign
intelligence,
but
that
is
what
the
spooks
have
told
us,
and
so
you
know
we
believe
them,
even
though,
of
course
they
deny
it.
We
know
now
that
fewer
than
100
customers
were
affected,
which
is
great,
because
we
originally
thought
that
it
could
have
been
up
to
16,
000.
A
and
solarwinds
was
almost
certainly
attacked
because
of
the
nature
of
the
orion
platform
itself,
like
any
system
that
does
large-scale
management
and
monitoring,
orion
has
to
have
a
lot
of
credentials
to
do
what
it
does.
Everything
within
this
product
category
has
to
do
that
kind
of
thing
right,
because
you're,
basically
saying
hey,
I've
got
a
system
that
can
log
into
some
other
system
and
pull
stats
or
manage
it
run.
A
Scripts
do
all
kinds
of
things,
so,
even
amongst
penetration
testers,
you
know
who
who
hack
for
a
living
good
guy
hackers
are
hacked
for
a
living.
Orion
has
for
a
long
time
been.
You
know
something
or
these
types
of
systems
have
been
something
that's
kind
of
nice
to
find
on
an
engagement,
because
if
you
can
get
in
from
there,
you
can
pivot
widely
through
the
network.
A
We
realized
also
that
we
were
going
to
need
to
completely
redo
all
of
our
build
infrastructure
and
kind
of
rebuild
it
on
a
much
more
contemporary
and
sort
of
bleeding
edge
technologies.
I
mean
that's
because
obviously
this
type
of
thing
is
not
going
to
go
away
like
a
an
attack.
That's
successful,
there's
going
to
be
more
of
them
and
folks
today
have
already
alluded
to
sort
of
the
wide-ranging
problem
of
supply
chain
kind
of
getting
bigger
and
bigger.
A
A
The
first
is,
we
need
to
move
to
ephemeral
infrastructure
and
the
advantages
of
ephemeral
infrastructure
are
not
unfamiliar
at
all
to
anybody
attending
a
conference.
That's
you
know
talking
about
cloud
native
tech
and
kubernetes,
so
I
think
people
understand
that,
but
just
moving
just
straight
up
moving
away
from
long
life,
build
agents
gets
you
quite
a
bit
more
security
than
you
would
have
had
in
the
past
if
you're
kind,
of
always
using
the
same
machines
that
are
always
on.
A
Obviously,
you
want
determinism
wherever
possible,
which
is
a
really
interesting
challenge,
because
most
or
quite
a
few
technologies
don't
actually
support
the
ability
for
you
to
build
something
from
the
same
exact
inputs
like
the
same
point
in
your
code
base
and
the
same
collection
of
dependencies
and
get
the
same
thing
twice
right.
For
instance,
if
you
want
to
build
a
container
image,
that
is,
the
exact.
Has
the
exact
same
shaw
from
the
exact
same
source
code?
You
can't
do
it
with
build
kit
right.
You
can't
do
it
with
docker.
A
You
need
to
go
and
find
some
other
thing
like
kneeco
or
one
of
these
other
technologies.
That
sort
of
supports
that
you
know
you
can
build
a
java
jar
this
way
with
maven
with
like
reproducible,
flag,
etc,
but
some
things
just
don't
support
it
yet
like,
for
instance,
net
you
cannot
yet
get
a
completely
deterministic
binary.
So
what
that
means
is
you
know
you
go.
For
instance,
you
want
to
download
firefox
you
download
firefox,
and
you
can
compare
the
sha
that
they
put
online
to
you.
A
Can
you
know,
run
open
ssl
and
you
can
generate
a
digest
and
you
can
say:
oh
the
binary
that
I
got
was
the
binary
that
they
they
expected
to
send
me.
These
bits
are
good,
but
it's
most
likely
that
I
don't
know
that's
for
certain
about
firefox,
but
it's
most
likely
that
the
system
that
produces
that
original
thing
with
that
particular
sha.
A
They
couldn't
get
that
same
shot
again
right,
but
wherever
possible,
you
need
to
try
to
go
and
get
determinism,
and
this
is
kind
of
funny
because
on
the
salsa
levels,
there's
a
little
asterisk
around
determinism
because
it's
just
sort
of
like
not
everything
can
do
it.
Yet
consensus
is
the
next
one,
and
that
means
basically
just
two
systems
agreeing
right.
If
you
have
determinism,
then
you
should
be
able
to
build
the
same
thing
twice
and
compare
them,
and
the
idea
here
is
that
you
know
if
this
had
been
possible
for
an
attack
like
sunburst.
A
A
Well,
I
should
say
this
helped
us
kind
of
decide
that
certain
systems
just
weren't
going
to
work
right
off
the
bat
in
particular
sas
offerings
so
right
away.
We
looked
at
circle
and
travis
and
github
actions
to
see
if
those
could
get
us.
What
we
needed,
as
I
mentioned
before,
solarwinds,
has
got
a
lot
of
different
kinds
of
products,
a
lot
of
different
kinds
of
build
systems
in
use,
but
we're
really
big
fans
of
sas
products
and
the
ability
for
developers
to
have
a
whole
lot
of
control
over
what
they're
doing.
A
So
we
decided
that
we
would
go
with
tekton
because
which
is
based
on
kubernetes,
because
that
would
give
us
the
desired
mechanics.
But
it
would
also
give
us
that
kind
of
all-important,
developer
experience
that
we
wanted
and
to
go
a
little
bit
deeper
on
that
on
why
it
won't
work.
If
you
think
about
how
circle
and
travis
and
github
actions-
and
I
think
most
of
the
cloud-
vendors
ci
solutions-
work-
the
entire
build
definition
from
top
to
bottom
lives
in
the
repo.
A
A
We
don't
want
the
developers
to
have
to
think
too
much
about
security
and
validation,
and
indeed
we
don't
want
to
put
any
of
those
controls
in
their
hands
right
per
the
golden
rule.
We
need
to
be
able
to
bring
those
receipts,
as
people
say,
for
every
bit
of
the
standardization
and
enforcement
that
we
do,
and
so
we
have
to
be
able
to
separate
this
out
right.
A
So
kubernetes
is
actually
really
good
at
architectures
that
involve
you
submitting
like
a
resource
definition
that
you've
written
and
then
have
that
thing
kind
of
get
augmented
and
modified
by
the
system.
Right.
You
can
think
about
things
like
how
istio
works
where
it
injects
an
envoy
sidecar
into
every
pod.
You
don't
have
to
think
about
that.
It
just
sort
of
happens
for
you,
so
the
idea
that
we
could
easily
mutate
user
supplied
definitions
was
really
attractive
to
us,
and
also
this
is
a
great
dev
experience.
A
A
You
write
yaml,
you
define
tasks
and
tasks,
have
steps,
steps
running
container
image,
there's
great
semantics
for
being
able
to
share
data
for
passing
data
around
surfacing
up
results.
You
can
have
tasks
inlined
or,
crucially,
you
can
reference
heavily
parameterized
tasks
that
live
in
a
separate
repo.
You
can
have
a
catalog
so
yeah.
That
makes
it
easy
to
not
repeat
yourself.
It
makes
it
easy
to
standardize.
A
Okay
cool,
so
we
have
technology
that
will
let
us
separate
concerns.
It
will
allow
us
to
follow
that
golden
rule,
making
sure
that
the
developers
write
their
own
stuff
and
they're
not
involved
in
making
sure
that
that
it's
validated
and
enforced
and
secured
we've
got
a
sas
experience
the
developer's
familiar
with.
So
the
next
thing
is
to
tie
it
to
github.
A
A
So
we
built
several
pieces
of
kit
here,
as
I
say,
to
kind
of
marry
up,
tecton
and
github.
The
first
is
we
have
a
github
app.
That
is
the
thing
that
actually
kicks
off
builds.
So
a
github,
a
github,
app
in
case
you
don't
know,
is
basically
just
going
to
catch
a
web
hook
right.
You
have
it
sitting
in
your
infrastructure
and
when
a
pr
is
opened
or
a
commit
is
added.
A
It
fires
a
web
hook
with
data
and
then
what
this
app
does
is.
It
will
fetch
the
repo
it
will
extract
the
pipeline
definition
from
a
standard
location
and
it
will
sort
of
validate
that
it's
ready
and
then
it
submits
it
to
the
text
on
pipelines,
controller
over
rest,
api
and
then,
of
course,
techton
starts
to
spawn
task
runs,
which
are
just
instantiations
of
the
tasks
defined
in
the
pipeline
and
as
it
does,
you
know
that
taskrun
might
want
to
talk
back
to
github.
A
We
keep
our
cluster
from
talking
to
anything
but
github,
but
we
don't
want
necessarily
any
given
task
to
be
able
to
just
do
whatever
it
wants
with
github.
So
we
ensure
that
it
can
only
talk
back
through
a
proxy
which
limits
the
api
access
that
it
can
that
it
can
hit.
It's
got
very
limited
access
to
the
github
api
surface.
So
even
if
there
was
some
kind
of
compromise,
it
couldn't
just
go
off
and
sort
of
send
things
back
to
github
under
the
aegis
of
the
token
as
it
wanted.
A
We
also
have
a
kubernetes
controller
called
our
pipeline
watcher,
which
reports
results
back
to
the
github
checks
api.
So
this
is
what
reads:
log
lines
and
status
information
from
the
task
runs
and
kind
of
kicks
it
back
in
real
time
to
github,
and
that
gives
you
this
experience.
That
feels
somewhat
like
github
actions,
like
you,
click
on
a
tab
and
you
can
see
all
the
different
checks
that
have
run
you
can
see
them
in
order.
You
can
click
to
each
of
them.
A
You
can
see
your
log
output
and,
if
you
want
to,
you,
can
click
a
link
down
from
there
and
jump
straight
over
to
the
tekton
dashboard,
but
you
can
really
do
quite
a
bit
of
your
work
just
directly
inside
github,
as
somebody
who's
just
kind
of
looking
at
your
pr
and
seeing
if
you
can
move
forward
or
not
okay.
So
this
is
what
the
system
looks
like
now
at
this
point
right
we've
got
github
the
repo,
and
then
it
comes
in.
It
hits
the
app
that
other
kit
box
up.
A
A
So
then,
of
course,
things
hit
the
tecton
pipeline
magic
happens
and
things
get
built
and
they
land,
if
they're
container
images
in
ecr
or
in
s3.
If
they're
built
binary,
artifacts,
okay.
So
now
we
know
that
we
can
build
something,
and
we
know
that
we
can
honor
the
dev
experience.
We
can
do
kind
of
a
good
job,
giving
an
easy
experience
to
devs
and
they
can
have
a
thing
that
feels
quite
a
bit
like
they're
used
to
with
circle
ci
or
actions.
A
So
now
we
need
to
make
sure
that
we
can
record
how
we
build
it.
We
need
to
know
the
provenance
of
everything
and
that's
a
word
that
you'll
probably
hear
a
lot
today.
It's
basically
just
means
where
something
comes
from
and
to
some
extent
what
it
contains
and
we
have
to
be
able
to
produce
records
that
hold
data.
That
tells
you
kind
of
what
a
build
step
did
and
what
was
kind
of
what
the
inputs
were
to
that
build
step,
and
those
records
also
need
some
kind
of
guarantee
right.
They
have
to
be
signed.
A
How
many
people
here
are
familiar
with
in
toto
already,
okay
cool?
So
that's
like
most
of
the
room
good
job,
so
that's
that's
good
and,
as
you
know,
then
it
solves
kind
of
a
lot
of
the
conceptual
problems
in
supply
chain
security
and
helps
give
sort
of
a
framework
for
thinking
about
a
lot
of
these
things.
A
So
we've
been
participating
quite
a
bit
within
toto
and
we
ended
up
implementing
some
specs
and
go
for
some
of
the
things
that
we
wanted
to
be
able
to
use,
and
the
main
thing
that
we
implemented
is
in
total
expansion
proposal.
Six
or
eight
six
proposes
an
attestation
format,
foreign
toto
based
on
salsa's
attestation
spec.
So
this
diagram
here
is
a
lot
nicer
than
my
diagrams
and
it
comes
from
the
salsa
github
page.
A
A
subject
is
the
thing
that
we're
actually
building
a
predicate
contains
like
it's
just
like
a
recipe
like
how
how
we
build
it
and
what
what
we
use
to
build
it
and
method
and
the
ingredients
and
then,
of
course
the
signature
is
the
guarantee
right
that
some
system
produced
this
and
and
it
is
signing
saying
hey.
This
is
me,
and
you
should
trust
me
because
I
signed
it
and
I
know
there's
lots
of
t-shirts
at
this
con.
A
A
Okay,
so
great
now
we
know
that
we've
got
like
a
format
for
how
we're
going
to
have
these
documents,
like
a
standardized
attestation
format-
and
you
know
again
the
go
back
to
that
golden
rule.
It
says
that
devs
can
build
what
they
want,
but
their
workflows,
their
authoring
won't
have
any
role
in
producing
those
attestations
right.
So
those
are
not
going
to
be
part
of
their
workflow
definitions.
A
So
now
our
diagram
has
this
extra
bit
right.
We've
got
tekton
chains
in
place
and
it's
sitting
there
using
a
kms
backend,
never
manage
your
encrypted
material.
If
you
can
avoid
it
and
it's
using
that
to
sign
and
stick
those
documents
into
a
document
database.
So
great
we've
got
a
pipeline
pipeline's,
easy
to
use,
it's
got
out
of
stations
and
it's
got
a
database
which
means
that
we
can
have
clients,
ask
questions
and
say
things
like
hey.
Did
I
get
an
attestation
from
from
pipeline?
A
A
So
this
is
the
basic
system,
but
of
course
we
need
more
than
one
system
to
agree
right.
As
we've
said
before,
a
unitary
system
is
bad.
We
had
a
unitary
system
with
sunburst
and
we
had
no
mechanism
of
validating
and
you
know
bad
things
happen.
So
we
really
want
consensus
to
get
our
consensus
to
tested
system.
We
need
more
than
one
system
in
place
and
those
need
to
agree.
A
So
this
is
our
final
diagram
and
you
can
see
here
that
on
the
bottom
we
have
now
a
validation
cluster.
So
we
have
two
completely
physically
separate
clusters
and
they
have
we
have
the
standard
cluster
which,
like
our
devs,
will
have
access
to
and
which
github
sends
its
web
hooks
to,
and
then
we
have
a
validation
cluster
which
looks
basically
exactly
the
same
other
than
it
gets
its
marching
orders
by
pulling
cloud
events
off
an
event
bus.
So
what
happens
is
a
web
hook?
A
Web
request
comes
in
to
the
github
app
the
github
app
does
all
of
its
validation
stuff.
It
will
send
one
copy
off
to
its
own
tecton
pipelines
controller,
and
then
it
will
take
its
thing,
wrap
it
up
into
a
cloud
event
and
emit
it
on
the
bus,
and
then
the
github
app
is
just
configured
with
a
in
a
different
way
on
the
other
in
the
validation
cluster.
A
So
that
gives
us
an
asynchronous
sort
of
parallel
system
to
do
or
to
do
these
builds
with,
and
you
can
see
here
that
each
of
them
has
a
copy
of
tecton
chains.
Each
of
them
has
their
own
key
and
they're,
both
writing
things
into
the
same
document
database
and
then
from
there.
We
actually
have
a
little
app
that
etl's
that
stuff
into
a
postgres
database
that
we
use
to
have
a
variety
of
other
clients.
Kind
of
ask
questions
of.
A
So
you
can
sort
of
imagine
that,
like
all
this
attestation
stuff
is
great
but
like
in
a
document,
it's
not
as
easy
to
kind
of
blow
it
out
into
like
a
relational
structure
and
understand.
You
know
like
what
builds
contain,
which
thing
or
how
many
builds
came
from
a
given
git
sha
or
you
know
eventually
we'll
go
a
little
deeper
and
be
able
to
have
some
marry
up
some
threat
intelligence
data
with
some
build
artifact
data
and
ask
sort
of
more
detailed
questions
like
that.
A
But
this
little
green
box
here
on
the
right
of
release
gates
is
basically
just
things
like
a
lambda
job,
for
instance,
that
might
ask
hey.
Can
I
pick
something
up
from
the
trebuchet
s3
bucket
and
put
it
in
the
on-prem
release
platform
release
bucket
no
s3
bucket,
and
if
you
know
it
would
say,
hey
do
I
have
two
systems
agreeing
on
this
thing?
Do
I
have
the
vulnerability
posture
that
I
want?
A
If
so,
yay
I'll
do
that
motion
another
one
would
be
like
a
kubernetes
validating
admission
web
hook
to
say
hey
this
container
image
that
you
want
me
to
launch.
Has
it
been
signed?
Has
it
been
signed
by
an
authority?
I
trust.
Does
it
have
an
attestation
from
each
of
two
pipelines?
If
so
yay
I
will
out
to
be
launched
in
the
cluster?
If
not,
then
I
won't.
A
And
this
is
we
have
this
need
because,
again,
like
we're
not
just
doing
containers,
I
should
give
you
a
little
bit
of
background
so
so
far
we
have
been.
We
had
kind
of
a
hard
deadline
to
ship,
a
bunch
of
things
out
of
the
new
environment
this
summer,
and
they
were
all
on-prem
projects
based
on
java
and
they
all
have
a
bunch
of
different
types
of
install
targets.
A
A
Some
miscellaneous
details
so
yeah
we
use
opa
to
do
some
vulnerability
analysis
and
this
is
kind
of
in
the
early
phases,
we're
thinking
about
just
sort
of
defining
policies,
per
project,
around
wild
vulnerabilities,
exceptions,
etc.
And
then
we've
actually
been
experimenting
with
producing
our
own
out-of-stations
in
a
slightly
different
format.
For
things
that
we
have
determined
to
have
a
decent
posture,
vis-a-vis
vulnerability
presence,
we're
pretty
heavy
users
of
a
tool
called
jfrog
artifactory,
which
is
where
we
keep
all
of
our
things
for
all
of
our
dependencies
for
node
and
for
java,
et
cetera.
A
So
we're
continually
scanning
that
with
one
of
their
scanning
tools
called
x-ray,
and
then
you
know
we'll
take
an
s-bom
file.
Look
at
everything
inside
that
s-bomb
file
and
produce
a
vulnerability
report
from
x-ray,
based
on
the
things
inside
there
and
then
compare
that
with
policy
for
that
project
in
opa,
and
you
kind
of
get
a
thumbs
up
or
a
thumbs
down
right.
And
so,
if
you.
A
We
keep
everything
locked
to
github
through
the
pretty
nifty
fact
that
github
continually
publishes
their
over
their
meta
api.
The
cider
ranges
that
all
of
their
their
web
books
are
going
to
come
from,
so
the
alb,
the
amazon
application
load
balancer
that
we
use
has
a
great
feature
where
it
allows
you
to
sort
of
dynamically
keep
updated
like
the
allowed
ip
ranges
that
you're
going
to
allow
to
talk,
so
we're
able
to
keep
it
constantly
locked
to
just
github.
A
Earlier
no
egress
out
of
the
vpc
whatsoever
other
than
to
github.
So
if
we
were
compromised,
that
vector
of
compromise
would
have
to
involve
in
some
way
talking
back
to
github,
maybe
pulling
some
bad
thing
from
github
or
whatever,
but
as
of
about
a
week
or
two
ago,
we've
actually
started
red
teaming
trebuchet,
which
is
pretty
fun.
So
I've
got
somebody
sitting
there
trying
to
see
if
he
can
escape
from
containers
and
things
like
that
and
we'll
see
we'll
see
what
he
comes
up
with
and
now
some
concluding
thoughts.
A
Like
you,
you
will,
I
should
have
said
you
will
not
you'll,
probably
like
you
will
experience
a
breach.
It
happens
constantly.
It
happens
more
and
more.
It's
really
hard
to
secure
software.
So
just
be
humble
about
this.
I
know
that
that's
kind
of
facile
advice,
but
it's
worth
mentioning
it's
it's
going
to
be
difficult
and
the
people
that
you're
going
to
work
with
are
going
to
be
from
all
over
your
company.
A
That's
something
that
we
on
the
trebuchet
team
have
tried
to
do
quite
a
bit
over
the
last
few
months,
we've
staffed
up,
as
you
can
imagine,
in
the
wake
of
this
breach
quite
a
bit
within
our
cso's
office,
and
all
of
those
people
are
now
pretty
pretty
well
read
into
everything
we're
doing.
They
understand
it
and
they're
involved
with
it,
and
that's
only
going
to
bear
dividends
for
us
over
time.
A
Move
security
left
as
much
as
you
can
put
things
into
the
hands
of
devs.
Devs
know
the
tools
they
know
the
product.
They
know
what
they're
building
give
them
the
means
and
the
training
to
understand
how
to
try
to
prevent
themselves
from
inserting
security
vulnerabilities
in
the
first
place,
train
people
on
secure
systems,
design,
you're
going
to
get
back
again
massive
dividends
compared
to
the
investment
that
you
do
and,
as
the
ancient
saying
goes,
be
excellent
to
each
other.
A
A
So
I
know
there's
people
that
I
missed,
but
yeah
everybody
everywhere,
whether
you're
at
solarwinds
or
whether
you
were
affected
by
this
or
whether
maybe
somebody
in
your
company
just
ran
in
with
a
waving
around
a
zd
net
article
saying
hey:
we
got
to
do
something
about
this
and
it
ate
your
world
yeah.
Thank
you,
too,
and
also
I
need
to
observe
that
today
is
indigenous
people's
day.
So
yes
go
out.
There
teach
peace
expect
justice.
B
A
Variants
across
the
pipelines
no
there,
so
the
only
variant
is
that
in
the
validation.
If
I
understand
the
question
correctly
in
the
validation
cluster
rather
than
getting
direct
communication
from
github,
it
gets
a
synchronous
communication
from
a
message
bus.
So
there's
nothing
going
into
that.
Second
cluster,
that
hasn't
at
some
level
already
been
validated
before
by
a
slightly
lower
assurance
system.
If
that
makes
sense,
so
we
double
produce
everything
right
like
we
did
originally
try
to
have
like
a
reduction
in
the
things
that
we
did
in
the
pipelines
in
the
validation
pipeline.
A
But
it
was
just
too
logistically
complex
to
try
to
think
about
how
to
mutate
out
certain
tasks
that
didn't
make
sense
within
the
validation
context.
So
eventually,
we
just
decided
we'll
do
the
exact
same
thing
in
both
places,
and
the
only
thing
that
we
do
is
we'll,
like
update,
we'll
insert
like
a
a
change
to
like
kaneko
to
like
not
push
a
container
but
it'll
still
it'll
still
build
an
image
and
calculate
checksums
and
produce
stations
around
that.
A
A
Yeah,
so
the
question
is
about
trade-offs,
conversations
around
trade-offs
in
time
versus
you
know,
trade-offs
and
security.
If
I
understand
it
right
so
that
really
has
never
been
a
thing
as
part
of
this,
because
the
nature
and
the
high
profile
level
of
this-
I
don't
know
if
anybody
caught
this,
but
you
know
our
ceo
and
our
former
ceo
both
had
to
testify
in
front
of
congress
twice,
and
they
literally
promised
this
system
to
the
united
states
congress
on
live
television.
A
So
there
is
no
one.
Who
really
cares?
I
mean
I
care
deeply
about
the
developer
experience,
obviously,
and-
and
we
do
our
best
to
make
it
as
fast
as
we
can,
but
frankly,
our
experience
so
far
has
been
that,
like
modulo,
some
temporary
problem
with
like
not
enough
kubernetes
resources,
these
are
as
fast
or
faster
on
tecton,
as
they
were
doing
similar
motions
on
team
city
or
jenkins.
A
So
we
we
absolutely.
We
instrument
everything.
You
know
we
do
all
the
things
that
you
do
to
try
to
make
sure
that
any
cloud
service
runs
as
efficiently
as
it
can,
and
we
don't
want
anybody
to
have
a
bad
time
with
this.
But
security
is
like
priority
one
two
and
three,
and
that's
just
the
way
it
has
to
be
right
now.
A
Is
what
are
we
using
for
kms,
and
so
the
the
real
answer
to
what
we're
using
from
kms
is
we're
using
sig
store.
So
my
colleague
here,
cody
soyland,
actually
added
some
support
for
aws
kms
back
end
it.
The
a
lot
of
the
stuff
in
from
sigstor
is
fairly
gcp
oriented,
so
there
was
already
support
for
kms
there
and
for
gcp
kms.
So
we
on
our
team
added
support
for
aws
kms
and
then
it's
you
know.
A
If
you
have
a
an
iam
role
that
allows
it
to
request
a
signature,
then
it
just
is
allowed
to
request
a
signature
of
some
data
and
that's
that's
basically,
you
know
what's
happening
there,
so
you
could
conceivably
run
this
on
any
anything.
You
know
as
long
as
you
wanted
to
do
the
work
to
make
sure.
Like
I
don't
know,
maybe
I
don't
think
six
store
supports
azure
or
kms,
but
or
maybe
it
does
now,
okay,
yeah,
so
you
could.
You
could
conceivably
build
this
pretty
easily
on
any
of
the
on
the
big.