►
Description
Air Marshal: No More Secrets In Your Manifests - Michael Brodhead, Stark & Wayne
In real-world use our BOSH and Concourse manifests are littered with secrets: passwords, AWS credentials, SSH keys, you name it. Dealing with these secrets is a pain in the neck. If we check secrets into source control then there are myriad ways to accidentally leak them. Without source control distribution becomes a hassle and we still have some risk of leaking. Ideally we would like to keep production secrets off staff workstations altogether.
Enter Air Marshal. Air Marshal acts as a proxy between staff workstations and BOSH or Concourse. Manifests checked into source control have placeholders wherever secrets are necessary. Air Marshal reads secrets from a secure back-end, adding them to manifests where needed.
Michael Brodhead
Stark & Wayne, LLC
Michael Brodhead, mkb to his pals, works for Stark & Wayne, a consultancy based in Palo Alto, CA. Mkb has worked for financial giants, tiny startups, and everything in between. At Stark & Wayne he helps clients secure and automate large-scale cloud deployments.
A
All
right
greetings
programs.
My
name
is
Michael
Brodhead
I
work
for
Stark
and
Wayne
and
I'm
here
to
get
some
input
from
you,
so
I'm
gonna
keep
my
prepared
remarks.
As
short
as
I
can
and
then
I'm
gonna
ask
you
a
lot
of
questions
at
Stark
and
Wayne.
We
often
find
ourselves
helping
our
clients
with
their
Cloud
Foundry
deployments
and
with
the
supporting
services
around
those
Cloud
Foundry
deployments.
A
So
in
practice
this
means
we
wind
up
working
a
lot
with
Bosch
and
a
lot
with
concours,
a
Bosch
and
concours,
of
course,
both
configured
with
big
Yamma
files
and
once
you
get
beyond
the
very
basics,
if
you
want
to
do
anything
interesting,
inevitably
you
need
secrets
in
those
files.
So
you
wind
up
with
database,
passwords,
SSH
keys,
AWS,
creds,
etc.
A
Many
organizations
succumb
to
temptation
and
just
check
those
configs
right
into
source
control
secrets
in
all
and
I
will
confess
here
publicly
for
the
first
time,
I
understand
that
temptation
I
felt
the
pull
many
times
myself,
but
it's
not
a
great
way
to
do
things.
For
starters.
These
days,
when
we
say
source
control,
most
of
the
time
we
mean
github.
So
now
you've
got
the
keys
to
the
kingdom,
hosted
on
a
public
service
online.
A
A
So
the
Concours
we
can
pull
in
from
that
extra
file
right
with
the
fly
command
as
where
as
we're
setting
up
our
pipeline
and
with
Bosch,
hopefully
you're
already
using
a
tool
like
spruce
or
spiff
to
assemble
your
manifest
from
multiple
files.
Now
this
is
great
and
it
gets
rid
of
a
good
chunk
of
our
risk,
but
it
still
leaves
some
issues
behind,
because
those
secrets
are
still
on
the
local
disk
of
of
all
your
engineers.
So
if
an
engineer
loses
a
laptop
or
a
laptop
gets
stolen,
now
you've
got
to
treat
those
creds.
A
It's
compromise.
You
can
reduce
the
risk
of
an
actual
compromise
by
enforcing
a
good
password
policy
by
making
sure
that
people
have
disk
encryption
turned
on,
but
as
a
practical
matter,
once
you
lose
physical
control
of
that
asset
and
it's
in
unknown
hands.
You
have
to
treat
all
the
data
on.
It
is
compromised.
A
Also,
engineers
will
sometimes
we
will
accidentally
check
files
into
source
control
that
don't
belong
there
and
we've
all
done
it.
Everybody
in
this
room
probably
has
done
it
and
some
sometimes
that
that's
those
other
credential
files,
and
so
if
you
accidentally
check
your
credentials
in
a
source
control.
Well,
now
we're
back
to
square
one.
Now,
if
you
accidentally
check
those
credentials
into
source
control
and
it's
a
public
repo,
then
things
get
very
unpleasant.
Indeed,
I
actually
found
that
totally
by
mistake,
but
it
I
it
had
to
go
in
right.
A
A
Ideally,
the
way
we
want
to
deal
with
mistakes
is
not
buy
it
by
gritting,
our
teeth
and
and
just
hoping
really
fervently
that
mistakes
don't
happen
but
to
seek
out
tools
and
processes
that
keep
human
error
in
mind,
so
they
either
make
it
harder
to
make
mistakes
or
easier
to
catch
mistakes,
or
they
limit
the
damage
that
mistakes
can
do
now.
In
addition
to
these
existing
problems
we
haven't
solved
yet
we
took
on
a
new
problem.
A
Distribution
previously
source
control
would
make
sure
that
all
your
engineers
had
an
update,
up-to-date
copy
of
all
the
creds
you've
taken
that
away
from
them.
So
now
the
engineers
have
to
figure
out
how
to
distribute
that
data
out-of-band,
and
we
just
hope
that
the
approaches
they
use
a
reason
in
practice
and-
and
many
of
us
have
lived
this
the
moment
you
discover
your
threads
are
out
of
date-
is
when
you're
in
the
middle
of
doing
something
else,
and
you
suddenly
realize
I've
just
broken
the
pipe
line.
A
I've
just
broken
the
deployment,
so
you
go
on
slack.
You
hope,
there's
somebody
around
who
has
an
up-to-date
copy,
the
file
they
have
to
go
to
get
up.
Then
the
two
of
you
work
out
how
they're
gonna
get
it
to
you
and
eventually
you're
back
in
the
saddle,
but
the
whole
time
that's
going
on
you're,
not
advancing
on
whatever
story.
You
were
working
on
now,
having
heard
all
these
limitations
of
the
approach
of
pulling
the
secrets
out
of
source
control,
but
still
having
them
on
the
local
disk.
A
You
might
think
that
it's
a
bad
idea
and
it's
not
actually
all
that
bad.
It
makes
sense
in
a
lot
of
situations
and
the
reason
comes
down
to
risk
management
insecurity.
We
don't
get
to
eliminate
all
of
our
risks.
We
would
love
to
do
that,
but
the
real
world
is
always
going
to
be
imperfect.
So
what
we
have
to
do
is
look
for
acceptable
trade-offs
and
we
shave
risk
where
we
can.
A
In
this
case
the
the
cost
of
implementation
is
really
low.
It's
really
easy
to
do.
The
the
added
downside
is
manageable
and
it
gets
rid
of
a
good
chunk
of
your
risk.
So
if
you
have
secrets
checked
into
source
control
for
your
Bosch
deployments
or
your
concours
pipelines
today,
I
urge
you
to
take
a
look
at
that
approach.
It's
probably
going
to
be
a
net
win.
A
A
A
The
reason
I
bring
up
Genesis
today
is
because
Genesis
talks
to
spruce
and
spruce
talks
to
vault
vault
is
an
open
source
data
store.
That
is
specifically
designed
for
storing
secrets,
and
my
experience
with
vault
has
been,
and
that
of
everybody.
I've
talked
to
who's
worked
with
vault
is
that
vault
works
as
advertised?
It's
easy
to
setup,
it's
easy
to
use,
and
it's
pretty
well
thought
out.
So
with
spruce.
Talking
to
vault,
when
you
write
your
manifests
instead
of
putting
in
a
secret,
you
can
put
it
in
placeholder
that
identifies
that
secret.
A
Now,
when
you
invoke
Genesis
and
Genesis
under
the
hood
is
invoking
spruce,
spruce
I
looks
for
all
those
identifiers.
Z'
looks
all
your
secrets
up
involved,
fills
them
in
and
then
the
master
manifest
that
it
produces
has
all
your
secrets
in
it
now,
if
you're
not
making
manual
changes
to
that
manifest
after
generating
it
that,
if
you
want
once
your
deployment
is
done,
you
can
delete
it.
It's
a
derived
artifact
and
it's
easy
to
recreate.
So
why
not?
A
So
now,
with
with
Genesis,
we
have
addressed
the
distribution
problem
for
secrets
and
we've
at
least
made
a
dent
in
the
problem
of
secrets
on
the
local
disk,
and
that
brings
us
to
air
marshal
and
means
it's
time
for
me
to
take
a
drink.
My
throat
is
getting
very
scratchy,
so
the
original
idea
for
air
marshal
came
from
our
fearless
leader,
dr.
A
Nick
Williams,
who
found
it
Stark
and
Wayne,
and
a
lot
of
the
work
on
on
Air
Marshal
has
been
done
by
my
colleague,
Norma
Brahm
of
its
I
want
to
make
clear
what
air
marshal
is.
Not
air
marshal
seeks
to
address
the
problems
of
Secrets
on
the
local
disk
secrets
and
source
control
and
the
distribution
of
Secrets.
It
does
not
attempt
to
solve
the
problem
of
a
a
malicious
engineer,
abusing
secrets,
and
the
reason
for
that
is
pretty
simple.
A
If
you
look
at
statistics
for
data
breaches
for
recent
years,
more
than
80
percent
of
successful
breachers
breaches
are
due
to
an
external
attack
and
so
well.
The
problem
of
the
insider
threat
is
worth
worrying
about.
We
have
to
take
our
best
crack
at
the
big
risks
before
we
start
to
take
on
the
medium-sized
ones.
A
So
air
marshal,
in
essence,
is
a
proxy
that
sits
between
your
command
line
and
the
backend
service
you're
dealing
with
so
either
the
the
Bosch
director
or
the
concours
ATC.
So
when
you
configure
your
CLI,
you
lie
to
it
and
you
tell
it
that
air
marshal
is
its
back-end.
You
create,
manifests
much
like
if
you
using
Genesis
in
spruce
with
placeholders
instead
of
the
secrets.
Now,
when
you
go
to
to
kick
off
your
deployment
or
configure
a
pipeline,
your
CLI
takes
that
configuration
sends
it
to
air
marshal
air
marshal
identifies
all
the
placeholders
looks
up.
A
A
So
therein
lies
the
the
the
basic
idea
of
air
marshal
and,
as
I
said,
we've
built
a
couple
prototypes
internally
and
we're
really
excited
about
it,
and
we
want
to
hear
from
you
about
how
to
take
that
further
and
I
thought
the
stage
was
going
to
be
righter,
so
it
brighter
and
I
had
these
these
questions
written
down.
The
first
starters
can
I
get
a
show
of
hands
how
many
of
you
are
at
an
organization
that
is
using
Bosch
today,
all
right,
so
large
proportion,
as
we
expect
it
all
right
now,
keep
your
hand
up.
A
If
you
personally
are
one
of
the
people
working
with
Bosch
at
your
org,
okay,
how
many
people
are
at
an
org
that
is
using
Concours
today,
yes,
gentlemen,
and
keep
your
hands
up
if
you
personally
are
working
with
concours
are
some
pretty
decent
representation?
I
was
I
was
really
wondering
what
the
Concours
uptake
would
be.
So
that's
pretty
so
those
of
you
using
Bosch
wondering
how
many,
how
many
deployments
you've
got
put
put
your
hand
up.
If
you
have
more
than
five
apps
being
deployed
with
Bosch
more
than
10,
more
than
20,
more
than
30.
A
A
Alright,
so
a
big
percentage
of
the
people
using
hog
horse
a
more
than
ten
more
than
twenty,
more
than
thirty
all
right,
and
that's
far
enough
there,
okay,
any
of
you
with
with
your
your
deployments
or
your
pipelines.
Do
you
have
actual
deployments
of
pipelines
that
you're
using
for
real
production
purposes
that
don't
have
secrets
somewhere
in
the
config?
A
That's
that's,
usually
how
it
goes
are
any
of
you
at
organisations
that
are
subject
to
compliance
requirements
for
something
like
sock,
2
or
HIPAA
or
PCI
any
of
the
above
all
right,
so
yeah
decent
number
of
people.
So
what
about
other
compliance
regimes
other
than
the
ones
I
mentioned
anybody
FedRAMP
good
times.
A
A
A
A
A
It
it
can,
and-
and
one
of
our
prototypes
is
doing
that-
we're
not
yet
fully
confident
enough,
because
we,
basically
we
have
to
be
parsing
the
output.
That's
coming
back
so
and
that's
why
we
say
at
least
for
now
the
the
intention
is
to
address
the
you
know:
mistakes
and
the
outsider
threat
and
and
not
worry
about
the
problem
of
a
malicious
insider,
but
there's
definitely
something
that
that
we're
looking
at
and
keen
to
do
and
I
would
love
to
be
able
to
get
up
here
and
say:
oh
yeah,
I
can
do
that
too.
A
No,
not
anything
other
than
than
big
pie-in-the-sky,
and
certainly
I
mean
conceptually
could
do
that
with
anything
I
mean
the
nice
thing
about.
Those
two
is
you've
got
a
local
CLI.
That's
making
HTTP
calls
out,
so
you
have
a
like
a
well-defined
interface
and
they
both
work
in
very
similar
ways,
but
but
yeah
you're
right.
You
know
that
concept
could
be
extended.
A
A
The
the
actual
requirements
of
something
like
like
PCI
or
saw
two
are
fairly
broad
because
they
know
that
they
have
to
apply
to
a
lot
of
different
places
and
so
the
how
you
choose
to
define
that
and
apply
that
to
your
environment,
winds
up
being
a
negotiation
between
your
staff
and
the
end
the
auditors,
so
I
mean
it
really
depends
on
the
relationship
you
have
with
them.
But
that's
a
good
idea.
I
should
run
that
by
some
folks,
hey.
A
Well,
so
there
are
two
different
pieces:
if,
if
we're
using
the
the
genesis
spruce
of
approach,
then
you
do
wind
up
with
a
local
file
that
has
those
creds.
If
you're
using
the
air
marshal
approach,
then
those
local
creds
don't
go
on
your
local
disk.
So
what
you
create
just
has
placeholders
you
send
it
to
air
marshal
with
placeholders
and
then
placeholder,
then
placeholder,
then
air
marshal
does
the
lookup
in
the
fill
in.
A
A
That
that
certainly
is
a
good
idea
and
it
means
you
have
fewer
moving
parts.
The
advantage
is,
we
have
there's
a
lot
of
things
in
common
between
the
problem
for
Concours
and
the
problem
for
bosch,
and
so
we
only
have
to
touch
one
code
base
rather
than
multiples
and
yeah.
It's
it's
at
least
easier
at
this
at
this
juncture
to
deal
with
the
thing
in
isolation,
but
yeah
that
might
make
some
sense
is
to
have
that
be
right
in
Bosch.
Yes,.
A
We
haven't
looked
at
how
you
do
it
with
you,
a
a
authenticated
box.
We
prob
I'm
sure
we
could
make
that
happen,
but
I
haven't
I,
haven't
thought
it
through.
That
I
mean.
Let
me
write
that
down,
because
I
may
forget
that
so
yeah,
but
that's
a
good
idea
and
that
and
that's
an
area
that
we've
been
kind
of
glossing
over
so
far
with
the
prototypes,
is
a
different
compliance
approaches
at
this
point.
We're
just
I'm,
sorry
that
the
different
authentication
approaches
at
this
at
this
point
we're
essentially
passing
authentication
through.
A
A
A
No,
but
there's
no
reason
we
couldn't
point
it
at
kms
being
the
the
once.
You
pointed
it
at
one
back-end
and
you
have
an
interface
and
interface.
You
could
plug
it
at
different
things
and
we've
done
custom
work
for
one
of
our
big
clients
that
wasn't
part
of
this,
but
was
related.
Where
we're
talking
to
another
big.
You
know:
enterprise
e
key
store
so
but
yeah
there's
no
reason
why
air
marshal
couldn't
be
configurable
to
be
to
multiples,
yeah,
so
Kate,
kms,
Amazon
kms
might
be
a
good
choice,
yeah
or
or
for
that
matter.