►
Description
Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Save Your Services from Sneaky Snoops With SPIFFE? - Daniel Feldman, Scytale
Lurking inside almost every cloud native project is a hidden threat: hardcoded credentials for services like external APIs and databases. While these credentials can be set to long random strings and encrypted, they still can be stolen by an intruder or accidentally misused by insiders. In this talk, we’ll demonstrate how to use CNCF’s SPIFFE and SPIRE Projects to securely authenticate to workloads such as PostgreSQL, MongoDB, and AWS from inside your services, all without any hardcoded credentials -- eliminating an entire class of security vulnerabilities while decreasing your work as a DevSecOps team.
https://sched.co/aDcB
A
Hello
welcome
to
the
cubecon
cloud
nativecon
europe
2020
virtual
event,
I'm
daniel
feldman
and
the
title
of
this
session
is
save
your
services
from
sneaky
snoops
with
spiffy.
First
of
all,
I'd
like
to
start
by
introducing
myself
again.
My
name
is
daniel
feldman.
I
live
in
minneapolis
in
the
u.s,
and
I've
been
working
on
cloud
security
engineering
for
most
of
my
career.
I
started
out
at
a
company
called
veritas
which
makes
the
number
one
used
enterprise,
backup
software,
which
has
a
lot
of
really
interesting
security
requirements.
A
So
I'm
really
excited
to
be
continuing
to
work
on
spiffy
inspire
at
hpe,
but
I'm
not
just
here
to
represent
myself
there's
a
huge
team
with
security
experience
at
a
variety
of
top
companies.
That's
working
on
spiffy
inspire.
In
fact,
one
of
our
founding
engineers,
evan
gilman,
was
the
author
of
the
book.
Zero
trust
networks
and
he'd
probably
be
here
today,
except
that
he's
out
on
paternity
leave.
So
congratulations,
evan
here
are
the
subjects
we'll
be
discussing
today.
First,
what
is
service
identity
and
why
is
it
an
interesting
problem
to
solve?
A
A
Why
is
service
identity?
An
interesting
problem?
Well,
you've
probably
seen
charts
like
this
before
each
one
of
these
dots
is
a
service
at
netflix.
All
of
these
services
are
chunks
of
code
that
are
then
deployed
in
multiple
clouds,
multiple
regions,
maybe
on
cdns
and
they're,
all
communicating
with
each
other
and
they're.
All
constantly
changing
developers
are
constantly
updating
the
code
and
changing
how
these
services
interact
with
each
other.
A
In
order
to
do
any
kind
of
security,
each
entity
that
you're
trying
to
secure
has
to
have
some
kind
of
identity,
for
example,
to
encrypt
something
you
have
to
know
the
identity
of
the
party
that
will
be
decrypting
it
in
order
to
ensure
data
integrity.
If
you're
adding
a
digital
signature,
you
have
to
know
who
will
be
verifying
the
digital
signature
and
how
they'll
access
your
public
key
in
order
to
do
authentication.
A
Obviously,
the
thing
that
you're
authenticating
has
to
have
some
kind
of
identity,
and
in
order
to
do
authorization
you
need
to
first
do
authentication
and
authentication
depends
on
identity
for
humans.
Identity
identity
is
mostly
straightforward.
Each
human
is
a
unique
individual
humans,
don't
clone
each
other,
they
don't
constantly
change,
they
have
memories.
You
can
give
a
human,
a
username.
You
can
give
a
human
a
password.
You
can
give
a
human,
a
two-factor
authentication
device
and
expect
them
to
to
retain
that
information
for
services.
The
situation
is
much
more
complicated.
A
Services
are
constantly
scaling
up
scaling
down
moving
around
changing
completely.
They
have
no
long-term
memory
when
the
service
starts
up.
How
is
it
supposed
to
know
what
its
username
and
password
is
so
solving
the
service
identity
problem
is
very
important
in
order
to
be
able
to
do
any
kind
of
security
for
connections
between
back
in
the
pre-cloud
days,
we
used
to
use
perimeter
security
in
the
perimeter
security
model.
Everything
inside
a
perimeter
is
trusted
and
everything
outside
is
untrusted,
so
you
have
a
firewall.
A
A
As
you
add,
multiple
data
centers,
multiple
clouds,
multiple
regions
within
clouds.
This
becomes
increasingly
complex
to
maintain
you
need
really
complicated
firewall
rules
to
allow
specific
connections
from
service
a
to
service
b
to
service
c,
and
then,
as
you
add
in
developers,
throwing
in
new
code
constantly
with
continuous
integration
and
continuous
delivery,
this
gets
impossible
to
maintain
because
the
the
connections
between
services
are
constantly
changing
and
as
these
systems
get
more
and
more
complicated,
there
are
more
and
more
holes
in
the
perimeter
that
a
hacker
can
exploit.
A
Can
we
just
use
shared
secrets?
Well,
this
is
what
you're,
probably
using
today,
if
you
haven't
heard
of
spiffy,
yet
in
the
shared
secrets
model,
every
service
has
credentials
or
certificates
that
are
used
to
access
other
services.
Hopefully
these
are
all
stored
in
a
secret
store.
Maybe
amazon
secret
store,
maybe
hashicorp
vault,
maybe
azure
secret
store.
But
maintaining
these
secrets
is
hard
work
you
have
to.
You
have
to
rotate
them
frequently.
You
have
to
make
sure
they're
not
being
misused.
A
You
have
to
make
sure
that
they
are
being
stolen
by
insiders
by
by
hackers
from
the
outside
and
and
then
you're
really
reducing
the
problem
to
a
different
question,
which
is
how
do
you
authenticate
your
service
to
the
secret
store
in
the
first
place,
and
there's
no
good
way
to
do
that?
Usually
you
just
have
some
kind
of
a
token.
That's
in
your
config
map
that
authenticates
your
service
to
the
secret
source,
so
it
can
get
other
secrets.
A
This
is
really
a
hack
mapping,
username
password
authentication
for
humans
onto
services,
but
it
doesn't
work
very
well
before
we
move
on
it's
worth
asking
why
we
can't
just
use
web
public
key
infrastructure
or
web
pki
to
secure
our
internal
services.
Web
pki
works
great
for
securing
access
to
our
bank
accounts,
so
it
should
work
for
internal
services
right.
Unfortunately,
web
pki
is
just
not
a
great
fit
for
internal
services
in
webdki,
your
browser
requests
a
hostname
from
dns
and
dns
gives
it
an
ip
address.
A
A
Unfortunately,
internal
services
typically
do
not
have
an
individual
globally
recognized
dns
name.
You
might
have
a
service
with
thousands
of
copies,
they
might
all
have
different
host
names
or
you
might
have
one
service
that
only
has
an
ip
address.
It
doesn't
have
a
host
name
at
all.
Even
if
we
could
solve
that
problem,
there
would
be
no
way
for
an
instance
of
a
service.
A
So,
if
you
think
about
the
solutions
we
just
discussed
perimeter
security,
shared
secrets
and
web
pki
you'll
realize
that
what
we
really
need
is
a
way
to
securely
distribute
identities
to
services
without,
depending
on
any
manual
effort,
without
depending
on
any
external
services
like
dns
or
a
secret
store,
and
that
method
is
spiffy.
The
secure
production
identity
framework
for
everyone.
That's
what
we're
here
to
talk
about
today
and
that's
what
I
work
on
spiffy
is
a
platform
agnostic
standard
for
implementing
service
identity.
A
It's
inspired
by
internal
software
that
was
developed
at
google,
netflix
and
facebook,
and
people
from
those
companies
got
together
way
back
in
2017
to
start
drafting
the
spiffy
standard.
Spiffy
gives
your
service
a
passport
for
communicating
with
other
services.
It
doesn't
actually
do
the
communication.
It's
not
a
service
mesh,
it's
not
an
overlay
network.
It
just
gives
your
service
the
identity
information.
It
needs
to
establish
connections
within
those
other
systems,
but
it
is
partially
implemented
in
several
different
service
meshes.
A
Then
the
full
implementation
is
called
spire
and
that's
the
the
reference
implementation.
That's
the
code.
I
actually
work
on,
but
other
other
software
is
free
to
implement.
Spiffy.
The
safety
standard
consists
of
four
different
pieces.
The
first
piece
is
spiffy
ids
spiffy
ids
are
a
standard
format
for
identifying
services.
A
It
starts
with
spiffy
colon
slash,
it
looks
like
a
url
and
then
there's
a
trust
domain
field.
That's
a
unique
identifier
for
your
organization.
Typically,
it's
just
your
organization's
top
level
domain,
then
there's
a
slash,
then
there's
an
identifier
for
the
service.
It's
very
simple.
The
next
piece
of
spiffy
is
spiffy
verifiable
identity
documents.
A
These
are
cryptographically
verifiable
documents,
asserting
that
a
service
has
a
specific
spiffy
id.
We
actually
support
two
different
formats
for
svids.
One
is
x,
509
certificates
with
certain
fields
filled
in
to
include
the
spiffy
id
and
the
other
is
javascript
web
tokens
again
with
certain
fields
filled
into
to
a
service
specific
spiffy
id,
and
you
may
want
the
x
509
certificate
or
the
javascript
web
token
in
different
situations.
A
A
A
A
Here's
another
diagram
of
the
spiffy
standard.
You
have
a
spiffy
provider
which
is
providing
those
four
elements
of
the
spiffy
standard
to
each
of
your
services.
Each
service
gets
a
spiffy
id
an
svid,
a
trust,
bundle
and
access
to
that
workload.
Api
which
lets
it
obtain
the
spiffy
id
svid
trust
bundle
and
also
notifications
whenever
any
of
those
elements
change.
A
So
there
are
a
couple
of
different
implementations
of
the
spiffy
standard
that
are
available
right
now.
Some
of
the
service
meshes,
including
kuma,
istio
and
network
service,
mesh,
implement
spiffy
to
varying
degrees,
and
then
the
official
reference
implementation
is
spire,
which
is
the
implementation
of
spiffy.
That
includes
the
entire
api
and
it
is
production,
quality
software.
A
It's
in
use
at
a
number
of
big
companies
right
now,
it's
at
the
incubation
level,
in
the
cncf
structure
so
similar
to
envoy
or
at
cd
we're
trying
to
get
it
up
to
to
the
graduated
level
in
cncf,
but
that
may
take
a
little
while
longer
and
it's
available
on
github.
You
can
download
it
right
now
and
see
the
open
issues
and
contribute
back
up
to
it.
I
and
many
others
work
on
spire.
A
Every
day
spire
consists
of
two
parts:
the
spire
server
is
actually
responsible
for
signing
the
svids
because
it
needs
to
be
highly
secure.
We
recommend
running
spire
server
on
an
isolated
virtual
machine
or
even
on
dedicated
hardware.
Within
your
data
center,
then
the
spire
agent
provides
the
workload
api
and
that
runs
on
all
the
alongside
your
services
on
individual
nodes.
A
The
first
step
in
getting
spire
running
is
called
no
data
station.
In
no
data
station,
the
node
proves
its
identity
to
the
spire
server
on
cloud
platforms
or
on
kubernetes.
The
node
provides
an
instance
identity
document
to
the
spire
server
and
then
the
spire
server
validates
that
instance
identity
document
and
grants
the
agent
its
own
svid.
A
It's
also
possible
to
perform
no
datastation
through
pre-distributed
certificates
or
using
join
token
strings,
because
the
node
attestation
process
is
automated.
As
you
scale
up
the
number
of
nodes,
each
node
can
be
attested
automatically,
so
you
don't
have
to
do
any
manual
effort
as
you
deploy
more
nodes.
A
Now
that
we
have
secure
communication
from
the
spire
agent
to
the
spire
server,
the
aspire
agent
can
provide
the
workload
api
over
a
local
connection
as
new
workloads
connect.
The
spire
agent
performs
another
process
called
workload
attestation
for
each
workload.
The
spire
agent
checks
that
the
workload's
uid
image,
checksum
or
other
characteristics
match
the
expected
values
to
give
that
workload
in
svid.
Remember,
the
spire
server
actually
creates
and
signs
the
svid,
but
the
spire
agent
is
local
to
the
workload
and
that's
the
only
place
where
those
detailed
characteristics
of
the
workload
can
be
checked.
A
Once
no
data
station
and
workload
attestation
are
done.
We
have
a
secure
chain
of
trust,
going
from
the
workload
all
the
way
back
to
the
spire
server,
and
it
looks
a
little
bit
like
this.
You
can
have
all
these
different
workloads,
they're,
all
communicating
with
aspire
server,
getting
up-to-date
essays
and
those
workloads
can
establish
secure
communication
between
each
other
using
their
secure
identities.
A
It's
important
to
remember
that
none
of
this
is
static.
The
bundles
and
svids
are
getting
updated.
Frequently,
workloads
are
starting
up
and
shutting
down
nodes
are
starting
up
and
getting
shut
and
shutting
down.
Typically,
we
actually
refresh
the
svids
every
four
hours.
So
it's
it's
very
frequent
and
there's
no
manual
effort
needed
everything
is
happening
behind
the
scenes
automated
by
the
spire.
A
A
Second,
if
you
have
multiple
clouds,
you
want
them
to
be
loosely
coupled
so
that
if,
if
one
can't
access
another,
the
entire
system
doesn't
go
down
immediately.
So
we
support
a
feature
called
federation
which
lets
you
have
multiple
spire
clusters
that
are
exchanging
their
bundles
periodically
so
that
they
can
all
exchange
all
the
aspens
are
compatible
within
both
systems,
but
they
are,
they
aren't
tightly
coupled
they
are
constantly
talking
to
each
other.
A
Now,
let's
talk
about
some
frequently
asked
questions
about
spire.
First
of
all,
why
do
we
need
one
spire
agent
per
node
and
what
counts
as
a
node
anyway?
Do
virtual
machine
host
counts
as
a
node?
Do
kubernetes
hosts
count
as
a
node,
and
the
answer
is
remember:
the
spire
agent
is
providing
the
spiffy
workload
api,
the
spiffy
workload.
Api
is
a
local
api,
it's
unauthenticated
and
then
it
the
implementation,
uses
features
of
the
operating
system
to
identify
the
process.
A
A
A
What
if
the
spire
agent
is
compromised.
So
I
didn't
discuss
this
in
the
presentation,
but
the
spire
agent
itself
has
really
minimal
trust.
It
can
only
issue
acids
that
are
allowed
to
be
issued
by
that
node.
So,
even
if
the
spire
agent
is
completely
compromised,
even
if
a
hacker
gets
in
and
tampers
with
the
spire
agent,
it
can't
simply
request
acids
for
any
possible
workload
within
your
organization
can
only
request
s-vids
for
the
processes
that
are
supposed
to
be
running
on
that
node.
A
A
A
A
A
A
A
The
registration
entries
map
the
characteristics
of
a
workload
to
a
specific
spiffy
id
and
then
spire
will
generate
an
svid
for
that
spiffy
id
when
it
needs
to.
So
here
is
an
example:
registration
entry
for
a
node.
It
says
if
this
node
is
starting
up
and
it
has
this
aws
tag
database
then
give
it
the
spiffy
id
corresponding
to
database.
A
A
Well,
every
estimate
is
configured
with
the
time
to
live
and
it
will
continue
working
for
that
length
of
time
in
our
default
configuration
we've
set
up
the
time
to
live
to
be
four
hours.
So
if
your
spire
server
goes
down,
you
have
four
full
hours
to
fix
it
before
s-vids
start
failing
in
a
lot
of
organizations.
Four
hours
isn't
long
enough,
so
they
make
that
time
to
live
longer.
A
A
The
team
at
bloomberg
has
written
a
plug-in
for
spire
that
uses
a
trusted
platform
module
to
do
no
data
station,
so
say
you're
running
physical
hardware
in
the
data
center,
and
you
order
a
thousand
new
servers.
You
can
do
that.
No
data
station
automatically
using
the
secure,
unique
identifiers
that
are
built
into
those
servers
at
the
factory
they
also
developed
vault
auth,
aspire
a
plug-in
for
hashicorp
vault
that
lets
you
connect
to
it
using
aspiresvid.
A
A
Finally,
github
has
released
emissary
a
sophisticated
sidecar
proxy
based
on
envoy
that
implements
a
client
for
the
workload
api
and
also
flexible
access
control
based
on
svids.
This
lets
you
wrap
your
existing
services
in
a
fully
featured
spiffy
client.
So
you
don't
have
to
change
a
lot
of
code.
I'm
really
looking
forward
to
using
this
one
in
our
own
projects.
A
So
this
is
very
widely
used
software.
At
this
point,
where
can
you
go
to
find
out
more
well,
you
can
go
to
our
website
spiffy.io.
We
also
host
regular
community
days.
These
are
quarterly
events,
they're
online,
these
days,
of
course,
where
members
of
the
spiffy
community
present
and
how
they're
using
the
standard
and
propose
various
improvements
and
enhancements
to
the
standard.
And
finally,
we
have
a
very
active
slack,
I'm
always
on
the
slack
answering
questions
about
spiffy
and
many
others
are
too
and
that's
where
you
should
go.
A
A
First,
you
can
see
an
introduction
to
spiffy
inspire
from
evan
gilman
again
he's
the
author
of
the
zero
trust
networks
books.
Then
we
have
another
presentation:
that's
part
of
kubecon
2020,
which
is
called
running
spire
in
large
scale,
enterprise
grade
environments
that
gets
more
into
the
federation
high
availability
upstream
certificate
authorities
that
kind
of
stuff
that
you
need
to
run
spire
for
for
truly
large
environments,
like
with
tens
of
thousands
of
agents.
A
Finally,
again,
we're
part
of
hpe
and
hpe
runs
its
own
annual
conference.
It
actually
just
happened.
It's
all
online,
these
days,
of
course,
and
there
are
a
number
of
inspired
deep
dives
at
that
conference.
That
will
will
be
very
helpful
if
you're
trying
to
learn
more
about
spire.
Finally,
again
we're
on
github,
you
can
just
go
to
github
see
what
the
open
issues
are.
Maybe
contribute
a
patch,
maybe
update
some
documentation.
A
So
again,
just
to
recap,
we
discussed
why
you
need
service
identity.
We
discussed
several
different
approaches
to
service
identity
that
don't
use
spiffy,
such
as
perimeter,
security,
shared
secrets,
web
pki,
things
that
that
aren't
a
perfect
fit
for
the
service
identity
challenge.
Then
we
discuss
spiffy,
which
is
the
standard
that
I
work
on.
We
discuss
spire,
which
is
the
actual
implementation
of
that
standard,
and
then
we
discuss
next
steps
for
adopting
spiffy
within
your
organization.
A
A
A
This
is
a
really
good
question.
We
actually
have
a
working
group
in
the
spiffy
standards,
org
working
on
something
called
transitive
identity,
which
is
going
to
be
a
way
for
one
authentication
service
to
essentially
prove
to
the
rest
of
your
system
that
a
user
has
a
particular
identity.
So
it'll
be
a
way
to
attach
user
information
to
svid
service
identity.
That's
work
in
progress
and
we're
discussing
a
bunch
of
different
ways
to
do
it
and
different
trade-offs.
A
So
I'd
really
encourage
you
to
get
involved
with
that,
but
we're
just
working
on
it
now
and
it's
not
quite
ready.
Yet
the
next
question
from
shyam
sundar
will
different
instances
have
different
spiffy
ids.
Will
the
same
application
have
a
unique
spiffy
id?
If
there's
a
deployment
with
three
replicas
will
have
three
different
spiffy
ids
and
that's
a
really
good
question
too.
The
easy
answer
is,
basically,
you
can
do
whatever
makes
sense
for
your
organization.
A
There's
no
rules
about
this
and
we
do
have
organizations
doing
a
bunch
of
different
things.
What
I'd
really
recommend
is
just
have
one
spiffy
id
for
all
the
instances
of
your
service
and
that
way,
when
you
scale
out
multiple
instances
of
the
service,
they
can
all
have
the
same
authentication
properties.
A
A
Next
question
from
nattie
on
slack
would
spiffy
inspire
work
for
ias,
workloads,
I.e
traditional
servers
and
vms,
and
the
answer
is
yes.
Actually
a
lot
of
what
makes
this
so
complicated
is
because
we're
trying
to
support
individual
vms
that
are
outside
of
kubernetes,
so
we
support
kubernetes,
but
we're
not
super
tightly
linked
to
kubernetes
and
we
also
work
on
bare
metal
servers.
We
work
on
all
kinds
of
different
platforms.
A
We
have
plugins
that
can
talk
to
all
kinds
of
different
platforms.
Again,
I
work
for
hpe
we're
a
server
company,
so
we're
working
on
some
features
that
integrate
with
hpe
bare
metal
servers
as
well.
Next,
oh
sorry,
that's
the
same
question.
Another
question
that
comes
up
really
frequently
is:
can
I
use
spiffy
ideas
to
authenticate
to
my
databases?
Maybe
databases
that
are
running
in
amazon,
rds
or
the
equivalent
on
azure
or
gcp,
and
the
answer
is
yes.
We
have
a
way
to
do
that.
A
A
That's
something
that
I
think
is
really
useful,
because
often
the
database
security
is
really
the
weak
point
in
a
lot
of
environments.
So
are
there
any
more
questions
you
can
feel
free
to
follow
up
with
me
on
slack
or
in
the
q
a
box
on
in
toronto?
I
guess
and
I'll
be
here
to
answer
answer
questions
for
a
while.
You
can
also
email
me
or
join
the
spiffy
slack
and
again.
Thank
you
very
much.
I'm
really
happy
to
be
here.
I
guess
we're
done
just
a
couple
minutes
early.