►
Description
Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Running SPIRE In Large Scale, Enterprise-Grade Environments - Andrew Harding, HPE
SPIRE started as the reference implementation of SPIFFE back in 2017, and is now part of the core security architecture of many large enterprises. That means it has to be flexible and reliable in a wide variety of deployment scenarios. We’ll discuss the practical logistics of running SPIRE in a large enterprise: high availability, nested deployment for multiple availability zones, integration with upstream certificate authorities, observability, and monitoring. This talk will be useful for anyone who is already using SPIRE at scale, as well as anyone who is thinking about using SPIRE and wants to know what to expect as their infrastructure grows.
https://sched.co/aDcf
A
Hey
good
morning
or
good
afternoon,
wherever
you
are
welcome
to
today's
deep
dive,
maintainer
track
for
spire,
my
name
is
andrew
harding
and
I
work
at
hewlett,
packard
enterprise
and
I'm
a
full-time
maintainer
of
spire
in
this
session,
we're
going
to
explore
a
few
ways
to
deploy
spire
and
build
a
reliable
identity
plane
for
your
infrastructure
before
we
get
into
it.
I
want
to
first
apologize
in
advance
for
any
mistakes
or
problems
due
to
the
pre-recorded
nature
of
this
presentation.
A
The
first
is
load
distribution.
Specifically,
how
can
we
just
deploy
spire
in
order
to
distribute
load
related
to
workload,
registration
and
identity
management
and,
as
we
scale
up
the
number
of
workloads
that
spire
is
servicing?
How
do
we
ensure
that
spire
and
the
issuance
of
service
identity
doesn't
itself
become
a
bottleneck,
as
we
scale
out
your
application?
A
A
Spire
is
a
reference
implementation
of
the
spiffy
specification.
Fundamentally,
what
this
means
is
that
spire,
particularly
the
spire
agent,
provides
the
spiffy
workload
api
now
workloads
leverage
the
workload
api
to
obtain
a
couple
of
different
things.
The
first
is
a
spiffy,
verifiable
identity
document
or
s-vid.
For
short,
an
s-vid
is
a
document
with
which
workloads
prove
their
identity.
A
A
A
A
You
could
define
a
trust
domain
for
each
business
unit,
for
example,
or
maybe
around
different
classes
of
software
within
a
business
unit.
The
key
takeaway
here
is
that
the
identities
issued
to
workloads
within
a
trust
domain
are
all
issued
by
the
same
set
of
authorities
and
can
be
verified
using
the
spiffy
bundle
of
the
trust
domain.
Also,
there's
no
requirement
on
how
many
trust
domains.
You
know
you
maintain
within
your
organization,
it's
all
dependent
on
on
what
your
needs
are
around
security
boundaries
in
isolation.
A
Now,
within
that
trust
domain,
we
have
a
machine,
virtual
or
otherwise,
which
is
hosting
some
workloads,
and
our
workloads
are,
of
course,
spiffy
enabled
workloads
that
leverage
the
spiffy
workload
api
hosted
by
the
spire
agent
through
which
they
obtain
their
service
identity
and
bundle.
Information
now
spireagent
does
not
mint
s-vids
itself,
but
rather
relies
on
the
spire
server
within
the
trust
domain.
A
Spire
servers
have
a
few
responsibilities,
but
the
primary
of
which
is
to
act
as
the
certificate
authorities
and
job
issuers
of
the
trust
domain.
They
maintain
and
rotate
key
pairs
for
signing
x-509
and
jot
svets
spire
agents,
query
the
servers
about
workloads
they
are
responsible
for,
and
they
also
issue
signing
requests
to
spire
server
to
obtain
identity
or
s-vids
on
behalf
of
those
workloads.
A
A
A
For
the
purpose
of
today's
discussion,
let's
focus
on
just
a
few
things
stored
by
the
database.
The
first
is
the
spiffy
bundle
like
we
discussed
earlier.
The
spiffy
bundle
contains
the
public
key
material
for
the
x509
and
jot
authorities
that
workloads
use
to
authenticate
svids
second
are
registrations.
A
These
registrations
form
the
policy
in
the
trust
domain,
about
what
workloads
exist
and
how
to
identify
them
and
where
they
are
authorized
to
run
the
synchronization
of
these
first
two
things.
You
know
the
spiffy
bundle
and
the
registrations,
along
with
the
signing
requests
for
sveds.
That
agents
do
constitute
the
majority
of
the
traffic
and
communication
efforts
that
take
place
between
the
spire
server
and
agent.
A
A
It's
important
to
note
that
this
database
layer
is
what
links
up
spire
servers
within
aspire
deployment.
Spire
servers
attached
to
the
same
database
have
a
shared
view
of
all
the
above
information
all
right.
So
now
that
we've
discussed
the
database
layer,
let's
return
to
our
basic
deployment,
we've
got
a
spire
agent,
servicing
workloads
with
s-vids
and
bundle
information
which
that
agent
obtained
through
coordination
with
aspire
server,
backed
by
a
database
that
holds
bundle
and
registration
information,
and
with
that
we've
got
a
complete
spire
deployment.
A
But
you
know
this:
this
deployment
is
nowhere
close
to
robust,
so
we're
going
to
look
at
a
few
ways
to
scale
this
up,
but
first,
let's
go
over
some
failure
scenarios,
for
example:
what
happens
when
the
spire
server
can
no
longer
talk
to
the
database
without
access
to
the
database?
There's
not
much
that
can
happen
within
your
deployment.
A
New
workloads
cannot
be
registered.
New
agents
cannot
attest
and
join
the
trust
domain
and,
like
we
mentioned
before,
the
database
contains
authorization
policy
for
agent
sveds.
Without
access
to
that
policy.
The
server
cannot
authenticate
requests
from
existing
agents
and
will
deny
access.
This
means
agents
cannot
mint
svids
for
workloads
or
maintain
up-to-date
bundle
and
registration
information
critically.
A
A
A
A
A
Now
in
this
failure
scenario,
the
outlook
is
still
pretty
bad.
In
fact,
from
an
agent
perspective,
this
scenario
more
or
less
mimics,
that
of
a
loss
of
availability
in
the
database,
no
new
workload,
s
vids,
no
updates
for
bundle
or
registration
information,
no
agent
s,
vid
rotation,
depending
on
the
nature
of
the
failure,
say,
a
network
failure
in
between
agents
and
servers.
A
A
A
The
natural
solution
here
is
to
scale
out
the
number
of
spire
servers
so
that
if
one
server
goes
down,
agents
have
another
place
to
go
operating.
Multiple
servers
at
scale
like
this
isn't
without
its
own
challenges.
For
example,
when
you
go
to
roll
out
new
software
configuration,
you
want
to
adopt
a
rolling
deployment
strategy
so
that
the
problem
with
the
updates
does
not
take
out
all
of
your
servers
at
once,
and
you
can
roll
back
failed
servers.
In
the
case,
there
is
a
problem.
A
A
If
we
dig
into
the
agent
internals,
we
see
that
the
agents
do
not
mint
workload
s-vids
on
demand.
Instead,
they
actively
maintain
an
s-vid
cache
the
agent
fetches
and
maintains
fresh
s-vids
for
all
workloads.
It
is
authorized
for
independent
of
whether
or
not
that
workload
has
attempted
to
contact
the
agent
through
the
workload
api.
A
A
A
Now
the
spire
agent
has
two
kinds
of
svids
cached.
The
one
we've
primarily
focused
on
so
far
are
the
workload
svids.
These
are
the
ones
that
are
served
to
workloads
over
the
workload
api,
and
you
know
they
are
primarily
a
concern
of
the
data
plane
when
a
workload
estimate
expires
or
is
otherwise
unavailable.
This
leads
to
workload
level
failures,
which
are
the
ones
that
are
most
apparent
to
our
customers.
A
Unfortunately,
the
second
is
the
agent's
own
s-vid.
This
s-vid
is
given
to
the
agent
when
it
enters
the
trust
domain
via
node
attestation.
It
is
maintained
and
rotated
by
the
agent.
The
agent
s-vid
is
used
to
authenticate
requests,
aspire,
servers
and
represents
the
agent's
membership
in
the
trust
domain.
A
A
A
The
ttls
of
these
two
svids
can
be
controlled
independently.
This
gives
operators
flexibility
with
the
security
reliability
trade-off,
for
example,
if
the
workload
aspect
lapses,
but
the
agent
s-vid
does
not.
After
the
outage
is
repaired,
the
agent
will
be
able
to
quickly
recover
since
its
membership
in
the
trust
domain
has
not
been
lost.
A
For
this
reason,
it
isn't
uncommon
for
agent
svids
to
be
given
a
higher
ttl
than
a
workload
svid,
especially
if
the
control
plane
segregation
makes
it
more
unlikely
that
an
agent
aspect
is
compromised
than
a
workload
asvid
sitting
in
the
data
plane.
The
takeaway
here
is
that
operators
are
given
a
choice.
A
A
The
agent
cache
is
the
first
line
of
defense
to
mitigate
infrastructure
outage.
This
is
because
the
cache
allows
spire
to
continue
in
its
fundamental
mission
to
provide
s
vids
and
bundles
over
the
workload
api,
while
the
infrastructure
is
being
repaired
and
last
ttls
are
a
powerful
knob
to
adjust
between
the
trade-off
of
security
and
reliability,
keeping
the
ttls
as
small
as
possible,
but
no
smaller
is
important
to
survive
infrastructure
outage,
while
reducing
the
window
of
impact
for
compromised
svids.
A
Of
course,
that
isn't
going
to
be
enough.
Our
business
is,
of
course,
thriving
and
load
against.
Our
workloads
is
climbing.
In
order
to
meet
customer
demand,
we
need
to
scale
up
the
number
of
workloads,
as
the
number
of
workloads
and
accompanying
agents
grow.
The
load
on
that
lonely
little
spire
server
is
going
to
go
up
and
up,
and
of
course,
if
that
spire
server
falls
over
agents
are
suddenly
unable
to
obtain
or
rotate
workload
s-vids
to
increase
the
reliability
of
our
system
and
to
scale
up
demand.
A
A
A
Like
we
mentioned
previously
s,
vids
within
a
single
trust
domain
are
minted
by
the
same
set
of
authorities.
This
means
that
each
of
these
zones
must
have
a
shared
view
of
the
spiffy
bundle
for
the
trust
domain,
so
the
workloads
minted
in
one
zone
can
authenticate
and
be
authenticated
by
workloads
in
other
zones.
Let's
explore
some
options.
A
A
A
A
A
A
However,
there
are
some
potential
downsides
to
consider.
First,
that
shared
database
becomes
a
single
point
of
failure
for
all
zones.
Second,
the
database
has
to
handle
all
of
the
queries
for
servers
from
all
zones
which
definitely
adds
up.
This
can
make
it
hard
to
distribute,
load
and
scale
the
database
appropriately
as
the
number
of
zones
or
workloads
in
each
zone's
grows.
A
A
A
However,
because
each
x509s
bit
is
signed
by
a
common
authority
or
authorities,
the
x-509
authorities
present
in
the
bundle
for
each
zone
ends
up
being
the
same.
This
again
enables
mutual
authentication
between
workloads
across
zones.
Let's
take
a
detailed
look
at
how
the
bundle
in
each
zone
is
built.
A
A
It
returns
that
intermediate
ca
certificate,
along
with
a
set
of
upstream
route
authorities
back
to
the
spire
server
spire
server,
then
appends
the
upstream
route
authority
to
the
bundle
in
the
database
since
the
spire
server
ca
is
an
intermediate
authority
off
of
that
root
authority.
An
s-vid
signed
by
that
server
has
a
valid
chain
of
trust
back
to
the
upstream
wind
authority
present
in
the
bundle
the
spire
servers
in
other
zones
likewise
get
an
intermediate
ca
signed
by
the
upstream
authority
and
append
the
upstream
authority
to
the
bundle
in
their
own
database.
A
A
A
A
A
A
This
process
is
repeated
for
each
spire
server
that
shares
the
upstream
authority
and,
after
that's
all
propagated.
Each
bundle
now
contains
the
complete
set
of
jot
signing
keys
for
aspire
servers
which
share
the
upstream
authority
before
we
go
on
I'd
like
to
point
out
that
this
ability
to
publish
jot,
signing
keys
upstream
is
an
optional
feature
of
upstream
authorities
and
is
not
widely
implemented.
So
you
have
to
be
careful
about
which
upstream
authority
you
choose
if
jot
s,
vids
are
important
to
you.
A
The
advantage
of
using
the
upstream
authority
option
is
that,
beyond
providing
a
shared
view
of
the
spiffy
bundle
of
the
trust
domain,
each
zone
has
its
own
database
instance
to
handle
the
queries
for
that
zone.
This
makes
it
easier
to
distribute,
load
and
scale
out
the
database
layer
within
a
single
zone.
A
Another
advantage
when
considering
tying
together,
multiple
spire
deployments
using
the
upstream
authority,
is
that
interactions
with
the
upstream
authority
are
relatively
infrequent.
Spire
servers
only
use
the
upstream
authority
when
they
rotate
their
x,
509
intermediate
ca
or
jot
signing
keys,
which
is
a
configurable
but
relatively
infrequent
event.
A
Spire
servers
rotate
the
signing
key
pairs
at
half
at
the
half
ttl
point
similar
to
the
agent
and
its
asvid
rotation
strategy,
the
ttl
for
the
ca
and
jot
signing
keys
defaults
to
24
hours,
but
it
isn't
abnormal
to
have
a
much
larger
lifetime
say
a
week
or
a
month
or
more.
This
means
that
if
the
upstream
authority
goes
down,
there's
quite
a
bit
of
leniency
in
how
fast
the
outage
needs
to
be
addressed,
the
disadvantage
is,
of
course,
that
it
is
a
more
complicated
setup
with
more
moving
parts
and
more
points
of
failure.
A
Spire
has
a
few
built-in
upstream
authority
implementations,
for
example,
one
that
leverages
aws
certificate,
manager's
private
ca
feature,
but
let's
turn
to
the
third
option,
which
is
a
refinement
of
the
upstream
authority
approach
where
the
spire
cluster,
where
aspire
cluster
itself,
is
acting
as
the
upstream
authority
for
a
downstream
spire
cluster.
This
is
something
that
we
call
nested
spire
the
spire
servers
in
each
zone
have
their
intermediate
ca,
signed
by
the
spire
cluster,
acting
as
the
upstream
authority.
A
A
Nested
spire
allows
you
to
organize
your
spire
deployments
into
clusters
that
are
largely
independent.
If
an
upstream
cluster
goes
down,
the
downstream
clusters
stay
relatively
operational.
For
the
reasons
we
mentioned
when
discussing
the
upstream
authority,
this
can
be
a
very
powerful
way
to
scale
out
and
isolate
your
deployments.
A
There
is
a
trade-off,
however,
remember
that
each
of
these
clusters
has
its
own
database
the
advantage.
Of
course
there
is
load,
distribution
and
fault
isolation.
However,
if
you
remember
our
earlier
discussion
about
what
the
database
is
in
charge
of
you'll
recall
that
workload
registrations
are
stored
in
the
database,
each
cluster
is
in
charge
of
its
own
registrations.
A
A
At
this
point,
it
may
make
sense
to
break
our
deployment
into
more
than
one
trust
domain,
and
how
do
we
enable
authentication
between
workloads
and
two
distinct
trust
domains?
Well,
the
answer
here
is
spiffy
federation
federation
is
the
process
by
which
a
trust
domain
gains
the
ability
to
authenticate
workloads
from
another
trust
domain.
It
is
a
one-way
relationship,
meaning
that
to
enable
mutual
authentication,
each
trust
domain
needs
to
federate
with
each
other.
A
A
This
mechanism
is
the
spiffy
bundle
endpoint,
which
is
an
https
endpoint
that
serves
the
spiffy
bundle
of
the
foreign
trust
domain.
The
endpoint
is
authenticated,
and
at
this
time
we
define
two
authentication
mechanisms.
The
first
is
via
web
pki
or
via
spiffy
authentication,
as
the
foreign
trust
domain
rotates
its
authorities.
A
Well,
that's
it
for
today,
that's
what
I
have
for
you
to
summarize.
Today:
we've
explored
deployments
from
the
very
simple
to
multi-zone,
to
nested
hierarchies
and
onto
federated
trust
domains.
We've
talked
a
little
bit
about
failure
modes
and
the
various
layers
that
require
redundancy
in
order
to
achieve
a
highly
available
deployment
that
can
scale
and
we've
also
dug
into
the
details
of
how
bundles
are
built
across
databases
or
shared
databases,
and
we've
touched
on
the
mechanisms
around
spiffy
federation
and
the
ability
for
two
trust
domains
to
federate
with
each
other.