►
From YouTube: 10 Lessons From Migrating to SPIFFE After 10 Years Of Service Identity at Square - Mat Byczkowski
Description
10 Lessons From Migrating to SPIFFE After 10 Years Of Service Identity at Square - Mat Byczkowski
At Square we have developed our own service identity system a few years ago that served us well in our datacenters, but as we increasingly started adopting the cloud, we decided to implement SPIFFE to provide seamless service identity system that would span many environments. In this talk I would like to briefly present how we built a migration process and what we learned from it.
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2021 Virtual from May 4–7, 2021. Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
A
A
A
So
square
you
know,
might
not
be
well
known
outside
of
us,
but
a
lot
of
people
recognize
us
as
the
little
white
reader
for
taking
payments,
but
square
has
seen
a
tremendous
growth
over
the
years.
So
as
many
other
companies,
we
started
from
a
monolith
and
these
days
we're
kind
of
getting
close
to
a
thousand
services
running
in
production,
we're
like
in
different
regions
around
the
world
and
so
on.
A
So
once
we
got
more
than
one
service
we
needed
to
make
the
service
call
secure,
and
you
know
a
long
time
ago
someone
decided
to
use
mutual
tls
since
there
weren't
many
apps,
because,
as
you
can
imagine,
you
have
a
monolith.
You
start
building
new
services,
maybe
break
off
a
little
bit
of
that
functionality
or
build
something
totally
on
the
side.
So
you
just
have
a
few
apps
right,
so
you
can
just
manually
issue
certificates,
client
certificates,
and
then
you
can
distribute
them
manually
to
the
host
that
you
have.
A
Probably
your
infrastructure
is
not
massive,
so
you
can
mostly
do
it
by
hand
or
using
some
scripts,
but
eventually,
as
we
were
growing
more
services
and
more
hosts,
it
was
you
know,
kind
of
becoming
brittle,
so
we
needed
a
better
way
to
manage
sensitive
material.
So
we
built
kiwis.
It's
a
project
that
we
later
open
sourced
and
q
is
basically
what
it
does.
It
stores
and
distributes
secrets
to
all
the
hosts.
A
And
then
once
we
had
that
we
also
had
to
look
into
authorization.
So
again
at
some
point,
someone
wrote
authorization
code
and
added
that
to
a
common
framework-
and
you
know
this
common
framework
or
a
library
would
be
used
by
other
applications.
Other
services
at
square
and
the
way
it
was
done
actually
was
that
you
know
there
was
some
code
that
would
basically
we
encoded
the
the
application
name
in
part
of
the
certificate,
and
for
that
we
specifically
picked
the
organizational
unit
part
of
the
subject
field.
A
Everyone
just
calls
it
ou
at
square
and
then
there's
this
common
framework.
That
would
you
know
get
the
client
certificate
would
parse.
It
extract
the
identity
and
based
on
that
would
perform
some
tracks,
and
then
you
know
we
kept
iterating
on
that
and
then
eventually
some
of
the
scripts
that
were
still
left
to
orchestrate
the
certificate
issuance
and
just
using
keywords,
and
so
on.
We
turned
that
into
a
more
cohesive
service
with
an
api
just
sort
of
like
a
orchestrator,
if
you
like.
A
So
you
know,
if
you
kind
of
think
about
this,
this
we
have
the
ca
you
have
keys.
We
have
some
other
services,
it
started
looking
a
little
bit
like,
maybe
you
would
get
with
spire
in
terms
of
you
know,
issuing
identity
system.
So
now
the
question
is:
why
would
we
want
spiffy?
We
have
to
something
that
seems
to
be
working
well
for
us,
and
you
know
the
the
big
part
of
our
service
identity
system
where
it
kind
of
comes
short,
is
it's
ties
to
keywords
and
it's
tied
to
our
on-prem
architecture?
A
A
So
we
would
have
to
build
all
those
different
things
that
we
have
in
order
to
bring
the
service
identity
with
us,
and
one
of
the
big
things
that
we
had
in
past
couple
years
was
that
we've
had
a
growing
adoption
of
cloud
services
at
square,
and
it
wasn't
just
that.
You
know
we
wanted
to
use
cloud,
but
we
had
to
because
we
had
to
keep
up
with
our
growth.
A
And
finally,
you
know
when
you
have.
This
kind
of
situation
comes
up,
it's
kind
of
an
opportunity
for
you
to
really
start
questioning
status
quo
and
being
like
hey.
Do
we
have
like
you
know
as
our
system
that
we've
built
is
it
best?
Does
it
serve
us
the
best
these
days
or
is
there
something
else
that
we
can
do
and
look
for
the
future?
A
A
A
Then
we
have
now
so
yeah
so
once
we
have
it
running
up
and
it
was
kind
of
dormant
state.
The
next
thing
was
to
you
know,
start
registering
entries
inspire
such
that
we
can
start
issuing
identities.
So
we
had
to
build
some
workload
registrar
because
that's
usually
a
custom
thing,
especially
if
you
have
a
custom
stack
and
another
thing
we
did.
A
We
started
populating
that
with
with,
like
actual
applications,
actual
identities
that
we
had
in
our
system,
and
we
slowly
were
trickling
that
until
we
get
to
100,
so
all
of
our
production
application,
all
of
the
services
are,
are
registered
in
inspire.
And
then
we
got
to
the
point
where
we
had
like
over
a
thousand
hundred
thousand
entries
and
we
could
rotate
certificates,
and
we
can,
you
know,
play
with
configs
and
seeing
like
how
spire
performs
does
the
rotations
and
everything
and
none
of
those
identities
were
used
by
production
workloads
yet.
A
But
we
could
already
see
a
great
deal
of
information
from
that.
So
then,
once
we
had
that,
we
wanted
to
make
sure
that
spire
could
be
hooked
into
envoy
or
andre.
Could
you
know
call
out
to
spire
agent,
since
we
have
a
custom
control
plane,
we
had
to
run
some
modifications
a
little
bit
there
to
the
config
and
maybe
made
a
change
to
initial
kind
of
config
that
comes
with
the
sidecar,
but
nothing
too
drastic,
especially
that
the
sds
api
was
implemented
by
spireagent.
A
So
once
we
had
that,
then
we're
getting
close
and
well
there's
another
catch.
It's
like
okay.
How
do
you
make
sure
that
all
the
sort
of
services
all
the
applications
are
in
production?
How
can
they
understand
those
new
identities?
And
this
is
where
we
had
I,
as
I
mentioned
before,
we
had
the
authorization
logic
and
for
for
historical
reason.
It
lived
in
the
common
framework
so
in
the
library-
and
it
was
in
each
supported
language.
A
So
we
had
to
go
ruby
and
java,
and
you
know
we
had
to
figure
out
how
to
make
sure
that
those
applications
understand
spiffy
ids,
because,
as
I
mentioned
before,
we
had
this
internal
way
of
looking
at
identities
and
we
had
to
kind
of
wrangle
the
code
a
little
bit
to
make
sure
we
had
some
backwards
compatibility
there,
and
I
think
it's
especially
important
because
we
had
accola.
So
you
know
access
control
list
in
and
it
was
in
a
code
base
of
each
application
and
it
was
calling
out
the
application
name.
A
But
now
you
can
get
spiffy
id.
So
we
had
to
figure
how
to
do
it
without
making
lots
of
changes
or
make
it
hard
on
app
owners
to
be.
You
know
so
that
they
could
adopt
spiffy
and
then
final
step.
Once
we
had
all
of
that
done
and
updated
was
to
really
you
know,
start
using
spf
ids,
and
for
that
we
built
feature
flags
into
the
onboard
control
plane,
and
then
we
had
a
real,
really
fast
way
of
switching
to
spiffy
and
back
so
like
any
new
connection.
A
A
How
do
you
map
this
new
system
onto
something
that
you
know
really
well,
but
it's
very
specific
to
you.
So,
if
anything
I
would
say
yeah,
you
know
manage
your
expectations
and
you
know
make
sure
you
staff
your
product
accordingly,
and
you
have
people
who
are
dedicated
to
this
project
and
not
necessarily
get
distracted
by
your
other
day-to-day
things.
A
So
you
know
the
days
might
be
long
but
they're
shorter
when
you
get
help.
So
I
just
cannot
say
enough
of
good
things
about
spiffy
slack
super
helpful
lots
of
smart
people
a
great
place
to
bounce
off
ideas,
brainstorm
people
who've
done
this
before
in
other
companies.
So
you're
not
alone.
You
know
you
don't
again
with
spire
and
open
source,
and
this
is
a
great
point
of
open
source.
You
don't
have
to
build
everything
yourself.
A
You
know
you
can
trip
in
features,
but
others
can
make
it
better.
You
know
you
can
ensure
the
burden
of
maintenance,
because
obviously
maintenance
is
is
a
big
part
of
products
in
in
order
to
you
know,
keep
them
stable.
It
takes
a
lot
of
work
after
you
actually
implement
features.
So
I
think
this
is
a
big
big
plus
for
spire
and
having
this
robust
community.
A
A
So
you
don't
want
to
end
up
in
a
situation
where
you're
going
to
production
with
a
small
subset
of
your
platform,
and
then
you
realize
that
you
can't
scale
to
all
the
nodes,
maybe
which,
by
the
way,
wasn't
the
case
with
spire.
It's
been,
it's
been
working
really
well
for
us
in
that
sense.
So
another
thing
you
don't
want
to
just
like
switch
low,
throughput
services
to
use,
say
spiffy
and
then
you
find
out
later
like.
Oh,
it
doesn't
work
with
some
hype.
A
You
know
throughput
and
highly
critical
applications
so
because
of
that
we
kind
of
uncover
a
couple
of
things.
For
example,
we
run
into
some
database
performance
issue
once
we
you
know
enable
it
for
all
our
workloads
and
we're
running
it
in
shadow
mode,
but
because
we
had
you
know
telemetry
and
internal
db
metrics.
We
could
share
that
with.
You
know
the
spire
core
team
and
we
actually
got
help
on
that,
and
you
know
they
managed
to.
A
A
So
if
you
want
to
have
this
adoption-
and
I
think
some
people
already
mentioned
this-
you
want
to
have
backwards
compatibility.
You
can
just
you
know
greenfield
this
and
just
hope
for
the
best
product
teams.
You
know
their
migrations
happen
all
the
time.
There's
infrastructure
changes
like
you
know,
someone
brings
new
logging,
there's
new,
like
service
mesh.
There
are
other
things
and
product
teams.
Just
don't
have
time
for
that.
You
need
to
focus
on,
like
you
know,
developing
features
and
you
won't
be
bogged
down
by
another,
yet
another
infrastructure
migration.
A
A
You
know
you
can
gradually
opt
into
spitty
rather
than
being
in
a
lockstep
where
it's
getting
really
complicated,
as
you
have
lots
of
upstream
and
downstream
dependencies,
so
the
actual
mesh
of
your
services
can
get
really
complicated.
Once
you
have
a
lot
of
them
in
production
and
of
course,
we
had
to
add
the
we
had
to
share
the
trust
bundle.
A
A
Right
and
when
you
build
this
backwards
compatibility,
I
think
you
want
to
define
limits
on
that,
because
at
some
point
you
have
you
know
the
law
of
diminishing
returns.
You
have
migrations
having
long
tail,
some
apps
will
get
decommissioned
before
you
get
to
them.
So
maybe
you
shouldn't
even
bother
building
support
for
that.
I
think
cutting
scope
is,
is
you
know
true
of
any
project
and
what
it
meant
for
us?
For
example,
was
we
had
the
envoy
service
mesh
that
was
taking
off?
A
Maybe
we
had
at
some
point
like
you
know,
50
or
100
apps
using
it,
and
I
I'm
talking
apps,
I
mean
services,
but
we
keep
calling
them
apps
at
square.
So
I'll
keep.
You
know,
probably
change
the
words
and,
and
then
we
said
like
okay,
why
don't
we
just
start
with
onward
service
mesh
first
and
ignore
everything
else,
so
any
sort
of
direct
tls
connections
between
services?
We
can
just
keep
them
as
as
is
and
don't
break
them,
hopefully
don't
break
them
and
and
then
what
it
meant.
A
A
So
you
know
now
again
talking
about
how
you
expand.
Spiffy
is
like
how
do
you
convince
folks
that
they
they
should
use
this
new
and
better
system?
And-
and
sadly
you
know,
security
is
not
necessarily
the
best
selling
point.
When
you
tell
people
like
oh
do
this,
because
it's
secure,
you
know
people
may
not
upgrade,
but
you
have
some
new
emojis
and
and
all
of
a
sudden
you
have
a
great
adoption
rate.
A
So
you
know
if
you
can
bundle
those
features
somehow
that
I
think
that's
really
helpful
and
unfortunately
we
missed
the
on
voice
service
mesh.
If
we
were
about
to
go
with
like,
for
example,
this,
you
know
this
new
service
mesh
and
we
could
from
day
one
say
like:
oh
the
server
smash
is
coming
with.
Spiffy
then
actually
would
be
great
for
us,
but
we
kind
of
missed
this.
A
But
in
order
for
that
to
happen,
you
need
to
have
documentation
in
place.
You
have
to
make
it
really
easy
for
someone
to
you
know,
do
it
themselves,
and
we
had
this
this
one
example
where
we
were
launching
this
kubernetes
cloud
cluster,
and
you
know
there
are
some
features
that
people
wanted
to
do,
or
people
wanted
to
deploy
to
the
cloud.
A
A
Like
you
know,
maybe
you
pre
like
keep
the
identity
of
this
app,
but
you
have
a
some
site
process
that
we'll
call
another
app
and
we
don't
even
may
not
know
about
that-
that
you
know
there's
such
use
case
right
then
frameworks
parse
certificates.
This
is
this
happened
early
on,
where
you
know,
as
I
mentioned
before,
the
certificate
decline
certificate
is
exposed
to
the
framework
to
the
library.
So
you
do
some
parsing
there's
some
code,
someone
wrote
it
in
java,
but
then
maybe
someone
brings
a
new
language
at
some
point
again
in
some
corner.
A
There's
a
need
for
I
don't
know
doing
stuff
in
python
right
and
then
people
be
like
okay.
How
do
I
make
it
work
with
other
services?
Now
I
need
to
get
the
call
and
they
look
at
the
code
and
it's
like.
Oh,
I
would
just
copy
this
code
and
roughly
replace
that
you
know
in
a
different
language,
but
then
there's
small
differences,
and
then
you
get
into
problems
with
that.
So
then
you
know
we
introduced
onboard
service
mesh,
it
started
terminating
tls
and
you
have
this.
A
You
know
plain
text
connection
over
unix
socket
to
the
to
the
application,
but
because
so
much
code
still
relies
on
the
client
certificate.
Now
you're
forwarding
the
client
certificate
to
the
app,
rather
than
just
maybe
passing
some
spiffy
id
in
the
header
or
something
else.
So
again,
it
kind
of
takes
time
to
clean
this
up
and
you
have
to
consciously
kind
of
take
the
effort
to
make
those
changes.
A
If
you
want
to
change
things,
so
I
already
mentioned
how
you
know
folks
parsing
certificates
later
they
all
want
to
parse
the
pids,
because
some
of
the
trust
domains
are
not
intuitive
anymore
and
then
we
have
those
apples
and
eclipse
only
has
the
application
name
and,
as
you
hopefully
can
see
on
the
slide,
it's
either.
You
know
application
name,
my
app
or
you
have
this
50
construct
this
uri,
where
it's
like.
I
don't
know
what
the
stress
domains
are.
A
So
you
know
that's
kind
of
confusing
another
thing
where
you
know
people
might
have
mental
models,
it's
like.
Oh,
we
have
staging
and
we
have
production.
If
I
allow
my
app
to
app
a
to
call
app
b,
then
that
means
it's
good
for
staging
in
production,
but
it
might
not
be
really
true
in
you
know,
once
you
have
more
trust
domains
and
with
you
know,
spf,
we
have
more
trust
domains.
A
We
have
staging
production
on-prem,
we
have
a
dev
staging
production
cloud
and
then
we
have
like
a
separate
one
for
lambda,
and
then
it
just
keeps
growing.
So
you
know
it's
not
the
same,
so
you
definitely
need
to
educate
and
promote
within
the
company.
I
mean
obviously
true
of
the
project
in
general,
in
the
in
the
broader
tech
community,
but
definitely
true
internally
as
well,
then
short-lived
certificates
are
great
until
they're
not-
and
this
is
probably
a
nightmare
scenario
you
want
to
avoid.
A
You
don't
want
to
get
in
a
place
where,
like
all
your
certificates
expire-
and
you
may
not
have
a
lot
of
time-
and
some
of
this
is-
is
totally
how
square
did
it.
Our
certificates
were
over
one
year
lifetime
before
the
homegrown
identity.
So
you
know
you
had
different
monitoring
systems
that
you
know
if
you
tripped
all
those
different
tracks,
even
in
the
worst
case,
you
should
have
get
to
the
few
days
or
you
can
respond
by
friday
and
be
like
okay.
I
can
get
to
it.
A
But
if
your
svits
are
a
24
hour
ttl,
then
you
know
and
you
refresh
them
at
half
of
the
lifetime.
Then
you
may
end
up
with,
like
you
know,
up
to
12
hours
to
fix
some
issues.
Of
course,
they're
different
failure
scenarios
and
we
were
actually
running
some
exercises
and
I
would
also
say
the
24-hour
ttl
is
purely
what
square
chose
and
it's
definitely
beneficial
for
us
in
many
ways
and
spire's
been
actually
really
really
good
about
the
rotations.
A
B
A
I
imagine
a
good
you
know,
pizza
size
crew
would
be
would
be
great.
I
guess
it
depends
on
your
needs.
One
of
the
things
that
we've
done
at
square
that
was
was
kind
of
helpful
was
that
we
actually
had
three
people
at
a
time
working
on
this
right,
and
that
was
really
good,
but
the
problem
we've
done
is
we
kind
of
ended
up
with
swim
lanes
that
one
person
was
looking
at
some
aws
cloud.
I
was
looking
at
on-prem
and
someone
else
was
looking
at
aws
native.
A
We
were
looking
into
building
mtls
into
lambdas,
so
we
ended
up.
You
know
very
much
separated,
and
I
think
that
in
hindsight
it
would
have
been
great
if
we
just
built
more
focus
on
just
tackling
one
problem
at
a
time
and
not
paralyzing
that
and
having
that
sort
of
like
a
shared
knowledge,
because
what
happened
later
is
my
team
is
more
like
you
know,
six,
seven
people,
so
we
actually
own
other
things,
not
just
you
know
service
like
our
homegrown
service
identity.
A
So
later
you
had
to
go
and
also
like,
educate
the
rest
of
your
team,
because
they've
been
just
busy
doing
other
stuff.
So
you
know
you
kind
of
end
up
in
a
situation
where
you
should
definitely
involve
more
people.
I
would
say
three
people
sounds
about
right
to
me,
depending
on
the
size
of
your
project.
B
A
Federation
right
so
federation,
we're
not
using
federation
yet
and-
and
this
is
where
it
gets
a
little
bit
tricky-
we
want
to
build
federation
next
year,
but
the
way
we
we
do
it
is.
We
have
different
data
centers.
So
staging
is
one
trust
domain
different
different
say.
Data
centers
in
staging
are
trust
our
one
trust
domain.
We
have
like
say
three
production,
on-prem
data,
centers,
they're,
one
trust
domain.
So
we
have
separate
clusters,
but
they're
under
one
trust.
Consider
one
trusted
domain,
actually
they're
backed
by
production,
is
backed
by
one
database.
A
That's
why
you
have
the
same
trust
bundle,
and
then
we
also
have
cloud
and
cloud
is
like
one
trust
domain
configured
but
like
separate
registration
for
each
cluster,
so
they're
kind
of
disconnected.
So
the
way
we've
done
it
to
solve
this
problem
is
we
essentially
ended
up
sharing
roots
manually
and
putting
them
in
a
truss
bundle
which
eventually,
we
want
to
move
into
the
federation,
but
this
was
sort
of
like
like
a
short-term
fix.
Until
we
got
to
the
point
where
you
know
we
can
build
a
whole
federation.