►
From YouTube: Uber x Security: Why and How We Built Our Workload Identity Platform - Tyler Julian & Daniel Feldman
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Uber x Security: Why and How We Built Our Workload Identity Platform - Tyler Julian, Uber & Daniel Feldman, Scytale
Since launching in 2009, Uber has become the poster child for explosive growth - in revenue, headcount, and operational complexity. This growth has created new challenges for Uber’s engineering team and the services they maintain. How can these services reliably identify and authenticate to each other? How can we ensure isolation for critical systems? How can we achieve compliance with regulations like GDPR and SOX? How can we efficiently incorporate identity into unsupported legacy systems? Most importantly, how can we do all of this in a way that doesn’t get in the way of engineers? This talk will explore what led Uber to build out an automated, API-driven, cloud native approach to workload identity with SPIFFE that provides the critical bridge of trust within Uber's next-gen infrastructure, and the impact it’s had on Uber’s internal security and developer efficiency.
https://sched.co/MPe6
B
B
So
goober
and
security,
so
the
title
like
I
mentioned
subject
is
how
and
why
we
built
out
our
workload
identity
platform.
I
really
want
to
focus
in
on
the.
Why
today
and
the
how
we'll
come
so
when
we
talk
so
I'm
Tyler,
Julian
I'm
at
right
now
on
our
Identity
and
Access
Management
team
and
actually
I
should
probably
just
grab
this
clicker
so
that
I
can
our
Identity
and
Access
Management
team
I've
been
working
here
for
a
couple
years
on
our
authentication
and
identity
frameworks.
A
I'm
Daniel
Feldman
I
am
currently
a
site.
L
have
been
there
for
about
a
year
and
we
do
were
a
whole
company
that
is
built
around
the
concept
of
workload,
authentication
and
workload
identity
and
in
my
past
life
I
was
a
company
called
Veritas
which
makes
backup
software,
and
they
did
all
the
work
load,
authentication
for
veritas,
backup
products
which
is
sort
of
how
I
developed
an
interest
in
this
subject.
A
Oh-
and
you
can
follow
me
at
@d-
underscore
Feldman,
not
D,
Feldman,
that
someone
else
and
Tyler
doesn't
have
a
Twitter,
but
he
does
have
a
Spotify
with
his
own
originals.
So
you
can
follow
him
there.
Alright.
So
today
we're
gonna
talk
about
four
sections
of
our
talk.
The
first
is
the
identity
problem
at
uber.
A
The
serve
you'll
get
an
idea
of
the
scale
of
the
infrastructure,
the
scale
of
the
problem
at
a
do,
BER,
which
is
all
news
to
me,
because
I
have
never
whoever
and
then
we're
gonna
talk
about
Sofia
inspire,
which
are
sigh,
tells
solutions
to
the
workload
the
identity
problem,
then,
in
part,
three
Tyler's
going
to
do
a
case,
study
sort
of
an
example
of
how
a
growing
company
can
end
up
with
a
big
workload,
identity
problem
that
company
may
or
may
not
be
over.
You
can't
talk
him
too
much
detail
about.
A
B
B
Yeah
well
wing
it
here:
alright,
so
identity
giver,
so
I
wanted
to
kind
of
set
the
stage
for
how
we
came
upon
this
problem
of
identity.
What
led
us
to
get
there?
What
sort
of
business
value
does
solving
identity
get
get
Eber?
So
just
to
give
you
a
rough
sense
of
the
scale,
we've
got
about
three
thousand
unique
services
that
we
run
Eddy
Bert.
All
of
these.
Not
all
of
these
are
specifically
for
ride-sharing,
but
these
are
sort
of
in
support
of
uber
as
a
business.
B
What
that
results
in
is
roughly
four
hundred
thousand
containers
running
at
any
time
to
support
those
three
thousand
services
State.
So
when
I'm
talking
about
services,
primarily
stateless
workloads,
we
run
about
a
hundred
million
batch
workloads
every
day,
and
this
doesn't
even
capture
all
the
the
stateful
storage
that
we
also
run
as
well,
and
so
we're
removing
a
tremendous
amount
of
data
right.
And
so,
where
does
this
data
run?
On
here's
a
little
picture
of
our
infrastructure?
B
We
have
deployments
in
both
the
cloud
and
on-premise
centers,
so
we
have
data
centers
in
the
US
and
we
also
have
cloud
zones
across
a
couple.
Different
providers,
the
way
that
data
moves
over
our
pcs
is
generally
G
RPC,
although
actually
the
lion's
share
of
our
traffic
is
HTTP
and
a
a
custom
protocol.
B
But
a
cloud
native
approach
actually
does
service
quite
well.
I'm
I
will
get
into
exactly
what
that
looks
like.
So
when
we
talk
about
identity,
there's
a
lot
of
things
come
to
people's
minds.
The
first,
of
course,
is
humans.
Human
identity.
When
we
talk
about
humans
were
talking
about
users,
users
are
writers,
drivers,
couriers,
customer
support
representatives,
I
mean
engineers,
managers
I
mean
it's,
it
runs
the
gamut,
people
think
about
OAuth,
2
factor
auth.
All
that
sort
of
thing.
B
When
we
talk
about
identity
for
machines,
we're
talking
about
routable
hosts
within
our
infrastructure,
we
talk
about
identity
for
workloads.
This
is
really
the
meat
and
potatoes
of
your
business.
This
is
these
are
the
things
that
are
the
applications
that
are
running
business
logic,
and
these
are
things
like
your
stateful.
B
Your
stateful
storage
is
your
stateless
services,
your
you
know,
Damon's
that
might
be
running
across
your
hosts
and
you
and
your
batch
jobs
and
so
workload
identity
is
really
what
what
the
focus
of
this
topic
is
and
we're
not
going
to
focus
too
much
on
the
other
things.
So
what
does
it
mean
to
have
a
workload
identity?
The
goal
in
my
eyes
is
to
be
able
to
uniquely
identify
a
particular
application,
but
say
you
have
one
service
and
you
have
a
hundred
instances
of
this
of
that
service.
B
Is
it
particularly
useful
to
know
one's
one
instance
versus
another
instance?
Well,
every
instance
has
roughly
the
same
permissions
so
differentiating
on
an
instance
by
instance,
basis
it's
probably
not
super
useful,
but
being
able
to
uniquely
identify
that
service.
That
seems
like
a
powerful
construct
right
for
controlling
access.
Now
you
might
have
that
same
exact
service
running
and
staging,
but
running
in
production
and
creating
a
different
ID.
B
What
identity
also
allows
you
to
do
is
to
protect
data
with
encryption
and
transit,
which
is
going
to
get
you
confidentiality
and
data
integrity.
You
know
nobody's
eavesdropping,
and
you
know
nobody
can
change
your
data
in
flight
or
super
critical.
It
also
allows
you
to
control
access
through
authenticity
and
authorization.
You
know
what
is
accessing
the
data
and
you
can
manage
that
access.
B
One
thing
to
note,
though,
is
identity
by
itself,
doesn't
necessarily
get
you
encryption,
but
when
we
talk
about
identity
and
identity
documents,
it's
often
manifests
as
an
x.509
certificate,
and
so
the
protocol
of
TLS
and
that
sort
of
thing
sort
of
come.
As
a
byproduct
of
that.
So
we
talked
about
the
business
value
of
identity.
One
of
the
first
things
that
comes
to
my
mind,
just
based
on
a
lot
of
conversations
that
have
been
in
is
compliance
and
kind
of
the
most
relevant
thing.
B
There
is
gdpr,
which
went
into
effect
last
May
yeah
about
a
year
ago
now,
and
this
gdpr
talks
a
lot
about
how
you
need
to
manage
access
to
user
data
for
the
purpose
of
privacy,
which
is
pretty
important
and
being
able
to
tell
a
coherent
story
around
data
access
for
you
for
your
user
data
or
for
any
data.
For
that
matter.
It
comes
down
to
having
a
base,
that's
rooted
in
a
strong
identity,
because
if
you
don't
have
that
it
becomes
very
hard
to
tell
who's
actually
controlling
data
who's
actually
accessing
that
data.
B
Sarbanes
oxley,
if
you're
not
familiar,
is
a
regulation
in
the
u.s.
that
applies
to
publicly
traded
companies
and
it's
very
similar.
But
the
data
story
is
more
around
financial
data.
That's
super
important
there.
Beyond
the
compliance
story.
There's
your
trust
and
your
security
model
of
your
of
your
systems
being
able
to
reduce
the
assumptions
of
your
system.
Behavior
is
very
important
for
developing
what
we
call
it
a
zero
trust
network.
B
It's
it
sort
of
been
the
common
practice
that
you
know
inside
your
secure
perimeter.
You
expect
that
things
behave
correctly,
but
this
expectation,
as
we've
started,
to
see
it's
not
always
accurate
and
threats
come
from
outside
your
your
secure
perimeter,
but
also
often
inside,
and
so
what
this
gets
you
is.
You
can
reduce
the
risk
of
unauthorized
access
in
that
case
of
a
compromise,
but
you
can
also
reduce
the
risk
of
bad
configuration.
B
It's
not
common
for
developers
to
to
test
against
production
and
very
kind
of
I'll,
say
I
won't,
say
unsafe,
but
it's
it's
a
bit
dubious.
The
way
that
a
lot
of
developers
these
days
talk
to
production
and
so
being
able
to
have
identity
that
can
can
control
access
to
your
data.
You
know
that
Deb
isn't
talking
to
fraud
or
if
it
is,
maybe
it's
only
has
access
to
to
read
certain
types
of
data.
So
with
that,
just
to
give
you
the
background
of
why
identity
is
important
to
us
to
do.
B
A
Sure
so
Tyler's
introduced
us
to
the
scale
of
the
identity
problem
in
uber
and
now
I'm,
going
to
talk
about
solutions
for
workload,
identity
that
we're
working
on
it
at
site,
l
and
Tyler's,
working
on
at
uber
and
those
solutions
are
called
spiffy
and
spire.
So
spiffy
is
this:
is
the
spec
for
a
way
to
do
workload,
identity
and
authentication
and
spire
is
an
implementation
of
that
spec?
Oh
I,
just
thought
louder.
So
spire
aspires
the
codes.
A
Spiffy
is
a
description
of
how
to
do
workload,
identity
in
a
particular
way
and
before
I
get
into
that.
We're
just
gonna
talk
for
a
moment
about
what
makes
a
really
good
workload
identity,
because
it's
important
to
really
think
about
what
makes
a
secure
identity
before
we
start
implementing
identities
all
over
the
place.
So
this
is
this
is
an
example
of
a
not
very
good
identity.
It's
it's
a
name
tag
which
could
have
any
name.
You
know
you
write
it
on
your
nametag
and
it's
just
a
first
name.
A
A
The
reason
we
would
rely
on
on
a
government
ID
is
first
of
all
it
has
some
unique
information
in
it.
So
it's
unique
to
to
a
person
which
is
the
the
unit
that
matters
each
person
has
has
one
identity
and
each
identity
corresponds
to
one
person
it's
stable
over
time.
You
can't
you
can't
change
your
government
ID
every
single
day.
No.
A
Identity
and
the
identity
comes
from
some
kind
of
source
that
we
trust
in.
For
a
human
identity,
it
might
be
the
government
because
we
we
believe
the
government
isn't
going
to
make
a
whole
bunch
of
fake
IDs,
but
for
an
organization
it
might
be
some
unit
within
that
organization.
That
has
the
power
to
grant
to
grant
identities
to
things.
A
So
how
do
we
implement
this
for
workloads?
Well?
First
of
all,
it
starts
with
an
ID
which
looks
a
lot
like
a
URL.
It's
called
the
spiffy
ID.
It
looks
like
a
URL,
it's
not
a
URL,
that's
not
a
resolvable
hostname
necessarily,
but
that
that
part
in
green
in
neon
green
is
a
it's
called
a
trust
domain.
So
it
could
be
the
domain
name
of
your
organization.
A
It
doesn't
have
to
be,
but
everything
under
that
trust
domain
is
a
single
unit
of
trust
that
is
traced
back
to
someone
granting
those
identities,
and
then
everything
in
blue,
that's
user.
That's
essentially
a
user-defined
identity
for
that
workload.
So
in
this
case
you
could
have
slash
billing,
slash
payments
that
there's
no
semantic
meaning
to
the
directory
structure
or
anything
like
that.
It's
just
a
name
that
makes
sense
for
your
organization
and
then
you
can
encode
these
IDs
in
two
different
ways.
You
can
encode
them
as
an
x.509
certificate
or
JWT
token.
A
If
you
want
these
to
be
interoperable,
so
we
have
this
thing
called
the
workload
api
that
gives
you
a
way
to
first
of
all,
get
your
own
identity
document
and
second
get
the
information,
that's
necessary
to
check
other
identity
documents
so
you're
running
in
in
concert
with
3,000
other
services.
You
need
to
first
of
all,
if
you're
one
of
those
services,
you
need
to
get
your
own
identity
and
then
you
need
to
get
the
information
that's
necessary
to
validate
any
of
those
other
identities.
The
public
key
information
to
be
able
to
do
that.
A
No
we're!
Just
gonna
stop
for
here
for
a
second.
This
is
all
about
authentication,
spiffy
and
spire,
and
everything
we're
talking
about
today-
we're
not
actually
making
a
decision
about
whether
one
service
is
allowed
to
talk
to
another
service,
we're
just
providing
the
information
to
the
service
of
who
they're
definitely
talking
to,
and
then
it's
up
to
you
to
figure
out
some
kind
of
out
the
authorization
that
makes
sense
your
organization.
A
So
maybe
the
users
database
can
only
talk
to
the
billing
database
and
it's
just
hard
coded.
That's
that's
one
way
to
do
it
or
you
could
have
a
proxy,
a
smart
proxy
that
maybe
is
talking
to
Hopa
and
making
some
kind
of
complicated
authorization
decision,
and
there
are
all
kinds
of
examples
of
different
ways
to
do
that.
But
that's
not
part
of
the
specification,
so
spiffy
is
seeing
a
little
bit
of
adoption,
pretty
rapid
at
doctrine.
A
Actually,
it
launched
officially
in
December
2017,
at
least
according
to
the
information
I
found,
and
he
was
accepted
into
the
CNC
F
sandbox
and
in
2018.
It's
already
been
integrated
into
a
number
of
different
open-source
projects,
or
at
least
is
being
worked
on
in
a
number
of
different
open-source
projects,
and
a
number
of
companies
are
working
on
deploying
it,
including
uber.
A
Now
we're
gonna
move
on
to
spire
spire.
Remember
is
the
reference
implementation
of
spiffy
so
someday.
Hopefully
there
will
be
many
different,
interoperable
implementations
of
spiffy
right
now.
Spire
is
the
one
that
we're
working
on
and
is
the
most
complete
implementation
of
spiffy
and
some
github.
You
can
check
it
out.
There's
a
link
at
the
end
of
this
presentation.
If
you
download
the
slides
and
what
spire
does
is,
it
has
a
number
of
different
kinds
of
plugins
that
will
all
talk
to
your
infrastructure
and
we
call
that
attestation.
A
We
talk
to
your
infrastructure
and
then
give
you
an
identity,
that's
based
on
that
infrastructure
and
I'll
go
into
what
that
means
in
a
second.
There
are
other
ways
you
could
do
that
that
would
might
also
work
with
spiffy
that
wouldn't
involve
attestation.
You
might
not
have
to
talk
to
the
infrastructure,
you
might
do
something
totally
different,
so
this
is.
This
is
just
gonna,
be
a
very
quick
architecture,
diagram
example:
let's
imagine
we
have
something
running
inside
kubernetes
and
that's
inside
AWS,
so
we
want.
A
A
C
A
A
Now
here
are
some
goals
of
the
spire
project.
These
are
these
are
all
in
progress.
That's
all
it's
all
work
in
progress,
but
it
should
be
fully
automated.
So
no
one
is
known
as
deploying
certificates
manually
in
this
whole
system,
there's
minimal
knowledge.
So
if
any
part
of
the
system
is
compromised,
hopefully
nothing
leaks
out
that
we
don't
absolutely
need
in
that
component.
A
So
one
of
the
major
areas
that
the
spire
contributors
are
working
on
is
differ
ways
to
make
it
more
reliable,
including
high
availability,
nested
spire
servers,
so
you
could
have
one
master,
spire
server
and
many
sub
spire
servers
in
order
to
handle
different
trust,
amines
and
Federation
between
spire
servers.
So
you
could
have
two
different
spire
servers
that
are
loosely
coupled
and
they
share
some
information,
but
not
everything,
and
that
way,
if
one
goes
down,
it
doesn't
necessarily
affect
your
entire
system.
B
B
Maybe
maybe
not
I
want
to
tell
the
story
about
a
system
that
starts
rather
small
and
simple
and
as
the
business
requirements
change
so
do
the
system
and
as
as
that
system,
evolves
sort
of
as
the
the
data
model
and
the
data
access,
and
as
that
data
access
changes,
sort
of
the
policies
and
the
security
policies
need
to
evolve.
What
identity
might
allow
you
to
do
is
to
evolve
more
quickly
with
that
security
policy.
So
we
start
in
the
beginning
in
the
early
stages
of
a
system.
It's
so
simple!
B
You
have
a
user's
database
that
might
include
riders
and
drivers.
You
have
a
couple
services
that
need
to
talk
to
it.
We've
got
a
matching
service
that
might
pair
a
rider
with
the
driver.
You've
got
a
trip
service
that
might
manage
the
lifecycle
of
a
trip
from
beginning
to
end
and
track
GPS
points
or
something
like
that,
and
you've
got
a
payment
service
that
might
charge
the
rider
and
pay
the
driver.
They
need
access
to
your
data
store.
B
So
in
the
beginning
the
simplest
solution
seems
to
be
the
tried-and-true,
username
and
password
that
seems
to
work
now.
What
do
you
do?
Do
you?
Do
you
have
a
unique
username
and
password
for
each
service
or
do
you
share
them?
Well,
there's
trade-offs
and
the
trade-offs
might
depend
on
on
what
your
your
business
wants
to
do.
If
you're
sharing
those
credentials,
man,
that's
pretty
it's
pretty
easy
on
your
service
owners.
B
A
number
of
a
credential
starts
to
scale
with
the
number
of
edges
in
your
in
your
the
graph
of
services
to
data
stores
and
not
with
the
number
of
nodes.
So
this
very
quickly
becomes
an
intractable
problem
right.
So
what
do
you
do?
You're
in
a
you
know
your
company's
growing
very
fast.
You
decide
to
implement
an
API
gateway.
B
Look
at
that.
You
only
have
one
credential.
Now,
that's
a
that's
a
very
tractable
problem.
All
the
folks
who
were
writing
garbage
queries
that
we're
making
your
database
fall
over.
You
can
now
limit
them
at
the
away.
You
you've
solved
your
instrumentation
problem.
Things
are
looking
quite
good
right.
All
the
access
is
coming
through
a
single
gateway.
B
You
can
even
start
to
fine-tune
your
your
ACLs
so
that
the
trip
service
can
access
payment
data,
but
you
now
have
a
new
problem,
which
is
how
do
all
these
services
authenticate
to
the
Gateway?
You
could
provision
them
all
with
public
keys,
but
that's
a
rather
that's
rather
difficult
task
when
you're
in
hyper-growth.
B
So
someone
suggests
well
we're
inside
our
own,
secure
perimeter.
Why
don't
we
use
the
name
tag
approach
that
seems
fine
every
service
could
just
say,
am
the
matching
service
and
we're
in
our
secure
perimeter,
and
you
know
we
have
bastions
and
all
that.
So
maybe
this
is
fine
and
it
seems
fine
right
until
you
start
to
realize
that
your
secure
perimeter
is
no
longer
really
a
secure
perimeter
number
one
either
because
you
started
to
see
insider
threats
because
the
number
of
services
have
grown
so
large.
B
You
really
can't
monitor
every
little
thing
or
you
go
through
a
migration
later
on.
You
know
you
realize
that
your
first
API
gateway
didn't
scale
so
well,
or
maybe
you've
run
out
of
compute
and
you
need
to
move
to
a
new
cloud
or
new
region
or
what
have
you?
But
these,
this
type
of
thing
is
not
uncommon.
B
B
What
happens
you
realize
you
got
a
missed
configuration
somewhere
or
perhaps
I
mean
you
know
as
bad
as
it
could
be.
You
know,
maybe
you
have
a
bad
actor
somewhere
I'd
like
to
prefer
a
bad
configuration,
because
that's
more
common
in
my
experience,
so
you
got
a
service
out
there.
That's
Santa
the
Panther,
a
payment
service,
but
they're,
not
the
payments,
and
this
kind
of
thing
can
be
very
hard
to
track
very
hard
to
track
down
to
solve.
So
what
does
spiffy
inspired?
Get
you?
B
Now
you
don't
have
to
worry
about,
assuming
that
your
services
are
acting
the
right
way.
You
have
the
proof,
you
have
cryptographic
tokens
or
you
have
cryptographic,
documents
that
prove
identity
and
so
by
reducing
the
amount
of
trust
in
your
system.
You've
reduced
your
assumption
of
behavior.
Now
that's
a
good
thing
for
the
reliability
of
your
system
and
the
security
of
your
data,
so
we're
Zuber
at
right.
Now,
we've
we've
progressed
quite
a
bit
from
from
the
early
days
right
now.
B
What
this
cloud
native
approach
allows
us
to
do
is
that,
regardless
of
the
infrastructure,
regardless
of
where
these
applications
are
running,
this
identity
acts
as
a
bridge
between
those
things.
So
this
is
very
powerful,
so
we
talked
a
lot
about
why
we
decided
to
implement
identity
at
uber.
I
wish
I
had
more
time
for
exactly
how,
but
what
we're
hoping
to
be
able
to
do
at
the
next
cube
con
is
actually
go
through
some
of
the
lessons
we've
learned
from
working
with
spiffy
over
the
last
year
and
deploying
spire
at
scale.
A
Tyler-
and
this
is
this-
is
our
last
slide
if
you
take
away
anything
from
this
talk,
this
is
what
we'd
like
you
to
take
away.
If
you
have
spiffy
inspire
that
enables
you
to
do
workload,
authentication,
and
if
you
can
do
workload,
nothing,
you
can
do
authorization.
There
are
lots
of
different
ways
to
do
that,
and
once
you
have
authorization,
your
services,
your
data
are
safe
and
you
can
check
out
spiffy
I/o
to
find
out
more
and
now
we
have
some
time
for
questions.