►
From YouTube: Securing Application Telemetry & Tracing with SPIFFE and Envoy - Sabree Blackmon, Docker
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Securing Application Telemetry & Tracing with SPIFFE and Envoy - Sabree Blackmon, Docker
Application telemetry, such as Prometheus metrics, distributed logs, and tracing, offer a surplus of information on how an app works, how it's performing, what it’s communicating with and how. However, even when these streams do not contain PII, this information can be invaluable to malicious actors in the days of highly distributed systems. In this talk, we will demonstrate how to use Envoy proxy and SPIRE to protect your telemetry endpoints, in both push and pull use cases, utilizing fluentd, Prometheus, and OpenTracing & Jaeger.
To learn more: https://sched.co/GrXa
A
A
A
Through
these
slides.
This
is
going
to
be
more
of
a
towel,
but
the
problem
is
you:
should
care
about
this
thing?
Kind
of
talk
more
than
an
instructional
type
of
tutorial
I
will
link
to
the
git
lab
github
to
the
demo
in
case
you
want
to
run
this
at
home
and
dig
into
the
configuration
and
stuff
like
that,
a
bit
more,
it's
pretty
straightforward
and
there
are
some
some
instructions
on
how
to
run
it.
Hopefully
this
will
work.
A
It
doesn't
work,
alright
hope,
you'll.
So,
let's
get
started
after
you've
kind
of
already
seen
our
apps
are
generating
a
lot
of
data
right.
This
isn't
a
new
problem,
but
as
we
sort
of
reach
into
these
microservices
paradigms,
we
are
generating
even
more
we're
making
the
burden
a
little
bit
harder
on
ourselves.
A
But
what
happens
when
this
information
leaks
right?
We
have
examples
with
us
recently.
This
could
be
to
the
public.
It
could
be
potentially
captured
by
bad
actors
to
this
damaged
reputation.
Or
can
this
information
be
valuable
for
your
competitors,
so
this
is
another
tweet
by
report
URI,
gentlemen,
it's
an
interesting
piece
on
how
get
github
and
Twitter
recently
have
been
leaking
past
where
information
through
their
logs.
A
A
This
is
so
common
that
there
is
a
cwe
written
out
there
for
these
concerns,
and
if
you
follow
this,
if
they
you
know
had
followed
this
cwe,
they
wouldn't
have
leaked
password
data.
But
this
isn't
an
extremely
common
concern.
There's
actually
quite
a
bit
of
information
there
for
anyone
who
wants
to
look
through
it.
It's
very
helpful.
It
even
goes
through
some
mitigating
factors
and
some
of
the
statistical
data
out
there.
A
So
we
have
you
know
a
pretty
large
set
of
absolutely
tools
available
available
to
us
in
the
CNC
F
Prometheus,
open,
tracing,
open
consensus,
I
think
is
now
a
thing
as
well,
but
there
isn't
a
whole
lot
of
guidance
on
how
to
protect
this
data
in
transit.
That
assumes
that
you
were
even
sort
of
hip
to
the
fact
that
you
probably
should
protect
this
information
and
transfer
in
the
first
place,
and
some
of
the
information
really
does
to
be
protected
at
rest,
but
that's
a
different
talk.
A
That
complexity
is
very
interesting,
feel
free
to
poke
me
afterwards.
Talk
about
that
so
now
that
we
kind
of
understand
why
it's
important,
we
do
need
to
treat
these
sorts
of
data
a
little
bit
differently.
So
what
are
some
of
these
types
of
observational
data?
So
we
have,
you
know,
I,
think
the
most
obvious
one
that
we
think
about
is
time-series,
metrics,
right,
Prometheus,
and
things
like
that.
So
this
is
often
what
we
prefer
to
buy
telemetry.
A
And
so
this
is
often
used
for
performance,
metrics
and
things
that
you
know
if
you're
using
a
Java,
Runtime
or,
for
instance,
things
that
you
can
often
get
for
free
memory,
usage,
CPU
usage,
things
like
that,
and
then
you
know.
This
is
also
use
as
useful
in
determining
performance
bottlenecks
and
things
like
that.
A
But
you
can
also
be
useful
in
determining
how
an
app
behaves
in
response
to
a
certain
API
event
or
a
set
of
events,
or
maybe
the
time
of
day.
You
know
the
load
increases
for
some
reason
and
we're
not
entirely
sure
and
then
there's
another
set
of
another
sort
of
esoteric
use
of
time
series
data
and
that
we
can
use
it
to
capture
any
sort
of
statistical
information
that
an
application
thinks
it's
important
in
our
app
example,
app
we're
using
a
time
series
scraped
for
access
and
policy
statistics.
A
A
Some
of
those
logs
is
relevant
for
debugging
purposes,
but
some
of
this
log
is
also
required
for
all
the
purposes.
If
you're
you
know
facing
compliance,
and
things
like
you
may
have
to
audit
access
or
audit
certain
transactions
or
database
reads
or
things
like
that,
and
then.
Lastly,
we
have
tracing.
A
There
are
other
types,
but
that's
all
we're
going
to
cover
today.
These
are
often
seen
as
an
extension
to
logging,
but
they're
really
not
that's
kind
of
fighting
words
in
some
places,
but
these
traces
are
a
little
bit
weird
they're,
a
combination
of
logging
but
they're
distributed
and
if
you've
never
used
something
or
look
at,
looked
at
a
trace
through,
say,
Jager
or
light
step,
or
something
like
that,
really
what
it
gives.
A
You
is
a
picture
of
a
single
API
request,
for
instance,
through
each
hop
of
your
back-end
system,
and
it
gets
information
from
each
one
of
those
hops
along
the
way,
probably
should
have
included
an
example
tracer.
That
would
be
helpful,
but
the
the
big
difference
here
between
traditional
logging
and
tracing
is
that,
because
each
parts
of
your
infrastructure
are
contributing
information
to
the
trace,
things
like
IP
addresses
or
host
information
can
be
added
and
exposed
to
that
trace.
A
Important
note,
if
you're
thinking
about
implementing
traces,
these
aren't
necessarily
the
best
place
to
add
extremely
detailed
application.
Specific
logging.
There
really
aren't
any
guarantees
of
information
or
detail
sanitation
across
applications,
and
so
each
hop
of
that
that
trace
potentially
on
how
it's
implemented.
You
can
see
all
the
information
from
all
the
previous
hops,
and
so
depending
on
your
data,
your
data
policy
that
might
be
a
no-go
and
the
very
too
long
didn't
read
version
of
this
is
their
art.
A
These
are
all
tools
that
are
great
for
debugging
and
operability
purposes,
but
there
are
important
application.
Security
and
information
security
concerns
around
all
of
this
data
and
the
unfortunate
truth
of
that.
The
potential
for
dangerous
information
leaks
increases
the
more
useful
we
make
this
data
right,
and
so,
as
we
add
more
detail
as
we
make
it
more
useful
for
engineers
and
our
operators,
it
actually
increases
the
value
of
this
information
and
so
from
I'm
security,
engineer
and
so
I
get
to
deal
with
the
Sundstrom
stuff.
A
A
So
this
goes
into
the
demo
portion.
However,
the
time
looking,
okay
and
so
because
I
intentionally
made
this
demo
a
little
bit
higher
level
I'm
not
going
to
get
into
too
much
details
about
how
this
stuff
is
implemented,
but
I
will
touch
on
some
important
points,
and
so
we're
going
to
look
into
a
case
study
of
this
really
small,
unrealistic
but
helpful
app.
It's
a
simple
secret
server
to
Longden
reversion.
It's
a
HTTP
API
in
front
of
EDD,
where
you
can
store
secrets,
it's
not
really
secret.
A
The
secrets
aren't
encrypted,
but
it's
good
enough
for
our
purposes
and
uses
the
simple,
username
and
login
password
password
login
and
it
has
a
token
based
API
access,
and
so
these
tokens
are
actually
stored
in
SCD
and
they
expire
after
fifteen
minutes
or
something
like
that
cool
and
so
excuse
me.
The
application
uses
Prometheus
metrics.
It
actually
creates
a
Prometheus
endpoint
around
some
interesting
sort
of
like
access
statistics,
the
number
of
valid
ganz
invalid
logins,
and
things
like
that
and,
let's
see
if
this
will
work
actually
I
will
just
do
this.
A
A
Here
we
have
successful
dog
encounters
on
successful,
walk
encounters
if
you're
familiar
with
Prometheus,
you
understand
counters
are
their
metrics,
that
you
can
increase,
but
you
can't
decrease,
and
so
throughout
the
lifetime
of
this
application
instance,
as
there
are
successful,
logins
or
unsuccessful
logins
or
as
a
secret
is
set
all
of
these
things
get
incremented
by
one.
That's
really
all
that
means
and
they're
exposed
through
a
slash,
metrics
endpoint
here
and
an
interesting
little
important
thing
to
look
at
here
is
that
the
metrics
endpoint
is
hosted
on
a
separate
HTTP
server
from
our
primary
API.
A
A
Some
HTTP
spamming
throughout
the
last
couple
of
hours
and
the
Prometheus
isn't
the
nicest
UI
in
the
world,
but
you
can
kind
of
see
over
time
if
this
were
a
more
realistic
example,
for
instance,
this
is
a
secret
set
access
denied
total.
You
would
expect
this
thing
to
increase
fairly
regularly
over
time.
If
there's
a
massive
spike,
you
can
create
an
alert
for
that
and
say
hey.
Something
is
going
on
so
a
little
bit
of
a
cleave,
but
in
a
more
realistic
example,
you
can
imagine
how
powerful
this
could
be
cool.
A
Cool-
and
so
it
also
in
implements
fluent
D
logging
and
so
a
little
bit
different
than
Prometheus
Prometheus
is
a
pull
service.
In
other
words,
we
have
to
create
an
endpoint
for
prometheus
to
hit
as
part
of
a
scrape.
Every
configurable
amount
of
time
fluent
day
is
a
little
bit
different.
We
actually
have
to
push
events
to
fluent
D
as
part
of
our
configuration,
and
so
this
example
app.
A
It
pushes
audible
events
like
access
events
and
system
errors
and
details,
but
as
such
valid
and
invalid
user
identities
are
exposed,
which
makes
it
easy
for
us,
and
if
we
want
to,
can
do
some
pretty
long,
interesting
log,
aggregation
and
analysis.
We
can
do
that,
but
it
does
make
the
contents
of
that
of
those
logs,
particularly
vulnerable,
so
I'm
going
to
go
into
kind
of
how
our
application
does
that
yeah.
That's
actually
right
above
here,
and
so
we
have.
A
This
audit
event
function
that
you
know
it's
used
in
quite
a
few
places
throughout
the
program,
including
here
we
know
you
can
see
when
we
verify
a
password.
We
create
a
server
event
if
it
the
day.
Log
in
is
invalid.
Password,
for
example,
nothing
terribly
terribly
interesting
here,
but
we
can
kind
of
see
what
that
looks
like
whoops
and
in
real
life,
and
so
this
is
the
sort
of
a
follow
up
via
the
fluency
logging
and
we
can
see
when
secrets
are
created.
We
can
see
when
secrets
are
fetched.
A
If
you
need
that
level
of
detail
for
auditing
purposes,
we
can
even
see
the
ID
of
the
session
tokens
that
are
created,
and
then
we
also
have
login
success
for
a
user
user
name
here
right.
You
can
imagine
an
engineer
wanting
to
know
what
user
logged
in,
but
now
we've
you
know,
we
verified
that
a
user
one
exists
right
and
that
can
be
potentially
useful
information.
A
Okay,
so
this
is
kind
of
like
a
high-level
overview
of
what
the
architecture
of
our
demo
looks
like.
We
have
three
instances
of
an
fcd
database,
our
simple
secret
server,
there's
their
load
balancing
or
anything
like
that.
The
only
important
things
to
note
here
that
we
are
using
envoy
as
a
sari
color
pattern
in
a
sidecar
pattern
to
create
TLS
connections
between
Prometheus
and
simple
secrets
and
or
fluency
and
simple
secrets.
A
The
arrows
here
are
important
because
we
can
actually
see
that
simple
secrets
is
pushing
information
to
fluent
D
and
Prometheus
is
pulling
information
from
sequencer
secrets
server.
If
you
ordered
you
something
like
an
open,
tracing
or
or
an
open
the
consensus
there,
they
also
have
push
in
pull
models
as
well.
I
think
Prometheus
also
has
a
push
model
now,
there's
an
aggregator
that
you
can
use
cool.
A
Here
in
our
example,
where
this
is
a
the
launch
of
the
spire
server,
that's
actually
issuing
this
beefy
IDs,
and
you
can
see
here
that
there's
50
IDs
being
issued
for
the
Prometheus
proxies
the
simple
secret
server
itself,
fluent
et
cetera,
et
cetera,
and
so
you
can
imagine
in
a
much
more
realistic
example
being
able
to
prove
that
all
of
the
points
of
communication
are
trusted
and
are
heard.
They
say
they
are,
can
be
extremely
valuable.
A
And,
as
I
said,
these
estimates
are
used
to
establish
TLS
connection
than
between
the
app
in
prometheus
and
fluent
deep
and
very
long
story
short.
These
apps
know
where
data
is
going
to
and
the
tooling
knows
what
app
instance
is
providing
the
telemetry,
and
so,
if
you're,
you
know
having
a
conversation
with
auditors-
and
you
know
they
need
to
know.
How
can
you
prove
that
this
access
information
is
correct?
A
Very,
very,
very
briefly,
because
it's
pretty
much
standard
envoy,
but
here's
sort
of
the
configuration
for
fluency
and
important
highlight
here
is
that
the
verify
subject,
alt
name
actually
contains
a
spec
for
the
ID
that
we're
expecting
fluent
dia
to
have,
and
so
envoy
is
using.
The
spiffy
ID
that
we
are
expecting
to
verify
of
the
other
end
is
indeed
fluent
D
so
before
it
pushes
any
sort
of
information.
Potentially
be
interesting
about
actors,
it
does
check.
A
I
want
to
have
enough
time
for
questions.
This
is
the
link
for
github
for
the
simple
secrets:
demo
there's
a
lot
of
meat
there
for
you
guys
to
dig
into
I.
Also
have
some
post
post
man
collections
in
there.
So
if
you
want
to
poke
around
with
the
API,
you
can
without
too
much
too
much
headache,
and
we
have
time
Wow
any
questions
on
that.
Yes,.
B
B
A
Right
right
and
so
I've
actually
there's
a
previous
demo.
That
actually
does
that,
because
we
were
using
the
baggage
itself
to
carry
spiffy,
IDs
right
and
I
know
there
is
a
the
question
was
with
open
tracing,
particularly
adding
you
know.
If
there's
a
is
there
a
danger
of
adding
sensitive
information
to
the
trace
itself?
If
the
contents
of
the
trace
isn't
carried
on
to
other
downstream
right.
A
Pretty
much
so
open
tracing
works
that
way
things
like
soap
can
or
a
little
bit
different
but
tracing
in
general.
If
you
can
ensure
that
the
connection
between
your
application
and
the
thing
that's
actually
assembling
the
trace,
if
you
know
confidently
what
aspects
of
that
information
is
only
going
to
you
know,
say
Yaeger
or
something
like
that,
then
you
can
make
the
determination
there.
A
A
A
A
A
You
need
that
information
in
order
to
make
that
alert,
but
once
that
information
has
been
sent
to
say
grey
log,
if
you
need
to
push
that
data
further
downstream
for
some
Kafka
job
or
something
like
data
warehousing,
then
you
could
do
that
as
part
of
that
workflow,
and
so
it
really
just
kind
of
depends
on
again
the
more
useful
we
make
of
information
the
more
valuable
becomes,
but
once
we
are
done
with
that
usefulness,
if
you
will,
then
you
can
further
sanitize
that
information
to
make
it
safer.
Basically,
any
other
questions,
yep.