►
Description
Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Scaling Telepresence Across Your Organization - Abhay Saxena, Datawire
Adopting Kubernetes often forces tradeoffs related to your cloud native software development lifecycle, particularly in relation to the inner development and debugging loops. End-to-end verification of a system consisting of 10+ microservices is simply not possible, so many developers use the CNCF-hosted Telepresence two-way proxy to solve this problem. This works very well, individually. But how do you scale these strategies across a whole organization? Learn from the experiences of the primary Telepresence maintainer: - Watch a demonstration of how Telepresence swap deployment does a good job for one developer who owns a cluster - Learn how this can be scaled with multiple clusters and namespaces, and understand the tradeoffs of this approach - Explore how the new Telepresence intercept enables a larger team to work together without needing multiple clusters and namespaces
https://sched.co/Zetp
A
A
Not
so
long
ago
we
were
developing
our
first
big
application
in
kubernetes.
Datawire
was
doing
this
for
the
first
time
and
we
followed
the
typical
journey
that
any
company
might.
When
we
started
developing
this
application
right,
we
wrote
some
microservices.
We
figured
out
how
to
get
them
to
talk
to
each
other,
how
to
get
data
in
from
the
outside
etcetera.
A
So
we
ran
into
this
problem
pretty
early
on
where
we
want
to
answer
this
basic
question.
I
have
this
piece
of
code.
It
does
something,
and
I
want
to
change
that
code
and
see
what
happens
right.
What
happens
if
I
change
this
code,
and
so
we
would
do
that.
We
would
edit
the
code
and
follow
the
process
that
you
use
to
get
this
code
running
in
kubernetes
right.
So
we
build
the
code.
Then
we
build
our
container.
A
Then
we
push
the
container
to
a
container
registry,
and
then
we
tell
kubernetes
to
turn
the
pods
over
so
that
the
new
image
gets
pulled
down
into
the
kubernetes
cluster
and
the
new
pod
spin
up,
and
then
we
can
start
sending
inputs
to
our
application
and
see
what
the
new
code
does
doing
this.
We
realize
the
first
problem
that
we
see
when
using
kubernetes
the
developer
loop
right.
The
developer
loop
is
this
idea
that
you
want
to
be
able
to
change
your
code
and
see
what
happens
for
normal
software.
A
Things
are
a
little
more
complicated.
You
have
to
take
that
executable,
put
into
a
container
image
and
then
somehow
get
kubernetes
to
load
that
container
image,
and
that
typically
means
pushing
it
into
a
container
registry
and
then
bringing
it
back
down
into
your
kubernetes
cluster,
where
it
can
then
run
continue
our
story.
The
next
question
we
would
often
want
to
answer
while
developing
the
microservices
that
comprise
our
application
is
well,
I
made
a
change
and
that
didn't
do
what
I
expected
it
to
do.
A
That
didn't
achieve
the
goal
I
wanted,
or
now
it
led
to
a
crash
or
the
behavior
is
completely
different
from
what
I
had
predicted
or
it's
just
subtly
different
or
in
summary.
Why
didn't
that
work?
The
way
I
expected
it
to
software
is
opaque
right.
Software
writing.
Software
is
hard
and
debugging
software
is
even
harder
than
that
software
doesn't
give
you
access
to
its
internals
unless
you
use
specialized
tools
to
look
inside
like
a
debugger
or
tracing
system
or
profiler,
etc,
and
so
being
software
engineers
over
the
over
the
decades.
A
We
have
developed
these
tools
to.
Let
us
look
inside
running
software
to
try
to
understand
what's
going
on,
but
when
your
software
is
running
in
a
kubernetes
cluster,
it's
far
away,
it
is
the
op.
The
opacity
of
your
software
is
magnified
by
the
fact
that
it's
not
even
running
on
your
own
computer
that
you
using
to
write
and
debug
the
software
it's
running
somewhere
else,
and
so
the
net
effect
is
you're
sort
of
stuck
with
using
coupe
control
logs
and
adding
effectively
adding
print
statements
to
debug
your
code.
A
A
In
essence,
what
telepresence
does
is
replace
the
deployment
or
the
pods
created
by
the
deployment
in
your
kubernetes
cluster
with
a
proxy
pod
and
then
modifies
your
local
machine,
your
laptop's
networking,
so
that
it's
directly
connected
to
that
proxy
pod.
The
net
effect
is
when
other
microservices
in
your
cluster
connect
to
the
pods
for
this
deployment.
They
instead
connect
to
the
proxy
pod
and
those
requests
come
down
to
your
laptop
and
the
software
running
on
your
laptop
can
respond
to
those
requests.
A
Let's
take
a
look
at
a
quick
demo,
so,
on
the
right
hand
screen
you
can
see.
I
have
the
hello
world
service
checked
out
and
there's
a
copy
of
the
full
text
of
the
service
right
in
front
of
you,
including
the
message
that
will
print
out
this
hello
world,
of
course,
and
on
the
left.
You
can
see
that
control
shows
you
that
hello
world
has
been
deployed
into
the
cluster
as
hello.
A
A
Okay,
on
the
right
hand,
side
you
see.
My
editor
is
now
open
to
the
hello
service
and
I
will
change
the
contents
of
this
message
and
hit.
Save
and
flask
is
going
to
reload
the
file
and
now,
if
I
do
this
curl
once
more
boom
hello
kubecon,
I
have
live
edited
the
code
and
without
having
to
build
and
push
an
image.
We
can
see
the
change
right
from
the
cluster
and
finally,
let's
hit
ctrl
c
to
quit.
A
A
We
get
a
fast
development
loop
because
we
don't
have
to
push
and
pull
build
and
push
and
pull
images
for
every
change.
We
make
to
see
what
happens.
We
can
simply
run
the
code
on
our
own
machine
and
access
it
using
the
cluster,
as
we
might
in
any
other
case
and
see
what
happens,
we
can
very
quickly
make
a
change
and
try
it
in
the
context
of
the
entire
application,
as
opposed
to
trying
to
run
a
unit
test
of
just
our
own
microservice
and
secondly,
because
the
code
is
running
on
your
laptop
all.
A
The
tools
that
you're
used
to
using
from
your
days,
developing
a
normal
application
now
are
applicable
to
your
kubernetes
application
as
well.
So
in
other
words,
you
can
run
your
microservice
through
your
ide
and
step
through
each
individual
line
of
code
in
your
debugger.
Just
as
you
always
would
because
it's
running
on
your
laptop
and
yet
it
still
works,
just
as
it
would
if
it
were
running
in
the
cluster
because
it
has
network
access
to
and
from
the
cluster
telepresence
swap
deployment,
does
a
good
job
for
one
developer,
who
owns
a
cluster
now?
A
What
I
mean
by
for
one
developer
basically
comes
down
to
this
when
using
the
swap
deployment
workflow,
all
the
pods
for
your
deployment
have
been
replaced
by
the
proxy
pod
that
sends
connections
and
requests
down
to
your
laptop.
This
is
how
telepresence
works,
because
every
connection
and
every
request
to
the
microservice
you've
swapped
goes
down
to
your
laptop
effectively.
That
microservice
is
now
owned
by
the
one
developer.
Who's
running
telepresence.
A
Other
developers
can't
work
on
that
microservice.
At
the
same
time,
telepresence
swap
deployment
does
a
good
job
for
one
developer,
who
owns
a
cluster
when
I
say
who
owns
a
cluster,
what
I
really
mean
is,
who
owns
the
particular
instance
of
the
application?
That's
been
deployed
to
the
cluster,
so
maybe
you
can
deploy
your
application
into
a
single
namespace
and
there
are
other
deployments
of
that
application
in
other
namespaces.
A
A
So
in
practice,
once
you're
running
swap
deployment,
other
users
can't
use
the
application,
it's
simply
owned
by
you.
So
let's
see
what
that
means
in
practice,
let's
say
you're,
given
the
directive
to
debug
a
problem
in
a
particular
microservice,
you
think
to
yourself.
Well,
this
is
fine.
I
can
step
through
this
code
in
telepresence
in
my
debugger,
using
telepresence
and
we'll
figure
this
out
and
we'll
get
it
debugged,
nice
and
quickly.
A
So
how
do
you
proceed
to
do
this?
Well,
one
approach
is,
you
can
say
well
I'll,
simply:
grab
the
staging
cluster
or
I'll
simply
grab
the
developer,
staging
installation
this
instance
of
the
application.
If
anybody
else
wants
to
use
the
application
at
the
same
time,
well,
they'll
just
have
to
wait,
and
you
do
this.
You
do
whatever
you
talk
to
your
fellow
developers
and
you
say:
hey
I'm
using
the
staging
application
right
now,
so
please
don't
do
anything
there,
and
then
you
go
off
and
run
telepresence.
A
You
swap
deployment
and
you
debug
your
problem
well
at
a
small
enough
scale.
This
is
fine.
You
might
be
able
to
debug
your
problem
in
an
hour
or
two
and
the
other
developers
can
keep
working,
but
they
won't
be
able
to
use
the
staging
cluster
to
try
out
their
changes
because
they
won't
be
able
to
use
the
application
another
way
to
approach.
This
is
to
say,
let
me
spin
up
the
app
in
a
brand
new
cluster.
Well,
let
me
spin
up
the
app
in
a
brand
new
namespace
or,
however,
that
might
work
for
your
application.
A
In
essence,
you're
saying
let
me
create
an
entire
duplicate
instance
of
the
application.
Just
so
I
can
own
it
and
I
can
run
swap
deployment
and
do
what
I
need
to
do
so.
This
is
fine
if
your
application
is
small,
if
your
application
is
easy
to
spin
up,
if
it's
not
using
resources
that
are
not
just
microservices
in
the
cluster
but
say
persistent
databases
and
things
like
that.
Basically,
if
it's
easy
to
spin
up
another
instance
of
the
app
and
it's
not
too
expensive
to
do
so
then
go
ahead.
A
And
so
how
do
you
do
that?
Well,
you
start
by
taking
the
microservice
you
want
to
debug
and
you
replicate
it.
You
make
a
copy
of
the
deployment,
and
now
you
can
use
telepresence
on
that
copy
because
well
nothing's
going
to
be
talking
to
it,
but
then
well,
how
do
you
use
this
copy?
How
do
you
test
this
copy?
A
Okay,
so
there's
some
sequence
of
of
microservices
that
a
request
might
go
through
where
it
eventually
reaches
the
microservice
you're
debugging,
so
just
make
a
copy
of
all
of
those
and
construct
an
entirely
new
chain
of
connections
such
that
you
can
insert
a
request
into
your
application
or
this
sort
of
partial
copy
of
your
application,
and
that
request
will
only
go
through
this
partial
copy
of
sequence
of
microservices
and
eventually
reach
the
copy
of
the
deployment
that
you
care
about.
So
you
can
debug.
A
A
What
I
really
want
is
some
agent
standing
in
front
of
the
microservice
that
I'm
trying
to
debug
basically
saying
is
this
request?
One
of
my
requests
that
I'm
sending
to
try
to
debug
this
code
in
that
case,
send
it
down
to
my
laptop.
Is
this
request?
Something
else?
Well,
then,
just
send
it
to
the
deployment
as
it
continues
to
run
in
the
in
the
cluster
and
it'll
be
business
as
usual
for
all
other
users.
So
let's
think
about
that,
this
agent,
somehow
somehow
has
to
figure
out
the
nature
of
a
given
connection
or
request.
A
Is
this
connection
or
request
mine
for
my
debugging
process
or
something
else
to
do
this?
We're
going
to
need
to
impose
some
restrictions
and
requirements
on
our
problem
domain
right?
This
is
too
broad
to
solve.
Generally,
the
theoretical
agent
that
sits
in
front
of
your
microservice
needs
to
be
able
to
make
decisions
about
the
requests
coming
into
the
microservice
so
that
it
can
do
something
smart.
A
So
if
this
agent
needs
to
understand
the
requests
going
into
the
microservice,
we
need
to
restrict
what
those
requests
can
be.
So,
let's
focus
ourselves
on
the
htp
family
of
protocols,
and
so
that
includes
http
2
and,
if
you're
willing
to
let
your
agent
terminate
tls
for
you,
then
it
can
even
include
https
because
again,
if
this
agent
needs
to
understand
the
request,
it
of
course
has
to
be
able
to
decrypt
it.
A
So
if
we
narrow
ourselves
to
just
http
requests,
we
now
have
the
possibility
for
an
agent
to
stand
in
front
of
your
microservice
understand
the
requests
that
are
coming
in
and
do
some
sort
of
redirecting
in
an
intelligent
manner,
and
that
brings
us
to
our
requirement
for
the
agent
to
decide
that
a
particular
request
is
one
of
my
requests
that
I
want
redirected
we're
going
to
need
to
require
a
special
header
that
tells
the
agent.
This
is
a
debugging
request.
It
needs
to
be
sent
down
to
the
laptop
now.
A
A
When
I
insert
that
initial
request,
I
will
include
the
header
that
I
need
to
mark
this
as
a
debugging
request,
and
then
every
intermediate
microservice
must
pass
that
header
along
unchanged,
so
that
when
the
request
finally
reaches
the
microservice,
I'm
debugging,
the
header
is
still
present
and
the
agent
can
act
on
it.
This
is
not
just
a
simple
curl
command
line
flag.
We
have
to
be
sure
that
every
microservice
in
the
chain
passes
along
this
header.
A
A
A
First
of
all,
everyone
can
continue
to
use
the
application,
as
is
without
any
disruption
when
a
developer
adds
an
intercept
to
the
agent.
Only
the
requests
marked
with
the
debug
header
will
flow
down
to
the
developer's
laptop.
All
other
requests
will
continue
to
flow
to
the
deployment
running
in
the
cluster.
A
Second,
because
the
agent
has
the
ability
to
make
decisions
about
individual
requests
based
on
the
headers.
It
becomes
possible
for
multiple
developers
to
intercept
the
same
service
developer.
Jane,
can
add
a
new
feature,
developer
bob
can
add
instrumentation
or
metrics,
and
I
can
go
in
and
fix
the
bug
I
introduced
in
the
last
release.
A
A
Service
preview
is
a
part
of
the
ambassador
edge
stack
and
it
is
available
right
now
at
get
ambassador
dot.
Io
now
service
preview
uses
envoy
proxy
to
process
and
redirect
http
requests.
As
you
may
know,
envoy
proxy
is
very
good,
very
fast
and
very
stable
service
preview
does
not
do
everything
that
telepresence
does.
A
It
doesn't
offer
some
of
the
advanced
features,
such
as
access
to
config
maps
and
secrets
and
other
file
system
volume
mounts,
nor
does
it
offer
environment
variables,
but
it
does
offer
the
core
features
of
letting
the
code
running
on
your
laptop
talk
to
other
microservices
in
the
cluster
and
letting
other
microservices
in
the
cluster.
Talk
to
the
code
running
on
your
laptop
now,
with
the
additional
feature
of
only
a
subset
of
requests
coming
down
to
your
laptop
based
on
the
intercepts
that
you've
set
up
service
preview.
A
A
A
I
have
looked
I've
seen
that
hello
is
available
for
intercept.
I
have
set
up
an
intercept
and
the
header
I'm
matching
is
the
dev
header
and
it
needs
to
be
set
to
this
value
ark
3,
and
when
that
header
value
combination
is
seen,
the
agent
is
going
to
redirect
that
traffic
to
my
own
machine
on
port
8000,
and
then
you
can
see
that
I
have
started
the
application
on
my
own
machine
and
it
is
running,
and
these
curls,
which
I
can
run
repeatedly,
do
not
cause
anything
to
happen
there.