►
From YouTube: From Network Engineer to K8s Developer: Lessons Learned via Telepresence - Peter ONeill, Ambassador
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
From Network Engineer to K8s Developer: Lessons Learned via Telepresence - Peter ONeill, Ambassador Labs
Kubernetes is every bit as complex as a globally distributed network, but all wrapped up in one cluster. From Ingresses, to load-balancers, to headless services, Kubernetes has more possible points of failure than an IP packet jetting around the world. Join in to hear a network engineer's journey into Kubernetes. As any network engineer would do, we will follow the networking path. We will start at the Kubernetes API, work our way through the services and connections, and eventually make our way into the pods and containers. Understanding how complex it is to reach backend services hosted in Kubernetes, it's surprising a tool like Telepresence hasn't come around sooner. Once you've experienced it, bridging your laptop's local network to a Kubernetes cluster makes sense. No spotty VPN tunnels or long-lived port-forwarding sessions. Just "telepresence connect" and start developing and debugging as if your laptop was in your cluster.
A
A
So,
as
I
said,
I'm
peter
o'neill.
You
can
find
me
just
about
everywhere
at
peter
o'neill.
Jr
tweet
me
connect
with
me
on
linkedin
I'd
love
to
hear
from
you
cool.
So,
let's
start
off
right,
simple
networking
path
here
once
upon
a
time,
websites,
networking
very
simple:
you
have
your
home
network
right,
connected
to
your
isp,
connected
to
some
unknown
number
of
middle
hops,
eventually
to
your
your
website's
isp
right,
very
simple
networking
path,
and
then
you
have
a
very
simple
troubleshooting
procedure.
A
Right
like
this
is
this
is
troubleshooting
before
cloud
native
right
like
so
you
have
your
your
simple.
Your
simple
website
hosted
on
a
public
ip,
probably
served
on
port
80
unencrypted
right.
Maybe
it's
a
simple
html
website
right
troubleshooting
consists
of.
Can
I
ping
this
house?
Does
netcat
say
that
the
port
is
open
and
available
to
connect?
To
does
nslookup
resolve
to
the
right
address.
A
Right,
like
is,
is
dns
working
for
the
site
right
and
then
your
typical,
your
typical
fix
for
this
is
either
you
fix
dns
or
you
trace
route
to
the
end
you
figure
out
which
hop
the
path
stops
at
and
then
you
call
your
isp
and
say:
hey
fix
this.
It's
broken,
I'm
a
network
engineer
and
I
can
see
exactly
where
the
path
broke
right,
so
very
simple
right.
Then
we,
then
we
move
into
the
cloud
native
application
right
now.
We
have
a
lot
more
things
right,
you're
hosted
in
a
big
cloud
provider.
A
You
might
have
hosted
dns,
you
might
have
some
elastic
ips
assigned
to
multiple
high
availability
load,
balancers
you're,
serving
on
443
you're,
doing
tls
termination
right,
you're,
passing
this
all
through
into
kubernetes,
which
then
might
hit
api
gateways
and
ingresses
going
through
a
service
message,
a
lot
going
on
now
right.
So
so
we
we
we've
increased
complexity,
quite
a
bit
right,
and
so
now
it's
not
as
simple
as
just
where
does
my
networking
path
stop
now
all
of
a
sudden?
It's?
A
Where
is
my
networking
path
getting
all
the
way
through
and
which
services
is
it
stopping
at
which
connections
are
breaking?
Which
things
are
not
working
as
expected?
So
the
the
troubleshooting
steps
here
are
not
as
simple
as
is
the
port
open
anymore.
Now
now
it's
like
which
port
on
which
service
in
which
place
right.
There
is
a
very
large
complex
system
now
and
so
once
again
kind
of
looking
back
at
the
network
path
for
the
cloud
native
application
right.
A
This
is
kind
of
the
first
first
half
of
the
journey
here
home
network
isp,
and
then
we
can
even
simplify
this
and
say
we
we're
paying
for
a
direct
connect
for
our
application,
connected
to
a
the
cloud
network
to
our
load,
balancers
to
our
kubernetes
service
right
and
then,
but
then
that's
just
the
first
half
of
the
of
the
networking
path.
Right
or
sorry.
I
jumped
ahead
of
myself
here
and
so
so
so
so
this
this
is
first
half
of
the
the
networking
path
right.
A
But
let's,
let's
cover
this
in
a
little
more
detail
right.
So
this
networking
path
to
the
cloud
native
application
right
and
so
so,
as
I
said,
right
home,
network
isp
direct
connect.
But
this
is
connecting
you
to
something
called
write.
Your
cloud
provider's
edge
network
right
this
edge
network
is,
is
where
a
lot
goes
on
right
and
we'll
cover
that
in
a
little
bit
more
detail
in
a
second
and
then
so
from
that
edge
network,
you're
gonna
be
passed
into
a
cloud
region.
A
So,
let's,
let's
cover
this
cloud
edge
thing
a
little
bit
in
a
little
more
detail
right
so
with
the
introduction
of
of
of
this
cloud
edge
right,
like
you're,
able
to
bring
your
application
closer
to
to
to
yourself
or
to
your
customers
to
your
end
users,
whatever
wherever
you're
trying
to
access
this
thing
right
and
so
with
this,
what
what
connecting
to
your
cloud's
edge
gives.
You
is
right.
A
What
does
a
network
engineer
do
at
a
small
startup
right
like
if
you're,
not
if
you're,
not
consistently
troubleshooting
broken,
broken
hops
and
nodes
right,
and
so
this
is
when
I
figured,
I
would
join
over
to
the
dark
side
and
become
the
kubernetes
developer
right
and
you
might
ask:
why?
Is
it
the
dark
side?
Have
you
ever
looked
at
the
sith
logo
and
compared
it
to
the
kubernetes
logo?
A
A
So
let's
look
at
this.
Let's
look
at
this
networking
path
to
the
cloud
application,
one
more
time
right,
and
so,
let's,
let's
like
I
said
this-
is
the
first
half
of
the
equation
right.
Getting
that
direct
connect
right
to
your
cloud
provider
and
getting
into
your
getting
to
your
cloud,
your
kubernetes
cluster
right,
but
that's
the
first
half,
then
we
have
the
second
half
right.
This
is
this
is
where
the
networking
path
continues,
but
then
it
is
still,
it
gets
more
complicated
right.
A
So
now
it's
just
it's
not
just
this
linear
path
right,
but
now
you
enter
your
kubernetes
cluster.
You
you're
at
your
kubernetes
api.
You
get
passed
to
maybe
an
api
gateway
which
is
which
is
handing
your
user
traffic
to
a
set
of
ingresses.
Those
ingresses
are
pointed
to
any
number
of
front-end
services.
A
Those
services
are
intertwined
into
a
mesh
with
back-end
services,
and
then
all
of
these
services
have
n
number
of
pods
running
n
number
of
containers
and
there's
a
lot
of
places
that
things
can
go
wrong
right
and
so
now
something
goes
wrong
right
now,
all
of
a
sudden
one
of
these
pods
are
one
of
these
containers.
One
of
these
things
fails.
A
So
what
do
you
do
right
now
sudden?
Is?
It
is
you're
looking
at
this
from
an
end
user
perspective,
you
have
no
idea,
what's
actually
happening
right,
like
you're
you're,
looking
at
a
lot
of
this
and
you're
saying
like
oh,
it's
just
not
working,
it's
kind
of
like
frustrating
right,
and
so
so
so
so
now,
right
now
we
have
to
do
some
network
plumbing.
Do
some
network
plumbing
right,
like
as
any
good
network
engineer?
A
Does
you
kind
of
just
start
at
one
end
fish
to
the
pipe
and
see
what's
happening
right
and
so
traditional
network
plumbing
right,
like
you,
you
kind
of
start
at
one
end
go
to
the
other
right.
These
are
the
tools
of
the
trade
trace
route,
mtr
ping,
nslookup,
netcat
and
ssh.
Right
like
these.
Are
these
are
helping
you
identify
where,
along
the
path
things
are
not
working,
then
you're,
seeing
how
to
connect
to
things
you're,
checking
your
dns
and
then
at
the
very
end.
Here
you
have
that
ssh
command.
A
Where
you
know
I
want
to
get
as
close
to
the
problem
as
possible.
Once
I'm,
there
write
ssh
to
that
box
and
I
can
probably
fix
it
I
can
fix.
I
can
open
up
the
ports
I
can
write
like
I
can.
I
can
restart
some
sort
of
host
level
daemon.
I
can
figure
it
all
out
once
I
can
get
to
ssh
right,
but
all
this
network
plumbing
doesn't
work
inside
application.
A
Routing
right
all
this
stuff
works
on
the
top
half,
but
now
that
we've
handed
off
to
a
kubernetes
cluster,
we
have
all
this
application
routing.
How
do
we
do
this
right?
So
tada
there's
a
whole
bunch
of
kubernetes
networking
plumbing
tools
right
so
coupe
ctl
is
the
base
command
here.
I'm
sure
most
of
us
have
experience
with
this
now
right
and
so
the
three
three
very
basic
three
very
basic
tools
that
that
that
we
use
pretty
much
every
day
when
we're
troubleshooting
or
working
with
kubernetes
are
kubectl
proxy.
A
To
connect
your
laptop
to
your
cluster
right.
If
you
want
to
access
that
api
directly
kubectl
port
forward,
if
you
want
to
do
something
a
little
more
specific
connect
to
a
pod
connect
to
a
service
or
something
and
then
execit
and
then
pipe
that
into
like
a
bash
shell
command
right,
and
so
that
kind
of
gives
you
something
similar
to
an
ssh.
A
A
Cool
that
looks
good
all
right,
so
here
I'm
just
setting
my
my
cubeconfig
file.
This
just
lets
me
access
my
cluster
k
here,
I'm
using
shortcuts,
so
k
stands
for
kubectl
and
I'll
be
using
that
for
the
rest
of
this
demo.
Let's
just
do
a
get
all
see
it's
running
on
this
cluster
should
be
empty,
cleared
it
right
before
this.
So
here
we
see
we
have
just
a
a
empty
cluster.
A
I'm
going
to
cd
into
my
git
repository,
then
I'm
going
to
be
looking
at,
which
is
it
cd
there
we
go.
Okay,
it's
example.
So
these
examples
are
the
examples
that
are
actually
on
the
kubernetes
website.
These
are
just
free
to
use.
I
was
just
looking
for
something
simple
so
to
work
with
here,
so
you
can
find
these
on
the
kubernetes
repo
in
the
kubernetes
repositories
under
examples,
and
so
let's
cd
into
this
guestbook
go
end
with
this
right.
A
So
I'm
just
going
to
cat
out
all
the
config
files
and
shove
them
right
into
my
cluster
right.
This
is
going
to
give
me
a
sample
application
to
work
with,
and
here
we
see
that
the
sample
application
has
has
three
services,
and
so
we
just
do
a
coupe
ctl
get
all
right
now
we
can
see
that
this-
oh,
that's,
not
very
formatted,
very
well
right!
So
now
we
can
see
that
this
application
is
up
and
running.
A
I
see
that
the
guestbook
has
has
the
front-end
service
has
three
pods
running
and
and
then
it
has
a
database
with
masters
and
slaves
or
mains
and
and
follow
followers
I
don't
but
anyway,
so
this
application
is
running
now,
let's
actually,
let's,
let's
start
working
with
some
of
these,
these
networking
tools
right.
So,
let's,
let's
do
a
cube,
ctl
proxy
on
port
8080
right,
and
so
this
is
up
and
running
and
remember
as
actually
so
yeah.
A
So,
as
I
mentioned
before,
the
proxy
here
is
going
to
connect
us
connect
our
cluster
to
our
our
our
local
laptop
right
like
so.
This
is
going
to
give
us
access
to
the
kubernetes
api
on
this
port
8080..
So
we
can
do
things
like
curl
on
localhost,
on
8080,
with
api
right
and
so
the
kubernetes
api
now
is
returning
things
on
localhost.
We
can
see
that
it's
showing
me
it's
showing
me
some
information
about
the
api.
Maybe
I
want
something
a
little
bit
more
specific.
A
Maybe
I
want
right.
Maybe
I
want
to
see
the
pods
right
so
name
spaces
default
pots
right
like
so
you
kind
of
have
to
kind
of
have
to
know
where
things
are,
though,
on
this
api
right
like
it's,
not
exactly
the
easiest
thing
to
do,
but
you
can
you
can
pipe
through
things
and
then
right.
What
this
gives
you
right.
Coop
ctl
does
a
lot
of
this
for
you
right.
A
It
masks
a
lot
of
working
with
the
api
and
and
when
you
want
to
do
a
get
pods
right
like
you
can
you
can
see
things.
So
let
me
actually
show
that
so,
let's
see
set
my
cube.
Config
get
pods
right
like
this
coupe
ctl
command,
it's
also
working
with
the
same
api
and
it's
returning.
It's
returning
data
about
returning
the
same
data.
It's
just
presented
in
a
way.
That's
supposed
to
be
very
easy
to
use,
but
sometimes
easy
to
use,
isn't
going
to
give
you
all
of
the
information
that
you
need
right.
A
A
Thank
you
appreciate
that,
and
so
right
here
we
can
see
like
this
is
going
to
pull
up
right,
that
same
list
of
pots
right
we're
accessing
the
same
information,
but
maybe
we
want
to
write,
see
more
information
about
this
right
like
so.
This
is
going
to
give
us
a
ton
more
right
and
so
right.
This
is
just
another
way
to
look
at
this
information.
It's
another
way
to
to
access
specific
information
that
coupe
ctl
might
might
be
changing
for
us
right.
So
so
with
that
right,
like
maybe
like.
A
Right
so
this
gives
us
some
information,
it's
kind
of
like
very
basic,
very
basic
stuff
of
what's
going
on
like
maybe
maybe
we
even
do
instead
of
a
describe,
we
do
a
get
pod
and
then
we
do
output
json
to
kind
of
really
really
see
more
information
about
this
right
like,
and
so
this
is
going
to
look
pretty
pretty
similar
to
right.
If
we
were
to
pass
this
in
and
say,
we
want
the
first
object
in
this
pod
list
right
because
we
grabbed
the
first
one
off
the
list
over
there.
A
Right
and
so
so,
port
forwarding,
we
talked
about
the
proxy
being
a
full
connection
between
the
cluster
and
your
laptop
right,
connecting
connecting
the
whole
api
to
to
to
your
laptop
and
now
with
port
four.
This
is
a
little
bit
more
specific
right.
This
is
this.
Is
I
want
to
forward
one
one
specific
pod
or
one
specific
deployment
or
one
specific
service
to
my
computer,
so
that
I
can
interact
with
interact
with
it
in
some
way?
Right
like
I
want
to
troubleshoot
that
thing.
A
Forward,
actually,
let's
do
a
ctl
get
services.
First
right,
let's,
let's
see
what's
what's
working
on
here,
what
exists
on
here
right.
So
I
see
the
I
see
here.
There's
a
guestbook
service
type
load
balancer,
which
means
it's
probably
our
front
end
right
and
it's
running
on
port
3000
right,
so
coop,
ctl,
port
forward
service,
guestbook,
3000
right!
So
now
this
now
this
now
this
forward.
Now
this
port
forward
is
running.
I'm
forwarding
that
service
for
a
3000
down
to
my
local
machine,
also
on
port
3000
right.
A
It's
it's
a
shorthand
3000..
If
you
don't
list
a
a
local
port,
it
just
mirrors
the
same
port.
So
with
that,
let's
go
back
over
to
other
terminal.
Here,
I'm
going
to
clear
this
clear
this
as
well
right
and
so
now
this
is
running.
I
can
look
at
what
this
looks
like
on
port
3000
right
http
is
just
a
a
way
to
do.
Curl
with
pretty
colors,
you
can
do
brew,
install
http
if
you
want
to
try
it
out
so
yeah.
A
So
I
see
this
is
working
okay,
cool
right,
but
maybe
maybe
I
actually
just
want
to
like
see
the
service.
So
I
can
pop
over
to
here
and
kind
of
just
pull
it
up.
Like
oh,
okay,
cool
this
looks
like
look
looks
like
this.
Service
is
working
right,
so
whatever
problem
was
being
reported
is
not
here
right,
so,
okay,
so
that
seems
fine
right.
Maybe
maybe
I
need
to
dive
in
a
little
bit
deeper.
Let's
so,
let's
go
back
to
our
port
forward
here
and
say:
okay,
the
service.
A
A
A
Right,
but
instead
of
a
cert,
the
overall
service,
like
let's
look
directly
at
one
of
these
pots
right,
and
so
this
is
actually
going
to
look
very
similar
because
it
is
it
is.
It
is
one
of
the
pods
in
the
service,
so
they
should
be
very
similar.
You
have
this,
this
nice
changing
color
effect,
but
you
see
it's
connected
the
same
thing:
it's
presenting
the
same
data,
but
we're
just
a
little
bit
further
into
the
application
now
and
so.
A
Right
yeah,
so
here,
let's
actually
so
last
last
last
kind
of
networking
command
here
we're
gonna
talk
about
is
the
or
the
last
group.
Ctl
command
is
ctl,
exec
right
and
so
with
coupe
ctl
exec.
What
this
is
is
kubectl,
execute
right,
coop,
cto,
execute
is,
is
going
to
be
how
we,
how
we
execute
commands
right.
Maybe
you
need
to
execute
a
single
command?
Maybe
you
need
to
execute
multiple
commands?
Maybe
you
need
to
bring
up
a
shell
terminal
right
so
that
right
this
is.
A
A
And,
let's
just
let's
just
check
this
out:
let's,
let's
run
some
commands,
okay
get
pods
right!
Let's
try
this
first
one
here,
so
coop,
cto,
exec,
pod
name
and
let's
pipe
in
just
like
in
ls
right,
let's
run
ls
on
this.
Okay
that
looks
good
well,
it's
pwd
see
what
directory
you're
in
okay-
let's
just
let's
try
to
actually
like
modify
this.
Let's
create
a
peter.txt
file
right.
Let's
run
that
ls
again
right,
so
we
can
see
that
we
can.
We
can
very
easily
run
individual
commands
on
any
of
these
pods
right.
A
This
gives
us
it
gives
us
a
simple
way
to
to
interact
with
it.
If
we
want
to
test
something
out,
if
we
think
a
connection
is
bad
right,
this.
This
is
one
one
way
to
get
get
into
the
pod
right
and
then,
if
we
want
to,
if
we
don't
like
to
do
it
as
a
single
command,
what
we
could
do
is
bring
up
our
our
ssh
style,
bring.
A
I
used
to
try
to
always
run
this
with
bash,
and
I
just
always
hit
this
problem
and
like
one
of
the
things
you
have
to
remember,
is
that
you're
running
containers,
these
containers
are
normally
stripped
down,
trying
to
be
as
lightweight
as
possible,
so
bash
normally
is
a
little
too
heavy,
and
so
it
gets
stripped
out
and
so
there's,
but
there's
always
going
to
be
a
shell
right.
So
changing
from
from
the
bash
shell
to
just
shell
will
typically
get
you
in
and
so
now
now
we
have.
A
Right
like
we
can,
we
can
figure
out
exactly
what's
going
on.
Maybe
we
want
to
like
ping
the
database
right
like
I
want
to
check
out
like
what
our
connections
are
like.
This
is
kind
of
giving
us
just
a
very
simple
way
to
interact
and
understand,
what's
happening
at
a
pod
level
and
so
while
well.
This
is
this
is
kind
of
a
very
traditional
way
of
doing
a
lot
of
the
kubernetes
troubleshooting
stuff
right,
like
maybe
maybe
there's
something
better
right.
A
So
let's
try
telepresence
telepresence
is
a
cncf
open
source
tool.
It
is
currently
in
the
sandbox
stage,
I
believe,
and
so
what
it
does.
It
is
a
networking
tool
that
allows
you
to
to
bridge
your
local
laptop's
network
with
the
kubernetes
cluster
right
and
so
so
similar
to
how
coop,
cdl
proxy
and
coup
cl
coop
ctl
port
forward
work
right,
like
you're,
being
able
to
get
closer
to
your
your
cluster's
network
and
you're
able
going
to
be
able
to
do
a
lot
more
things.
A
And
so
it's
going
to
ask
me
for
my
password
here
why?
Why
need
your
password
it's
starting
up
a
a
local
local
daemon
on
your
computer,
which
creates
a
networking
tunnel
to
the
to
the
cluster
right.
So
we
can
see
this
with
telepresence
status
right,
and
so
here
we
see
like
this
is
the
the
tunnel
that
is
up.
We
see
that
it's
running
see
the
version.
We
can
see
the
the
remote
the
remote
clusters
ip
address.
A
A
A
I'm
able
to
see
these
things
most
most
of
these
don't
respond
to
ping,
but
it's
an
easy
way,
easiest
way
that
I
know
to
just
check
an
address,
real
quick
so,
but
then
that
also
lets
us
do
things
like
guessbook.default,
knowing
that
it's
running
on
port
3000
right
so
now,
instead
of
setting
up
that
setting
up
a
port
forward
or
setting
up
other
another
way
right
now,
this
is
just
working
right.
So
it's
not
just
not
just
this
one
thing:
either
right,
we're
connected
to
all
things.
A
So
now
like,
if
we
wanted
to
pull
up
or
yeah
so
get
pods
like
if
we
want
to
do
connect
one
of
these
pods
now
like.
Let's
describe
this
okay,
try
a
pod
right.
We
can
also
just
dive
directly
into
one
of
these
pod,
ips
and
so
pod.
Ips
are
have
strange,
host
name
rules
and
don't
just
don't
just
exist,
you
can't
use
the
host
name
because
you
have
to
specifically
append
a
host
name
in
your
config
file,
so
the
ip
address
ends
up
being
easier
at
the
pod
level.
A
Right
and
so
right
now
we
can
see
oh
much
much
easier
than
going
back
and
forth
if
you
are
troubleshooting
and
you're
consistently
needing
to
check
a
bunch
of
things
right
being
able
to
not
create
multiple
port
forwards
and
multiple
things
to
hop
back
and
forth.
This
is
kind
of
just
a
time,
saver
right
being
able
to
just
use
one
tool
that
bridges
the
whole
network
at
once.
A
So
with
that
right,
we
also
right-
and
we
can
also
pull
this
up
right.
This
dns
works
anywhere,
so
let's
write
the
service
name
and
then
the
the
namespace
here
right
on
port
3000
gives
us
the
same
thing
and
right
like
maybe
maybe
we
want
to
do
something
like
like,
like,
oh
once
again
like
we
want
to
check
check
that
the
database
is
working
right.
So,
let's
see
so,
we
see
that's
there.
Maybe
we
want
to
do
redis
cli
right.
A
Oh,
I
forgot
the
name
space
right,
so
we
see
that
that's
operational.
Maybe
we
want
to
actually
like
get
some
information
back
right.
Oh
there's
that
there's
the
entry
right
that
that
I
did
right.
Those
are
the
actual
information
from
from
from
this
database
service.
That
shows
what
I
had
inputted
right.
So
this
is
the
information
on
the
screen
right
just
interacting
directly
with
the
kubernetes
services
on
my
command
line.