►
From YouTube: Proxy Service: A New Network Traffic Abstraction in Kubernetes - Walter Fender & Yongkun Gui, Google
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Proxy Service: A New Network Traffic Abstraction in Kubernetes - Walter Fender & Yongkun Gui, Google
Kubernetes master-to-cluster communication doesn’t get as much attention as the opposite direction, yet many critical features (kubectl proxy, logs, exec, …) rely on it to function. In order to support secure communications from Kube API Server running on the control network to nodes running on a cluster network, SSH Tunnels were developed. This technology complicates the API Server in a manner which is neither extensible nor popular. The new proposed gRPC based proxy service abstracts this complexity away from the API Server, while providing a greater degree of extensibility. In this talk, we will see how SSH tunnels are implemented right now, what the new proxy service looks like, and how it opens the door to future extensions for use cases like auditing and multi-network support.
https://sched.co/Nrn7
A
A
So
cube
API
server,
we're
adding
a
new
proxy
service,
it's
an
extensibility
point
and
the
questions
we're
gonna
be
focusing
on
for
this
or
why
are
we
doing?
How
does
it
work?
How
do
you
use
it
what's
next,
and
can
you
help
us
and
by
the
way
you
know
spoiler?
For
the
last
question?
Yes,
you
can
help
us
so
the
cube,
API
server.
You
know
we
were
adding
this
this
network
proxy.
We
also
called
it
connectivity
service
and
one
of
the
first
questions
I
tend
to
get
is.
A
A
Another
big
thing
is
the
cube:
api
server
should
always
be
able
to
talk
to
the
cluster.
It's
serve
it
now.
Most
of
the
time
communication
is
in
the
other
door.
Most
of
the
time.
The
cluster
talks
to
the
QB
API
server,
but
I'll
get
into
details
of
when
the
cube,
API
server
needs
to
talk
to
the
cluster
and,
if
you're
thinking
a
standard,
easy
configuration.
What
you
do
on
your
box.
That's
pretty
easy.
It
just
talks,
but
there
are
interesting
scenarios
such
as
hybrid
clouds,
where
it's
actually
not
that
even
easy
to
have
the
cube.
A
Api
server
talk
to
the
cluster
and
then
the
last
thing
on
the
requirements
is.
There
are
some
interesting
vulnerabilities
that
exists
today
that
we
would
like
to
close.
So
if
I
look
at
certain
functions
of
the
cube,
API
server
like
proxy,
it's
actually
possible
to
configure
the
proxy
to
act
as
a
channel
that
can
then
be
used
for
an
attack
vector
against
the
control
plane.
A
So
if
I
know
where
the
traffic
is
intended
to
go
its
intended
to
go
to
the
cluster
and
I,
stick
a
firewall
between
the
cluster
and
things
that
the
cluster
shouldn't
be
talking
to
on
the
control
plane
like
the
Etsy
D
server,
then
I'm
providing
a
significant
increase
in
the
protection
that
is
provided
to
kubernetes
so
requirements,
but
do
I
want
it.
I
mean
we
need
it,
but
do
I
want
it
and
I'm
gonna
claim
that
yes,
I
actually,
in
addition
to
needing
this
solution,
I
want
this
solution
and
you
should
want
the
solution
too.
A
One
thing
is
SSH
tunnels.
So
today,
when
the
cube
API
server
wants
to
talk
to
the
cluster,
there
are
two
solutions:
you
either
just
drop
the
traffic
onto
the
network
and
it
gets
there
or
you
use
this
very
archaic,
unpleasant
solution
called
SSH
tunnels,
which
requires
that
you
run
an
SSH
daemon
on
your
cluster
that
you
generate
certs
to
those
and
it's
a
mess.
You
don't
want
to
do
it.
This
is
a
much
cleaner
solution.
It's
also
an
extensibility
point.
A
So
if
you,
you
know,
if
I
want
to
be
able
to
do
something
like
talk
between
the
cube
API
server
and
my
cluster
on
a
VPN
that
my
particular
IT
department
tells
me
I
need
to
use.
This
is
a
mechanism
that
will
allow
me
to
do
that
in
which
the
SSH
tunnel
solution
doesn't
have,
and
there
is
no
other
way
to
do
it.
With
this,
we
begin
we
give
you
the
extensibility
that
would
allow
you
to
do
that.
A
So
I
promise
to
tell
you
about
when
the
cube
API
server
wanted
to
reach
out
and
talk
to
something
else.
So
the
first
and
primary
thing
we
think
of
that
it
wants
to
talk
to
on
the
master.
Is
that
CD?
That's
where
it
stores
its
data?
That's
the
primary
thing
it's
going
to
talk
to,
but
also
on
the
API,
sir
yeah,
also
on
the
control
plane.
Other
things
the
API
server
needs
to
talk
to,
including
quite
a
few
things
web
hooks,
aggregated,
API,
servers,
etc.
A
A
It
turns
out
that
the
configuration
of
the
web
hook
will
tell
me
whether
I
should
expect
that,
with
the
admission
web
hook
to
be
running
in
the
control
plane,
ie,
it's
the
sort
of
web
hook
that
the
cloud
provider
is
putting
in
or
it's
the
sort
of
web
hook
that
the
customer
is
putting
in
and
the
this
net.
This
configuration
on
the
cube,
API
server
will
route
the
traffic
appropriately
to
either
the
control
plane
or
the
data
plane
for
the
cluster.
A
B
So,
let's
talk:
let's
talk
into
some
of
the
technical
details
of
how
we
implement
this
process
server.
So
if
you
look
at
the
top
box,
we
have
a
proxy
server
and
in
proxy
server
we
basically
expose
two
interfaces
for
the
copy
API
server
to
rather
traffic
into
the
cluster
Network.
So
one
of
them
is
HD
vignette.
The
other
is
G
RPC.
So
for
HTTP
connect,
it's
actually
HTTP
standard.
So
just
like
get
potent
post,
it's
part,
it's
not.
Connect
is
another
HTTP
method,
so
connect
comes
with
the
perimeter,
IP
and
port.
B
B
So
basically
it's
a
it's
a
tcp
over
HTTP
proxy
and
it's
exactly
what
you
want
and
the
good
thing
about
that
is
it's
since
it's
a
standard
so
most
likely
you
have
it
native
support
for
a
lot
of
different
languages,
and
we
don't
go
that
habit
even
support
for
that.
So
that's!
That's
the
only
thing
we
care,
so
that's
good
for
us,
and
another
thing
is
that
it
doesn't
support
a
UDP
tau.
We
would
not
really
care
too
much
about
that.
B
So
that's,
basically
what
we
use
right
now
and
for
G
RPC,
it's
based
on
PC
streaming
API.
So
it's
basically
a
TCP
/
G
RPC
it
support
UDP,
but
it
doesn't,
but
you
require
a
specific
client
installed
on
the
client
side
so
that
they
translate
your
local
connection
and
socket
connection
into
the
jar
pc-based
implantation.
So
that's
a
little
bit
complex
and
obviously
it's
still
there.
If
you
want
to
use
it,
you
can
still
use
it
and
of
course
both
of
the
connections
are
both
most
of
the
interfaces
are
secured
with
mtrs.
B
So
that
means
your
payloads
occurred
encrypted
and
also
both
parties.
The
client-server
are
correctly
authenticated
with
each
other.
Now
now
you
have
that.
So
the
question
becomes
how
the
project
server
do,
how
to
route
the
traffic
into
the
cluster
network
right.
So
one
of
the
options
is
that
we
do
for
SSH
tunnel.
Basically,
it
just
go
into
the
cluster
Network
right,
but
we're
not
ready
to
make
that
assumption
that
the
the
the
processor
always
knows
have
a
routable
connection
to
the
classes
or
network
I
know
that
most
cases
it's
true,
but
it's
not
always
true.
B
B
B
So,
of
course,
the
agent
is
responsible
for
distributed
stream
to
different
endpoints
and
also
responsible
for
open
a
close
connection
to
the
endpoints,
like
no
port
and
web
hook.
So
if
you
look
at
the
48
of
flow
and
the
target
red
arrows,
so
the
requests
coming
from
the
HTTP
connect
or
G
RPC
interface
from
the
API
server.
It
goes
into
the
product
server
and
over
the
agent
to
proceed
on
Oh
eventually
get
to
the
agent
agent
distributed
the
stream
to
the
endpoints
northern
ports.
B
Yeah,
if
you
resume
me
a
little,
if
a
little
bit
more,
they
you
see
the
agent
to
proxy
server
tunnel
and
it's
basically
also
implemented
with
the
G
RPC,
bi-directional
stream
API.
And
yes
that
and
of
course
it's
a
fool,
put
your
price
connection,
what
it
means
that,
of
course,
it's
a
photo
back.
There's
a.
B
B
Also,
we
need
to
multiplex
over
over
the
one
channel.
What
it
means
is
that
if
you
have
a
multiple
connections
in
the
tunnel,
it
must
be
supported.
That
means
that
if
you
have
multiple
connections
over
the
same
channel
and
it
read
right
the
same
time,
it
also
needs
it
also
works,
so
how
we
implement.
That
is
that
we
have
this
protocol
to
find
the
data
structure
called
packet,
a
packet
that
we
have
two
types
of
packet.
One
of
them
is
called
data
packet.
B
It's
basically
the
data
you
want
to
transfer
over
the
wire
and
also
the
control
packet
is
basically
where
you
want
to
open
the
connection
close
the
connection,
etc
and
also
response
of
that
control
signal
and
some
of
the
connection,
ID
etc.
And
with
this
implementation,
we
have
support
for
resumable
connection,
what
it
means
that,
if,
if
the
JRPG
stream,
a
stream
will
get
reset
it
the
crashing
dead,
as
long
as
the
agent
can
connect
back
to
the
proxy
server,
all
the
connections
will
be
alive
and
with
the
edges
which
Esther
tunnel
you
can
do
that.
B
If
a
knows
that
and
all
the
connections
are
is
dead
and
of
course,
we
can
support
more
advanced
features
like
if
you
have
a
multiple
agents
now
back
to
the
proxy
server.
All
sudden,
your
proxy
server
have
multiple
choices,
then
it
become
the
load
balancer
and
we
can't
comment
different
load,
balancer
algorithm
to
to
share
the
load
across
different
agent
and
also
we
can
implement
monitoring
auditing.
You
want
the
auditor,
the
specific
user
and
also
throught
when
you,
for
example,
you
don't
want
to
you,
want
to
be
fair
to
all
the
connections.
B
You
don't
want
to
a
one
connection
to
to
drink
all
the
other
traffic
and
all
the
other
connections
are
not
as
fear.
So,
of
course,
all
of
these
are
not
implemented
yet
the
project
still
the
early
stage.
So
that's
why
we're
here-
and
we
ask
you
for
help
if
you
are
new
to
the
community
or
you're,
a
veteran
you're
welcome
to
join
we'll,
have
a
slides
that
have
a
lot
of
links
on
it.
The
code
is
in
github
and
you're
welcome
to
contribute
and
I.
A
We've
taken
a
look
at
this,
and
one
of
the
distinctions
I
would
like
to
make
here
is
part
of
what
you're
seeing
is
part
of
the
cap
and
part
of
the
definite
implementation
and
part
of
it
is
part
of
the
reference
implementation.
So
if
you
think
back
to
what
Duncan
was
just
describing
everything
he
was
describing
as
part
of
the
reference
implementation
and
we'll
get
into
it,
we
have
working
code,
it
is,
but
it
is
a
reference
implementation.
A
The
reference
is
there
for
you
to
either
use
or
to
take
and
modify
as
you
need
it
in
your
cloud.
So
how
do
I
configure
my
cube,
API
server?
To
then
do
these
wonderful
things
I've
been
describing,
so
we
have
a
new
flag,
the
network
proxy
configuration
file
flag
and
it
takes
a
yeah
mol
file
and
that
Yama
file
is
a
connectivity
service
configuration
object.
It
looks
just
like
a
standard
resource.
It
isn't,
however,
a
standard
resource,
and
the
reason
is
that
we
don't
necessarily
to
allow
customers
to
modify
this
at
runtime.
A
We
see
one
that
says
I'm
a
cluster
I'm
going
to
connect
via
HTTP
connect,
I'm
going
to
be
connecting
via
HTTP
on
port
80
131
and
here's
all
of
my
credential
information
and
on
the
master
I'm,
actually
not
going
to
worry
I'm
just
going
to
do
Direct
Connect
using
tcp/ip
like
normal,
so
the
question
would
become
hey.
What
is
this
configuration
that
I
just
described?
Give
us
and
hey?
If
you
say
this
works,
where
can
I
find
the
code?
A
So
what
it
gives
us
is
actually
this
image
we
have
here.
So
we
see
a
cube,
API
server
connecting
to
the
the
cluster
connectivity
proxy,
and
we
see
we
have
two
connectivity
agents,
one
per
node,
presumably
connecting
via
our
G
RPC,
also
to
the
connectivity
proxy.
But
the
cube
API
server
is
actually
doing
direct
to
reach
out
to
both
the
SC
D
and
things
like
the
web.
A
So
that's
kind
of
what
I
just
described
and
let
me
go
down
just
a
little
bit
so,
where
can
I
find
this
code
I
now
before
I
get
into
the
details
here,
I
will
point
out
that
all
of
these
slides
are
available
on
the
cube
con
site.
For
this
talk,
so
you
can
download
this
PDF
and
you
can
find
these
slides
there.
This
slide
there
and
get
the
links
from
that.
So
I've
got
three
links
up
here.
A
A
So
a
lot
of
mine,
and
especially
young
Kunz
work
on
getting
the
proxy
server
and
a
proxy
agent
working
all
exists
under
that
and
we'll
get
later
on
how
you
can
help
us
make
interesting
changes
to
that
code,
and
the
last
link
is
if
you
want
to
get
much
finer
grained
detail
on
exactly
what
is
approved.
We
have
the
approved
cap
for
this
entire
feature.
Under
that
last
link.
A
All
right,
so
what
are
the
cube?
Api
server
code
changes,
look
like
so
the
first
one
is
that
we
start
out
with
this
idea
of
a
network
context,
and
you
want
to
be
able
to
look.
You
set
a
little
network
context.
You
ask
for
the
look
up
on
that
network
context
and
what
you
get
is
a
context
to
where
dialer
and
if
we
look
at
the
network
context,
we're
gonna
see
that
right
now
all
it
contains
is
a
contact,
a
connectivity
service
name,
that's
presumably
going
to
be
either
at
CD
master
or
cluster
very
simple.
A
So
why
did
we
worry
about
having
a
connected
network
context
object
to
hold
what
amounts
to
one
string
or
enum?
The
reason
is
that
we
have
ideas
on
how
we
want
to
enhance
this
I'll,
get
into
a
lot
of
options
on
how
this
might
get
into
an
slider.
But
an
obvious
thing
might
be
that
if
I
want
to
do
a
multi-tenant
system,
I
might
want
to
be
able
to
indicate
my
tenants
in
the
network
context
and
have
the
lookup
provide
back
the
dialer
that
talks
to
that
tenants
cluster.
A
So
that's
just
one
example
of
how
this
might
work
and
then
for
the
concrete
usage.
Whether
this
is
a
log
call
or
a
web
book
call
at
the
appropriate
place.
You
would
do
as
you'd
set
up
your
contacts.
Saying:
hey
I
need
to
talk
to
the
cluster.
Do
the
lookup
you
get
back
to
your
context,
dialer
or
an
error?
Hopefully
not,
and
then
you're
going
to
do
a
wrap
transport
to
to
get
back
the
transport
that
uses
that
dialer
to
actually
forward
the
traffic.
A
All
right
so
I
sort
of
hinted
at
possible
futures
earlier,
but
there's
quite
a
few
possible
futures
and
ways
that
we
can
really
improve
this
feature.
So
the
alternate
communication.
This
is
an
interesting
one,
because
this
isn't
even
an
in
kubernetes
future
when
I
talked
about
extensibility.
This
is
the
extensibility
future.
So
it's
actually
relatively
easy.
A
So
that's
fine
I
kind
of
hinted
at
the
next
one,
but
connectivity
connectivity,
service
configurations
beyond
the
simple
master
cluster
at
CD,
so
I
might
want
to
do
connectivity
by
label
if
I
know
that
I
have
a
cluster
in
multiple
zones,
I
may
want
to
label
the
zone
and
then,
instead
of
just
going
to
cluster
I,
can
send
to
the
traffic
to
the
right
zone
and
that's
very
nice,
because
then
I
don't
get
to
that
zone
and
then
have
to
route
a
hop.
You
know
across
the
Pacific.
A
A
A
A
I
can
also
allow
better
connections
than
HTTP
connect
from
cube
API
server
connectivity
server,
so
HTTP
connect
had
one
very
nice
advantage
right
now.
That
advantage
is
that
it
is
as
close
to
a
standard
as
we
could
come
up
with
as
an
open
standard
for
the
communication
between
the
cube,
API
server
and
the
proxy
server
G
RPC
we
tend
to
like
it.
It
has
very
nice
multiplexing
behaviors
nicer
than
what
you
get
out
of
HTTP
connect.
A
On
the
other
hand,
I
can
prove
to
you
that
the
client
that
we
use
for
HTTP
Connect
is
a
standard,
because
curl
will
do
every
bit
exactly
the
same
thing,
so
we
can
look
at
exactly.
You
know
how
important
one
is,
the
other.
We
have
a
trade-off
there.
I
know.
Amazon
is
very
interested
in
this
last
one.
We
can
allow
the
cube,
API
server
to
not
be
public
so
today,
if
I
have
just.
A
What
this
can
do
is
once
I've
set
up
this
tunnel,
then
the
procs,
then
the
eight
you've
been
nodes
and
things
running
on
nodes.
Instead
of
talking
directly
to
the
cube
API
server,
they
can
send
to
their
traffic
through
the
agent
reversing
everything
we've
shown.
You
so
far
have
their
traffic
securely
land
on
the
control
plane
where
it
will
go
to
the
cube
API
server,
and
we
can
have
special
logic
in
the
agent
to
make
sure
that
that
traffic
is
only
destined
for
the
cube
API
server.
A
So
this
is
another
example
of
the
sort
of
things
that
this
speech
sure
buys
us
all
right.
So
I'm
hoping
there
are
a
few
food
folks
here
being
convinced.
This
is
an
interesting
problem
and
that
they'd
like
to
contribute.
So
how
can
you
get
contribute
so
young
and
I
being
members
of
API
machinery?
Networking
cloud
provider
will
of
course
be
suggesting
that
a
great
SIG's
to
work
in
our
API
machinery,
networking
and
cloud
provider
there
are
home
SIG's.
A
We
all
do
interesting
work
and
I
highly
recommend
them,
especially
if
you're
interested
in
this
sort
of
problem.
I'm
also
gonna
say
that
the
API
server
Network
proxy
is
a
special
project
that
young,
coda
and
I
are
the
main
maintainer
of
under
kubernetes
SIG's.
It's
a
small
project.
It's
a
much
less
intimidating
codebase
than
the
main
kubernetes
one
and
you're
gonna
get
a
lot
more
attention.
B
Yeah
I
know
it's,
so
it's
a
very
long
day
and
it's
a
last
session,
and
not
only
you
want
to
be
really
want
to
be
here
so
yeah
I
want
to
add
that
the
contribution
is
not
just
limited
to
a
poor
request.
It
could
be
a
bug,
report,
future
requests
and
some
use
cases
discussion
and
it's
all
welcome
and
yeah.