►
From YouTube: K8SNIff - End-to-End Encryption Till the Pod [A] - Sebastian Scheele & Jason Murray, Loodse GmbH
Description
K8SNIff - End-to-End Encryption Till the Pod [A] - Sebastian Scheele & Jason Murray, Loodse GmbH
When running Kubernetes on Kubernetes, in order to ensure end-to-end encryption, we were confronted with the challenge to route TLS traffic directly to the pods. With one ingress only per cluster, that was not possible with the existing solutions.
To solve this problem, we created K8SNIff as an open source project on github: https://github.com/kubermatic/k8sniff. K8SNIff is a small ingress server that will accept incoming TLS connections, and parse TLS Client Hello messages for the SNI Extension. If one is found, K8Sniff will forward that connection to the pod.
In this talk, participants will learn how easy it is to implement your own logic on top of the Kubernetes API. Moreover, we will give practical examples for the use of K8SNIff.
About Jason Murray
Jason Murray is a Senior Infrastructure Architect at Loodse. He has contributed to both Kubernetes and Container Linux, focusing on large scale bare metal deployments. Prior to joining Loodse, Jason worked as an Operations Engineer at Collins and was Head of Development at Contetto.
About Sebastian Scheele
Sebastian Scheele is a co-founder of Loodse, a software company that has developed a solution for the management of multiple container clusters and provides consulting and training services in the area of cloud native strategies. He has been a major contributor to the development of K8SNIff. Prior to founding Loodse, Sebastian worked as a software developer for SAP. He holds a degree in Computer Science from the University of Applied Science and Arts of Dortmund.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Okay,
hello,
everyone,
so
we
want
to
talk
to
day
about
end-to-end
encryption
to
support.
So
when
we
set
up
our
system,
we
had
some
challenges
to
do
this,
but
yeah,
let's
first
start
with
introducing
us,
so
we
are
Lutz.
We
are
loser.
We
are
a
state
up,
startup
from
Hamburg
and
I'm
Sebastian
as
a
founder,
the
co-founder
and
CTO
and
I'm.
B
It
makes
it
really
easy
for
you
to
stand
up
kubernetes
clusters
and
makes
it
very
very
simple
for
you
to
do
upgrades
and
all
those
kinds
of
things,
but
not
everybody
can
use
Google,
especially
here
in
Germany,
where
you
have
things
like
data
privacy
laws
and
can't
necessarily
export
your
data
to
a
u.s.
provider,
so
you're
kind
of
restricted
on
the
hosting
providers
that
you
can
utilize,
and
that
gives
us
some
issues.
B
A
Yeah
and
when
we
start
building
q
emetic,
we
get
inspiration
like
how
Google
is
running
it.
So
what
we
are
doing
is
running
each
master
cluster
in
a
no-name
space,
with
all
the
components
like
an
API
server
and
controller
scheduler
and
HCD.
But
here
is
the
main
challenge
about
networking
and
traffic.
So,
and
this
is
where
we
want
to
talk
today,
so
how
we
get
traffic
into
the
cluster
and
how
we
solve
this
with
ka
sniff.
A
So
on
our
side,
we
have
Q
biometric
running
where
we
have
our
own
API
server
and
own
controller
to
really
control
what
we
like.
Now,
it's
more
called
operator
a
Cuba
needs
operator,
then
for
all
the
web
traffic
we
use
nginx,
but
we
still
had
some
yeah.
We
need
like
encrypted
traffic
to
support
and
when
we
started
it
was
not
really
possible.
We
couldn't
use
like
ingress.
A
B
So
the
the
big
issue
that
we
had
was
especially
around
HCV
so
HCD.
We
previously
had
to
HCV
instances
we
weren't
using
CNI
as
of
yet
see
and
I
was
still
kinda
kind
of
new
when
we
first
deployed
this
and
so
to
configure
flannel
for
the
nodes
we
had
to
have
a
public
at
CD
and
we
didn't
really
feel
comfortable
with
just
terminating
TLS
certs
right
at
the
nginx
controller.
B
That
required
a
lot
of
configuration
on
our
side,
and
so
we
tried
to
figure
out
a
better
way
and
that's
where
we
came
up
with
cage
sniff.
So
we
wanted
to
be
able
to
get
traffic
straight
from
wherever
the
source
is
right
to
the
pod
and
allow
the
pod
to
handle
the
TLS
offloading
and
to
do
that
in
a
transparent
way,
where
we
could
still
you
leverage
the
power
of
the
ingress
resources
that
are
available
in
kubernetes,
so
covered
some
of
the
problems.
B
You
know
we
we
really
want
to
maximize
that
utilization
of
TLS
and
it's
not
just
for
at
CD.
You
can
use
that
for
the
API
server.
You
can
use
it
in
a
lot
of
other
use
cases
which
we'll
go
into
a
little
bit
later,
but
it's
a
challenge
that
hasn't
been
addressed
as
of
yet
in
the
kubernetes
ecosystem.
So
we
went
okay.
Well,
we've
got
some
time,
let's
do
something
fun,
so
we
have
ingress
right
now,
and
this
is
what
we
wanted
to
see.
B
We
wanted
to
be
able
to
take
the
TLS
and
the
encrypted
traffic
feed
it.
You
know
a
hostname
have
that
feed
through
the
ingress
right
into
the
service
right
to
the
pod
and
have
the
pods
take
care
of
that
TLS
offloading.
The
problem
is,
with
all
of
the
existing
ingress
controllers
that
were
available,
whether
that
be
the
gke
ingress
controller.
The
nginx
ingress
controller
are
a
couple
of
the
other
implementations
that
came
out
earlier.
A
And
in
the
future
we
will
use
this
for
the
API
server
of
cue
bonitas.
So
when
we
started
building
this
up,
we
use
this
for
HDD,
especially
to
secure,
communicate
to
HDD.
We
use
client
certificates
and
was
client
certificate.
It's
not
possible
to
do
the
TLS
termination
on
the
ingress,
so
it
really
must
go
to
the
ports
that
you
can
authenticate.
We
supplanted
a
certificate
on
the
pot
and
can
ensure
ok,
I'm
I'm
a
valid
connection
and
you
cannot
terminate
and
we
encrypt
the
traffic.
B
This
was
our
goal
TLS
to
the
pods,
and
this
is
how
we
solved
it
was
with
Cade
sniff,
so
it
does
layer,
3
forwarding
of
the
traffic
into
the
service.
But
what
we
do
is
we
look
at
the
hello
headers,
the
TLS,
hello
headers
that
are
coming
in
from
you
know
the
SNI
stuff,
and
so
we
can
take
a
look
at
that
figure
out
where
we're
supposed
to
be
directing
that
traffic
using
the
ingress
resources
that
are
available
in
kubernetes.
B
So
for
you
using
k
sniff,
all
you
have
to
do
is
treat
it
just
like
any
other
ingress
controller,
except
for
knowing
that,
instead
of
it
doing
the
TLS
offloading,
all
it
will
do
is
take
that
and
shove
it
right
to
your
pod.
So
you
don't
have
to
worry
about
anything.
You
don't
have
to
do
any
magic.
You
don't
have
to
do
any
configuration
beyond.
You
know
your
simple
deployment
to
tell
Kate
sniff
to
exist
in
your
cluster
and
use
the
ingress
resources
that
are
already
available.
B
So
semantically
we
try
to
be
as
adherent
to
all
of
the
other
ingress
controllers
that
were
available
in
the
ecosystem
but
provide
you
know
a
little
bit
of
a
different
layer
of
where
we
pass
the
traffic
through.
So
this
is
what
it
actually
looks
like.
So
you
have
a
client.
The
client
will
pass
TLS
traffic
it'll
hit
Cade
sniff,
Cade
sniff
will
read
these
sni
headers.
It
will
pass
it
through
to
the
service.
B
B
The
problem
is
that
we
aren't
always
on
a
cloud
provider
that
will
provide
us
with
a
load
balancer,
and
then
we
start
going
into
the
the
magic
port
number
issue
where
we
just
have
to
figure
out.
Ok,
what
ports
are
already
allocated
start
using
a
node
port
that
we
can
dynamically
assign,
and
we
didn't
really
like
that
semantics.
It
makes
it
really
hard
to
start
figuring
out
how
to
deal
with
stuff,
and
that
puts
pain
not
only
on
the
cluster
operators.
A
The
only
thing,
what
would
you
mainly
must
address
on
your
part?
You
must
ensure
that
you
have
certificates
there
and
then
everything
is
more
or
less
transparent.
You
deploy
your
ingress
kay
sniff
will
reach
all
your
English's
and
then
forward
it
to
the
services
and
it
going
forward
to
the
pot.
So
no
really
much
to
do
extra.
Only
deploy
certificates
with
your
deployment
or
with
your
ports.
B
So
in
and
this
works
really
well
for
native
applications
that
want
to
use
certificates
on
their
own
in
the
application.
So
if,
if
you
have
applications
that
really
really
don't
like
having
the
TLS
offloaded
somewhere
else,
there's
a
couple
of
things
and
like
the
at
last
name
stack
where
they
really
want
to
hold
those
certs.
B
And
so,
if
you
want
to
utilize
those
things,
but
you
don't
have
something
like
Cade
sniff
available,
you
start
running
into
problems
really
quickly
and
you
have
to
start
doing
some
magic,
pushing
things
around
to
different
port
numbers,
and
that's
just
not
fun.
So
you
know
this.
This
really
allows
you
to
kind
of
leverage
the
power
of
sni
and
leverage
the
power
of
ingress
without
having
to
you
know,
deal
with
doing
that,
TLS
offloading
at
the
point
of
ingress.
B
So
we'll
do
a
quick
demo
here
and
we'll
go
through
a
couple
of
the
configuration
files
to
kind
of
show
you
guys
what
we're
looking
at
so
I'm,
totally
cheating
by
the
way
I'm
a
copy-paste
ninja.
So
the
first
thing
we'll
do
is
we'll
go
ahead
and
just
deploy
the
initial
stuff
so
nope
wrong.
One.
So,
just
to
be
fair,
I've
done
some
contrib
work
on
the
nginx
ingress
controller,
so
I'm
not
bashing
on
those
guys
at
all.
B
They
do
a
great
job
and
it's
an
awesome
ingress
controller
if
you're
doing
layer
7
stuff,
so
just
so
that
it's
very
clear
that
we're
not
trying
to
say
you
know
use
this
instead
of
nginx
or
something
else
all
I'm
trying
to
say
is.
If
you
have
these
kind
of
problems,
definitely
leverage
that,
if
you
can
work
at
the
layer,
7
layer
level
definitely
use
that
because
it
is
way
like
ingress
is
awesome
right
and.
A
So
what
why
we
build
a
separate
tool
is
like
because
it's
when
you
look
at
the
coding,
it's
really
a
small
go
binary
and
it
doesn't
do
much.
It's
only
really
looking
at
the
ingress,
listen
on
the
so
the
ingress
resources
and
configuring
itself
and
then
forwarding
it
and
the
first
thing
ok,
should
be
enhanced
some
of
the
existing
tools.
B
For
sure,
so
all
I've
done
here
is
I've
just
pushed
in
an
engine
action
grass
controller,
so
we'll
go
ahead
and
use
that
in
an
insecure
manner
with
that
CD,
because
that's
just
an
easy
way
to
do
this
so
magically
we
have
at
CD
in
our
cluster
up
and
running.
It
should
theoretically
have
some
data
in
there
if
I
haven't
completely
broken
my
demo,
so
Oh.
B
The
Internet
has
been
shaky
today,
alright,
so
there
we
go.
We
just
went
through
the
nginx
ingress
controller
port
80.
It
did
its
stuff
and
pass
that
along
so
now
great
fun,
we're
very
excited,
but
let's
go
ahead.
I'm
gonna
jump
over
and
let's
see
what
this
looks
like
if
we
go
ahead
and
start
using
sed
with
its
certs.
So
all
we're
actually
doing
here
if
I
can
find
the
deployment
is
we're
just
setting
up
here.
B
The
client
start
off
giving
it
the
certs
and
saying
hey,
validate
this
stuff.
We're
gonna
do
TLS
in
both
directions:
fabulous
from
start
happening
here,
so
we'll
go
ahead
and
hit
this
on
port
8000
window
again
on
port
80.
Obviously,
that's
going
to
fail.
It's
gonna
freak
out
right,
so
let's
actually
hit
it
on
4
for
3,
where
nginx
is
serving
its
TLS
stuff.
B
Well,
it's
gonna
have
a
problem
too,
because
the
right
resources
aren't
available
and
the
right
resources
can't
be
available
at
this
level
because
we're
doing
the
TLS
offloading,
where
we're
telling
@cd
to
do
the
TLS
offloading
and
to
validate
those
client
certs.
But
we
can't
actually
pass
any
of
that
through,
because
nginx
insists
on
doing
the
TLS
offloading
at
this
point.
B
So
this
is
where
kate
sniff
comes
into
play
and
we
can
actually
leverage
that
so
we'll
go
ahead
and
push
Kade
sniff
into
the
cluster
and
while
we're
doing
that,
I'll
just
show
you
guys
what
kate
sniff
kind
of
looks
like
so
we're
pushing
it
into
its
own
namespace.
It
has
a
service
that
we
aren't
actually
leveraging
because
we're
just
using
host
network,
and
this
is
what
the
deployment
looks
like
so
we're
just
pushing
one
of
them
in
and
we
run
this
fabulous
little
binary
with.
B
You
know
they
a
JSON
configuration
file,
and
this
is
what
that
JSON
configuration
file,
looks
like
bind
everything
serve
on
8
4,
4
3
talk
to
kubernetes
and
if
you
need
to
put
in
a
like,
provide
a
special
kubernetes
config',
it
just
takes
the
standard,
kubernetes,
coop,
config
format.
You
can
mash
that
in
and
away
you
go
so
very,
very
simple,
very
easy
to
operate.
B
There's
not
really
any
magic
going
on
here,
except
for
the
host
network
stuff,
just
because
I'm
not
using
a
service
on
on
a
cloud
provider
just
using
a
node
on
digital
ocean,
so
we've
gone
ahead
and
deployed
Kade,
sniff
and
I
do
need
to
apply
this
again.
We're
gonna,
theoretically,
there's
potentially
a
bug
in
1.6
that
I
found
this
morning.
So
we
might
have
some
problems,
but
well
we'll
see
if
this
actually
works
so.
B
B
B
Theoretically,
magically
stuff
will
work.
Oh
there
we
go
so
now
we're
talking
to
@cd
we're
doing
client
auth
using
the
certs.
It's
talking
back
to
us
again,
using
TLS
we're
doing
all
the
certificate
verification
everything
is
actually
being
passed
through
on
layer
3,
and
we
don't
have
to
worry
about
the
layer,
7
complexities
of
like
trying
to
do
all
the
offloading
and
drop
it
back
down.
Everything
just
magically
happens
right
at
layer.
3
we
push
that
traffic
forward
and
away
you
go
so
it
makes
it
really
really
easy.
B
We
we
also
leverage
this
for
the
API
server.
Has
a
couple
of
calls
I
can
think
off
the
top
of
my
head
I
think
it's
any
of
the
streaming
stuff,
so
coop
cuddle,
exact,
coop,
kind
of
logs
and
port
forward,
I
believe
and
potentially
proxy
I.
Don't
remember
if
proxy
uses
that
or
not,
but
you
actually
have
to
use
layer,
3
traffic
for
that.
B
So
if
you're
putting
your
API
server
behind
an
ingress
controller
like
nginx,
that's
working
at
layer
7,
you
immediately
break
all
of
those
functions
for
us
where
we're
running
you
know
if
you're
running
just
one,
you
can
bind
those
ports
and
be
happy
with
that,
but
we're
we're
running
multiple
clusters.
Theoretically,
on
the
same
nodes,
we
don't
always
have
a
load
balancer
available
for
us,
it's
very
important
for
us
to
let
you
utilize
those
functions,
because
you
know
when
breaks
you
want
to
have
the
logs.
B
B
We
do
all
of
the
rest
based
forwarding
for
the
API
server
using
the
ingress
con
that
enginex
ingress
controller.
Currently,
do
we
I
think
we
switched
that
actually
last
week,
so
now
everything's
actually
running
for
the
API
server
through
Kade
sniff,
but
it
used
to
be
that
the
rest
stuff
would
go
through
one
port
and
then
you
could
tell
the
API
server
to
tell
its
clients
to
do
the
functions
that
required
streams
on
another
port
and
those
were
all
going
through.
B
A
I
mean
for
HTTP
traffic,
but
there
are
definitely
a
lot
of
other
use
cases,
so
in
general
you
it
should
be
possible
to
use
Kate
sniff
for
every
TCP
traffic,
where
an
S
and
I
had
a
set,
because
it
only
looks
into
the
TCP
header
and
checks
if
there's
an
S
ni
header
checks
for
the
domain
and
forwarded
to
the
specific
service.
So,
theoretically
we
didn't
try,
orders
it's
open
to.
A
You
would
like
to
get
feedback,
but
if
you
want
to
like
run
pop
server,
SMTP
server
or
some
other
things
on
a
queue,
bonitas
cluster
and
one
heroes's,
and
want
to
secure
this
on
the
pot
level,
it
should
be
possible
with
ka
sniff,
who
really
forward
the
traffic
to
the
port
and
it's
after
we
started
this.
We
use
this
for
our
own
infrastructure.
A
One
month
later
we
had
the
same
problem
at
our
customer,
so
they
run
on
certificates
for
some
test
environment
and
said:
ok
now
we
need
it
on
on
the
pot
level,
the
certificates-
and
so
we
said,
ok
yeah.
We
have
something:
let's
try
ka
sniff
for
this
and
forwards
the
traffic
to
the
pot
and
yeah
test.
It
out
it's
on
github,
it's
on
yeah.
B
So
to
find
it
on
get
it
alright,
ICO,
closed
chrome
for
very
good
reasons.
I
didn't
want
to
get
notifications.
While
we
were
having
this
chat,
so
you
can
find
us
at
github,
slash,
kaboom,
attic,
Cade
sniff
is
there
you
totally
can't
see
any
of
this?
Apparently
that's
awesome,
oh
I
totally
could
have
left
notifications.
Open
would
have
figured
out
if
I
got
tinder
matches
yeah
all
right,
so
you
can
find
the
project
here.
So
it's
github.com,
/q,
o
matic,
slash
Cade,
sniff
yeah.
It's
really
simple.
B
We've
got
some
example
ingress
some
example
configurations
or
mash
an
example
folder
in
there
with
some
deployments
probably
later
on
today
and
yeah.
If
you
guys
have
a
need
for
this,
please
take
a
look
at
it.
If
you
run
into
any
issues,
let
us
know
hop
on
our
slack
or
you
know
just
to
open
an
issue
and
we're
happy
to
work
with
you
on
it.
B
If
you
guys
have
any
pull
requests
for
may
be
using
it
in
another
use
case
that
might
have
some
special
considerations
well
totally
open
to
those
and
happy
to
hear
how
you
guys
want
to
use
that.
But
it's
just
an
the
big
thing
about
it.
Is
it's
just
another
tool
in
your
toolbox,
so
you
know
please
happily
come
use.
It
contribute
we'd
love
to
have
you
guys
around
and
if
you
have
questions
get
us
up
so.
A
And
I
mean
it's:
it's
a
good
point:
how
easy
it
is
to
extend
queue
benitez
with
your
own
requirements.
So
when
you're
looking
at
through
the
coding,
it's
only
a
few
lines
where
we
do
things
so
reading
the
ingress.
So
it's
really
a
really
good
example.
When
you
have
specific
requirements
and
cuban
eaters
or
the
current
set
up
doesn't
support
this,
you
can
build
up
your
own
tool
based
on
cuban
eaters
and
integrate
this
and
run
this,
and
you
are
free
to
use
it.
Yeah.
B
So
yeah,
like
I,
said
you
can
find
us
on
our
slack,
so
we've
got
a
slack
in
at
slack
Cuba
madaky!
Oh,
you
can
also
find
me
pretty
much
everywhere
at
chaos
alpha
and
your
s
Sheila
yeah.
Okay,
so
you
can
find
him
through
me
because
his
name
is
hard
to
spell
I
still,
don't
get
it
right.
So
do
you
guys
have
any
questions
at
all
all
right,
we'll
start
from
the
front
and
work
our
way
back,
because
that's
how
I
arbitrarily
decided
to
do
it?
So
you,
sir.
B
Yeah,
so
the
the
question
for
those
of
you
in
the
back
was:
is
there
an
advantage
of
using
cage
sniff
over
something
else
that
will
do
the
TLS
offloading
and
then
re
encrypt
and
send
it
to
the
pods?
So
one
of
the
big
ones
is
client.
Auth,
like
you
mentioned,
the
other
one
is
speed,
so
this
is
a
little
tiny
go
Lane
binary
and
it
just
shelves
traffic
through.
B
A
Other
thing
is
security.
Was
this?
You
can
really
ensure
that
you
have
an
end-to-end
encryption
without
some
men
in
the
middle
who
doing
some
stuff.
So
it's
really
also
when
you
need
secure
communication
between
your
client
and
your
pot
or
your
container.
There
is
no
way
to
do
this
or
to
do
this
on
the
ingress
layer
level.
B
B
B
Yeah,
so
what
we
usually
do
so
because
we
have
like
a
UI
and
an
API
and
stuff
that
are
running
in
our
clusters.
We
run
Kaede
sniff
to
handle
this
specific
traffic
and
we
also
use
the
nginx
ingress
controller
to
handle
any
of
our
layer,
7
traffic.
So
this
isn't
to
replace
anything
else.
We
use
this
to
yes,
so
the
big
thing
that
that
kinda
is
awesome
about
kubernetes
is
we've
got
this
fabulous
little
ingress
class,
so
that
tells
the
ingress
controllers
and
as
far
as
I
know,
at
least
they
all
respect
this.
B
So
even
if
you're
running
on
gke,
which
comes
with
the
embedded
gke
controller,
you
can
run
the
nginx
ingress
controller
and
bind
certain
things
to
the
nginx
ingress
controller.
If
you
want
to
do
things
like
client,
cert,
attenti
keishon,
but
terminate
that
at
the
nginx
ingress
controller,
it
allows
you
to
do
all
of
that
stuff.