►
From YouTube: Compliance the Easy Way: Zero-conf mTLS for Dev and smooth Day-2 f... Christian Hüning & Lutz Behnke
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Compliance the Easy Way: Zero-conf mTLS for Dev and smooth Day-2 for Ops - Christian Hüning & Lutz Behnke, finleap connect
Finleap Connect operates in a strong financial context under which regulatory requirements demand that all internal traffic between critical applications be encrypted. In this case study Christian Hüning and Lutz Behnke will review the decision making process that led to selecting Linkerd for the task of securing traffic for the Finleap application infrastructure. Christian will also provide insights into the integration process with the application teams and how Linkerd positively affected observability and reliability of the application services. Lutz will present learnings of operating workloads with Linkerd for more than a year, including how to establish secure communication through the service mesh. He will discuss the required operations work to continuously run the cluster despite updates, more than 2500 injected pods or root CA rotations, using cert-manager.
A
This
talk
is
organized
around
the
idea
of
the
challenges
we
faced
and
how
we
solved
them.
So
here's
the
first
challenge
the
challenge
involved,
how
to
encrypt
all
traffic
between
applications
in
our
cluster
in
no
time
and
with
zero
developer
overhead
to
understand
why
we
had
this
particular
challenge
of
doing
it
with
no
time
and
zero
developer
overhead,
it's
probably
important
to
understand
where
it
all
started.
It
all
started
in
2018
when
lutz
and
me,
together
with
some
other
friends
of
ours,
joined
the
company
finley
connect
to
rebuild
the
present
internal
cloud
platform.
A
A
This
whole
project
faced
several
challenges,
including
a
very
tight
timeline
of
only
five
to
six
months
to
implement
the
solution.
We
had
some
time
to
plan
it
beforehand.
We
had
a
cloud
team
of
around
about
five
people.
Some
were
only
able
to
join
a
bit
later,
so
it
was.
We
started
with
two
people
and
in
the
end,
we
ended
up
with
being
six
people
and
a
new
platform
before
going
live,
had
to
pass
compliance
checks
from
customers
and
regulators
alike.
A
Now,
when
it
came
to
the
particular
bit
for
this
talk,
the
road
to
service
mesh
and
the
challenge
of
encrypting
all
of
the
traffic
in
the
cluster,
we
basically
needed
this
before
the
go
live,
and
we
we
sat
down
at
discussing
this
matter,
looking
at
what
we
already
had
in
the
infrastructure
in
the
current
application,
landscape
and
so
option
a
was,
as
some
services
did,
it
already
to
build
it
into
each
service,
a
full
mtls
encryption
between
the
services
we
discussed.
This
read
up
a
bit
about
it
and
then
quickly
realized.
A
This
is
a
no-go
as
the
threat
from
having
to
maintain
all
these
different
implementations
in
different
languages
and
us
all
just
being
humans,
and
by
that
actually
making
potential
errors
in
the
implementation
was
too
too
high
a
risk.
From
that
perspective,
also,
the
time
perspective
simply
didn't
permit
it
so
option
a
was
off
the
table
quickly.
A
Secondly,
we
came
up
with
the
idea
to
use
haproxy
or
nginx
as
a
reverse
proxy
configured
for
mtls
based
on
an
internal
pki
and
it
basically
the
minute
we
put
this
on
the
table
and
thought
about
it
for
another
minute.
We
realized
that
probably
somebody
on
the
internet
had
already
done
this
and
probably
even
better
than
what
we
could
achieve,
potentially
in
the
left
time,
so
that
left
that
was
option
c
and
without
really
knowing
it.
We
were
looking
for
the
service
mesh
landscape
on
the
internet.
A
Doing
so
back
in
2018
quickly
brought
up
istio
as
it
has
had
hit
version
1.0
in
july.
2018.,
looking
at
the
web
page
of
istio
quickly
showed
a
very
rich
feature
toolkit
that
was
quite
compelling
to
us,
including
mtls,
and
it
also
appeared
to
be
already
been
implemented
by
a
good
number
of
organizations
which
also
spoke
for
the
project,
so
we
went
ahead
and
installed
sdo
into
our
cluster.
A
This
worked.
It
was
not
as
simple
as
we
might
have
thought
in
the
beginning,
but
it
required.
Furthermore,
it's
required
to
provide
a
specific
configuration
for
each
service,
and
that
was
something
where,
when
we
looked
at
what
it
actually
meant,
we
thought.
Okay.
This
is
a
bit
too
tough
for
developers
that
already
had
to
do
a
lot
of
things,
so
we
had
to
sustain
the
learning
curve
for
everybody
and
also
make
it
somewhat
doable.
So
we
decided
okay,
this
is
not
going
to
work.
A
Sadly,
we
can't
use
the
solution
and
that's
when
we
looked
further
and
found
another
project
called
conduit,
it
was
supposed
to
be
a
super
lightweight
and
quite
simple
to
install
service
mesh
and
we
tried
it
and
indeed
it
just
worked.
Sadly,
we
quickly
discovered
it
did
not
yet
support
mtls.
So
we
were
kind
of
back
to
the
beginning,
and
at
this
point
in
time
we
said.
Okay,
we
need
to
talk
to
some
people
in
the
community,
as
this
is
usually
what
works
best
and
especially
in
the
awesome
cloud
native
community.
A
So
we
joined
the
slack
channel
of
the
conduit
team
and
quickly
discovered
an
excellent
community
with
excellent
support
for
every
question
you
had
and
especially
that
mtls
was
promised
to
be
one
of
the
next
items
on
the
roadmap,
so
that
combined,
we
said.
Okay,
let's
take
our
bet
on
the
conduit
project
to
eventually
deliver
this
in
time.
A
A
The
7th,
with
this
new
version
that
had
fixed
this
resource
shortage
shortly
after
our
go
live
then
also
mtls
turned
into
a
general
available
state
with
link
id
2.3
on
april,
the
16th
so
long
story
short
after
this
intense
five
to
six
month
period.
Linker
d2
basically
helped
us
to
solve
this
large
challenge
of
implementing
a
full,
auto
mtls
between
all
services
in
the
cluster,
without
actually
having
to
involve
the
development
teams.
A
There
was
one
minor
thing
where
they
had
to
set
some
configurations
for
a
particular
service,
but
that
was
just
to
tweak
and
tune
something,
so
that
was
really
a
great
success
in
that
end,
so
we
went
on
migrating
our
customer
environments
onto
the
cluster,
and
here
it's
probably
important
to
mention
that
our
cluster
is
a
multi-tenant
setup
where
a
tenant
is
an
internal
team
or
business
unit.
So
we
are
not
offering
this
externally,
it's
all
in
the
same
trust
zone
if
you
want,
but
we
have
to
separate
these
environments.
A
So
the
first
thing
that
at
scaling
up
we
found
was
the
question:
how
to
scale
link
id
to
5000
pods.
Is
this
actually
an
issue,
or
does
this
just
work,
and
we
can
conclude
that
for
the
most
part
it
just
works?
Naturally,
you
have
to
increase
and
set
some
resource
requests
and
limits
appropriately,
which
we
did
so.
A
So,
basically,
these
values
became
a
challenge
and
the
fact
that
we
wanted
to
make
use
of
these
metrics
naturally
and
we
did,
but
it
was
only
always
limited,
and
it
was
always
a
bit
hard
to
work
with
those
values.
So
we
ended
up
in
taking
our
tenancy
concept
and
the
fact
that
we
we
had
some
work
to
do
here
to
turn
this
into
a
proper
monitoring
solution
that
I'm
going
to
explain.
Now.
A
It's
based
around
the
idea
to
slice
and
dice
the
linkade
metrics
alongside
our
tenant
deployments.
For
that
to
understand
the
first
thing
we
did
is
to
disable
the
linker
d
prometheus.
You
can
by
now
also
do
this
just
in
the
helm
chart
when
deploying
link
id.
So
I
think
id
itself
has
no
prometheus
and
then
our
tenancy
system
works
like
that.
A
Each
tenant
has
a
dash
system
namespace
here
in
the
middle
and
then
the
respective
workload
namespaces
belonging
to
that
particular
tenant
and
as
we
follow
basically
a
concept
of
eating
our
own
dog
food
in
the
cloud
team.
Also,
the
cloud
staff
is
implemented
as
a
tenant
and,
as
you
can
see
here,
below
the
affinity
cloud
system.
A
Namespace
contains
a
prometheus
deployment
that
scrapes
the
linker
the
control
plane
metrics,
together
with
all
the
other
kubernetes
and
infrastructure
metrics,
and
then
offloads
them
to
a
thanos
deployment
for
long-term
metric
storage,
and
we
also,
of
course,
use
the
prometheus
then
for
alerting
based
on
these
metrics
tenant
system.
Namespace,
on
the
other
hand,
has
an
additional
prometheus
with
a
one
day.
A
Retention
time
also
only
used
for
alerting
and
a
also
a
thanos
sidecar
for
offloading,
and
it
scrapes
all
of
the
linkade
proxy
metric
endpoints
from
the
tenant's
workload
namespaces
by
that
reducing
the
overall
amount
of
metrics
in
each
prometheus
for
each
tenant
and
optionally.
These
teams
are
able
to
federate
with
a
prometheus
proxy,
which
only
exposes
some
selected
linker
d
control,
plane
and
also
general
kubernetes
control,
plane
metrics
to
the
tenants
if
they
are
interested,
but
that
also
further
reducing
the
required
amount
of
metrics
and
resources
per
prometheus
per
tenant.
A
B
B
B
But
there
are
certain
small
details
that
make
us
hard
on
us,
because
one,
the
home
update,
is
not
really
item.
Potent
every
single
helm,
update
run
will
create
a
new
set
of
certificates
and
private
keys
for
each
of
the
web
hooks
and
because
they
are
self-signed,
you
cannot
rotate
them
in
a
stable
manner,
stable
manner
being
that
you
create
a
new
set
of
new.
B
This
does
work
or
did
work
in
version
2.7
already
and
was
properly
documented
for
the
identity
chain,
but
not
for
the
web.
Hooks
in
linker
d,
you
have
to
separate
or
distinguish
between
two
basic
groups
of
certificates,
one's
the
actual
control
plane
that
generates
the
proxy
certificates
for
the
automatic
mtls
between
all
the
the
parts
in
your
your
applications
and
the
other
is
the
initially
four
today
only
three
web
hooks
and
api
services
that
pose
the
kubernetes
api
integration.
A
B
The
changes
we
introduced
start
initially
with
improving
the
compatibility
with
cert
manager.
Some
of
these
were
small
things
like
improving
or
changing
formats
of
certain
kubernetes
secrets
and
others
where
the
support
for
a
separate
say
I
cert
for
the
web
hooks
as
well
as
allowing
tools
external
to
the
helm,
chart
to
set
the
certificates
within
kubernetes
secrets,
so
that
external
tools
could
then
access
them
for
rotation.
B
B
Chain
the
control
plate
chain
and
one
for
the
web
hooks,
which
are
then
used
by
the
circ
manager
to
generate
the
intermediate
certificates,
which
then
still
is
used
by
the
identity
issuer
to
create
the
proxy
certificates
and
in
finley
connect.
We
go
one
step
further,
using
a
not
yet
released
as
open
source
volt
operator
that
pulls
secrets
from
hashicorp
vault
into
our
cluster,
because
we
believe
in
not
granting
or
not
making
certain
secrets
and
and
credentials
visible
to
any
human
being.