►
From YouTube: Manage secrets in K8s using GitOps without compromising security • Rajith Attapattu & Andre Adam
Description
Managing secrets is an essential consideration when you adopt Kubernetes.
There are several approaches to storing and correctly using secrets while following a GitOps approach. These improve the coordination between your developers and platform / SRE teams without compromising security and while getting the full benefits of GitOps.
Webinar participants will learn:
- Different options available when handling secrets in Kubernetes
- How we evaluated the options, the pros and cons of each and why we selected the External Secrets Operator
- How we implemented the External Secrets Operator with HashiCorp Vault and the lessons we learned in doing so
Presentation slides: https://bit.ly/3SowXGd
__________
- Interested in more great talks? Check the upcoming ones here: https://platformengineering.org/groups-upcoming-talks
- Join PlatformCon 2023: https://platformcon.com/
- Join our Platform Engineering Slack community: https://platformengineering.org/slack-rd
A
A
B
A
C
Awesome
all
right:
okay,
first
of
all,
Andre
and
I-
are
really
excited
to
have
the
opportunity
to
share
our
humble
learnings
with
the
platform
engineering
community.
All
right,
I
also
hope
I
know
my
voice
will
hold
over
for
the
next
couple
of
minutes.
I
have
a
pretty
bad
cough,
but
I
think
we
can
get
through
and
let's
hit
the
next
slide.
I
want
to
spend
a
minute
talking
about.
C
You,
know,
kind
of
how
we
got
started
in
the
platform
engineering
space
and
how
we
ended
up.
Talking
talking,
you
know
getting
more
information
or
knowledge
about
the
topic
that
we
are
going
to
talk
today.
Over
the
last
couple
of
years.
We've
migrated
a
lot
of
applications,
workloads
onto
kubernetes,
mostly
open
shift,
because
we
haven't
had
Partners,
but
also
some
other
kubernetes
flavors
right
and
managing
secrets
was
always
a
topic
of
discussion
right
and
we
worked
with
some
financial
institutions.
C
Government
agencies
all
the
way
down
to
some
smaller
startups,
but
this
was
a
pain
point
across
the
board
and
then,
when
we
work
with
these
customers.
The
second
thing
we
kind
of
noticed
was
that
almost
like,
you
know
like
most
of
these
customers
that
we
worked
with
had
a
team
that
that
kind
of
you
know
try
to
make
it
easy
for
the
rest
of
the
developers
in
that
company
and
they
built
tooling
and
scaffolding
and
platforms.
C
C
So
when
we,
when
we're
talking
about
this,
we
are
coming
from
both
angles,
one
as
a
you
know,
High
experience
with
Consulting
working
with
other
customers
who
had
similar
challenges
in
managing
secrets
and
as
well
as
our
own
experience,
trying
to
manage
sequence
in
our
own
production,
environment
and
obviously
in
other
environments
as
well.
So
that's
a
kind
of
story.
Hopefully
that
gives
you
a
bit
of
context
behind.
You
know
the
the
next
set
of
slides
that
we're
gonna.
You
know
talk
about
right.
C
Let's
start
with
a
quick
primer
on
kubernetes
Secrets
I,
I'm
I'm,
assuming
most
of
you
are
already
familiar
with
it,
but
just
in
case,
if
someone's
new
I,
just
don't
want
you
to
be
left
behind.
So
let
me
take
a
a
quick
30
seconds
on
this,
and
number
one
to
realize
is
that
kubernetes
Secrets?
Although
the
word
secret
is
used,
it
is
not
secure,
it's
basically
encoded
very
easily
reversible
and
that's
that's.
That's
some
of
the
that's
one
of
the
things
that
I've
noticed.
C
C
This
problem
at
all
the
other
thing
I
wanted
to
quickly
talk
about,
which
is
really
a
little
bit
out
of
scope
from
today's
webinar,
but
kind
of
important
to
mention
is
that
once
the
secret
is
on
the
kubernetes
cluster
there's
a
couple
of
things
you
need
to
pay
attention
to.
One
is
you
need
to
in
you
know
encrypted
CD,
most
managed
kubernetes
Services
out
there
do
this
by
default,
if
you're,
installing
your
own
communities
clusters,
something
to
think
about
and
the
second
one
is
permissions.
C
Many
places
that
I've
been
to
I've
seen
that
they
they
forget
to
limit
the
role.
So
sometimes
they
only
want
the
developers
to
see
have
access
to
see
the
deployments
or
I
mean
the
service
Ingress
whatever,
but
forget
to
take
away
the
secret.
So
they
can
easily
you
do
a
group
CTL
again
and
see
the
secret
so
very
important
to
make
sure
that
the
correct
R
back
is
used.
If
you
want
to
efficiently
manage
the
these
secrets
in
a
secure
way,
right,
I
think
we
are
good.
C
Now,
let's
get
started
on
the
main
topic
right.
Yes,
two
ways
of
managing
secrets
that
are
that
is
gift
shops
friendly.
Now
one
option
is
to
store
an
encrypted
secret.
The
second
one
is
to
still
a
reference
to
the
secret,
so
I'll
first
kind
of
explain
the
two
and
then
I'll
give
you
my
own
perspective
and
opinion.
C
When
you
store
the
encrypted
speaker,
you
erected,
you
so
use
something
like
you
know.
If
you
see
Vietnamese
co6
use
Google,
you
know
you
created
a
seal
Secret,
that's
the
one
that
gets
committed
to
get
when
it
gets
applied
to
the
kubernetes
cluster,
the
COC
software
that
looks
at
it
and
and
creates
the
unencrypted
and
creates
the
actual
secret
right.
Because
you
know
the
key
is
not
there.
C
Even
if
someone
gets
into
your
git,
they
can
unencrypt
right
and-
and
this
is
a
fairly
simple
approach
in
that
you
only
kind
of
need-
the
the
the
the
the
the
the
co
Secrets
operator.
I
haven't
personally
tried
most
less
hubs,
but
a
similar
approach,
and
you
know
you
have
to
kind
of
worry
about
key
rotation
and
things
like
that
as
well.
This
was
the
first
approach
we
tried
and
I'll
come
back
to
comparing
the
two
in
a
bit.
C
C
Now
let
me
compare
the
two
when
I'm
carrying
I'm
looking
when
I'm,
comparing
I'm
looking
at
it
from
two
angles,
both
from
a
a
developer
perspective,
I
still
try
to
be
Hands-On
as
much
as
I
can.
Although
my
time
is
getting
limited
and
limited,
but
I'm
always
a
big
believer
in
that
the
more
you
Empower
and
enable
your
developers
the
better,
because
I
spent
majority
of
my
life
as
a
developer
writing
code,
and
the
second
angle
I'm
looking
at
here,
is
from
a
management
point
of
view
as
a
CTO.
C
How
do
I
ensure
security
and
scalability
and
ensuring
that
you
know
I
I
I,
take
care
of
the
management
aspects
of
keeping
things
secure,
so
my
personal
preference
I
have
no
issue
with
the
the
encrypted
secret
type
I've
known
customers
who've,
who
used
both
approaches
very
successfully.
But
my
personal
opinion
is
I
prefer
the
second
approach,
because
it
give
two
things
one
is.
C
What
I
like
about
this
approach
is
that
the
secrets
are
stored
and
managed
in
a
purpose-built
tool
for
that
and
there's
a
lot
of
value
behind
that,
like
whether
you
know
whether
you're
using
walls
where
they're
using
Azure
or
AWS,
you
know
you
know,
keywords
Solutions,
there
is
the
security
there.
There
is
audit
logs,
there's
rotations,
there's
a
lot
of
things
built.
A
C
C
I'm
really
glad
I
got
through
these
three
slides,
without
falling
apart
I'm
gonna
hand
over
to
Andre,
who
would
who
will
explain
all
the
details
and
how
we
ended
up
putting
all
of
these
things
together
and
and
our
approach
and
we'll
be
on
that
on
the
chat
later
on.
To
answer
your
questions
as
well,
what
you're
wondering.
A
A
One
thing
that
it
wasn't
really
nice
from
this
approach
was
that
our
experience
with
kubernetes
would
not
translate
really
well,
so
we
had
to
configure
Vault
directly
the
files
and
then
restart
the
service.
It
lets
some
issues
and
one
of
the
things
was
that
the
paths
we
we
did
the
initial
structure
for
the
past
to
the
secrets
we
did
in
a
way
where
we
would
separate
by
cluster
regions,
and
it
would
become
quite
a
bit
complicated
to
manage
all
of
the
the
permissions
needed
to
access
those
paths.
A
They
were
becoming
quite
large
and
not
really
human
friendly
to
read
so
when
a
developer
needed
a
path.
Just
one
single
missing
path
would
be
enough
to
to
make
it
not
work.
So
then,
you
know
next
approach.
When
we
had
the
time
to
take
a
re-look
at
it,
we
chose
to
go
with
instead
of
having
outside
of
the
kubernetes
cluster.
We
we
used
the
helm
chart
to
deploy
Vault
within
the
kubernetes
cluster
itself.
A
A
We
did
end
up
using
more
external
storage,
but
in
the
end
it
was
worth
it.
Although
in
this
approach
we
couldn't
use
common
share
some
values
between
the
Clusters,
our
classes
were
different
enough.
Where
that
wasn't
that
much
of
an
issue,
and
so
this
was
the
best
approach
that
we
do.
We
we
could
come
up
with.
For
our
case,
then
I'm
going
to
talk
a
bit
more
about
external
Secrets,
which
is
the
thing
that
connects
to
Vault.
So
it
can,
in
the
end,
generate
the
secret.
A
You
configure
that
once
and
then
you
can
create
a
crd
for
external
secret,
pointing
to
some
storage
backend
and
the
operator
self
will
handle
all
the
creation
for
the
secret.
So
in
the
end,
what
you
actually
store
and
get
it's
an
external
secret
crd
and
not
SD
card
itself.
So
it's
just
the
reference
as
Reggie
said.
A
So
this
is
more
of
a
look
into
the
the
whole
integration
within
the
cluster.
So
the
only
outbound
connection
for
this
to
work
is
to
the
external
storage,
which
has
some
Adventures
that
I'm
going
to
mention
later,
but
within
the
class
itself
we
also
have
a
zero
trust
policy.
So
the
only
thing
that
Vault
actually
talks
to
or
communicates
with
is
external
secret.
A
A
So
yeah,
Excel
Secrets
is
the
only
one
that
has
access
to
Vault.
That
creates
the
secrets
for
us
in
the
end
I'm
going
to
talk
a
bit
about
the
authentication
with
fault
within
kubernetes,
there's
a
few
options
for
it.
We,
since
we
had
more
expensive
with
kubernetes,
we
just
thought
it
was
logical
to
choose
the
kubernetes
authentication,
but,
as
you
can
see,
it
can
be
a
bit
complicated.
This
is
a
diagram
that
that
they
they
have
on
the
blog
post
from
them.
A
A
It
does
take
a
bit
to
learn,
but
I
would
say
it's
definitely
worth
it
and
it's
definitely
something
that
we'll
most
likely
do
most
of
the
things
once
so,
once
we
get
a
new
cluster,
we
do
the
initial
configuration
and
then
we
just
do
a
bit
of
adjustments
and
here
and
there
that
don't
consume
a
lot
of
our
time.
It
is
it's
very
simple,
so
you
can
see
here.
We
we
need
a
few
things.
So,
of
course
we
need
to
enable
the
authentication
in
the
the
first
approach
that
we
took.
A
A
Another
good
thing
by
having
the
cluster
within
you
don't
have
to
make
any
outbound,
so
you
just
make
a
a
reference
to
to
the
local
domain
of
the
kubernetes
host
and
also
this
the
the
cas.
Then
you
create
policies.
The
policies
with
involve
is
basically
what
gives
you
access
to
paths
and
and
the
path
is
where
the
secrets
are
stored.
So
it's
a
secret
sequence
of
paths
you
can.
C
Quick
question
here,
I
think
worth
clarifying
is
that
if
you
have
a
name
Secret
in
a
particular
name
space
and
and
if
I
don't
have
access
to
the
second
namespace
I
shouldn't
be
able
to
use
external
Secrets
as
a
backdo
to
X
grabbed
grab
the
values,
even
though
I
don't
have
access
to
the
namespace
I.
Think
that's
the
part
that
you
want
to
to
highlight
correct.
Yes,.
A
I
will
highlight
that
in
here,
as
you
can
see,
so
the
rules
is
what
links
to
a
policy
which
then
it's
what
actually
gives
you
permission
and
we
bound
that
to
a
service
account
in
a
specific
namespace.
So
in
our
case
we
normally
use
service
accounts
from
services,
and
then
those
Services
have
specific
access
to
only
the
secrets
that
it
needs
to
have
to
be
able
to
run
that
application.
A
A
So,
as
you
can
see,
we
have
a
secret
store,
which
is
something
that
I
haven't
mentioned
yet,
but
it's
basically
a
reference
to
how
excellent
secret
can
communicate
with
the
back
end.
They
we.
When
we
first
started,
we
actually
used
the
old
operators
that
didn't
have
this
yet.
So
this
is
something
that
was
recently
added
with
the
newer
operator.
A
A
So,
as
you
can
see
here,
the
only
things
that
we
are
now
storing
in
get
are
the
Secret
store
for
some
applications
and
then
the
external
sequence
that
references,
those
Secret
store,
normally
Secret
store.
We
we
have,
of
course,
the
name
space
spaces,
and
only
within
that
namespaces,
where
we
have
access
to
it.
A
A
A
For
example,
one
of
the
secrets
that
it
that
is
needed
before
is
one
for
the
back
end
access
for
Vault
itself,
so
without
it
we
there
would
be
no
way
to
that.
Vault
would
be
able
to
connect
with
its
back
end
external
backend
and
yeah.
So
that's
one
of
the
ways
the
that
we
do
it
another
one
that
I
I
want
to
bring
up
is
a
very
useful
thing,
which
is
if
a
vault
is
down
or
sealed.
A
So
if
that
region
is
unavailable,
we
cannot
reach
anymore.
If
we
bring
another
cluster
up-
and
we
point
to
the
same
external
storage,
all
the
configuration
will
still
be
there,
but
there's
only
a
few
other
ones
that
that
we
need
to
update
the
like
the
the
cluster
reference,
but
that
actually
really
helps
us
make
the
process
quicker
to
come
back
up,
and
that
is
actually
really
helpful
for
us.
A
A
A
So
one
thing
to
think
conversation
is
yeah
the
time
zones
and
who
do
you
share
it
with
for
external
Secrets?
One
thing
that
really
helps
us.
So
since
we
already
have
more
knowledge
with
kubernetes,
we
use
customize
to
help
us
with
the
crds
for
external
secrets
and
the
Secret
store.
Basically,
we
make
a
few
generics
templates
that
most
of
our
application
uses.
A
Another
one
was
at
some
point:
external
speakers
was
having
some
issues
with
thinking.
It
would
just
stop
and
we
would
not
really
know
much.
Why
would
it
stopped?
There
was
no
ugly,
we
tried
the
debugging,
but
I
would
say
that
after
they
fix
it
in
the
later
version.
So
if
you're
having
that
issue,
I
would
suggest
to
update
your
external
super
operator,
we
had
to
deal
with
it
for
for
a
bit
of
a
long
time
and
yeah.
It
was
really
nice
when
they
fixed
it.
B
B
Yeah
you
can,
you
can
stop
sharing
your
screen,
so
I
can
see
some
questions
in
the
chat
and
I'll
just
like
start
reading
them
out.
But
please,
let
me
know
if
it
was
answered
before
so
the
first
question:
if
you
allow
your
developers
to
sell
into
containers,
they
can
see
the
secrets
there
easily.
I
guess
that's
more
of
the
comment,
but
do
you
have
anything
to
add
to
that.
B
A
Yeah,
but
normally
the
developers
do
have
more
access
to
lower
environments
which
the
secrets,
although
they
are
secrets,
they
don't
really
have
that
much
of
impact.
We
do
run
ruthless
container,
but
we
do
have
enough
trust
in
our
developers
that
we
do
let
them
use
it
as
they
want.
Since
those
values.
Don't
it's
it's.
Okay,
but
in
higher
environments,
is
it's
a
more
restrict
environment,
so
less
people
do
have
access,
only
people
that
we
actually
trust
and
that's
why
that's,
how
I
would
say
hint
to
handle
that
I?
C
No,
no,
that's
exactly
what
it
is.
I
think
we
want
developers
to
be
productive
and
and
a
certain
amount
of
trust
has
to
be
given,
and
so
we
are
fine
with.
We
also
have
you
know,
separate
databases
and
other
resources
for
the
for
the
lower
environments,
and
so
that's
fine
I
mean
they
have
access
to
database
for
testing
anyways.
So,
and
you
know
we
have
to
share
that
yeah.
So
that's
not
an
issue
for
us
at
all.
Yeah.
B
C
I
didn't
quite
understand
the
question
itself
I.
Maybe
if
you
can
follow
it
up
with
like
what?
What
exactly
are
you
trying
to
do
as
as
part
of
your
pipeline?
Are
you
trying
to
generate
the
because
it's
a
one-time
thing
right,
it's
more
more
on
the
CD
side
of
things,
not
on
the
CI
side
of
things
right
like
it's,
it's
part
of
your
deployments.
So
that's
why
I'm
a
little
bit
confused?
What
are
you
trying
to
do
with
the
as
part
of
your
pipeline
itself
by
the
way?
C
B
A
A
B
A
A
A
There's
some
tools
to
import
as
well.
I,
don't
remember
the
name
now,
but
in
some
of
our
cases
where
we
have
to
export
all
the
data,
we
do
so
there's
a
tool
for
it.
I
think
if
you
just
I,
can
look
it
up
later
and
if
you
ask
again
on
slack
I
can
I
can
send
you
the
two,
but
that
is
also
some
yeah.
Some
other
developers
already
probably
have
done
some
tools
to
to
handle
the
same.
The
same
issue.
B
Well,
if
you
can
send
me
the
toolbar,
we
can
also
include
it
in
the
follow
up
to
to
the
Bible
for
everyone
next
question:
what
on
a
VM
is
an
example?
Isn't
it
other
cloud-based
secret
managers
KMS
as
well
play
well
with
the
design
explained
I,
don't
know
if
s
is
like
as
a
reward
or
that's
a
different
yeah
yeah.
A
C
A
C
Yeah
I
also
think
that's
that's
a
bit
of
an
advantage
too.
A
lot
of
you
know,
customers,
you
know
they.
They
may
switch
Cloud
vendors.
They
may
switch
other
things
as
well
with
external
Secrets,
it's
a
it's.
A
different
I
mean
it
makes
the
process
at
least
seamless
right
I
mean
there
is
some
work
to
be
done.
C
Obviously,
but
at
least
there's
some
consistency
on
the
front
end
right
like
it's
a
little
bit
like
in
the
Java
like
what
do
you
call
slfpj
kind
of
thing
you
switch
to
back
in
today?
Some
okay,
so
that's
to
me,
that's
a
bit
of
a
benefit
but
I
believe
it
so
far.
Based
on
what
we've
experienced,
it
plays
very
well
with
these
backhands,
and
you
know,
there's
more
and
more
people
using
it
testing
it
filing
bug,
reports,
it's
getting
better
and
better
I
think
there's
some
improvement.
C
B
C
You
can
add
your
perspective
as
well:
the
one
cities
on
the
kubernetes
cluster.
It
is
always
a
kubernetes
secret
and
you
can
get
around
that.
The
the
the
difference
is
in
how
you
store
it
in
in
git
right
when,
while
once
it
is
in
the
cluster,
you
need
to
work.
You
need
to
figure
out
how
to
protect
that
using
you
know,
HD
encryption,
our
back
and
other
mechanisms,
but
the
the
value
of
something
like
external,
Secrets
or
or
the
sales
sequences
that
how
you
store
it
in
in
git.
C
A
It
no
no
I
agree.
Yes,
it's
mostly
how
we
can
better
store
it
without
having
any
with
more
security
right.
There's
other
things,
of
course,
that
we
can
do
with
Secrets
like
encryption
and
changing
the
defaults
that
comes
with
kubernetes,
but
yeah,
it's
all
in
the
end,
it
will
still
all
be
there
still.
B
A
C
C
Yeah,
we'll
discussing
a
bacteria,
it's
also
like
I,
always
believe
in
Sharing
knowledge
is
the
you
know.
We
gotta
put
something
together
in
a
hurry,
but
we
we
might
be
able
to
do
at
least
some
some
aspects
of
it.
Then
we'll
write
a
blog
post
and
we
can
share
the
link
for
sure.
Okay,
give
us
the
probably
a
couple
of
days
and
we'll
we'll
get
get
back
to
you
with
the
link.
B
C
Personally,
not
a
big
fan
of
the
that
the
way
they
evolved
does
it.
We
did
evaluate
that
option
again,
I'm,
not
saying
it
shouldn't
be
used
or
anything
like
that.
There's
also,
you
know
some
other
first
class
integration
in
some
languages
too,
to
like
I
think
springboard
has
something
where
you
can
actually
grab
the
values
directly
from
what
right.
So
it
really
depends
on
your
company
standards
and
security
situations
stuff
like
that
yeah.
C
So
we
decided
to
kind
of
stay
away
from
what
using
Vault
that
way
and
instead
want
to
stick
with
the
the
vanilla
kubernetes
option,
plus,
but
still
use
Vault
through
external
Secrets,
because
we
really
love
Walt
in
some
other
aspects
right.
So
that
was
that's.
That
would
be
my
I
guess
my
honest
opinion
on
it.
B
I
didn't
see
any
other
questions
as
of
now,
but
but,
as
we
mentioned
before
guys,
you
can
hit
Andre
and
rotate
on
platform
engineering,
slack
Channel
and
if
you
think,
through
other
questions,
I'll
just
put
the
links
to
our
upcoming
events,
because
we
have
a
few
planned
in
different
cities
and
hit
me
up.
If
you
want
invent
in
your
city
and
there's
no
planned
for
now.
We
also
have
a
bunch
of
webinars
coming
up.
B
If
something
is
interesting
for
you
and
for
those
who
will
be
attending
cubic
and
Amsterdam,
we
are
planning
House
of
cube
again
bigger
variations
of
that,
so
feel
free
to
sign
up.
If
you
will
be
attending
and
I
see
one
question:
what
is
the
advantage
of
using
external
Secrets
operator
versus
helmtort.
A
A
B
C
Let's
answer
it:
yeah,
that's
actually
a
very
good
question
to
ask
I
think
to
be
honest
once
once
it
does
seem.
Okay,
if
you
grab
the
values
directly
into
a
into
a
application
it
it
is,
you
know
more
secure,
but
at
the
same
time
you
know
it
really
depends
on
the
kind
of
like
the
level
of
trust
you
have
right
like.
C
C
We
we
prefer
to
use
the
more
kubernetes
native
way
of
doing
things,
because
it
helps
us
to
evolve
as
we
you
know,
build
our
applications
and
grow
our
applications
instead
of
being
very
specific
and
tied
to
a
very
particular
tool.
So
that's
the
drawback
right,
like
you're,
very
tied
to
Vault
itself
in
in
you
know
in
how
you
grab
your
secrets.
We
prefer
to
keep
it
generic.
C
B
A
But
I
guess
the
question
would
be
if
it's
business
as
usual,
so
I
guess
they
want
to
update
the
the
configuration
without
stopping
but
yeah
reloader
would
be
one
with
which
is
another
tool
that
you
can
specify.
If
any
other
resources
for
any
resource
gets
updated,
then
the
Pod
gets
refreshed.
So
you
can
use
a
reloader
to
say
for
this
deployment.
If
there's
an
update
to
this
secret,
we
need
to
bounce
the
pods,
so
that
would
be
a
way
of
doing
it,
but
to
yeah
I
would
say
that
and.