►
From YouTube: The Insider Threat: Third-Party Applications In Your Cluster - Dagan Henderson & Will Kline
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from April 17-21, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
The Insider Threat: Third-Party Applications In Your Cluster - Dagan Henderson, Raft, LLC & Will Kline, Dark Wolf Solutions
Speakers: Dagan Henderson, Will Kline
As powerful as Kubernetes is out-of-the-box, it’s a reasonable bet that your organization’s baseline cluster includes more than just the core Kubernetes components. Service meshes, CSI drivers, admission controllers, and database engines are nearly ubiquitous additions to production-ready clusters. Crucially, these applications allow your organization’s development teams to focus on solving the organization’s unique challenges by building on top of robust third-party solutions that solve common industry problems, but vulnerabilities in third-party code can put the security of your clusters at risk. In this talk, the speakers will briefly review a few examples of real-world vulnerabilities in third-party applications commonly found in large Kubernetes clusters and describe just how they were discovered; demonstrate how critical some vulnerabilities can be; and then review clear, actionable steps your organization can take to help prevent third-party vulnerabilities from being the weak link in your clusters’ security.
A
Hi,
my
name
is
Degen
I'm,
a
developer
and
occasionally
a
security
researcher.
For
the
last
couple
of
years,
I've
been
working
primarily
as
a
kubernetes
platform
engineer
and
I
really
enjoy
working
in
kubernetes
and
compliant
restricted
environments.
So
I
think
things
like
sock,
2
and
fedramp
sort
of
my
happy
place.
B
My
name
is
Will
Klein
I'm,
also
a
platform
security
engineer.
You
know
it's
been
really
exciting
for
me
to
watch
kubernetes
grow
over
the
last
couple
years.
You
know
I
kind
of
started
out
doing
containers
in
kind
of
similar
environments.
This
is
what
day
is
described
and
we're
you
know
orchestrating
them
with
salt
stack.
I,
don't
know
if
you
guys,
remember.
A
Today,
we're
going
to
be
talking
about
third-party
applications,
why
we
love
them
some
of
the
dangers
that
they
impose.
So
what
do
we
mean
when
we
talk
about
a
third-party
application
and,
basically,
ever
since
subroutines
were
invented
in
the
1940s,
people
have
been
running
other
people's
code.
So
third
party
is
code
that
you
didn't
write
and
the
question
is
not
do
I
use
third-party
applications.
It
is
an
absolute
surety
that
you
are
going
to
use
a
third-party
application.
B
Yeah
and
I
think
one
of
the
best
things
about
kubernetes
in
my
experience
has
been
you
know
going
from
that
world,
where
you
know
vendors
would
deliver
you
a
random
RPM
file
or
even
like
a
tarball,
and
then
you
eventually
it
turned
into
like
a
VM
that
they
would
deliver
to
you
and
you'd
have
to
figure
out
what's
in
this
VM,
what
does
it
need
to
run?
How
do
I
need
to
network
it
together,
like
what
is
it?
What
is
it
bringing
to
my
my
network
here?
What
about
when
I
install
it?
B
How
do
I
meet
my
security
baselines
and
now,
with
kubernetes
we're
delivering
applications
and
getting
applications
from
third-party
vendors
which
have
a
full
description
of
you
know?
What
kind
of
containers
does
it
need?
What
is
the
networking
requirements?
What
kind
of
service
account
access
does
it
need,
and
it
brings
all
that
together
in
a
really
beautiful
API
that
I
can
use?
You
know
whether
I'm
deploying
an
Ingress
on
albs
and
AWS
or
I'm,
using
metal
lb
on
my
bare
metal
servers.
B
A
In
addition
to
kubernetes
itself,
what
do
we
add
on
top
of
it?
Well,
pretty
generally
we're
going
to
add
observability
we're
going
to
potentially
add
a
get
Ops
agent
of
some
sort,
if
you're
not
using
that.
Yet
you
should
be
you're
going
to
have
your
database
engines
which
might
be
running
in
the
cluster.
They
might
not
be
that
it
sort
of
depends.
What
about
your
base
containers
right
if
you're
running
a
language
like
Java,
then
you're
bringing
in
an
open,
jdk
or
maybe
something
direct
from
Oracle,
a
Java
based
container.
A
What
about
all
of
the
packages
that
are
running
on
that
base,
container
or
node.js
also
has
a
lot
of
extra
packages
in
that
base
container
if
any
of
those
have,
if
any
of
that's
exposed
externally,
like
let's
say,
grafana
and
grafana,
has
a
vulnerability
in
the
web
UI.
Your
cluster
now
has
a
vulnerability.
If
there
is
a
vulnerability
inside
the
an
you
know,
cluster
Network,
a
service
level
thing,
then
your
cluster
has
a
vulnerability
that
might
be
used
either
as
a
pivot
or
a
escalation
of
privilege.
Sorry
space
for
a
second
there.
A
So
we
need
to
be
really
aware
of
all
those
and
we
need
to
think
about
those
possibilities
as
we're
deploying
our
applications.
Essentially,
when
code
is
vulnerable,
your
cluster
is
vulnerable.
I
like
this
little
picture,
because
I
picture
the
third
party
application
being
the
ladder
that
you're
walking
across
right.
A
You
can
sit
there
and
be
like
well
I
didn't
write
this
ladder,
but
if
it's,
if
it
fails,
it's
a
long
way
down
for
you
back
in
August
will
and
I
gave
a
presentation
at
Defcon
and
we're
going
to
give
you
a
short
snippet
of
that
presentation
here
in
just
a
minute
and
before
I
do
it's
sort
of
sped
up,
because
that's
not
what
this
conversation
today
is
about,
but
it's
a
I
think
it
sort
of
sets
the
tone
of
why
today's
conversation
is
so
important.
So
what
we
did
at
Defcon
is.
A
We
took
three
vulnerabilities
that
we
had
found
in
common
kubernetes
third-party
applications,
specifically
chiali
Fleet,
which
is
a
get
Ops
agent
from
the
folks
at
Rancher
and
Longhorn,
which
is
a
distributed
storage
mechanism
for
kubernetes,
also
from
the
folks
at
Rancher,
and
we
had
found
various
vulnerabilities
over
the
years
they're
all
patched.
They
were
all
responsibly
disclosed
well
before
we
started
talking
about
them
and
and
demonstrating
how
to
take
advantage
of
them.
But
what
we've?
A
We
realized
from
looking
at
some
logs
from
Fleet
that
when
fleet
has
an
error
trying
to
reach
a
repository,
it
logs
the
URI
that
it
passed
to
a
third-party
library
to
access
that
and
that
URI
includes
embedded
in
it
base64
encoded
the
private
SSH
key
that
was
used
to
access
the
git
repository,
which
means
that
if
you
can
get
access
to
that
log,
you
can
then
access
that
repository.
You
know,
as
that
git
Ops
user
and
then
finally
Longhorn
in
Longhorn
we
identified
back
in
was
it
20?
A
It
was
just
last
year
it's
20
20
21.,
so
last
December
there
is
a
API
that's
exposed
inside
the
cluster
that
allows
you
to
allows
the
longhorn
manager
to
specify
a
binary
and
a
string
of
arguments
that
it
wants
the
agent
to
run
in
the
context
of
of
that
container.
It's.
It
was
a
completely
unauthenticated
API.
A
So
if
you
could
discover
that
running
in
the
cluster,
which
was
effectively
running
as
a
demon
set,
so
any
node
that
was
was
listening
on
Port
8500,
you
could
you
know
Say
Hey
I
want
you
to
run
Echo
out
to
a
file,
and
it
just
doesn't.
You
can
also
use
that
to
pop
a
reverse
shell,
which
is
what
we
did
and
additionally
because
of
the
Privileges
that
longhorn
needs
to
do
its
work,
it's
running
as
root,
and
it
had
several
host
paths
mounted
that
we
were
then
able
to
use
for
a
full
container
Escape.
A
A
One
of
the
things
that
kiali
does
is
it
allows
you
to
access
logs
of
running
workloads,
which
is
super
useful
when
you're
trying
to
you
know,
diagnose,
what's
going
on
within
your
service
mesh
that
includes
logs
from
Fleet
that
may
actually
may
include
SSH,
private
key
and
the
URL
to
that
private
or
to
the
repository
that
that
private
key
can
be
used
as
so
here.
What
we're
going
to
do
is
we're
going
to
set
up
my
terminal
to
use
that
SSH
key
to
log
in
to
get
and
see.
What's
in
this
Repository.
A
All
right,
so
it's
just
a
pretty
bare
Helm
chart.
All
I
need
to
do
is
add
a
job.
That's
going
to
give
me
whatever
my
attack.
Payload
is
and
push
that
back
up
and
then
get
Ops
is
going
to
do
get
Ops
it's
going
to
deploy
my
job,
so
I
started
listening
on
Port
one
two,
three,
four
five
for
my
reverse
shell
and
I,
went
ahead
and
started
that.
A
There
we
go
so
now
I'm
running
as
root
on
the
the
node,
that's
running
Longhorn,
which
is
pretty
devastating.
So
with
all
that
you
know.
How
can
we
prevent
that
I
mean
this
was
a
complex
attack
right,
we
chained
together
three
different
vulnerabilities,
but
it
really
didn't
need
to
be.
If
you
look
at
just
chiali,
if
chiali's
not
running
in
read
only
mode,
you
can
do
devastating
attacks
just
on
that
very
first
phase:
complete
traffic
management,
fun
things
like
that,
the
with
Fleet
it's
running
as
get
Ops.
A
You
know,
there's
probably
not
a
lot
of
restrictions
on
what
you're
allowing
your
get-offs
service
account
to
install
into
your
cluster,
so
I
didn't
need
to
bother
pivoting
to
Longhorn.
I
could
have
just
installed.
You
know
my
own
demon
set
from
the
get-go
and
then
finally,
you
know
with
Longhorn.
You
know
that
was
you
you,
let
your
imagination
go
wild
right
and
you,
if
you
already
are
running
in
a
multi-tenidented
environment,
then
something
like
LongHorn's
vulnerability
is
truly
devastating.
A
Because
then
you
know
anybody,
that's
running
a
workload
in
your
cluster
is
going
to
be
able
to
take
advantage
of
that
see.
One
thing
I
do
want
to
point
out
with
these
three
also
is:
these
are
not
poorly
written
applications
right,
so
Sousa
and
red
hat
know
how
to
develop
software,
and
they
worked
really
well
with
us
when
we
reached
out
to
to
disclose
the
vulnerabilities.
So
this
isn't
about
saying,
like
oh
there's,
bad
software
out
there,
it's
more
about
saying.
B
Yeah
so,
basically
at
while
we
were
putting
that
together
together
that
Defcon
talk
and
immediately
after
this,
these
vulnerabilities
were
disclosed
and
we're
kind
of
reevaluating
things.
You
know
Dave
and
I
started
having
conversation
about.
You
know.
How
did
we
end
up
getting
here
because
you
know,
obviously
third-party
apps
are
necessary
to
be
running
your
cluster.
You
know
I,
don't
who
Among
Us
wrote
kubernetes
right
like
it
is
all
third
party.
B
Person
did
not
write
it
and
you
know
we're
all
we're
all
using
it
together.
So
kind
of
the
question
that
we
had
is
like
as
an
organization.
How
can
we
be
looking
at
third-party
applications
as
we
bring
them
into
our
cluster,
and
how
do
we
make
sure
that
we
can
make
find
things
that
align
to
our
internal
development
practices
so
at
the
job
that
we
were
working
with
together
on
this?
B
There
were
all
these
CI
CD
requirements,
SAS
Das,
all
this
stuff
fuzzing
in
place,
a
lot
of
merge,
request,
controls
and
one
of
the
questions
we
had
was
like.
How
can
we
make
sure
that
we
are
looking
at
the
whole
application
life
cycle,
because
when
it
came
to
third-party
applications,
our
shop
and
I'm
sure
many
many
other
places
we're
looking
at
that
cve
scan
and
coming
back
like
no
CVS?
No
problems
right,
so
we
started
talking
about
like
how
do
we?
How
do
we
start
looking
at
applications
to
really
figure
out?
A
A
The
first
thing,
I
think
is
is
important
to
understand
is
this
is
not
about
a
single
point
in
time,
pretty
much
any
application.
That's
been
around
long
enough.
Is
that
and
used
widely
enough
is
at
some
point
in
time.
Gonna
have
a
cve,
eventually
it'll
probably
have
a
devastating
CPE.
So
it's
not
about
saying
what
is
the
state
of
this
application
right
now?
I
now
make
a
permanent
decision
for
all
of
time
that
this
is
safe
and
it
can
move
forward
it's
more
about.
What
is
the
project
look
like?
A
A
Yes,
security
policies
as
well,
you
know,
is
there
a
security
policy
is:
do
they
publish
security
advisories?
Do
they
even
publish
cves,
if
you,
if
an
application
is
not
using
GitHub
security,
advisories
or
cves?
Your
scanners
are
not
going
to
find
anything
because
they're
pulling
from
a
vulnerability
database
that
has
no
entries
for
that
application.
That
doesn't
mean
the
application.
Has
no
vulnerabilities
just
means
they're,
not
telling
us
about
them.
So
it's
really
important
that
we
look
for
applications
that
do
or
projects
that
do
publish
security,
advisories.
B
So
anyway,
so
we're
working
on
these
projects
together
and
you
know
we're
one
of
the
examples
that
came
up
in
our
development
cycle.
Immediately
after
the
longhorn
incident,
we
were
looking
at
bitnami,
sealed
Secrets
at
the
time
really
the
bitnami
sealed
Secrets
didn't
fit.
You
know
it's
a
fine
project,
it
just
didn't
fit
into
the
security
model
that
we
were
internally
developing
where
Vault
was
a
single
source
of
Truth.
B
So
we
started
looking
around
for
alternatives
to
benombi,
sealed
secrets,
and
you
know
we
started
looking
around
and
coming
up
with
a
list
of
Alternatives
and
we
decided
at
that
point.
We
should
be
applying
things
to
this
search
so
that
we
are
evaluating
not
just
the
the
point
in
time.
Does
this
have
cves?
Is
it
publishing
those
cves?
B
Does
the
specific
container
have
CDs,
but
instead
look
more
holistically
at
the
whole
project?
And
you
know
we
worked
together
to
assemble
this
list
of
potential
candidates
and
then
we
started
analyzing
them.
You
know
really
trolling
through
their
GitHub
projects
to
see
what
they
were
doing
internally
and
see
if
that
kind
of
aligned
with
our
values.
B
It's
checking
to
make
sure
that
they're,
mer,
they've
got
good.
Merge
hygiene
in
terms
of
you
know
who
are
people
going
through
merger
Quest
merging
to
the
master.
So
from
our
perspective,
these
are
all
of
the
things
that
we
were
requiring
the
developers
within
our
program
to
do
before.
They
could
run
their
code
in
our
production
cluster
and
it
made
sense
to
us
to
apply
those
same
requirements
to
third-party
applications
that
we,
as
the
platform
maintainers,
were
bringing
into
the
platform.
B
So
we're
gonna
got
a
little
quick.
Hopefully
people
can
read
this
so
this
is,
can
can
everyone
read
that
I
think
we
got
a
review
earlier
Yeah?
So
basically
we
have
here's
kind
of
the
first
half
of
the
scorecard.
We
broke
it
up.
So
it's
looking
at
whether
the
binary
artifacts
are
being
published.
That's
pretty
self-explanatory.
B
They're
looking
at
the
settings
within
GitHub
to
make
sure
that
Branch's
protection
is
enabled
and
the
main
production
branch
is
not
being
just
randomly
committed
to
and
pushed
to
by
Outsiders
it's
looking
at
the
CI
CD
tests.
It's
looking
for
CII
best
practices.
You
know
the
CIA
best
practices
is
actually
the
projects
themselves
will
begin.
Embedding
this
and
you'll
you'll
once
you
know
to
look
for
it.
A
lot
of
projects
are
including
this
I
know.
B
Psyllium
has
it
I
know
Argo
has
it
on
their
GitHub
page,
where
there's
the
badge
that
kind
of
shows
their
ossf
scorecard
and
what
we
take.
That
to
mean
is
that
the
the
application
developers
themselves
are
now
conscious
that
we
are
all
looking
for
these
best
practices.
You
know
I've
I've,
been
sitting
in
this
room
all
week
here,
listening
a
lot
of
really
great
talks
about
different
tools
to
enable
s-bomb
management,
but
at
the
end
of
the
day,
the
thing
as
a
maintainer,
you
know
someone
who's,
not
deploying.
B
B
You
know
maybe
that
person
gets
sick.
Maybe
they
change
jobs.
Maybe
they
get
uninterested
right.
If,
if
there's
one
company
that's
maintaining
the
future
of
this
project,
then
it's
hard
to
rely
on
them
and
not
not.
It's
a
lot
hard
to
rely
on
them
long
term,
because
it
you
don't
know
what
their
interests
are
going
to
be
you.
Maybe
they
start
making
decisions
that
are
orthogonal
to
your
security
plans,
seeing
that
a
project
has
lots
of
contributors
and
maintainers
helps.
You
understand
that
you
know
there
are
a
lot
of
stakeholders
here.
B
That
will
try
to
keep
it
in
line,
so
you
can
believe
in
the
vision,
long
term,
let's
see
I,
think
those
are
all
yeah.
You
know
it's
looking
for
workflow
patterns
that
are
dangerous,
making
sure
it's
using
some
sort
of
like
dependipot
type
tool
to
looking
at
the
dependencies
making
sure
it's
using
fuzzing
in
the
project.
The
licenses
yeah
I'm,
not
gonna,
go
through
all
this
and
I.
Think
one
of
the
things
that's
interesting
with
ossf
scorecard.
B
B
That's
actually
a
relatively
good
score,
because
a
lot
of
these
stuff,
you
know,
as
teams,
start
adopting
the
OSS
scorecard
model
and
hosting
this
on
their
GitHub
page
they're,
going
to
start
being
aware
of
it
and
once
they're
aware
of
it,
we
can
trust
that
they're
going
to
be
improving
those
scores
over
time.
We
thought
this
was
a
really
cool
tool.
B
So
once
you
have
that
scorecard
in
hand.
That's
that's
not
everything.
There
are
a
couple
other
things
that
I
think
are
harder
to
assess
automatically.
You
know
depreciation
policies
in
place.
You
you
never
want
to
end
up
in
a
situation
where
something
happens
upstream
and
you're
suddenly
depreciated.
But
you
don't
have
time
to
integrate
it
in
with
your
project,
making
sure
that
they're
showing
like
you
know
if
the
application
says
it's
in
beta
trust
that
they're
in
beta,
like
the
only
the
developers,
can
make
that
call.
Don't
trust
anyone.
A
All
right
so
at
this
point,
I'd
venture
to
guess
between
open
source
security,
Foundation
scorecards
and
all
the
added
research
that
will
had
done
on
external
secrets.
That's
far
more
work
that
has
gone
into
a
third-party
application
than
most
cluster
administrators
put
in
I'd
love
to
be
proven
wrong
on
that
I'd
love
for
the
industry
to
move
towards
something.
That's
that's
much
more
diligent
about
the
review
process.
So
are
we
done,
did
we
do
you
know,
did
we
high-five?
Do
we
install
external
Secrets
operator
and
call
it
a
day?
B
So
you
know
once
we
once
we
kind
of
settled
on
the
external
Secrets
operator
as
the
way
we
wanted
to
go
forward
this,
because
we
liked
it
as
a
project.
You
know
at
some
point
you
do
actually
have
to
do
those
scans
and
gather
that
point
in
time.
Analysis
of
what
is
the
exact
security
of
the
container
in
the
helm
threat
that
I'm
about
to
bring
in
we're
not
going
to
talk
too
much
about
hardening.
B
There's
been
a
lot
of
talks
today
that
cover
it
far
more
in
depth
than
we
want
to,
and
you
know,
I
definitely
recommend
you
look
at
our
GitHub,
our
our
Defcon
talk
about
specifics
of
how
we
were
hardening
the
longhorn
containers.
We
uncovered
all
these
vulnerabilities,
but
basically
we
want
to
say
that
you
know
this
is
when
you
start
bringing
something
in.
B
This
was
one
of
the
things
that
we
came
into
with
Longhorn,
that
we'll
talk
in
a
bit
like
where
we
were
applying
a
lot
of
hardening
changes
on
it.
That
became
very
fragile
long
term
because
it
was
not
meeting
the
Upstream
projects,
guidance
and
direction
that
they
were
going
and
as
that
Delta
grew
so
did
our
work.
B
Now
we
want
to
talk
about
what
happens
when
the
next
version
gets
released.
So
you
know
a
week
from
now
new
version
drops
new
container
what
we
were
doing
internally
at
that
point
is
we
were
building
taking
Docker
files
and
having
like
a
from
line
of
the
upstream
and
then
layering
on
our
changes,
you
know
changing
it,
so
it
doesn't
run
as
root
removing
unnecessary
packages,
one
by
one
building
that
process
out.
B
So
it
was
relatively
automated
and
then
using
the
Upstream
projects,
automated
tests
to
know
whether
or
not
our
hardening
was
breaking
it,
because
you
know
we
don't
want
to
have
to
run
through
the
manual
test.
Every
time
we
apply
our
changes.
We
just
want
to
apply
our
hardening
changes
to
their
container
and
then
run
their
automated
test,
Suite
to
confirm
that
it's
still
going
to
work
the
way
the
maintainer
intended
it
to.
A
One
thing
that
I'd
want
to
add
here
as
well,
is
just
how
difficult
it
becomes
to
harden
an
application
is
I.
Think
it's
set
itself
a
little
bit
of
a
smell
about
the
overall
security
of
the
application
it
less
mature
applications
on
average
are
far
less
secure.
We
do
have
you
know
some
very
security,
conscious,
open
source
application,
the
creators
out
there
that
start
strong,
but
not
everybody,
does
a
lot
of
the
time
things
come
in
over
time.
A
There
was
a
great
talk
here
last
night
from
the
folks
that
are
maintaining
Argo
CD
talking
about
what
happened
when
they
went
through
that
step.
Up
from
you
know,
incubation
to
a
fully
released
cncf
project
and
the
security
team
that
they
worked
with
to
understand
some
of
those
things
so
as
you're
going
through
this,
if
you're
fighting
that
application
a
lot,
it's
really
truly
is
something
to
consider.
Is
this
worth
it
do?
I
do
I
need
to
find
a
different
solution.
A
So,
for
in
a
one
of
the
nice
things
that
we
found
is,
is
the
the
folks
at
aquasec
have
just
been
absolutely
knocking
it
out
of
the
park
with
trivia.
It
has
become
such
an
invaluable
tool
that
we
use
it
for
almost
everything
that
it'll
do
it'll
scan
your
your
images
and
let
you
know
you
know:
not
only
are
there
vulnerable
packages
installed
by
the
OS,
but
there's
you
know
we
found
these
static.
Libraries
embedded
in
this
go
Application
as
well.
So
that's
really
useful.
You
know
it's
sort
of
a
starting
point
right.
A
There
here's
a
quick
walkthrough
of
using
trivia
I
like
to
run
it
in
Docker
myself,
just
so
that
I
don't
have
all
these
different
versions
of
the
tools
running
around
on
my
local
machine.
It's
that
fast
I
didn't
speed
that
up
at
all,
and
that's
just
like
regular
playback,
and
you
can
see
from
this.
There
are
some
issues
right.
It
looks
like
really
they
need.
The
external
Secrets
needs
to
update
the
AWS
SDK
library
that
they're
using
assuming
that
there's
a
newer
version
available.
B
A
Replay
yeah
there
we
go
so
in
that
particular
case
there
weren't
any
vulnerabilities,
nothing.
You
know
that
we
really
needed
to
worry
about,
but
what
about
when
there
are
vulnerabilities
that
we
need
to
worry
about,
so
here's
a
use
of
the
Tamarin
base,
image
for
running
JRE
applications
and
when
I
scanned
that
it
did
have
a
couple
of
vulnerabilities
in
them.
Trivia's
output
included
a
the
fixed
version,
so
I'm
just
really
easily
able
to
run
apt
to
to
do
a
pinned
upgrade
to
the
fixed
version.
A
If
it's
a
if
it
is
a
source
code
like
a
language
dependency,
though
I'm
going
to
tell
you,
you
do
not
want
to
get
into
the
business
of
forking
every
application
that
you're
running
every
third
party
application.
So
in
that
particular
case,
I
I
really
recommend
you
just
make
a
go
no-go
decision.
It's
either
not
going
to
be
something
that
is
super.
You
know
traumatic
and
you're.
Eventually,
it'll
get
patched
upstream
or
yank.
The
application
out
of
your
cluster
don't
try
to
fix
the
application
unless
you
are
actually
a
maintainer
contributor
to
that
application.
B
Yeah,
so
a
lot
of
places
that
I
work
kind
of
have
this,
this
mentality,
where
we
do
that
scan
at
the
beginning
and
no
no
CVS,
no
problems.
Meanwhile,
the
application
itself
is
surrounded
by
this,
like
nebulous
group
with
a
Helm
chart.
That
is,
like
you
know.
We
run
into
this
all
the
time,
and
you
know
we're
not
here
to
like
bash
on
a
lot
of
companies,
the
stuff
that
we
see
where
you
know,
applications
have
I
run
into
literal.
A
A
A
B
Same
I
think
we've
just
yeah,
so
as
you
can
see,
with
the
trivia
output
in
hand,
you
know
we
now
have
the
point
in
time
snapshot
of
the
application
and
running
inside
the
container.
That
seems
to
have
you
know
some
things
in
it,
but
we're
willing
to
accept
some
risk
on
this.
You
know
it's
absolutely
we
never
it's,
never
an
all
or
nothing
right.
You've
got
to
balance
the
value
something
brings
to
your
organization
versus
the
potential
risk
of
bringing
it
into
your
cluster.
Now
we
have
the
helm
chart
scan.
B
You
know
that
we
can
decide
again.
Is
that
something
we'll
want
to
do?
Is
it
something
we
don't?
But
you
know
now
we
can
see
exactly
you
know,
and
this
goes
back
to
my
thing
about
why
kubernetes
is
so
powerful
and
how
I
sell
it
to
all
the
security
folks
that
are
out
there,
like
you
know,
saying
no
to
everything.
Right.
B
Kubernetes
gives
us
the
opportunity
to
run
standardized
tools
like
this
to
understand
exactly
how
an
application
is
going
to
be
deployed
into
our
environment
and
the
way
that
we've
never
really
had
that
sort
of
visibility
for
like
VMS
and
things
like
that
coming
into
our
environment,
all
right,
so
we've
decided
that
we,
like
external
secret
operator,
based
on
the
helm,
charts
on
the
containers
development
practices.
All
that
we
felt
really
good
about
it.
So
kind
of
the
next
step
is
forking.
B
A
A
You
need
to
have
every
one
of
those
security
constraints
enabled
from
the
beginning,
so
that
a
Helm
install
gives
you
the
most
secure
option
and
you
actually
have
to
take
action
in
order
to
let
in
order
to
make
it
less
secure
in
a
development
environment,
for
example.
So
what
I
find
is
you
can
use
their
Helm
chart
as
a
starting
point,
but
you
tend
to
end
up
having
to
run
it
yourself
or
write
the
helm.
Chart
yourself.
A
All
right
so,
once
you've
got
the
a
hardened
image
you
have
a
hardened
or
Rewritten
Helm
chart.
What
do
you
do?
Well,
that's
the
easy
part.
All
you
have
to
do
is
run
a
Helm
install
or
go
through
your
git
Ops
process.
Whatever,
whatever
is
your
organization's
methodology
here,
we
again,
we
definitely
want
to
make
sure
that
our
clusters
themselves
are
hardened
and
that
our
practices
meet
those
or
work
alongside
those
hardening
requirements.
A
So,
for
example,
we
don't
run
our
kubernetes
clusters
with
the
API
exposed
to
the
public
internet,
which
means
that
if
a
developer
is
working
remotely,
unless
they're
coming
in
through
a
VPN
they're
not
going
to
be
able
to
just
run,
you
know
K8,
Supply
or
sorry,
Cube
cuddle
apply
or
Helm
install,
and
nor
should
they
in
most
environments.
So
that
means
that
you
need
to
have
a
get
Ops
workflow,
and
then
you
want
to
also
continue
to
monitor
those
applications.
A
You
should
know
what
the
meaningful
metrics
are
and
you
should
have
alerting
thresholds
that,
let
you
know
if
an
application
has
gone
down
because
remember
availability
is
a
security
control
right.
A
denial
of
service
is
a
threat,
so
you
want
to
make
sure
that
you're
aware
of
when
that
application
goes
down.
A
And
then
you
want
to
constantly
monitor
as
it
sits
there
deployed,
and
you
want
to
reassess
that
application
on
a
regular
basis.
You
should
be
running
vulnerability
scans
on
a
daily
basis
daily
basis.
Ideally,
you
should
have
some
sort
of
a
security
operator
running
in
your
cluster.
That
is
alerting
you
when
it
finds
a
vulnerability
running
in
your
on.
You
know,
in
an
application
that
you're
running
you
need
to
be
able
to
patch
critical
vulnerabilities.
Typically
less
than
seven
days
is.
What
is
what
the
compliance
Frameworks
will
tell
you,
depending
on
what
it
is.
A
Sometimes
you
get
that
phone
call
of
like
you
got
to
go
now.
We
all
remember
you
know
log
for
shell
and
Heartbleed,
where
everything
just
hit
the
fan
and
everybody
had
a
terrible
Christmas
and
then
also
you
really
need
good,
intra
organization
communication.
If,
if
you're,
using
something
like
a
base
container
where
you
your
team,
is
like,
oh
hey,
we'll
go
ahead
and
we'll
Harden
this
base
container
for
Java,
and
then
everybody
else
can
use
it,
and
you
know
you
can
really
scale
out
our
efforts
there.
A
A
All
right,
so
here's
the
questions
that
we
want
you
to
answer
at
the
end
of
the
day
is
you
know,
on
a
regular
basis,
is
the
project
still
actively
maintained?
We
were
working
with
Kim
to
get
AWS
credentials
into
applications
and
the
maintainers
had
pivoted
away
from
it
because
they
were
like.
Oh,
the
oidc
provider
for
eks
works
wonders
we
weren't
running
a
cast,
so
we
couldn't
use
it.
So
it
was
a
deprecated
project,
but
there
was
nothing
we
can
do
about
it.
So
is
that
project
still
being
actively
maintained?
A
If
there's
any
security
incidents
have
happened,
how
were
they
addressed?
Was
there
good
communication
from
the
maintainers?
Was
there
a
patch
made
readily
available?
Was
there
security
advisories?
You
know
all
of
those
things
internally.
Do
the
current
owners
still
want
to
own
it?
If
they've
pivoted
on
to
golang
from
java,
then
who's
going
to
take
over
that
that
JRE
base
image
that
everybody
else
is
relying
on
overall?
Is
it
meeting
your
organization's
needs
and
you
know,
if
not
what
else
is
in
the
ecosystem
today?
B
Yeah
and
I
think
that's
really
important
to
make
sure
that
we're
also
reviewing
that
Upstream.
As
he
said,
we've
run
into
multiple
examples
where
the
Upstream
started
diverging
away
from
what
we
were
intending
to
use
the
project
for
at
least
from
a
security
Baseline
perspective.
So
you
know
understanding
that
Upstream
process
and
being
very
active
there
working
in
our
spaces
that
we've
had
trouble
getting
things
upstreamed
patch
wise.
But
that's
always
the
dream.
Is
you
know
if
you're
making
something
more
secure?
If
you
can
get
that
upstream?
A
That
is
our
time
we'll
be
hanging
out.
If
anybody
has
any
questions
feel
free
to
come
up,
say
hi,
we'll
do
our
best
to
answer
them.
We
do
love
feedback
as
presenters.
It's
it's
invaluable
to
us.
So
there's
a
barcode
in
the
bottom
right
corner
there.
If
you
can
scan
that
and
offer
any
feedback,
positive
negative,
it
all
helps.
Thank
you
both.
Thank
you
all.
So
much
for
your
time.
I
hope
you
enjoy
the
rest
of
kubecon.