►
From YouTube: Deploying Kubernetes Without Scaring Off Your Security Team - Paul CzarkowskI & Major Hayden
Description
Deploying Kubernetes Without Scaring Off Your Security Team [I] - Paul CzarkowskI, Pivotal & Major Hayden, Rackspace
subtitle: "The Major Hayden Center For Kubernauts Who Can't Security Good And Wanna Learn To Do Other Stuff Good Too"
One of the larger roadblocks we face in the enterprise when trying to adopt new technologies is getting the security and compliance teams onboard.
Tools like kubicorn and kubeadm are likely the foundation on which Kubernetes deployments will be performed in the future as they help simplify the deployment and operations of Kubernetes a very complex distributed system.
However concerns about security and compliance, which are not as yet addressed by those tools, may act as inhibitors and road blocks to using these them and thus Kubernetes in the enterprise.
Thankfully the techniques and tools for deploying Enterprise Linux distributions, securing them, and ensuring compliance already exist and can be very easily combined with kubernetes.
In this talk we’ll expand upon these enterprise requirements and use cases and show how we can use existing Ansible tooling to deploy kubernetes on bare metal or the cloud, monitor it with common enterprise monitoring tools, secure it with a 2fa SSH bastion, and ensure [DISA STIG] compliance.
About Paul Czarkowski
Paul Czarkowski is a recovering Systems Administrator who has run infrastructure for longer than he cares to admit. After cutting his teeth in the ISP and Gaming industries Paul changed his focus to using (and contributing to) Open Source Software to improve the Operability of complex distributed systems such as Kubernetes and OpenStack. Oh and he makes the best Queso in Texas.
About Major Hayden
Major Hayden is a principal architect at Rackspace and lives just outside San Antonio, Texas. He automates infrastructure deployments while improving security along the way. He runs icanhazip.com and blogs on major.io. Don't give him new ideas for domain names -- he owns far too many.
When he's not at work, he tinkers with amateur radio (W5WUT), reads lots of books, and runs 5Ks from time to time.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Alright,
let's
get
started
hi
everyone,
so
we're
gonna
be
talking
about
deploying
kubernetes
in
a
secure
ish
way
and
we're
gonna
be
talking
about
like
the
under
the
kubernetes
stuff
and
the
kubernetes
install
itself.
So
if
you
hear
to
learn
about
how
to
like
run
an
application,
secure
link
you
securely
on
kubernetes,
this
isn't
that
talk,
and
so,
if
it's
not
what
you're
here
for
feel
free
to
leave,
we
won't
be
offended.
B
So
my
name
is
major
Hayden
I
work
at
Rackspace,
and
a
lot
of
my
focus
is
around
deploying
OpenStack
clouds
and
deploying
kubernetes
on
top
of
it
and
trying
to
do
all
that
in
a
secure,
documented
way,
and
so
we'll
kick
this
off.
This
is
what
it
feels
like
when
you
go
to
cube
con.
You
kind
of
feel
like
and
the
Sun
is
rising.
The
fog
is
starting
to
clear.
B
Maybe
the
fog
is
like
a
bad
outage
or
operational
problems,
or
things
like
that,
but
you
just
have
that
feeling
of
euphoria,
like
things
are,
maybe
gonna
get
better
like
how
many
went
and
saw
Kelsey's
keynote
this
morning,
I
felt,
like
things
were
awesome
after
that
I
wasn't
entirely
sure
what
he
did,
but
it
was
really
exciting,
and
so
this
is
what
it's
like
when
you're
back
at
the
office,
and
you
talk
to
your
friends
and
co-workers
about
what
you
saw
a
coupe
man.
You
be
amazed
by
this.
Look
at
all
this
cool
technology.
B
B
So
really
what
it
comes
down
to
is
enterprise
security
teams,
care
about
a
small
subset
of
things,
and
if
you
boil
it
down,
want
changes
that
are
not
gonna
get
in
the
way
have
some
kind
of
value
that
are
well
documented,
that
they
can
actually
go
back
and
audit
or
you
can
audit
and
provide
proof
of
compliance
and
that
are
easily
understood
when
you
go
into
a
corporate
security
team
and
say:
oh
we've
got
this
replication
controller.
That
you
know
takes
care
of
this
this
and
that
they
don't
hear
any
of
that.
B
But
if
you
say,
hey
you're
concerned
about
availability
right,
the
security
team
says
yes,
you
know
it
must
be
up
all
the
time.
Okay.
Well,
we
have
a
system
that
makes
sure
we
have
X
amount
of
copies
available
at
all
times
and
here's
how
our
networking
is
set
up,
you
kind
of
have
to
learn
to
speak
their
language,
and
so
really
what
you
want
to
do
is
find
a
way
to
get
here.
B
So,
if
you're
already
doing
DevOps
and
you're
looking
at
automating
your
infrastructure,
whether
that's
kubernetes
or
OpenStack,
or
Cloud,
Foundry
or
whatever,
have
you
you
want
to
balance
that
with
security,
you
want
to
find
some
way
to
make
that
relatable
for
your
security
team
and
so
to
do
this.
You
have
to
push
back
on
your
security
team
a
little
bit
and
ask
for
what
the
guardrails
are
so
a
lot
of
times.
B
When
you
have
those
interactions
with
your
security
team,
it
feels
like
you,
hit
a
roadblock
like
they're,
just
constantly
throwing
things
up,
that
you
have
to
jump
over
or
break
through,
but
if
you've
ever
heard
of
that
concept
of
managing
your
manager,
you
know
help
your
manager
understand
what
you
need.
It's
the
same
way
with
a
corporate
security
team
and
saying
hey,
look
what
you're
giving
me
a
roadblocks
give
me
guardrails,
show
me
where
I
cannot
go.
Tell
me
where
the
edges
are
and
then
I'll
make
sure
I
stay
in
the
middle
of
that
path.
B
You'll
find
over
time
that
most
corporate
security
teams
will
start
to
expand
those
guardrails
out
a
little
bit,
and
so
a
quick
public
service
announcement
always
enable
Linux
security
modules.
In
your
container
deployments,
I
went
to
a
talk
yesterday
and
someone
said
a
farmer
was
giving
us
trouble,
so
we
turned
it
off,
don't
so
seriously
stop
to
save
Niguel
ceilings.
If
you
go
to
stop
disabling
selinux
comm,
there's
some
more
information
you
can
review
there,
I'm.
B
B
A
review
for
some
of
us,
oh
we'll,
go
pretty
quick,
so
ansible
does
everything
from
deploying
your
software
restarting
services.
Installing
packages
running
commands
changing
config
files,
you
name
it'll,
take
care
of
it
if
you've
never
seen
ansible
before
I'll
explain
it
in
three
bullets.
You
have
tasks,
they
each
do
one
thing:
maybe
change
a
config
file,
restart
a
service
clone
something
from
get.
Then
you
take
all
those
tasks
you
group
them
together
in
a
roll
and
you
say:
hey
I
have
a
web
server
and
a
web
server.
Has
these
set
of
tasks?
B
And
then,
finally,
you
say:
okay,
well,
I'm
gonna
write
a
playbook
that
deploys
my
web
servers
and
then
my
database
servers
and
then
maybe
something
else,
and
so
you
group
those
roles
together
in
a
playbook,
and
so
that's
essentially
ansible
explained
in
a
quick
minute
and
why
we
love
ansible
is
that
it's
simple
everything's
written
in
y
amyl,
it's
very
easy
to
read
top-down
even
for
security
teams.
They
can
take
that
apart
and
understand
it.
There's
not
a
whole
lot
of
code
and
markup
and
craziness
to
understand.
The
inventory
system
is
very
easy.
B
If
you
can
understand
flat
files
or
JSON,
you
should
be
fine,
it's
also
very
versatile.
You
can
use
it
on
containers,
you
can
use
it
in
VMs.
You
can
use
it
to
make
your
containers,
you
don't
need
any
Damons
installed
in
your
nodes.
There's
not
a
security
concern
around
as
soon
as
you
go
to
your
corporate
security
team
and
say:
I'm
gonna
run
a
daemon
on
every
node.
You'll
watch
them
start
to
get
nervous.
B
They
want
to
know
how
its
configured,
who
has
access
to
it
and
then
finally,
it's
repeatable,
so
you
can
run
a
playbook
over
and
over
and
over
and
over
again
and
you'll
get
the
same
results.
So
this
is
also
handy
when
your
corporate
security
team
comes
knockin
and
they
say
hey.
Are
you
doing
all
the
things
you
said
you
were
doing?
You
could
just
run
the
PlayBook
right
in
front
of
them
and
say
see
nothing
changed.
Everything
is
applied
and
so
auditing
becomes
very
easy.
B
It
also
has
an
auditing
system
and,
like
a
dry,
run
capability,
so
that
maybe
you
come
to
a
new
system
and
you
want
to
know
what
ansible
is
gonna
do
on
it.
You
can
just
run
it
in
check
mode
and
ansible
will
say:
hey
look.
You
asked
me
to
do
50
things.
40
of
these
are
already
complete
or
I.
Don't
need
to
take
action,
but
these
10
I
need
to
take
action
on
yeah.
A
It's
basically
install
a
package,
run
a
template
to
configure
it
and
then
make
sure
it
started
or
restarted
if
that
template
changes
and
we've
done
a
ton
of
ansible
and
we've
done
a
lot
of
complicated,
stupid
things
with
ansible
and
we
keep
finding
ourselves
trying
to
simplify
it
back
to
this
and
making
it
super
readable
super
easy
for
anyone
to
pick
up
and
understand.
You
know
even
like
the
the
NOC
team
at
like
3
a.m.
if
they
can
read,
understand
and
run
a
playbook.
A
They
don't
have
to
wake
me
up,
because
I
thought
I
was
being
smart
and
added
in
a
bunch
of
logic
that
wasn't
really
necessary.
So
this
kind
of
pattern
of
these
three
tasks
like
per
thing,
you're
deploying
it's
super
powerful
and
you
don't
normally
need
to
do
a
ton
more
and
then
you
can
do
other
stuff.
You
you
can
you
can
what
am
I
doing
here?
Oh
yeah,
you
can
configure
Cisco
switches
and
other
other
switches
which
is
really
cool
right.
A
So
you
can
work
with
your
security
team
and
actually
like
I
need
to
change
into
this
via
VLAN
and
set
my
MTU
down.
So
I
can
pick
cebu
or
we
can
actually
do
that
straight
from
ansible
and
then
it
just
works
and
we're
all
happy.
You
can
deploy
cloud
instances.
So
this
is
deploying
some
Google
compute
instances
and
it's
using
variables
that
are
pulled
from
an
inventory
which
is
like
a
big
yeah,
Mille
dictionary
of
values
and
and
settings.
A
So
I
can
like
this
will
actually
loop
over,
like
an
array
and
like
I
can
say:
I
want
15
VMs
with
these
settings
and
we'll
go
and
do
that.
This
is
an
instance
where
you
start
to
add
a
little
bit
of
complexity
there
to
ansible
and
you
maybe
lose
a
little
bit
of
readability.
But
you
get
a
little
it's
that
extra
functionality.
A
That
makes
it
worth
it
and
then
you
can
do
really
complex
stuff,
like
pixie
booting
servers
and
it'll
go
through
and
do
like
your
ipmi
commands
to
reboot
drop
files
in
the
right
place
to
leave
them
once
it's
rebooted
and
all
that
sort
of
stuff.
So
you
can
get
very,
very
hands-off
for
like
doing
pretty
much
anything.
You
need
to
deploy
bare
metal
cloud
from
the
infrastructure
to
the
network
to
the
software.
B
And
so
we
talked
a
little
bit
about
ansible
and
kind
of
give
an
overview
and
some
of
the
things
that
you
could
do
or
even
give
your
security
team
access
to
run
for
you.
So
one
of
these
things
that
that
we've
worked
hard
on
is
ansible
hardening,
and
so
it's
just
it's
a
ansible
role
that
you
can
go
and
apply
to
any
server.
It
got
its
start
within
the
realm
of
OpenStack,
and
then
we
asked
the
question
of.
Why
couldn't
we
just
do
this
on
any
host?
B
Then
we
expand
it
into
that
and
then
finally,
someone
actually
came
to
the
IRC
Channel
and
says:
well,
have
you
done
this
on
the
kubernetes
host
and
we
said
well
shoot
we
haven't
done
that,
and
so
we
went
and
tried
it
and
now
for
a
couple
of
tweaks
that
actually
worked
fine
there
as
well.
So
we've
had
a
couple
of
folks
who
have
come
by
and
used
projects
like
cube
spray
which
we'll
get
to
in
a
little
bit
and
then
they've
said:
hey,
like
I've
just
run
ansible
hardening
with
that,
and
it
just
worked.
B
B
B
A
Then
I
realized
that
actually
my
job
there
was
to
help
them
so
I
had
to
stop
making
fun
of
them
and
actually
help
them,
and
that's
where
I
got
into
in
spec
I'm
like
hey
guys.
This
is
pretty
cool
we
can
go
from.
We
can
take
all
of
your
rules
in
these
excel
spreadsheets
and
we
can
write
them
in
a
fairly
simple
DSL
which
looks
like
this
and
then
our
monitoring
software
sense.
A
So
inspect
rules
with
ansible
is
super
easy.
You
can
just
kind
of
get
clone
a
repo
of
the
of
the
stig
rules
and
we
actually
have
up
on
a
git
repo,
all
of
the
rules
to
in
for
to
order
basically
use,
dig
settings
and
then
so,
when
we've
gotten
to
this
point,
I
sort
of
saw
the
security
hardening
bit
and
we're
like.
Oh,
we
can
use
security
hardening
combined
with
this,
combined
with
our
and
playbooks,
and
suddenly
we
have
a
method
to
secure
our
servers
and
also
a
method
to
audit
our
servers
and
yeah.
A
So
we
had
sensu
was
monitoring
it,
so
we'd
get
alerts
and
also
because
we've
had
a
centralized
logging
service.
Like
once
a
day,
we
had
a
report
in
our
logging
service
of
what
all
servers
were
compliant
and
what
weren't,
and
so
like
all
of
that
auditing
and
compliance
and
auditing
and
checking
just
magically,
became
automated
and
saved
a
ton
of
time
and
allowed
those
folks
that
were
keyboard
warriors
before
to
actually
work
on
things.
We're
adding
a
lot
more
value
than
just
manually
checking
files
on
servers
and
then
another
thing
we
worked
on.
A
I
was
called
cuddle,
which
was
originally
called
site
controller,
and
that
was
basically
all
the
other
bits
to
give
us
a
fairly
secure
environment,
fairly,
secure
infrastructure.
So
it
was
a
stage
Bastion
that
had
two-factor
authentication
and
some
robust
role
based
access.
It
gave
us
an
OAuth
web
portal
and
it
gave
us
our
centralized
logging
monitoring
and
about
a
bunch
of
other
stuff.
A
We
open
sourced
it,
and
it's
actually
just
a
big
monolithic,
ansible
repo
that
installs
sent
to
log
stash
set
up
the
bastion
and
that
kind
of
stuff-
and
it
kind
of
looks
like
this.
I,
don't
really
need
to
go
through
the
architecture,
but
it
kind
of
you
can
just
kind
of
run
it
locally
or
it
can
spread
out
and
you
can
VPN
between
and
it
forms
like
a
cut
like
a
mesh,
and
you
have
like
one
central
dashboard
to
go
across.
A
Maybe
30
data
centers,
which
is
what
we
had
I,
was
super
useful
because
it
meant
the
operations
team.
The
security
team
during
the
auditing
just
went
to
one
place
and
they
could
see
all
of
our
infrastructure
spread
across.
You
know
30
data
centers
and
like
they
had
one
Bastion
host,
they
needed
to
remember
that
could
get
them
to
any
nation
to
any
server
they
had.
One
web
host
had
remembered
and
they
could
get
to
like
the
logs
or
monitoring
or
whatever
for
any
of
the
servers.
A
A
Anyone
logged
into
this
machine,
it
just
logged
their
entire
console,
Internet
I
can
put
and
output
and
shipped
it
off,
and
because
that
was
their
entry
point
even
when
they
SSH
it
into
an
different
server
that
recording
still
still
still
happen
and
went
through
and
then
SSH
agent
would
basically
give
them
acts.
It
was.
It
would
fake
an
SSH
agent
and
give
them
access
to
a
key
to
SSH
to
another
machine
without
actually
letting
them
see
or
edit
or
change
that
key.
A
So
it
meant
that
once
they
one
if
the
SSA,
if
their
Bastion
knew
about
them
and
it
knew
what
servers
they're
allowed
to
go
to,
it-
gave
them
the
keys
in
their
agent
to
get
to
those
servers
and
then
from
then
on.
You
could
have
a
shared,
a
shared
username
amongst
them,
and
all
the
auditing
happened
right
there.
We
could
see
what
individual
user
was
doing.
What
and
we
didn't
have
to
do
a
ton
of
crazy
user
management
on
every
single
server
and.
B
B
Various
amounts
of
hosts,
it's
very
easy
to
set
up
your
inventory
and
get
going,
and
then
you
can
actually
apply
security
hardening
before
and
after
to
be
able
to
prove
to
your
security
team
that
you
have
a
deployment,
that's
meeting
certain
standards,
and
so
it's
I
think
it's
still
an
incubation
status
right
now,
but
you
can
use
this
in
production.
It
has
upgrades
built
in
h.a
all
that
kind
of
thing.
Yeah.
A
Then
you,
then
you
run
cuddle
again
on
your
actual
host
that
will
be
kubernetes
and
that
will
set
up
like
the
the
the
users
and
stuff
that
you
want.
You
need
it'll
set
up
the
sensitive
client
log,
stash,
client
bits
and
pieces
in
truck,
interact
with
cuddle,
and
then
you
run
coupe
spray
to
get
kubernetes
running
and
any
run
security,
hardening
Anniston
hard
hardens.
That
and
I
actually
did
a
bunch
of
experimentation
with
hardening
and
coupe
spray
and
found
if
I
ran
it
before.
A
If
I
ran
hardening
before
then
ran
coupe
spray
and
then
ran
in
hardening
again,
it
didn't
change
anything
that
second
time
so
I
was
I,
had
a
very
high
amount
of
confidence
that
the
two
playbooks
weren't
gonna
fight
with
each
other
and
like
one
set
one
setting
one
change
it
and
kind
of
go
like
that.
I
said
that
was
really
cool,
so
I
think
you
must
have
already
worked
on
that
to
make
that
work
right.
Yeah.
A
So
basically,
that
gave
us
a
pretty
secure
infrastructure.
The
the
security
team
was
happy
with
it,
because
it
was
the
same
set
of
tools
that
we've
both
used
to
do
open
staff
that
our
security
teams
are
happy
with,
but
we
didn't,
we
haven't,
really
touched
on
how
we
secure
stuff
on
top
of
kubernetes
and
that
was
kind
of
on
purpose,
because
when
they
have
25
minutes
or
whatever
to
talk,
but
you
know
there's
a
lot
of
stuff
still
to
consider.
You
know
what
did
you
build
pipeline?
A
Look
like
where
are
you
gonna
be
hosting
your
containers?
How
do
you
make
sure
that
the
image
you
want
is
actually
the
image
you
built,
and
you
know
image
scanning
all
that
sort
of
stuff
secret
management
authentication
to
kubernetes
are
you're
gonna
do
like
you
know,
Kelsey
was
talking
earlier.
Do
you
give
each
person
or
each
group
their
own
kubernetes,
or
do
you
want
to
go
down
there?
The
are
back
route,
the
tons
of
tons
of
other
considerations
to
still
think
about
and
to
work
with
the
security
team
about.
A
But
what
we
found
is
that,
by
going
to
the
security
team
and
bringing
these
tools
to
them
a
they
were
they're,
not
necessarily
familiar
with
the
sort
of
tools
we
use
in
the
I.
Guess
the
DevOps
space
right,
because
they
have
different
content,
cares
and
concerns,
and
as
we
bring
these
tools
to
them
and
we
say
hey,
we
can
help
solve
the
problems
that
you're
trying
to
solve,
using
the
tools
that
we
we
know
and
then,
as
major
said
earlier,
it
becomes
a
discussion
about
the
guardrails
versus
a
can.
We
know.
A
Can
we
no
kind
of
conversation
which
you
know
nobody
really
enjoys
like
we
don't
enjoy
being
told.
No
security
team
usually
doesn't
enjoy
saying
no,
and
so
you
know
you
join
their
conversation
together
and
you
get.
You
know
everyone
involved
in
that
kind
of
DevOps
mentality
of
like
let's
just
improve
everything
together.
Some
people
would
like
to
call
that
like
have
sex
and
sector
votes
and
stuff,
but
really
just
DevOps
itself
is
kind
of
describing
that
everyone
working
together
to
improve
like
what
we're
doing
with
a
business,
improve
the
culture
and
stuff
like
that.
B
A
So
you
can
do
you
be
keys
and
we
have
a
little
service
that
runs,
so
it
makes
it
syncs
between
the.
If
you
have
more
than
one
Bastion
and
you
you
in
your
animal
inventory,
you
have
like
the
Yubikey
ID
and
a
couple
other
pieces
of
information
per
user
and
every
time
you
run
ansible,
it
just
makes
sure
those
are
set,
and
then
we
also
have
if
using
the
Google
Authenticator
they
had.
A
They
have
a
put
that
Pam
module
so
installs
that
Pam
module
and
again
you
have
like
a
Google,
Authenticator,
ID
and
a
couple
of
other
pieces
of
information
and
that
it
needs-
and
it
just
runs
that
on
every
bet
drops
out
on
every
Bastion
server.
And
then
the
SSH
agent
marks
tool
has
a
small
little
sequel,
light
database
that
tracks
the
users
and
what
groups
they
have
and
gives
them
a
key
based
on
what
group
they're
in
so
they
can
then
get
on
to
the
other
servers.
B
Pretty
good
questions,
okay,
so
the
first
comment
was
around:
you
can
get
ansible
tower
for
free
as
awx,
so
you
can
go
download
that
today
and
the
second
part
was
ansible
hardening.
Why
it
wasn't
part
of
the
mine
point.
So
we
originally
had
a
discussion
in
the
beginning,
so
our
our
first
fry
was
into
translating
the
steak
for
a
bun
too.
So
that
was
the
first
thing
we
went
to
because
the
finding
well
there's
now
a
Stig
for
a
bun
too.
B
It
still
needs
a
little
bit
more
work,
it's
getting
there,
but
at
the
time
there
was
not
one,
and
so
all
of
our
deployments
internally
at
Rackspace
were
on
a
bun
too.
So
we
had
to
kind
of
translate
that
over
to
a
bun
too.
So
that's
where
we
went
first
and
then
we
kind
of
came
back
to
sent
to
us
and
Red
Hat
after
that,
and
then
it
expanded
to
Susa
and
then
debian
and
then
fedora
and
we
expand
further.
B
A
Yeah,
so
we
were
using
service,
BEC,
initially
and
then
inspect
inspect
came
along
and
it
was
much
stronger.
Dsl
and
the
DSL
is
Ruby
ish,
but
you
don't
really
need
to
know
Ruby
to
do
it
unless
you're
gonna
start
doing
loops
and
stuff
like
that,
and
we
didn't
have
any
issues
with
anyone
from
a
security
team
learning
enough
to
to
write,
write
all
of
the
rules
for
Stieg
and
then
all
of
the
other
IBM
related
compliance
that
we
had
to
make
sure
we
were
having.
So
it
was.
A
B
About
about
the
oh,
the
audit
tools,
okay,
so
yeah!
The
question
was
around:
what
are
the
audit
tools
that
we're
using?
So
we
talked
a
little
bit
about
using
ansible
kind
of
as
an
auditing
tool.
It
works
okay
for
that
and
then
then
we
talked
about
inspect
yeah
as
another
auditing
tool,
of
course,
on
the
system
itself
that
you
have
audit
d
as
well,
so
you
can
audit
syscalls
and
things
like
that
and.
A
That's
security,
hardening
turns
audit
D
and
some
of
those
Linux
auditing
tools
on
by
default
and
if
you've
got
a
central
log
service
and
if
you're
on
cuttle,
it
will
set
up
a
log
stash
forwarder
to
send
all
of
those
logs
to
your
elke
service
or
whatever
you
centralize
all
services,
and
so
that
way
from
verifying
the
side
of
things
you've
got.
Your
monitoring
is
gonna
alert
you
as
soon
as
something
goes
out
of
spec,
and
then
you've
got
your
your
elke
logs.
A
That
actually
say
everything
like
everything
that
inspect
round
came
up
good
or
didn't
at
a
particular
time
or
date,
and
so
we
would
just
basically
at
first
we
were
just
taking
a
screenshot
of
like
logged
logs
and
then
eventually
wrote
some
elastic
search,
queries
to
be
a
little
bit
smarter
about
that
and
we're
dumping
them
to
a
file
system
so
that
they
could
give
them
to
like.
If
someone
said,
oh,
we
need
to
check
a
server
from
this
date
burn.
You
know
like
a
random
spot
check
which
they
would
do.
B
Auditing
can
also
be
helpful
how
many
folks
in
here
have
used
or
tried
set
comp
at
some
point,
all
right
yeah.
So
all
of
us
have
great
here
from
that
experience.
So
the
scary
part
about
set
comp
is
that
you
might
actually
block
something
that
you
need.
Then
you
have
a
container,
that's
not
working
or
an
application.
That's
not
working!
So
one
of
the
things
I
generally
prefer
to
do
is
have
audit
D
audit.
B
The
SIS
calls
that
I
know
we're
gonna
be
problematic
for
me
that
I
might
go
and
block
was
set
comp
and
have
audit
d
audit
them
for
awhile
and
trigger
alerts
on
that.
And
then,
if
I,
don't
you
know
if
we
deploy
an
application
and
I,
don't
see
an
alert
on
that
says,
call
for
a
week,
I'm
like
okay.
Well,
then,
maybe
we
could
add
that
and
just
remove
that
capability
from
that
not
capability.
Sorry
remove
that
says,
call
from
the
container.
B
A
So
we
had,
we
did
exactly
that
with
cuddle
right,
so
you
had
a
like
a
list
of
all
your
users
and
their
public
keys
and
it
would
drop
them
in
so
when
they're
logging
in
they
would
have
their
public
key
and
the
second
factor.
And
then
you
can
simply
look
at
like
the
the
give
change
log
and
some
other
stuff
to
figure
out
if
it's
too
old
or
not
and
say,
okay,
everyone
time
to
have
time
to
roll
us
up
some
new
keys
and
they
would
just
make
pull
requests
to
her.
A
To
the
answer
ball
to
say:
here's
my
new
key,
and
so
we
didn't
have
to
it-
was
fairly
self-service.
I
mean
they.
They
couldn't
run
ansible
because
only
a
few
people
could
run
ants
all
across
the
bastions.
But
anyone
could
at
any
point
update
their
key
and
say:
hey.
Can
you
just
roll
the
playbook
for
me,
so
my
new
key
shows
up
and
then
the
same
across
all
of
the
servers.
A
So
we
had
basically
one
or
two
access
keys
for
all
of
our
servers
being
managed
by
cuddled,
because
we
had
the
are
back
at
the
bastion,
and
so
it
was
a
very
simple
drop-down.
Some
new
new
keys
in
the
Bastion
put
the
public
keys
and
authorized
files
on
the
on
every
other
server
and
just
run
ansible,
and
we
just
like
run
it
all
out
everywhere,
and
you
know
anyone
how
many
servers
we
were
doing
it
on.
That
would
take
like
five
minutes
to
20-30
minutes.
Yeah.
A
B
B
Okay,
so
the
comment
was
using:
SSH
keys
is
kind
of
the
old
way
and
then
we've
got
509
shook
it
support
and
why
don't
we
use
that
or
509
eh
sorry,
yeah
I
had
to
be
specific.
I
would
definitely
be
up
for
I.
Just
think
some
of
the
challenges
I've
seen
in
the
past
were
like
you.
You
got
to
make
sure
your
revocations
on
point
and
all
that
kind
of
stuff
I
know
there's
some
technology
to
help
with
that.
Yep.