►
Description
Securing Shopify's PaaS on GKE [I] - Jonathan Pulsifer, Shopify
Shopify has leveraged Kubernetes through Google Container Engine (GKE) to build its new cloud platform. This PaaS is currently serving the majority of the company's internal tools as well as business-critical production workloads. Moving to Kubernetes and a public cloud is no easy task, especially for a security team.
Unfortunately for us, a hosted solution does not offer all the features we've come to love in Kubernetes including NetworkPolicies, PodSecurityPolicies, and admission controllers among others. Given this, the security team has created a number of Kubernetes controllers and other cloud platform solutions to maintain an effective security posture on our new platform.
In this talk we'll introduce our cloud platform, explore the tools we've created to bridge the security gaps, detail the struggles we've encountered using Google Cloud Platform and GKE, and discuss our growing pains with Kubernetes multi-tenancy. Attendees will gain an understanding of the current state of Kubernetes security controls on GKE, a familiarity with some of the products available on Google Cloud Platform, and insight on how to integrate security controls into their development pipelines.
About Jonathan Pulsifer
Jonathan is a Senior Security Engineer at Shopify working on securing their new platform using Kubernetes on GKE. Previously, he was a SANS mentor, network defense instructor, and a team lead at the Canadian Forces Network Operations Centre in Ottawa. Find Jonathan on Twitter @JonPulsifer
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Everyone
welcome
to
my
talk.
Welcome
to
the
last
day
of
coop.
Con
I'm
super
excited
to
be
here
and
sitting
on
the
other
side
of
the
podium
is
really
curious
for
me.
So
thank
you
all
for
having
me
here
today.
So
Who
am
I.
My
name
is
Jonathan
Pulsifer
I'm,
an
infrastructure
security
engineer
at
Shopify
certified
kubernetes
administrator
number
89.
If
that
means
anything,
you
can
find
me
on
github
and
and
twitter
at
john
Pulsifer
previously
was
a
team
leader
at
the
Canadian
Forces
Network
Operations
Center.
A
B
A
Am
a
certified
packet
ninja.
So
thank
you
for
getting
for
having
me
on
stage
today.
My
first
coop
con
was
back
earlier
this
year
in
Berlin,
where
I
was
sitting
on
the
floor,
watching
Jessie,
Frizzell
and
Alex
more
talk
about
securing
Jiki
clusters
or
securing
the
kubernetes
clusters,
so
I'm
really
glad
that
the
venue
has
a
little
bit
larger
room
than
than
they
did
in
Berlin.
So
that's
fantastic
fun
fact!
For
those
who
didn't
know
the
kubernetes
store,
the
CNC
F
store
actually
runs
on
kubernetes
at
Shopify.
So
that's
really
interesting.
A
I
went
and
talked
to
the
folks
who
are
running
the
booth
down
there
and
I.
Let
them
know
that
they
should
be
letting
you
know
that
if
you
purchase
anything
you're
actually
purchasing
something.
That's
your
kubernetes!
It
was
one
of
the
actual
first
doors
that
we
moved
to
our
cloud
platform
at
Shopify.
A
So
before
start
talking
about
security
controls
and
how
we
do
this
first,
we
need
to
gather
some
more
context
about
how
we
run
services
at
Shopify.
So
we
have
what's
called
a
service
tier
model
where,
as
an
application,
sort
of
moves
through
the
software
development
lifecycle,
we
start
off
at
tier
4,
where
we
have
a
lot
few
requirements.
A
You
know
it
encourages
that
rapid
prototyping
really
gets
agility
in
there
and
as
we
move
through
through
business
importance,
we
we
impose
things
like
service
level
objectives
and
the
traditional
sort
of
software
development,
lifecycle
and
controls
and
impose
things
like
page
rotations.
The
CI
alerting
redundancy
Incident
Response,
these
sorts
of
things,
and
so
when
we
were
moving
to
kubernetes
setting
up
all
of
our
clusters
and
I
was
thinking.
Why
don't?
We
do
the
same
thing
with
security,
so
I've
coined
what
I've
called
the
security
tiers.
A
So,
first
and
foremost
when,
when
you
create
an
application
on
our
cloud
platform
on
tier
4,
we
impose
things
like
strict
auerbach
creedy's
at
two
stations.
We'll
talk
a
little
bit
more
about
what
those
are
later,
but
as
we
move
through
the
tiers
I
have
this
vision
if
I
were
to
have
one
of
restricting
the
the
runtime
of
containers
with
seccomp
a
primer,
the
network
policies,
security
context,
dropping
all
the
container
capabilities
and
these
sorts
of
things.
A
So
this
is
the
vision
that
I
have
for
doing
this
on
my
team,
so
we're
gonna
explore
a
little
bit
of
how
we
do
this.
So
how
many
namespaces
do
we
have
how
many
clusters
do
we
have?
What
does
it
look
like
so
kubernetes
namespaces
by
tier,
we
have
with
60
namespaces
in
tier
one
across
a
number
of
clusters:
you're
we're
about
70
total
of
about
450
namespaces
operationally
for
services
running
at
Shopify
on
gke
right
now.
So
that's
pretty
outstanding.
A
How
does
a
team
of
seven
cloud
security,
individuals
maintain
sanity
and
sleep
at
night
with
this
many
namespaces
running
amuck,
and
it's
important
to
know
that
as
we
move
to
kubernetes
we're
not
there
yet
we're
still
in
I,
guess
an
exploratory
phase.
We've
moved
most
of
our
internal
applications
over,
but
it
just
should
be
noted
that
we
still
have
two
data
centers,
which
are
very
much
hot
and
we're
exploring
some
kubernetes
there,
but
we're
actually
going
to
be
focusing
on
GK
for
the
portion
of
this
talk
so
I'd
like
to
introduce
our
cloud
platform.
A
Our
cloud
platform
is
running
on
GK
and
it's
it
again.
We
have
about
600
projects
in
Google
cloud
platform
today
maintained
by
15,
folders,
700,
Google,
Groups
and
20
G
key
clusters.
So
this
isn't
like
the
running
one
service
per
cluster
idea.
We
have
a
way
of
a
number
of
clusters
in
a
number
of
production
projects
which
serve
a
lot
of
traffic.
So
that's
some
members
for
you
with
that
many
clusters
and
this
many
resources
that
we
have
to
control
the
security
work
when
you
have
to
rely
on
people
to
do
things
correctly.
A
Obvious
answer
here
is
a
big
No
right.
So
it's
important
that
as
we
as
we
scale
our
operations
as
we
scale
our
clusters,
the
scale
they'll
all
this.
We
need
to
introduce
these
these
these
this
automation
into
our
into
our
platform.
So
we
can
actually
maintain
agility
with
our
developers
with
a
thousand
active
developers
at
Shopify,
and
how
do
we
give
them
that
Heroku
sort
of
feel?
How
do
we
had
we,
let
them
push
code
and
how
do
we
then
secure
those
running
containers?
A
So
this
is
a
general
overview
of
our
cloud
platform
architecture
and
what
we're
going
to
do
is
we're
going
to
step
through
a
number
of
components
here
and
we're
going
to
introduce
not
the
operational
patterns,
the
operational
automation,
because
it's
not
the
focus
of
this
talk.
It's
going
to
be
the
security
automation
here,
so
at
each
sort
of
stage.
In
this
process
we
have
our
traditional
I.
Guess
everybody
has
a
builder
right.
Cin
CD
is
not
new.
A
There's
been
a
lot
of
talks
on
this,
but
there
has
been
a
lot
of
talks
on
introducing
security
controls
into
those
pipelines,
so
we're
gonna
step
through
those
and
see
what
they
mean
so
we're
gonna
start
off
with
services
DB,
so
services
DB
is
this
sort
of
front-end
for
service
as
a
Shopify,
where
we
have
about
300
400
services
that
run
and
how
do
we
visualize
or
how
do
we
make
sure
that
everybody
has
their
their
their
PlayBook
set
up?
How
do
we
make
sure
that
their
logs
are
there?
You
know
we're?
A
How
can
we
visualize
this,
and
this
is
the
tool
call
it
services
DB
I'm,
not
going
to
focus
much
on
the
actual
front
ending
of
operations
again,
but
with
services
DB,
because
it
has
sort
of
the
application
context.
This
is
the
first,
the
first
place
where
an
application
sort
of
begins
its
life
cycling.
You
have
a
repo
and
you
sign
up
for
services
DB,
and
then
we
can
do
some
do
some
things
like
automatic
patching.
A
So,
on
the
right
hand,
side
here,
you
see
we
have
this,
this
robot
who's,
creating
a
pull
request
on
my
repository.
That
says:
hey
John,
your
Shopify
Cloud
Jim,
is
on
a
date,
so
you
should
probably
get
that
patched.
We're
a
big
ruby
shop
Shopify.
So
most
of
the
most
of
the
automation
we
have
a
built
in
for
our
dependencies
revolve
around
Ruby.
So
this
is
an
example
of
one
of
those
where
we
actually
provide
our
developers,
who
are
building
Ruby
on
Rails
applications
with
a
gem.
A
The
Shopify
Cloud
gem,
which
bill
did
some
more
operational
automation
as
well
as
things
like
providing
good
patterns
around
say.
Storage,
for
example,
is
a
good
example
of
that.
So
with
this
we
actually
when
we
have
a
repository
and
we
clicks
our
create
service,
runtime
Creek
Runtime.
It's
actually
going
to
do
a
lot
of
magic
in
the
back
end,
and
this
is
ground.
Control
and
ground
control
is
really
curious
because
we
use
this
to
assign
sort
of
identity
to
our
namespaces.
A
While
we
wait
for
and
we've
heard
of,
the
container
identity,
working
group
and
spiffy
and
we're
getting
to
a
point
where
we
can
then
attach
identity
to
a
pod
or
our
container,
we
can't
do
that
today.
So
how
did
we
do
this?
At
Shopify
is
where
we
actually
give
identity
to
the
namespace.
So
when
you
create
your
cloud
runtime
with
services,
DB,
what's
going
to
happen,
is
we're
going
to
create
you
a
GCP
service
account.
That's
gonna
be
bound
to
your
sort
of
name,
space
environment
pair.
A
Then
we're
going
to
mount
that
secret
for
you
inside
your
kubernetes
namespace.
At
the
same
time,
we're
gonna
provision
you
an
adjacent
key
pair.
So
what
see
jason
so
go
check
it
out?
It's
actually
open-source.
It's
called
encrypted
Jason.
It
allows
us
to
use
asymmetric
encryption
techniques
to
keep
our
secrets
inside
of
git
repository
so
that
we
can
actually
have
the
secret
encrypted
in
there
and
when
we
deploy
to
our
kubernetes
namespace
it
actually
decrypt
sat
runtime.
A
So
we
don't
need
to
worry
about
these
secrets
being
baked
into
containers
right
that
that
bad
anti-pattern,
that
sort
of
sort
of
has
come
up
over
over
the
years
and
we
can
avoid
this
by
using
encrypted
Jason.
So
if
we
have
an
encrypted
file
full
of
secrets
in
there
who
cares
right,
but
with
this
we
actually
get
the
auditing
for
version
control
and
all
the
secrets.
So
if
somebody
goes
and
checks
in
a
secret,
we
can
see
who
did
that
win.
A
A
So
now
that
we've
taken
a
repository
and
we've
sort
of
built,
its
kubernetes
groundwork
with
ground
control,
how
do
we
get
your
container
built?
So
this
is
the
fun
piece
at
Shopify
we
do
about
6,000
builds
average
and
weekday.
This
is
not
like
container
builds
going
into
production.
This
is
everything
from
CI.
This
is
this.
Is
our
overall
builder
stats,
so
we
so
we
build
about
6,000
containers
for
a
weekday
life.
A
Cycling
is
really
hard,
so
we
have
about
450,000
images
and
GCR
today,
and
that
has
given
us
and
the
folks
doing
the
GC
our
vulnerability
scanning
a
little
bit
of
load
testing
help,
because
this
there
is
they're
building
these
these
products
to
help
us
out
the
amount
of
of
containers
that
are
actually
shoving
in
there
is
actually
giving
us
a
hard
time.
So
there's
a
little
bit
of
a
latency
problem.
There,
but
we're
gonna
chat
about
that
a
little
bit
later,
so
our
builder
is
called
PIPA
and
our
builders
run
on
built
kite.
A
You
saw
previously
so
PIPA
and
Portuguese
actually
means
kite.
So
that's
where
that
name
came
from
and
it's
a
docker
in
docker
golang
creature.
That
does
a
lot
of
discovery.
So
what
I
mean
by
that
is,
if
you
have
a
repository
as
a
developer,
should
you
understand
explicitly
how
to
build
a
docker
container,
I
mean
arguably
now.
So
what
can
we
do
here
to
help
get
a
secure
container
into
production
so
with
people
what
it
actually
does?
Is
it
discovers
using
the
same
Heroku
buildpack
model?
A
What
code
lives
inside
your
repository
and
will
provision
you
the
appropriate,
build
packs
needed
for
your
application
to
run
so
on
the
right
hand,
side
here
you
see
the
sort
of
build
kite
screenshot
there
and
a
couple
different
different
stages
of
our
pipeline
to
a
pipeline
set
up.
We
trigger
a
validation
build
and
we
build
your
container
at
the
very
bottom.
A
You
see
this
face
and
Creedy
stuff
we're
gonna
chat
about
that
next,
but
with
our
builder,
it's
important
to
know
that
when
it
comes
to
building
a
container
and
actually
building,
that
container
is
when
you
have
the
most
context
as
to
what's
going
to
be
running
inside
of
that
container
right,
you
see
the
packages
being
installed.
You
see
the
versions,
the
build
packs.
A
You
know
you,
you
have
the
most
context,
so
this
is
where
we
can
actually
do
a
lot
of
our
discovery,
initial
discovery
and
an
auditing
and
actually
sort
of
check
out
what's
going
on
right,
so
we
can
actually
do
some
audits
like
does
this?
Can
this
is
image
run
his
route?
This
is
image
continually
vulnerable
packages
and
container
ACTA
stations,
so
what
our
container
into
stations?
What
does
this
mean
I'm
going
to
talk
about
Griffis
and
creedy's?
A
You
can
have
a
dead
in
our
PM,
these
sorts
of
things,
but
isn't
a
container
just
an
image
that
we
could
also
for
absolutely
right,
so
using
the
the
namespace
and
image
and
the
digest
as
the
key
we
can
actually
produce.
What's
called
the
note
that
build,
so
we
can
sort
of
start
this.
This
the
this
chain
of
provenance
is
who
who
created
this
container.
When
was
it
built?
You
know
what
what
what
other
is
a
deployed
in
production
right
now
and
these
other
pieces
that
we
have
contacts
for
at
build
time.
A
So
we
produce
these
notes,
I
build
and
if
you
actually
use
Google
container
builder,
you
get
all
this
stuff
for
free
on
the
inside,
which
is
sort
of
neat.
So
as
a
as
a
personal
customer
I've.
Actually,
I
didn't
know
this,
but
you
know
building
my
own
containers
and
my
own
personal
projects.
It
said:
ok,
well,
well,
what
is
this
stuff?
A
What
does
this
look
like
on
the
right
hand,
side
here,
we
see
what's
called
an
occurrence
of
or
a
note
of
a
package
vulnerability,
so
with
Griffis,
it's
just
metadata
right,
so
we
can
actually
start
pushing
things
like
vulnerabilities
there.
So
you
see
this
a
TCP
dump
before
and
I
know
package
and
that's
vulnerable
and
it's
fixed
in
version
492.
A
So
if
we
have
this
information
about
this
TCP
dump
package
a
container
build,
we
know
that
we're
installing
a
version
of
TCP
dump
in
a
container
on
the
other
side,
then
we
can
say:
hey
look
this
container.
This
specific
container
contains
this
package
with
this
vulnerability
right.
It's
that
it's
that
container
introspection
and
that's
where
Griffis
is
really
powerful
for
keeping
that
container
metadata.
A
So
if
you
want
to
learn
more
about
that
check
out,
Google,
Cloud,
blog
or
sharp
eyes,
engineering
blog
or
a
little
post
about
how
we
use
that
metadata
is
one
thing
but
creedy's
or
as
you'll
hear.
If
you
hit
the
graphics
repository
binary,
authorization
is
really
the
cool
piece.
So
it's
using
this
metadata,
that's
stored
in
your
face
to
actually
create
and
enforce
policies
at
runtime.
So
this
binary
authorization
means
that
we
must
step
back
here
and
show
you
at
the
very
bottom.
You
see
this
graph
face
and
Creedy
stuff.
A
So
now
that
we
have
our
repository
checked
in
our
kubernetes
primitives
started,
we
have
a
container
bill.
How
do
we
actually
deploy
this
to
production?
So
kubernetes
deploy
is
really
cool.
It's
it's
a
it's
a
gem
that
provides
us
a
lot
of
automation
around
deployment
to
kubernetes.
It's
actually
open
source,
so
go
check
it
out.
It's
a
essentially
a
plugin
for
our
our
build
tool,
called
shipit,
which
is
also
open
source.
A
So
you
can
go
check
those
things
out
and
see
how
we
deploy
to
kubernetes,
but
the
the
main
features
that
are
really
exciting
for
me
and
for
our
developers
is
that
we
have
the
clear
actionable.
Pass/Fail
results.
Right,
Green
is
good.
Red
is
bad
and
we
we
do
things
like
the
pre
deployment
of
certain
resources.
Right,
like
you
have
to
deploy
a
service
before
a
workload
so
that
you
can
have
the
appropriate
environment
variables.
A
These
sorts
of
things
that
we
do
a
little
bit
of
that
and
the
decryption
of
the
II
Jason
is
what's
what's
happening
here,
so
what
we
can
do
with
kubernetes
deploy
since
we
built
all
these
tools
say
all
of
the
context
is.
We
can
actually
put
this
other
structure
inside
of
you
Jason
to
say,
create
me
a
kubernetes
secret
from
this
blob
named
this,
and
this
is
this-
is
really
cool,
because
somebody
can
keep
these
secrets
inside
of
version
control
and
then
deploy
them,
decrypt
them
at
runtime.
A
So
we
don't
need
to
worry
about
any
of
that
plain
text,
secrets
and
stuff
and
there's
a
few
other
checks
in
there
as
well.
So
like
we
can't
overwrite
a
cube
system
for
example,
or
delete
key
public.
You
know
these
sorts
of
things,
some
some
checks
and
some
checks
in
there
too,
to
help
us
out,
but
there's
an
example.
Love
me
deploying
a
little
go
app,
so
you
deployed
in
4.2
seconds
it's
good,
it's
green
success
and
we're
all
happy
sorry.
B
A
My
fault
always
thank
you,
so
much
so.
Stepping
back
to
Cooper
days,
deploy
Cooper
days,
deploy
TLDR
gives
us
clear,
actionable
pass/fail
results,
also
decrypts
us
our
secrets
from
me
Jason
into
kubernetes,
and
really
helps
us
with
that
developer
experience.
So
if
we
have
robots
deploying
for
us,
this
is
actually
prevented
us
from
having
to
give
our
developers
keep
CTL
access,
which
is
really
unique.
A
So
with
a
thousand
active
developers,
then
wanting
control
over
their
infrastructure
is
is
a
really
big
deal,
and
you
know,
given
that
with
Heroku,
you
sort
of
you
can
you
can
get
a
little
bit
more
intimate
with
your
application.
We
we
built
tools
like
kubernetes,
deploy
and
another
one
which
I'm
not
going
to
show,
but
I'll
talk
about
is
is
cube,
shell,
which
will
actually
sort
of
clued
us
from
having
to
give
cube
CTL
access
to
our
developers,
which
is
really
awesome
with
cube
shell.
A
What
we
can
do
is
actually
with
TTY
gjs
and
a
little
bit
of
OAuth.
We
can
actually
give
them
a
terminal
into
their
own
namespaces,
with
access
provisioned
by
config
map,
so
we
don't
actually
want
to
want
to
explicitly
give
Oliver
users
that
admin
access
and
deal
with
with
our
back
in
that
way.
So
with
cube
shell,
we
can
actually
provision
a
little
bit
differently.
So
this
is
where
the
phone
automation
begins.
We're
going
to
talk
about
our
cloud
buddies,
so
cloud
buddies
internally
marketed
as
friendly
kubernetes
controllers,
keeping
the
cloud
fluffy.
A
Our
is
our
is
our
brand
for
our
kubernetes
controllers
that
we
built
there's
a
lot
of
them
about
15
of
them
run
per
cluster
and
depends
given
the
type
of
services
that
run
there.
But
this
is
where
we
can
introduce
our
service,
automation
or
security
automation.
So
we
have
things
like
Redis
buddy,
for
example,
memcachedb
buddy.
A
You
know
that
would
provision
you
in
appropriate
resource,
under
whatever
circumstance,
now
that
we've
decided
to
build,
but
the
ones
that
we've
decided
to
build
for
security
and
or
implement
security
controls
in
and
I
want
to
talk
about
our
accountabilibuddy
bucket
buddy,
an
apple
buddy,
our
back
buddy
and
secret
buddy.
It's
it's
pretty
fun.
Here's
an
example
of
what
a
kind
of
Billa
buddy
is
is
designed
to
do
so
remember
when
we
created
our
kubernetes
namespace
and
assigned
identity
to
it
with
the
GCP
service
account
it's.
A
How
do
we
give
access
to
a
given
Google
API
to
that
service
account
without
somebody
pinging
me
and
having
me
to
go
into
manually
sort
of
click,
some
buttons
there?
That's
not
very
scalable,
as
we
deploy
a
lot
of
applications
to
our
cluster.
So
if
they
just
drop
a
config
map
in
the
repo
and
say
hey
I
want
to
give
this
service
account.
Now
it's
a
container
viewer.
Well,
we
can
do
this
and
we
do
have
some
checks
in
accountabilibuddy.
Oh
you
can't.
A
You
can't
give
yourself
sort
of
God
note
on
the
entire
project
right
like
this,
so
this
is
something
that
we've
accounted
for
and
we
don't
want
to
do,
but
we
do
have
some
api's,
which
we
will
very
happily
allow
you
to
check
into
diversion,
control
and
allowance
to
provision
that
access
for
you.
So
this
is
how
the
seven
of
us
sort
of
maintain
sanity
moving
forward
with
this
with
storage.
A
Again,
every
developer
would
like
access
to
their
storage,
and
since
we
use
G
suite
for
our
actual
sort
of
business
products,
we
can
actually
leverage
that
identity
on
Google
cloud
platform
and
then
with
with
buck
Bundy
in
this
example,
is
an
example
of
a
custom
resource
where
we
are
giving
storage
admin
to
robot
into
a
group
and
giving
object
creator
to
you
at
Shopify
com.
So
again,
with
all
of
this
in
version
control,
this
allows
our
security
team
to
actually
then
audit
who've
requested
these
permissions
win,
and
we
have
that.
A
We
have
that
trail
right,
of
course,
there's
that
ole
anybody
can
sign
it.
You
change
your
identity
and
get,
and
that's
all
good.
You
know
this
is
a
way
that
that
that
helps
us
early
on
build
that
auditing
into
our
platform
and
which
is
really
neat
so
with
this,
then
we
can.
We
can
assign
appropriate
permissions.
A
All
of
Inc
Luster's
are
hard,
especially
when
we
have
20
of
them.
So
how
do
we
ensure
that
all
of
our
workloads
are
running
running
securely
when
I
started
down
this
road?
I
said
well,
the
tools
didn't
really
exist
at
the
time
is
that
how
do
I
effectively
audit
this
cluster
to
make
sure
that
my
workloads
are
are
running
securely,
and
this
is
where
Q
bada
came
in.
We
actually
have
a
couple
dedicated
developers
on
this.
A
Now
we
open
sourced
aver
recently-
and
this
is
just
a
little
golang
application
that
allows
us
to
perform
audits
against
arbitrary
security
controls
on
kubernetes.
So
we
cannot
afford
a
mouth
service
account
token.
We
can
make
sure
that
the
appropriate
image
and
tags
are
running.
You
can
see
an
example
in
the
top
right
there.
I
just
took
a
screen
cap
of
the
terminal.
We
do
have
some
human
readable
output,
but
we
also
have
put
in
JSON,
so
we
can
automate
this.
A
We
can
audit
for
network
policies
to
make
sure
that
you
have
appropriate
default,
denies
policies
in
there
and
every
control
under
security
context.
We
can
also
audit
for
so.
If
somebody
comes
to
me
and
says,
hey
John,
this
image
has
been
determined
to
be
vulnerable.
What
can
we
do?
Well,
we
can
just
quickly
run
out
a
little
key
bada
check
to
make
sure
that
we're
not
running
that
image
anywhere.
A
There's
this
other
continuous
security
monitoring
piece
that
that
we're
missing
so
with
600
projects.
Scaling
is
really
hard.
So
we
have
developed
a
couple
tools
inside
and
help
us
out
with
this.
So
this
is
an
tool
called
nosy
bastard,
which
does
a
scheduled
scanning.
It's
there
like
our
traditional
sort
of
VA
or
a
Zed
map
and
map
Nessus
cans.
It
does
some
discovery
of
cloud
resources
on
GCP
on
AWS,
on
Heroku
and
our
other
operating
environments,
which
is
pretty
cool,
and
we
also
decided,
as
we
moved,
towards
a
really
strict
our
back
control
clusters.
A
We
needed
some
sort
of
visualization
there
to
help
us
out
and
look
at
look
at
this
to
see
which,
which
service
accounts
are
mapped
to
which
roles
and
these
sorts
of
things.
Then
we
built
that
into
nosy
bastard,
so
we
can
actually
go
and
clickety-click
and
see
which
are
back.
Roles
are
assigned
to
which
servers
account,
which
is
really
awesome
for
sending
security.
A
If
y'all
haven't
checked
that
out
and
they're
using
TCP,
please
do
it
is
just
really
really
good,
comprehensive,
GCP
inventory
and
there's
been
a
lot
of
horsepower
behind
it
over
the
last
little
while
and
wheezes
for
the
discovery
of
other
resources
and
also
the
forcement
of
IEM
policies,
which
is
awesome.
So
we
can
make
sure
that
if
you
or
a
developer
at
Shopify
creates
a
new
project,
they
can
add
any
external
identities
to
that
project.
A
For
an
example,
we
found
that
using
gke
and
the
concept
of
the
or
the
management's
in
this
group
that
these
nodes
come
up
and
go
down
a
lot
and
moving
from
traditional
sort
of
config
managed
chef,
ansible,
sort
of
environment.
You
know
where
we
had
to
ship
around
SSH
keys
and
these
sorts
of
things-
that's
not
really
something
that
we
want
to
do
so.
The
Google
account
statement
is
actually
running
on
container
optimized.
A
Os
will
actually
provision
you
access
via
SSH
by
taking
the
SSH
keys
or
the
project
metadata
and
adding
them
to
the
authorized
key.
So
you
can
SSH
into
it
and
if
you
haven't
done
a
g-cloud,
compute,
SSH,
ticka,
ticka,
ticka
and
Banger
in
the
server
right.
But
where
are
these
keys
held?
How
can
we
manage
them?
Do
we
have
to
a
sort
of
arguable
right,
so
our
Google
accounts
today
are
actually
backed
by
multi-factor
authentication
right.
A
So
if
you
want
to
talk
to
Google
cloud
platform
API,
you
got
a
you
gotta
sign
it,
so
we
already
have
some
controls
around
identity
there,
so
accessing
a
cluster
via
SSH
or
exiting
a
node
by
SSH.
This
is
all
backed
by
our
G
suite
identities
today,
which
is
again
back
by
MFA.
So
do
we
have
to
ship
these
entities
around
these
SSH
keys?
A
We
were
missing
at
the
time
that
I
wrote.
This
talk
a
lot
of
things,
but
kubernetes
move
so
fast
and
given
that
like
gke,
is,
is
open,
source,
kubernetes,
plus
a
couple
bells
and
whistles.
You
know
we
haven't:
we've
actually
scratched
a
lot
of
these
off.
So
for
the
longest
time
we
actually
didn't
have
access
to
the
API
server
logs,
because
Google
hosts
the
master
for
us
right.
A
Given
that,
how
do
you
get
these
logs?
Keep
CTL
proxy
curl
the
keep
API
server
log
endpoint
and
pipe
it
up
to
a
file
and
do
some
other
things
you
know
that
wasn't
available
to
us,
but
as
of
GK
173,
it's
actually
in
cloud
audit
logging
now,
which
is
phenomenal.
Network
policies
are
finally
here
with
Calico,
which
is
great
as
a
GK
176.
Our
egress
network
policies
work
as
of
184,
which
is
really
good
for
us.
We're
still
missing
pod
security
policies,
I've
been
poking
for
a
long
time.
A
I
can't
set
my
cluster
wide
pod
security
policy.
Now.
Why
is
that
important?
Why
is
that
important
enough
to
grab
a
slide,
because
we
have
what
300
services
may
be
deployed
there
now?
So
if
in
kubernetes
1.8,
we
have
the
new
the
new
flag
and
security
contest
called
a
lob
privilege
escalation
right.
So
how
do
I
ensure
that
allow
privilege
escalation
is
set
to
false
and
all
my
containers
and
all
my
workload
manifests?
Well,
that's
a
couple
hundred
PRS
I
would
really
just
like
to
do
that
once
so.
A
That's
really
where
that
comes
from
the
I
am
in
our
back.
Synchronization
has
been
really
really
fun
for
us,
not
so
on
gke
we
have
a
couple
different
authorizers.
We
have
the
web
hooker
the
IM
authorizer
and
we
have
the
are
back
authorizer.
So
for
those
who
don't
know,
when
you
make
a
request
against
the
api
server,
we
have
this
long
list
of
authorizers
and
if
any
one
of
them
succeeds,
the
request
will
succeed.
A
So
if
I
don't
have
an
appropriate
auerbach
role,
matching
Jonathan
DePaul's
for
Shopify
comm,
but
I've
provisioned
myself,
access
through
I
am
my
request,
will
succeed.
I
can
get
my
pods
I
can
list
my
list,
my
nodes,
these
sorts
of
things,
but
I'm
gonna
see
our
back
failures,
keep
showing
up
in
the
API
server
lock
we
page
on
our
back
failures
in
the
cloud
security
team-
and
this
is
this-
has
been
quite
the
journey
for
us
for
us
in
this.
So
we're
really
missing
that
I
am
in
our
back
synchronization.
A
That
would
be
super
awesome
if
that
could
exist.
The
GL
BC
configuration
office
for
the
identity
where
proxy
I
know
stuffs
and
the
work
there,
but
I
would
really
like
to
enable
the
identity
aware
proxy,
because
we
are
a
big
users
of
the
Google
cloud
load
balancer.
So
the
GLB
see
if
I
could
configure
some
eiope
there
that'd
be
sweet
and
container
identity,
I
cross
it
out
it.
A
Yeah
so
I
mentioned
we
have.
We
have
four
service
tiers
right
here.
One
two,
three
four
there's
actually
a
couple
of
the
other
ones
that
I
didn't
really
mention
one
is
tier,
shared
and
tier
share.
Does
that
sort
of
providing
access
like
Splunk
and
zookeepers,
and
these
sorts
of
things
Kafka?
And
how
do
we?
How
do
we
connect
to
that?
We
don't
actually
connect
any
other
way
than
we
would
normally
so
we'll
still
front
it
with
an
appropriate
low
balancer
and
we'll
still
do
TLS
and
the
front
and
clients
are
often
these
sorts
of
things.
A
So
we
still
using
like
the
same
traditional
login
methods
that
we
would
have
used
before.
If
this
is
running
in
the
DC,
we'd
use
the
same
method,
maybe
an
extra
it's
sort
of
TCP
tunnel
on
there
just
for
funsies,
but
that's
we
haven't
really
played
with
that.
Yet
so
we're
really
looking
forward
to
the
what's
called
IP
aliased
clusters,
right
where
we
can
actually
make
the
pods
and
service
and
first-class
citizens
on
Andromeda
like
that
that
that's
really
cool
it's.
A
A
We
protect
those
keys,
the
same
way
that
we
protect
it
any
other
key,
and
that
would
be
a
trouble
hard.
We
have
what's
called
the
default
to
open
culture,
so
typically,
your
engineers
have
access
to
all
the
things
and
given
that,
like
they're,
still
held
in
encrypted
Jason
and
deployed
the
same
way
that
we
would
deploy
any
other
app,
so
it's
it's
just
it's
just
another
key.
A
secret
is
a
secret
or
secret
I
mean
we
do
have
auditing
around
this.
So
you
know
if
we
would
determine
that
it
could
have
been
compromised.
A
A
A
It's
not
like,
so
the
question
was:
if
we
have
like
future
parodius
or
sort
of
cloud
trail
there,
it's
it's
not
the
same,
but
our
models
like
very
much
different
now,
like
moving
in
gke,
like
all
of
our
all
of
our
cluster
operations,
are
logged
appropriately
through
cloud
audit
logging
right
and
we're
satisfied
with
this,
but,
like
other,
like
in
cluster
sort
of
operations,
app
level
operations,
that's
all
kubernetes
now.
So
it's
it's
sort
of
like
build
your
own
auditing,
I!
A
A
Operationally
I
can't
really
speak
to
this,
but
security-wise.
It's
it's
all
about
the
snowflakes
right.
So
not
every
app
is
the
same.
A
Python
application
is
built
differently
than
a
Ruby
on
Rails
applications.
So
we've
we've
tried
to
stick
to
that
Bill
packs
model
and
try
to
introduce
as
many
controls
as
we
can
there.
A
But
some
applications
are
different,
so
we've
actually
had
to
walk
teams
through
appropriately
building
a
docker
container
or
just
keeping
keeping
on
top
of
these
things,
because
not
every
apps
the
same
I,
it's
more
of
like
an
operational
question,
I
guess:
sort
of
migration
there,
but
as
an
infrastructure
security
engineer
like
traditionally
everything
below
the
app
in
a
stack.
Now,
it's
like
everything
below
the
app
in
a
docker
container
right.
So
it's
almost
almost
not
my
problem
that
make
sense
cool.
B
A
So
the
question
is:
how
is
the
GCP
sort
of
identity
that
we
provision
for
this
namespace
actually
consumed
by
services?
So
when
we
at
the
beginning
in
the
services
DB,
when
we
click
create
cloud
runtime
and
that's
where
and
all
that
stuff
is
provisioned,
we
actually
mount
that
secret
in
kubernetes
and
then
provide
that
sort
of
as
a
volume
mount
you
know,
sort
of
secret
volume
and
inside
the
container,
google
application
credentials.
You
know
the
typical
place
for
this.
This
credential
file
is
where
we
would.
A
We
word
mount
the
path
to
that
and
we
provide
this
in
the
workload
manifest
sort
of
for
everybody.
We
have
like
a
number
of
blessed
stacks
right,
so
you
know
for
the
snowflakes.
It
doesn't
really
work,
but
if
you're
deploying
like
a
rails,
application
and
everything's
gonna
work
magically
for
you
and
that's
so
we
get
the
we
haven't
done
any
really
like
handle
templating
like
any
templating
like
I've
mentioned
before.
So
that's
a
little
bit
hairy.
Some
manifests
are
quite
long,
but
this
is
where
we
can
mount
those
for
developers.
A
A
For
yes,
so
the
question
is
like:
when
we
do
our
automated
patching,
do
we
actually
is
it
fully
automated
or
does
the
developer
have
to
actually
like
a
button,
and
sometimes
they
do
is
some.
You
know
some,
for
example,
like
a
gem
bump
or
something
like
this,
something
that
we're
more
confident
with
automatically
patching
yeah
they're
gonna
go
through
and
be
merged
automatically,
but
that's
that's
a
risk
right
if
we're
serving
production
off
there.
So
here's
your
patch
boom
boom
push
it
through
like
and
it
breaks.
Well,
that's
I!
A
So
R
is
accountabilibuddy
open-sourced
and
we
have
a
plan
for
it,
a
lot
of
the
buddies
that
we
actually
have
are
really
sort
of
contextual
e
bound
to
Shopify
and
our
patterns.
So
it
doesn't
really
make
much
sense
for
us
to
open-source
those
right.
Now
we
thought
about
it.
You
may
see
somebody's
show
up
in
the
future,
but
right
now
they're
sort
of
all
mono
repoed,
so
it
doesn't
make
sense
for
us
to
open
source
or
really
componentize
those
right
now.
A
A
Yes,
so
the
question
was
going
to
explain
a
little
bit
more
about
why
we
don't
give
a
key
petal
access
to
the
devs
and
it's
more
about
management.
I,
guess
at
this
time
so
remind
back
a
year,
and
it
was
myself
and
a
couple
others
doing
the
cloud
security
stuff.
So
how
do
we
do
effective
auerbach
for
a
thousand
devs
at
the
same
time
sort
of
automatically,
and
that
was
sort
of
like
a
design
decision
that
we
had
to
make
and
did
we
want
to
expose
or
teach
kubernetes
to
developers?
A
Maybe
next
year
you
know,
but
but
to
keep
us
moving
fast
and
to
keep
actually
services
being
created.
You
know
and
not
having
to
pause
and
learn.
Kubernetes
has
actually
been
a
really
big
win
for
us,
so
we
over
the
last
couple
days
we
had
our
company
hack
days
and
which
is
two
days
where
we
just
you
know,
build
the
things
that
are
cool
and
I.
A
Think
we
have
like
50
or
60
different
applications
now
deployed
to
that
cluster,
and
that's
it's
pretty
phenomenal
that
they
can
do
that
without
having
to
worry
about
the
kubernetes
patterns
right.
They
can
just
click
through
it
and
have
their
storage
and
have
the
Redis,
and
you
know,
do
all
the
things
we
do
actually
spend
a
lot
of
time
supporting
deployments.
A
We
have
what's
called
the
cloud
help
rotation,
so
we
go
through
and
some
of
us
will
swap
out
and
actually
provide
sort
of
front-end
support
to
the
devs
who
are
onboarding
onto
the
cloud
platform
and
us
on
the
security
team.
Actually
do
that
as
well.
So
we
can
actually
course
correct
if
necessary,
of
receiving
sort
of
anti
patterns
throwing
up
there,
which
is
which
has
been
really
helpful
for
us,
but
yeah.
We
do
provide
sort
of
cloud
platform,
front-end
support.
A
Oh
sorry,
very
funny:
how
are
we
handling
the
image
scans
is
all
GC.
Our
vulnerability
scanner
is
the
name
tool
that
we
use
for
that
and
right
now
we're
just
we
can
pull
out
if
containers
are
vulnerable
or
not
via
the
g-code
command
line,
chilla
D
or
hit
the
API
or
in
the
GUI.
So
that's
all,
but
again
it's
all
reactive
right
now
because
of
the
latency
second
I
would
love
to
do
it
on
deploy
it's
just
again.
A
A
Yeah,
the
question
is:
do
we
see
a
lot
like
duplicate
images
or
like
the
same
sort
of
services?
And
absolutely
we
do
yeah
like
a
lot
of
applications,
aren't
too
dissimilar
right,
so
we
do
actually
provide
blessed
base
images
to
start
with
service
off
and
that
sort
of
wherever
you
make
those
decisions,
but
yeah
there's
a
lot
of
duplication.
A
A
Yes,
so
how
are
we
handling,
alerting,
alerting
on
what
I
guess
the
the
thing
that
I
mentioned?
Was
we
page
on
our
back
failures,
so
we're
actually
looking
for,
like
returns
like
on
round-trip
or
for
our
back
failures
and
we're
digesting
the
application-level
logs
that
have
anything
that
contacts
kubernetes
api,
for
example,
and
just
alerting
standard
League
through
Pedro
duty,
as
we
would
yeah
a
regular
logging
pipeline
sort
of
source
done.