►
Description
Hybrid-Cloud, HIPAA Compliant Enterprise with Kubernetes - Steve Sloka, Heptio
This talk will outline how UPMC Enterprises utilizes Kubernetes on-premises and in a public cloud (AWS). We’ll see how a large enterprise balances SaaS offerings vs Kubernetes hosted services. We will walk through our approach to meet HIPAA compliance and how our deployments and underlying infrastructure changed to meet those requirements.
We'll also look at the Elasticsearch Operator which is an example of how we implement stateful applications. The operator ensures encryption at rest, in transit and provides a managed cloud offering inside Kubernetes. Also, we’ll look at how we implement Kong, an API Gateway in combination with Kubernetes Network Policies to ensure applications are limited to what they can access as well as how security is implemented outside of code.
Healthcare systems have a history of being large and complex, but Kubernetes has allowed UPMC Enterprises to be more agile and bring startup innovations to the enterprise.
About Steve Sloka
Steve Sloka is a Sr. Systems Software Engineer from Pittsburgh, PA currently working at Heptio dealing with all things Cloud, Containers, and Kubernetes. At UPMC Enterprises he managed the open source initiative and has been working with k8s since early 2015.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Okay,
so
good
afternoon
welcome
to
hybrid
cloud
the
hipaa-compliant
enterprise
with
kubernetes.
So
today
we're
going
to
talk
about
how
a
a
large
healthcare
provider
in
Southwest
Pennsylvania
manages
their
HIPAA
compliance
on
on-premise
systems
in
the
cloud
and
we'll
learn
a
little
bit
a
little
bit
of
our
journey
along
the
way
of
how
we
went
from
you
know
from
our
on-premise
systems
to
the
cloud.
So
if
you
do
miss
my
swag
I
work
for
hep
tio
now,
but
when
I
wrote,
this
talk,
I
actually
worked
for
UPMC.
A
So
in
this
point
on
I'll
be
at
UPMC
employee
until
the
last
slide,
so
you
can
see,
I
was
a
Software
Architect
I
was
responsible
for
bringing
containers
and
kubernetes
to
UPMC
and
UPMC
enterprises.
So
there
my
job
was
to
help
move.
Applications
from
developers
workstations
to
the
cloud
I
mean
do
that
in
a
HIPAA
compliant
way.
A
I've
done
a
lot
of
open
source
and
you'll
see
that
throughout
this
talk,
and
if
you
check
out
our
enterprises,
github
repo
you'll
see
lots
of
things
that
we've
written
so
who
is
UPMC
well
they're
the
University
of
Pittsburgh
Medical
Center.
So
there
are
16
billion,
enter
dollar
enterprise,
eighty
thousand
employees,
more
than
30
hospitals,
3.2
million
member
health
plan,
as
well
as
some
other
stuff
and
the
other
stuff
is
where
UPMC
enterprises
comes
into
play.
So
UPMC
enterprises
is
outside
of
UPMC
and
their
job
is
to
create
revenue
streams
for
the
hospital.
A
So
we
do
that
in
a
number
of
ways.
One
is
to
write
software
and
then
commercialize.
It
will
also
write
software.
It's
been
a
company
off
of
that
and
have
that
company
be
successful
or
will
help
invest
in
companies
and
do
the
same
kind
of
thing,
so
that's
kind
of
where
I
work.
That's
with
my
job,
doing
R&D
kind
of
work
and
you
know
sort
of
more
things
out
front
in
front
of
the
the
traditional
Hospital
sense.
A
So
a
lot
those
applications
needed
to
access
clinical
data
and,
if
you
think,
of
an
enterprise
with
80,000
employees,
there's
lots
of
systems,
but
when
you
think
of
clinical
data
in
a
healthcare
environment,
you
get
a
picture
like
this.
So
this
slide
is
about
a
year
old,
but
there's
over
a
thousand
different
applications
within
UPMC
and
has
varying
data
sources.
So
things
from
you
know,
let
Stronach
health
record
systems
to
radiology
to
just
you
know
the
lab
lab
works
because
on
and
on
and
on
data
now
is
counted
in
petabytes
per
year.
A
There's
a
big
diversity
in
the
behavior
of
the
data,
the
type
of
data
that
is
contained
within
the
performance
of
accessing
that
data,
as
well
as
the
access
patterns
of
how
you
get
to
it,
as
well
as
the
sensitivity,
so
everything
from
mental
health
status
to
HIV
status,
to
evidence
of
child
abuse.
You
can
imagine
this
is
the
most
sensitive
data
you
could
you
could
deal
with
and
we
don't
need
to
treat
it
correctly.
So,
with
that
background,
let's
talk
about
what
HIPAA
is
so
hip
is
the
Health
Insurance,
Portability
and
Accountability
Act.
A
Now,
if
you've
read
through
HIPAA,
there's
lots
of
different
pieces
and
parts
there's
lots
of
white
papers
you
can
dig
into
so.
The
one
I
want
to
focus
on
today
is
the
piece
that
talks
about
prohibiting
the
disclosure
or
misuse
of
information
about
private
individuals
so
TLDR.
We
want
to
keep
your
patient
data
healthy
and
safe,
and
only
letting
the
people
who
should
access
it
access
it.
We're
talking
about
things
like
Social,
Security
numbers,
medical
record,
numbers
or
Emer
ends
which
are
IDs
within
your
EHR
to
help
you
find
you
uniquely.
A
Even
your
name
is
pH
I.
Basically
anything
that
can
describe
you
specifically
we're
going
to
protect,
because
with
that
data
we
can
go
back
and
find
out
who
you
are
uniquely,
so
why
crate
HIPPA
might
make
up
these
rules.
These
regulations
and
these
things,
ideas
we
wanna
come
up
with
sort
of
standards,
have
standards
in
quotes
because
there's
not
a
standard,
but
we
want
to
figure
out
ways
that
we
can
talk
about
this
data.
A
So
there's
one
way
is
to
have
us
manage
our
own
data
correctly,
but
then
you
have
to
think
of
when
we
move
to
public
clouds
or
other
places,
they're
managing
our
data
now
or
if
you
have
vendors
that
have
feeds
of
your
data
right
you're,
only
as
strong
as
your
weakest
link.
So
if
I
have
the
most
secure
network,
but
my
vendor
is
pulling
data
and
they're
weak,
then
the
not
then
I'm
still
having
an
issue
with
data,
so
so
we're
gonna
talk
about
some
things
around
standards
and
how
this
data
is
get
put
together.
A
So
again,
if
you
don't
want
to
read
through
all
of
that,
I'll
summarize
the
main
topics
that
we
want
to
cover
in
terms
of
meeting
HIPAA
compliance
and
the
first
one
is
encryption
at
rest.
So,
while
data
is
sitting
on
a
disk
not
moving
sitting
in
a
cache,
you
need
to
have
that
data
be
encrypted.
It
can't
be
in
plain
text:
you'd
have
encryption
in
transit,
so
you're
moving
from
point
A
to
point
B,
but
that
needs
to
be
over
a
TLS
connection
or
whatever
protocol
you're
gonna
manage
that
through.
A
A
Those
aren't
good
enough
right,
knowing
that
you
know
service
account,
one-two-three-four
accessed
Steve
slokas
data
doesn't
help
me
in
terms
of
auditing
who
looked
at
what
data
so
need
to
have
auditing
down
to
that
level
and
they
need
to
have
a
BIA,
and
this
is
with
vendors
and
other
places.
So
that's
a
business
associate
agreement
so
basically
a
contract
between
you
and
a
third
party
to
decide
how
we're
gonna
manage
this
data
in
a
HIPAA
compliant
kind
of
way
cool.
A
So
those
are
the
kind
of
rules
that
we're
gonna
focus
on
today
in
terms
of
meeting
compliance
so
for
on
premises,
we'll
give
you
an
I'm,
gonna,
try
and
be
open
and
clear
about
what
we're
doing.
You
may
see
things
that
you
could
do
better,
it's
just
the
reality
of
where
we've
grown
from
our
from
our
our
lives
of
kubernetes
and
everything.
So
for
our
infrastructure.
Right
now
we
run
some
Red
Hat
atomic
and
we're
looking
at
moving
to
some
core
OS
tectonic
as
our
solutions.
A
This
is
all
running
on
VMware
servers
within
our
corporate
data
centers.
We
started
our
kubernetes
journey
in
january
of
2015,
so
then
nodes
were
called
minions
still.
It
was
still
data,
so
there's
only
six
months
old.
It
didn't
GA
until
that
that
summer,
so
we
were
sort
of
on
the
bleeding
edge
of
red
Tomic
as
well
as
kubernetes,
and
then
our
own
learning
of
how
this
stuff
worked
for
storage.
We
use
things
with
NFS
for
good
or
bad,
and
we
also
use
local
storage
some
applications.
A
We
have
may
have
20
users
right,
so
the
complexity
of
being
cloud
native
and
having
dynamic,
persistent
volumes
and
all
those
you
know
fun
topics,
sometimes
that
complexity
outweighs
the
simplicity
of
just
running
data
locally
to
a
disk
for
those
small
kind
of
applications.
So
sometimes
you
know
do
the
simplest
thing
to
solve
a
problem
and
don't
get
too
crazy
for
load
balancers
on
Prem
we
use
a
lot
of
f5,
so
f5
can
route
to
node
ports,
we're
looking
at
doing
some
ingress
routing
as
well.
A
We
also
do
just
straight
node
ports,
so
an
engineer
engineer
may
push
an
application
expose
at
our
node
port,
and
then
you
hit
one
of
those
nodes
and
just
access.
It
that
way
in
terms
of
workloads,
we
run
a
lot
of
CI
CD,
so
we
have
some
rules
or
our
source
control
systems
must
live
on
Prem,
so
our
gait
lab
runners
and
our
Jenkins
workers.
Those
will
build
images
and
push
images
out
to
Amazon,
where
we're
actually
gonna
run
this
things
in
production.
A
We
also
have
some
local
agents,
so
things
like
data
collection
and
we'll
get
into
in
a
few
slides.
Those
things
run
in
our
internal
kubernetes
clusters,
so
I
have.
This
slide
is
to
show
you
that
you
know
on-prem
is
harder.
It's
not
impossible.
I
think
the
one
thing
that
on
premise
misses
that
we're
missing,
at
least
in
our
environments,
are
a
lot
of
api's.
So
you've,
given
us
a
good
API
you
can.
You
can
do
anything
right.
That's
what
cloud
environments
typically
give
us.
A
You
know,
be
able
to
script
out
machines
and
script
out,
load,
balancers
and
those
sort
of
things
I'm,
not
saying
it's
impossible,
I'm
proud!
It's
just
for
the
infrastructure
we
have.
Those
are
the
struggles
that
we
ran
into
and
that's
kind
of
why
we
did
this
move
to
the
cloud
just
because
it
was
simpler
and
faster
for
us
to
get
get
things
done
and
get
things
moving,
and
you
probably
at
story
over
and
over
I'm
sure
so
again
we're
in
a
public
cloud
mode
now.
A
So
our
cloud
infrastructure
looks
like
this:
we
run
everything
on
AWS.
We
have
a
few
services
on
Azure
for
authentication
to
our
back-end
ad
systems,
but
mostly
things
on
AWS
I
run
single
region
within
multiple
availability
zone.
So
we
run
in
the
US
East
one
Virginia
across
three
zones,
typically
we'll
run
multiple
worker,
auto
scaling
groups.
So
an
auto
scaling
group
is
kind
of
like
your
deployment,
so
it
says:
I
want
to
have
n
number
of
these
machines
running
and
then
those
reference
launch
configurations
which
is
your
spec
file
and
your
deployment.
A
So
that
tells
you
hey,
I,
want
you
know
ten
machines
of
this
type
and
we'll
mix
those
up.
So
you
can
have
you
know
a
general-purpose
kind
of
machine
running.
You
can
have
machines
with
GPUs,
maybe
and
maybe
high
memory
machines
again.
So
you
can
mix
and
match
those
environments
to
meet
the
capacity
of
our
users.
A
Everything
is
running
on
container
Linux
from
core
OS
and
it's
some
custom
CloudFormation
that
we've
kind
of
pushed
together
to
help
deploy
them
deploy
those
environments.
Again,
I
mentioned
the
CI
pipelines,
those
push
to
elastic
container
registry
in
AWS.
We
were
in
classic
Yale,
B's
and
we'd
a
lot
of
VPNs
from
to
the
on-premise
systems
so
from
on
pen
to
cloudy
pcs
that
both
create
those
VPNs.
A
Here's
a
look
at
what
our
coop
cluster
looks
like
when
we
deploy
it
to
AWS,
so
we
deploy
on
following
the
NIST
templates,
which
is
the
National
Institute
of
Standards
and
Technology.
So
this
outlines
basically
two
V
pcs,
so
you'll
see
on
the
left.
You
have
an
application,
V
PC
and
the
right
there's
the
management
and
they're
paired
together.
So
everything
in
the
application
VPC
is
private
to
the
world.
A
There's
no
public
addresses
the
only
X
access
is
via
the
bastion
hosts
and
that
management
PPC
will
manage
those
users
on
those
Bastion
hosts
with
off
sparks
and,
if
you're,
an
AWS
guru.
That
sounds
funny,
but
we're
using
the
ability
to
drop
SSH
keys
onto
those
boxes.
So
the
cool
thing
about
opsworks
does
it'll
manage
SSH
keys
across
regions,
so
you
can
have
a
machine
in
every
region.
A
I
can
drop
an
SSH
key
in
there
and
it'll
deploy
that
for
me,
if
I
kill
that
key
it'll
revoke
that
from
all
the
machines
so
now
I
kind
of
have
you
know
key
management
at
the
edge
for
free
from
from
AWS
again.
This
is
open
source
out
on
our
github
for
enterprises.
So
I
can
curious
about
how
that
looks,
feel
free
to
dig
in
there.
A
This
screen
is
gonna,
be
super
hard
to
read,
so
don't
try,
but
this
shows
you
all
of
the
HIPAA
compliant
offerings
in
AWS.
So
again
in
that
VA
agreement.
Any
of
us
today
will
certify
these
things
that
you
can
use,
and
the
idea
is
that
we
can
only
put
pH
I
in
a
service
that
AWS
lets
us
do
it,
and
if
we
don't,
if
they
don't
have
it
on
that
list,
then
we
can't
put
data
there.
A
It
doesn't
mean
you
can't
use
a
service
if
it's
not
in
that
list.
It
just
means
you
can't
put
pH
I
in
it.
So
at
one
point
ElastiCache,
which
is
like
Redis
for
the
cloud
was
not
compliant,
so
we
couldn't
use
it
for
restoring
patient
data,
but
we
could
use
it
for
say
you
know
a
phone
number
lookup
or
a
zip
code.
Lookup
long
as
you
don't
put
pH
I
in
there
systems,
then
you'll
still
be
compliant.
You
won't
miss
that
up
as
long
as
you
can
manage
that
cleanly
within
your
from
your
environments.
A
So
not
everything
is
compliant,
like
you
said
so.
Typically,
new
services
aren't
so
things
that
came
out
in
reinvent
last
week,
mostly
those
are
not
going
to
be
compliant
when
they're
new,
like
ETS,
that
came
out,
probably
won't
be
compliant
for
a
while
I'm,
even
being
a
still,
you
know
still
in
preview,
but
things
like
like
there's
a
spot
between
an
Internet
gateway
and
ELB.
That's
not
so.
A
If
I
have
an
on-prem
system
and
a
VPN
tunnel
to
my
V
PC
that
spot
right,
there
is
not
encrypted
so
I
have
a
full
tunnel
in
that
little
spot.
Basically,
doesn't
it
Matan
doesn't
meet
my
encryption
and
transit
story,
so
we
have
systems
on
Prem
that
don't
do
TLS
right.
They
just
don't
do
that.
So
if
we
fire
those
streams
up
to
the
cloud
through
this
encrypted
VPN
tunnel,
we're
gonna
miss
HIPAA
comply
because
that
little
spot
there
doesn't
does
it
me.
Compliance
nearly
goes
AWS
cloud.
A
So
these
are
the
kind
of
gotchas
you
have
to
watch
out
for
it's
not
always
super
simple
and
that's
why
you
pull
in
a
good
Solutions
Architect
in
AWS.
So
this
is
a
good
pump
to
my
Koons.
So
if
everyone
got
tweets,
announce
a
good
job,
Mike
you're
appreciated,
so
he's
been
a
great
help
from
us.
So
he
helps
you.
You
know
build
out.
A
So
let's
talk
about
some
workloads,
so
things
on
kubernetes
we
run
so
we
find
some
stateless
applications
right.
Let
me
think
of
stateless
application.
You
think
of
they're
easy
peasy
right.
Stateless
Apps
was
the
first
workload
you've
ever
run.
That
was
my
first
hello
world
demo
on
kubernetes.
You
know
they're
self-contained,
there's,
no
storage,
there's
no
dependencies.
We
can
scale
them
with
deployments.
A
They're
super
simple,
but
all
you
have
to
manage.
There
is
encryption
and
transit
and
you're
pretty
much
sold
right.
So
those
are
easy
and
we'll
talk
too
much
about
those.
But
what
gets
interesting
your
stateful
applications
right
I
need
to
store
state
somewhere
in
my
application.
So
how
do
I
do
that?
Well,
their
struggles
up
you
run
into
so
things
like.
How
do
you
manage
persistent
storage?
A
How
do
you
resize
an
upgrade
so
I?
Have
this
application
running
and
I
want
to
upgrade
it
and
make
it
bigger?
How
do
I
do
that?
How
do
I
reconfigure
it
am
I
templating
so
I
have
this
thing
expect
out
somehow
and
I
scale
it
up
to
some
different
number
of
replicas?
How
do
I
store
that
back
in
my
configuration?
So
I
can
redeploy
that
easily
how
do
I
back
it
up
and
make
sure
that
I
can
restore
in
case
of
disasters.
A
Well,
typically,
we
do
stateful
sets
for
that.
Right
and
staple
sets.
There's
kind
of
a
story
today
of
how
we
do
things,
but
the
problem
is:
is
that
they?
What
what
they
do?
Is
they
allow
you
to
have
persistent
storage
attached
to
a
volume
and
I?
Think
of
the
my
buddy,
so
they
went
from
the
80s
grew
up
knees.
It's
my
buddy
and
me
right.
Wherever
your
pod
goes,
your
volume
is
gonna
follow
when
you
get
basically
one
volume
per
claim,
and
this
solves
one
problem
right.
A
A
But
the
problem
there
is
that
we
still
have
you
know
the
pieces
that
are
missing
right.
How
do
we
resize
how
to
reconfigure?
How
we
back
it
up?
We
haven't
solved
that
story
with
stateful
sets,
but
so
the
answer
kind
of
then,
is
maybe
an
operator
so
quorum
s
kind
of
coin.
This
turn
about
a
year
or
so
ago.
Obviously
an
operator
is
a
tool
that
lets
you
manage
application,
specific
knowledge
within
a
complex
app
in
in
toad.
So
now,
instead
of
having
a
mo
file
describe
your
application,
you
have
an
operator
on
the
operators.
A
Job
is
to
know
how
to
do
those
things,
how
to
back
it
up
how
to
scale
it.
How
to
you
know
resize
it.
What
kind
of
I
simplify
this
and
think
of
it?
This
way,
let's
just
make
it
work
like
a
cloud
provided
offering
right
I,
don't
care
that
it's
an
operator
and
that
has
CR
DS
and
has
you
know
all
these
crazy
things
that
are
fun
and
interesting
as
a
user?
A
All
I
care
about
is
that
I
can
deploy
this
and
make
it
look
like
my
AWS
resources,
anytime
I
can
offload
work
to
AWS
I'm
gonna.
Do
that
right,
because
it's
easy
things
like
an
RDS
database,
which
is
you
know,
say
my
sequel.
I
can
tell
AWS
hey,
build
me
on
my
sequel
database
I
can
check
a
box
to
scale
to
cross
zones
back
it
up
at
night
and
do
patches
at
3:00
a.m.
and
then
I'm
hands-off.
What
I
want
to
make
is
these
operators
work?
A
The
same
way:
I'm
gonna
have
a
bit
of
the
ammo
of
my
serie
DITA
to
describe
I
want
a
cluster
of
this
size
and
I
want
to
scale
it
across
zones
and
have
it
back
backed
up
automatically,
and
that
leads
us
to
the
elastic
operator.
So
this
is
an
open-source
tool
that
are
written.
It
fills
the
need
of
us
needing
elastic
search.
So
we
have
a
large
system
which
is
pulling
in
documents
from
the
hospital,
and
then
we
can
order
to
have
full
text
search
on
that
system.
A
Amazon
offers
elastic
search,
but
it's
not
HIPAA
compliant.
So
again
we
had
to
go
build
around.
So
that's
where
this
kind
of
got
born.
Out
of
so
you
lassika
operator,
basically
mimics
the
cloud
offering,
so
it
has
full
TLS,
meaning
it
has
automatic
search
generation.
So
if
you
don't
provide
certs
to
it,
it'll
go
build
its
own
again.
We
want
to
have
full
encryption
and
transit
for
encryption
at
rest.
We're
gonna
use
encrypted
EBS
volumes,
so
it'll
create
storage
classes
behind
the
scenes
and
that
storage
classes
will
have
that
checkbox
to
encrypt
the
volumes.
A
So
within
kms
you'll
be
encrypting.
Those
EBS
volumes
automatically
and
it
implements
you
know.
Encryption
and
transit
via
search.
Guard
search
guide
is
a
tool
from
floor
gun,
so
they
have
a
commercial
offering.
Let
me
have
an
open-source
version
as
well,
and
that's
worked
really
well
for
us
over
the
over
the
last
few
months.
A
A
It
has
automatic
snapshots
to
s3
and
it
deploys
ions
as
well,
so
things
like
Cabana,
which
is
a
tool
used
to
manage
your
data
or
visualize
your
data
and
there's
three
Brad,
which
is
a
tool
to
manage
elastic
itself,
so
you
can
get
stats
off
of
that.
So
again,
if
you
look
at
this,
this
is
kind
of
thing.
The
needs
that
I
wanted
to
have
in
terms
of
the
pieces
that
were
missing
for
staple
applications.
A
But
there
are
some
gotchas
right.
There's
some
work
to
do
some
things
that
we
haven't
be
able
to
to
run
through.
So
things
like,
like
shard
allocation
zone
allocation,
so
I
mentioned
we're
distributing
this
across
the
zones,
but
it
doesn't
have.
We
have
to
tell
lastik
how
to
do
that
properly.
Something's,
like
the
elastic
cluster
status,
we
don't
have
implemented
yet
and
that
will
help
us
enable
basically
rolling,
restarts
and
upgrades.
A
If
things
go
wrong,
they
may
be
different
stack,
someone,
maybe
rubia
more,
maybe
Java
and
again,
if
there's
a
vulnerability,
you
may
fix
one
or
not
the
other,
and
it's
just
harder
to
manage.
Sometimes
we
want
to
get
to
a
world
that
looks
like
this
right.
So
at
the
bottom,
the
teams
will
have
to
deal
with
all
those
things
in
the
middle
box.
So
basically,
all
traffic
is
gonna
route
from
the
from
the
client
through
comm,
API
gateway
and
then
that
thing's
gonna
do
the
logging
and
authentication
all
those
kind
of
bits.
A
So
here
we've
greatly
simplified
kind
of
the
developer
experience
so
right
now
this
this
is
still
using
third-party
resources,
because
we're
still
have
a
one
six
cluster,
but
you
can
sort
configuration
basically
as
that
amo
file
still,
which
means
we
can
check
that
in
a
source
and
we
can
code
review
it.
Kong
uses
two
program,
calling
you
it's
an
it's
a
restful
api,
so
you
can't
check
a
restful
api
in
the
source.
A
Here's
what
this
might
look
like.
So
here
we
have
two
namespaces
right
and
say
each
namespace
is
an
application.
So
namespace
a
is
application.
One
namespace
B
is
application.
You'll
see
that
we've
deployed
a
Kong
pod
in
every
namespace.
An
idea
is
that
traffic
again
we'll
hit
Kong
and
then
it'll
authenticate
and
then
it'll
get
passed
to
the
application.
A
So
we
wouldn't
want
to
do
that.
What
we
want
to
do
is
enforce
that.
We
have
traffic
route
from
the
application.
Pod
through
common
again
so
now,
an
application
talking
to
another
application
is
going
to
be
enforced.
There's
gonna
be
forced
to
go
through
the
same
authentication
mechanisms
that
we've
that
we've
implemented
and
how
we
can
enforce
this
with
things
called
Network
policies,
so
number
of
policies
were
implemented
implemented
by
a
policy
controller
and
that
controller
will
write
things
to
IP
tables
to
enforce
them
behind
the
scenes.
A
But
what
you
have
to
do
is
you
create
label
queries
you
say
hey
only
you
know
an
application
will
only
accept
requests
from
maybe
a
label
type
Kong,
and
that
will
make
sure
that
only
Kong
pods
will
be
able
to
talk
to
applications.
So
I'm
going
to
great
detail
about
network
policies
work.
Basically,
you
can
limit
the
traffic
you
can.
You
can
send
around
your
truck
your
cluster
and
1-8.
Now
you
can
also
limit
your
egress
traffic,
so
traffic
that
leaves
the
cluster
and
you
can
also
set
network
policies
around
that
as
well.
A
A
A
Sure
yeah,
so
the
question
was:
how
long
ago
did
we
build
this?
And
what
would
we
do
differently
now?
Now
that
we
kind
of
know
what's
going
on
so
yeah,
the
the
service
stuff
I'm
like
like
sto
know,
sort
of
things
are
interesting
because
the
problem
is
having
the
full
TLS
between
services.
We
end
up
writing
a
lot
of
self-signed
cert
s--.
We
pass
these
sorts
all
around
and
it
just
becomes
a
mess
because
it's
just
hard
to
manage.
A
We've
looked
at
doing
things
like
using,
say
vault
for
key
LS
management
to
manage
like
become
our
own
central
CA,
but
that
we
just
never
did
cuz.
Now
we've
got
to
manage
another
layer
of
storage
across
the
zones,
but
yeah
I.
Think
something
like
like
a
mesh
would
be
helpful
because
now
you
can
turn
on
turn
on
TLS
between
all
the
pipes
and
then
also
you
know
enhance
network
policies
in
the
sense
of
like
the
demo.
A
Today,
with
what
I
guerra
it
was
like:
hey
yeah,
let's,
let's
put
policies
in
every
single
pod
and
then
we
can
manage
the
traffic
yeah
for
sure.
So
I
think
that's
that
web.
It
would
do
I,
just
don't
think
SEO
is
there
yet
in
terms
of
being
production
ready,
so
I
think
it's
coming,
I
think
it's
neat,
but
I
don't
know
that
we
could
go
there
tomorrow
if
we
could
yeah.
A
A
A
Yeah
so
yeah,
this
guy
was
saying
that
you
know
it
should
be
okay,
because
you're
not
going
over
the
wire,
so
things
are
gonna
route
yeah
and
Lana's
too,
like
nothing
in
AWS.
Stops
you
from
using
a
non
compliant
service,
just
your
own,
your
own
legality.
Does
that
you
know
so
they
don't
like
my
console,
doesn't
look
small
or
have
less
things?
It's
all
still
there.
It's
just
an
up
to
you
to
make
sure
you
don't
don't
do
the
wrong
thing,
yes,
which
is
kind
of
cool
but
kind
of
scary,
and
it's
always
changing.
A
A
A
A
A
Yeah
we
were
thinking
about
having
you
know,
one
kong
namespace
and
then
writing
network
policies
out
from
there,
but
it
just
kind
of
got
tricky
and
our
initial
version
was:
let's
keep
it
simple
still
and
deploy
it
this
way
and
just
fight
the
overhead
of
Kong,
which
is
just
basically
engine
active
you're,
not
coming
just
run
another
nginx
ingress
kind
of
POD.
So
it's
not
really
a
high
overhead.
Yes,
yes,.
A
A
So
yeah,
the
question
is:
is
that
there's
things
in
Google
is
using?
That's
not
HIPAA
compliant
yet,
but
you
want
to
it
is
encrypted
I
think
you
have
to
check
with
Google
on
that
to
see.
If,
if
you'd
be
an
illegal
issue,
I
think
I
mean
you
can
you
can
do
whatever
you
want?
I
think
there's
nothing
stopping
you,
but
it's
a
matter
of
if
you
had
a
data
breach,
I
think
Google
is
gonna,
say:
hey,
that's
not
our
job
yeah
yeah.
A
Yeah,
that's
compliant
yeah,
so
yeah
that
is
still
compliant.
It's
just
I
guess
the
matter
of.
If
there's
a
legal
issue
or
not
that
you
want
to
deal
with,
but
because,
if
Google's
not
certifying
that
as
HIPAA
compliant,
then
then
they're
not
going
to
help
you
out
if
there's
a
legal
profit
with
them,
if
there
is
a
breach,
but
you
are
doing
the
right
things
you
mean
like
I
could
store
data
I,
don't
know
what
a
service
it's
like
like
API
gateways,
and
is
it
a
service
in
AWS
that
it's
compliant?
A
But
caching
is
not
so
I
guess
if
I
stored,
cached,
pH
I
in
that
API
gateway.
That
sounds
weird
I
know,
but
I
think
they
still
wouldn't
help
me
if
I
had
a
breach
because
they
said
that's
not
compliant.
So
you
shouldn't
do
that,
but
you,
but
you
are
doing
the
right
things
in
terms
of
encrypting
data
for
sure.
A
A
Sort
of
not
really
so
we
have
like
flu
Andy
running
everywhere
and
we
typically
just
write
standard
out
to
make
that
simpler,
so
so
be
like
append
the
the
header
of
the
log
with
something
like
HIPAA
there's,
some
some
unique
value,
and
then
we
can
put
a
filter
on
that
and
then
we
can.
You
know
when
that
flu
indeed
picks
up
that
those
records
they
can
dump
them
to
an
s3
bucket
somewhere,
that's
yeah.
A
We
also
had
one
team,
so
we
have
lots
of
teams
doing
all
different
ways,
which
is
probably
not
not
word,
but
one
team
wrote
like
a
little
sidecar
to
dump
it
to
s3,
so
they
just
write
to
like
a
place
on
disk,
and
that
was
just
because
they
were
running
on
pram
and
they
were
writing
to
like
an
NFS
share.
And
then
let
me
switch
to
the
cloud
and
an
s3
was
the
right
place
for
that.
So
that's
the
best
to
me
clump
so
far,
yeah
yeah,
it
is
it's
always
hard
yeah.
It's.
B
A
C
A
Yeah
that
is
interesting,
yeah
I,
know,
I
know
it's.
We
had
lots
of
people
in
our
environment
wanting
to
run
security
scans
and
all
these
things
and
they
find
out
they're
running
in
core
OS
and
then
it's
just
they
don't
deploy
because
there's
no
agent,
you
can't
install
anything
so
yeah.
We
don't
have
any
like
that,
and
at
least
on
our
kubernetes
infrastructure
I
know
on
the
Windows
machine.
There
are
a
few
windows
instances
that
people
run
it
and
they
have
those
on
there.
But
yeah.
B
A
A
So
you
may
scan
an
image
upfront
and
then
you
know
next
day.
There's
invulnerability
that
comes
out.
So
kate
will
basically
say
hey.
This
thing
was
good
yesterday.
Now
it
just
spun
up
so
it's
about
there's,
you
know
things
issues
with
it.
We
didn't
actually
implement
any
of
that
yet,
but
yeah
those
are
things.
I
would
do.
I
guess.
A
B
A
It's
it's
goofy
I
mean
HIPAA
compliant,
it's
not
like
roll
cut-and-dry,
it's
just
the
better.
You
make
your
networks
and
it's
more
secure.
You
are
so
all
the
best
practices
come
into
play,
so
the
I
think
the
it's
great.
So
the
less
phone
abilities
you
have,
the
less
room
for
error,
you're
gonna
have
so
I
think
it's
always
a
good
thing
to
do
that
way.
Yeah
else,
yeah.
A
So
the
question
was:
where
network
policies
enough
to
satisfy
our
compliance,
I
think
they
were
at
least
the
last
time
we
did
this.
We
typically
deployed
an
environment
port
per
an
AWS
account
per
application
or
per
team,
so
it
just
seems
very
small.
So
we
didn't
have
big
multi
tenant
clusters,
where
we
had
to
deal
with
lots
of
lots
of
lots
of
different
applications
so
yeah.
Those
are.
Those
were
enough.
I
think
this
satisfied
satisfy
our
folks
yeah.
A
Yeah,
so
the
question
was:
how
do
I
deal
with
incidents
response
so
say
we
get
compromised.
What
do
we
do
with
that,
and
so
we
have
a
whole
cloud
kind
of
team
that
manages
these
environments
and
you
can
see
they
have
a
big
split
between
development
and
deployment
so
that
engineers
can't
access
production,
so
at
least
they're
off
the
hook
in
terms
of
working
with
that
I
know
they
have
ticketing
systems
and
the
architecture
has
those
bastion
boxes.
A
A
So
if
I
could
figure
out
what
container
yeah
so
if
I
could
figure
out
what
the
container
was
I
would
just
remove
those
labels
from
it.
So
it
would
come
out
of
the
deployment
and
it
would
still
up
stay
running
but
and
cube
will
spin
up
a
new
and
new
version
of
that
pie,
and
this
will
still
stay
there
sitting
in
the
cloud
environments.
We
don't
have
any
any
local
data,
they're
all
okay,
so
we've
mounted
the
EBS
volumes.
If
we
had
that,
that's
how
we
would
isolate
that
yeah.