►
Description
Jon Deeming, VP at Experian, presents how automation can ensure that security policy is consistently deployed into production OpenShift container environments. He shares some of the lessons learned in Experian’s move to containers. The dynamic nature of containers requires thinking through all the attack vectors in a new environment and creates the need for behavior-based security technology. This new technology needs to work with the system primitives, labels, resources and other metadata to maintain the desired security posture. At the same time, it’s critical to dive deeply into the Kubernetes architecture to understand how to properly deploy resilient services. Understanding the network and system behavior of pods and containers will enable DevOps and security teams to detect suspicious behaviors. This session provides a framework for evaluating container security and advice for securing business critical services and protecting sensitive data.
A
Cool
good
afternoon,
everybody
just
quick
showers,
how
many
people
kind
of
flew
in
last
night
feeling
a
bit
jet-lagged,
yeah
I
highly
recommend
standing
on
one
of
these
stages.
The
adrenaline
really
just
gets
you
over
the
jet
lag
quite
quickly,
so
hi
I'm,
John
Deming
I'm,
the
VP
for
the
platforms
of
service
center
of
expertise
to
Experian
I'm,
going
to
talk
today
a
little
bit
about
some
of
the
thinking.
We've
had
to
do
around
container
security.
So,
as
we've
rolled
out,
containers
experience
security
very
seriously
and
there's
a
few
different
areas.
A
We've
had
to
think
about
in
terms
of
inspection
and
controls,
and
things
like
that.
So
I'm
going
to
take
you
through
that
I'm
going
to
start
off
explain
a
little
bit
about
experienced.
If
people
don't
aren't
familiar
with
Experian
I'm
gonna
explain
a
little
bit
about
our
platform
as
a
service
initiative.
What
is
it
we're
setting
out
to
do
and
then
I'm
going
to
talk
through
some
of
the
the
sort
of
security
framework
pieces
that
we've
got
and
hopefully
have
some
time
at
the
end
for
questions
as
well?
A
So,
if
you're
not
familiar
with
Experian,
we
are
primarily
a
data
analytics
company,
so
we
are
really
helping.
Consumers
and
businesses
connect
better,
make
better
decisions.
So,
from
a
personal
perspective,
helping
you
understand
your
own
credit
portfolio,
the
make
helping
you
make
better
financial
decisions,
get
that
access
to
better
financial
products
from
a
business
perspective.
Helping
you
understand
your
risk,
protect
yourself
and
also
generate
new
new
market
opportunities.
A
Okay,
so
probably
60%
40%.
Now,
okay,
in
terms
of
the
sort
of
the
side
of
things
from
a
consumer
side
of
things,
these
are
the
sorts
of
interactions
that
we
help
with
solves
the
mortgage
vehicles,
insurance
and
the
like.
So
it
is
a
really
wide
range
of
products
that
we
offer
and
it's
also
obviously
a
significant
amount
of
data
that
we
we
look
after
as
well,
and.
A
Just
in
terms
of
really
the
it's
that
we
have
offerings
that
are
both
consumer
focused
and
I'm,
business
focused
I,
think
most
people
when
you
talk
about
Experian,
to
think
of
it
as
a
credit
reference
agency,
which
is
part
of
what
we
do.
But
it's
certainly
not
the
entirety
of
what
we
do.
We
have
a
an
entire
consumer
business.
That's
absolutely
focused
on
helping
individuals
and
does
a
lot
of
work
with
individual
credit,
education
and
things
as
well.
A
So
part
of
what
we're
doing
over
the
next
few
years
is
we're
going
through
a
large
digital
transformation
program,
and
one
of
the
pieces
of
digital
transformation
program
is
our
platform
as
a
service
and
that's
the
responsibility
of
my
team,
our
center
of
expertise.
What
we're
alternately
aiming
to
do
is
exactly
what
it
says
here.
So
historically,
Experian
has
been
a
very
acquisitive
company.
A
We've
had
a
number
of
different
acquisitions
over
the
years
across
a
number
of
different
geographies,
and
what
that's
led
to
is
a
significant
amount
of
variance
with
the
different
application
platforms
that
we
have
in
place.
The
different
programming
languages
that
we
have
in
place,
the
different
architectures
and
one
of
the
things
that
we
have
identified
as
a
real
target
opportunity
for
us
is:
what
can
we
do
to
simplify
that?
A
There's
a
significant
value
for
us
in
being
able
to
get
to
a
much
more
automatable
platform
to
have
something
much
more
elastic
so
that
we
can
be
cloud
agnostic
as
well.
We
have
some
plants,
some
areas
of
the
business
that
are
using
cloud,
but
they're
really
coupled
to
specific
clouds,
in
particular
simple
regions,
for
example,
in
the
UK
under
the
the
financial
conduct
authority,
it's
mandated
that
we
have
the
capability
to
work
with
multiple
clouds.
A
We
can't
couple
ourselves
to
a
specific
cloud,
and
so
what
we're
looking
at
doing
from
my
team's
perspective,
is
putting
a
reference
architecture
and
the
reference
implementation
together
of
an
elastic
platform,
so
something
developers
can
target
for
their
application
development
that
we
can
run
elastically.
We
can
use
on-prem
resources,
we
can
use
colocation
facilities
and
we
can
also
use
public
cloud
different
public
clouds,
as
need
be
at
the
core
of
that
we've.
A
We've
picked
Red
Hat
OpenShift
as
the
the
kubernetes
platform
that
we
utilize,
so
the
intent
is
that
once
we
have
an
application
built
for
the
platform,
the
developer
can
then
roll
it
out
very
quickly
between
development
and
production,
because
the
two
platforms
are
functionally
identical.
We
can
also
migrate
applications
more
easily
between
regions
because
we
standardized
the
application.
A
Sorry,
the
platform
deployment
between
regions,
so
if
I
have
an
application
that
is
built
and
operating
in
North
America,
for
example,
and
the
business
identifies
that
there
is
a
similar
target
opportunity,
a
market
we
could
go
after
in
SE,
Asia
Pacific.
We
want
to
be
able
to
move
that
application
over
to
say
Singapore
relatively
quickly.
You
know
there
are
some
things
that,
obviously
we
can't
do
the
dark
magic,
so
localization
UI
changes,
but
there
is
a
lot
that
can
be
done
just
in
terms
of
using
something
like
open
shift
to
abstract
away
the
underlying
platform.
A
But
one
of
the
things
that
that
has
lead
you
to
is
this.
You
containerized
architecture
and
containers
obviously
brings
with
it
a
different
workflow,
different
tools,
different
capabilities
and
obviously,
as
part
of
that,
it
also
brings
with
it
new
things
you
have
to
think
about
from
a
security
perspective,
and
we've
we've
spent
a
fair
amount
of
time
as
part
of
this.
Looking
at
what
tools
we
want
to
utilize
and
where
the
the
attack
vectors,
what
the
things
that
we
think
need
to
be
thought
about,
but
as
well
as
building
that
out.
A
What
we've
also
been
trying
to
do
is
to
embed
the
tools
into
the
applicant
into
the
platform
itself
as
far
as
possible,
because
what
we
don't
want
to
do
is
to
particularly
couple
ourselves
again
to
a
specific
cloud
provider.
So
AWS
might
have
a
fantastic
tool,
but
if
I
can't
then
get
that
on
as
your,
for
example,
that
doesn't
really
do
me
much
good
and
they
also
need
it
to
be
available
in
the
data
center.
So
we've
also,
you
know
when
we
talk
about
what
we
selected.
A
You
just
bear
in
mind
our
use
case
on
what
it
is.
I
think
that
the
framework
we've
used
will
still
be
useful
to
everybody.
In
terms
of
thinking
about
the
different
sort
of
products
and
capabilities
you
might
need
to
build
in,
but
just
bear
in
mind,
our
use
case
may
not
be
exactly
the
same
as
yours.
I
am
going
to
talk
about
some
specific
vendors
that
we've
we've
chosen,
I'm
not
going
to
go
into
exact
detail
configurations,
because,
obviously
you
know
we're
talking
about
security,
so
I'm
not
going
to
start
giving
out.
A
You
know
passwords
and
things
or
exact
conflicts,
but
I
am
good.
I
will
talk
about
some
of
the
vendors.
What
we've
used
why
we've
used
them
in
the
stack
I
am,
however,
gonna
skip
over
a
whole
bunch
of
basic
security.
Stuff
I
am
assuming
that
you
have
all
of
these
sensible
security
practices
in
place.
So
don't
assume
that
I
am
just
talking
about
security,
as
you
know
that
we
talk.
What
we
talk
about
today
is
magic
and
it's
the
be-all
and
end-all.
A
You
still
need
to
have
things
like
a
good
information
security
policy,
proper
architecture,
governors
all
of
the
things
and
more
and
more
and
more
it's
an
ever
growing
field.
So
please
don't
think
that
just
because
I
stand
up
here
and
I,
don't
for
example,
say:
oh
yeah,
let's
talk
about
web
application.
Firewalls
means
that
you
don't
need
a
web
application
firewall
I'm,
absolutely
not
saying
that.
A
A
Also
container
security
means
really
really
really
quickly,
as
I'm
sure.
Probably
most
people
know
the
you
know
the
release
cadence
of
kubernetes,
of
what
for
about
four
releases
a
year
is
it's
pretty
quick.
Open
shift
is
obviously
following
on
behind
that,
as
the
new
releases
come
along,
there
are
new
new
capabilities,
come
on
from
the
various
vendors
new
pieces
of
software
launch.
A
We
also
obviously
see
different
things
happening
from
an
attack
perspective.
So
recently
there
was
the
the
community's
vulnerability
around
their
PVC
end
team
empty
mount
that
that
came
up,
which
hopefully
everybody's
seen
the
cv4
and
patched
and
and
it's
all
been
addressed,
but
that
was
a
couple
of
months
ago.
You
know
that
was
a
brand
new.
So
again
the
basics
of
security.
You've
got
to
keep
an
eye
on
the
software
that
you're
running
keep
an
eye
on
the
exploits
that
are
out
there.
None
of
this
is,
you
know.
A
So
before
I
go
any
further,
how
many
people
are
running
OpenShift,
openshift
or
kubernetes
at
the
moment?
Okay,
and
how
many
people
that
aren't
know
what
a
pod
is?
Okay,
so
we
got
a
mix.
Okay,
so
I'll
cover
what
what
we
mean
by
a
pod
for
a
second,
because
it's
worth
talking
about
the
pod
as
the
core
of
what
we're
going
to
describe.
So
a
part
is
a
kubernetes
construct
and
for
anyone
that's
done
a
lot
of
kubernetes
work
in
the
audience.
Yes,
I'm
about
to
simplify
some
stuff.
A
Please
don't
beat
me
up
after
I
leave
the
stage
I
cut
a
pod
is
basically
a
mechanism
of
running
one
or
more
containers
inside
of
its
name's
own
namespace,
with
an
IP
address
allocated
to
it.
Roughly
speaking,
so
you
can
have
one
or
more
container
images
running
if
they
can
be
collaborating
with
each
other
providing
different
pieces
of
functionality.
A
One
might
be
a
monitoring
agent,
one
might
be
the
actual
application
itself.
You
might
have
one
that
does
initialization
and
setup
for
you,
but
fundamentally
from
a
kubernetes
perspective.
The
pod
is
your
unit
of
scheduling.
So
when
you
set
up
your
application,
it
will
be
comprised
of
one
or
more
pods,
hopefully
more
than
one,
hopefully,
because
you're
doing
something
with
high
availability,
you
will
have
a
minimum
of
two
pods.
A
So
when
we
were
starting
out
what
we
started
with
was
we
basically
took
a
look
at
the
pod
and
said:
okay,
so
from
the
lifecycle
of
the
pod?
What
are
the
various
things
that
we
need
to
think
about
in
terms
of
the
various
things
we
need
to
be
aware
of
to
monitor,
to
control,
to
put
governance
around,
and
that's
that's
basically
what
I'm
going
to
talk
through
over
the
next
to
the
15
minutes,
or
so
so,
let's
start
off
with
images
so
container
images.
A
The
container
image
is
obviously
the
thing
about
is
fetched
from
a
registry
is
loaded
into
one
of
the
containers.
When
your
pod
starts
up
and
is
then
the
golden
master
container,
all
containers
is
running
from
it's
going
to
be
pulled
in
from
somewhere.
Maybe
it's
built
in
house
and
it's
just
living
inside
your
own
registry.
Maybe
it's
coming
in
externally,
there's
a
number
of
different
ways.
It
could
come
in
so
start
off
with
image.
Provenance
image.
Provenance
is
something
that
gets
a
lot
of
conversation,
honest
I'm,
more
skeptical
about
image
provenance
than
I.
A
A
The
issue
with
it,
for
me,
is
that
anyone
that
is
relatively
competence
as
a
malicious
actor
can
probably
find
a
way
of
getting
around
image
provenance.
It's
not
very
hard
to
say,
I've
taken
our
limit
externally
I'm,
going
to
deconstruct
it
I'm,
going
to
put
it
back
together
myself
internally,
so
that
it
looks
dude
has
a
different,
checksum
and
I'm
going
to
load
it
up
into
the
registry.
So
it
looks
like
it
was
an
internally
produced
image:
it's
not
a
perfect
solution,
it's
absolutely
a
call
step.
A
It's
something
you
should
absolutely
look
at,
but
I
did
I
would
say,
don't
look
at
image
provenance
as
any
kind
of
panacea.
It's
a
really
sensible
control,
but
it
is
only
going
to
get
you
so
far.
So
for
us
with
image
provenance,
we
use
a
combination
of
Red,
Hat
cloud
forms
and
ansible
Tower.
So
cloud
forms
is
basically
looking
at
our
registry
and
helping
us
understand
where
the
image
is
coming
from
so
which
registries
are
they
using?
What
sort
of
what
containers
are
actually
using?
A
What
sorry
which
projects
using
those
images
which
pods
are
using
those
images?
So
it
helps
us
understand
what
we
have
in
the
environment.
That's
that's
kind
of
rolled
out.
Ansible
Tower
enables
us
to
wrap
some
more
governance
around
some
of
the
the
image
importing
an
artifact
workflows,
so
we
can
enable
developers
to
to
bring
in
external
images.
A
You
may
have
a
set
of
external
registries
that
you
trust
that
you
will
developers
pull
images
in
from.
You
may
have
a
set
of
registries
that
are
totally
untrusted
in
your
block.
You
might
say
you're
going
to
have
a
process
whereby
you
that
every
image
before
it's
before
it's
imported
in
your
lonely
you'll
use
something
like
sansabelt
our
to
put
together
a
workflow.
You
need
to
think
about
that.
You
need
to
think
about
where
you're
going
to
get
this
stuff
coming
into
your
environment.
A
From
that
said,
even
if
you
decide
to
block
the
registry
again,
a
lot
of
this
stuff
is
available
on
github.
You
know
you
can
get
a
docker
file.
You
can
go
download
the
libraries
you
can
build
this
stuff
yourself.
It's
not!
If
someone
wanted
to
go
and
get
some
of
this
stuff
and
inject
it
maliciously.
It's
not
beyond
the
wit
of
man
to
go
and
do
this,
so
you
do
like
I
say
be
careful
about
relying
too
much
on
image.
Provenance
is
a
perfectly
sensible
step,
but
it
will
only
get
you
so
far.
A
Similarly,
there's
there's
a
whole
piece
around
digital
signature,
verification,
so
understanding
and
starting
to
sign
doctor
images.
So,
for
example,
I
can
know
that
this
particular
image
is
digitally
signed
by
red
hand,
hasn't
been
altered.
Since
it
came
to
me
that
is
that's
again
useful,
because
I
can
know
that
something
hasn't
been
tampered
with
in
the
interim.
A
So
the
next
thing
is
your
docker
registry.
Absolutely
so
the
way
are
you
scanning
this?
So
there's
this
few
different
products
we
go
with
black
duck
and
that
gives
us
metrics
around
CDs
code
quality.
What's
actually
going
on
in
that
registry,
so
we've
got
an
understanding
of
the
the
changes
that
happening.
We
can
do
reporting.
A
One
of
the
nice
things
I
actually
like
about
using
the
openshift
internal
registry
is
because
we
use
an
integrated
OAuth
I
can
track
any
image
back
to
a
developer,
because
every
image
is
to
a
project.
Every
project
has
a
requester,
so
worst
case
scenario.
If
we
find
a
particular
CVE,
I
can
track
it
back
to
you.
Somebody
from
an
Active
Directory
account
perspective
and
say
you've
got
a
problem
that
needs
to
be
addressed
here.
You
know,
there's
you,
you
can
get
some
some
attribution
of
image
image
ownership
back,
so
that's
really
really
useful.
A
Think
about
ways
that
you
can
minimize
the
number
of
changes
to
the
layers
so
that
you're,
if
you're
submitting
a
new
container
image,
you
can
reuse
as
many
images
as
many
layers
as
possible,
so
think
about
not
creating
too
many
layers,
because
that
indirection
can
cause
problems
from
a
performance
perspective,
but
also
think
about
other
a
cool
set
of
layers
that
you
can
lay
down
first,
and
you
can
then
lay
your
code
down
above
it.
There
are
things
like
the
Red.
Hat
has
two
eye
system
without
B
shift
that
helps
with
a
lot
of
that.
A
If
you're
manually
constructing
docker
file,
then
you
know
you
need
to
give
that
some
thought.
But
the
reason
for
saying
that
is
that
can
also
make
your
system
much
more
efficient.
It
can
also
make
the
security
scanning
a
lot
easier,
because
if
you've
scanned
a
base
set
of
layers-
and
they
don't
change-
you
then
know
you
know
it's
a
lot
easier
to
then
just
scan
the
deltas,
so
it
can
make
that
performant
that
whole
process
much
more
performant.
A
So
the
next
thing,
obviously,
then,
is
monitoring.
So
monitoring
Feroze
means
really
couple
of
key
things.
One
is
the
log
messages,
so
hopefully
your
during
the
standard,
a
standard
error,
heavy
container,
if
not
the
the
bit
inbuilt
a
stack
for
openshift
works,
really
nicely
for
us.
One
of
the
things
we
have
supplemented.
That
with,
though,
is
as
well
as
the
Africa
stack
itself.
A
You
know
we
don't
use
the
ethic
stack
for
long-term
security
logging,
long
term
security
analysis,
it's
really
to
help
us
with
troubleshooting
and
short
term
trend
analysis,
but
it
allows
us
to
then
have
something
in
Splunk
to
be
able
to
go
back
to
the
long
term
trend
analysis,
security
off.
It's
also
really
important
because
then
you're
it's
outside
of
like
the
platform
administrators
hands.
So
from
a
platform
support
perspective.
A
If
someone
compromises
the
platform,
they
can't
you
know
they
might
be
able
to
get
into
the
elasticsearch
and
oops
sorry,
we
lost
our
elasticsearch
database
and
suddenly
normal
log
messages.
There
anymore
doesn't
matter
they're
still
going
to
Splunk,
so
we've
got
an
extra
layer
of
protection
and
from
that
perspective
as
well
performance,
metrics
performance
metrics
are
obviously
really
important.
So
again,
we
use
the
the
in
Bill
openshift
heat,
stir,
korcula
setup,
and
they
are
they're
really
useful.
Actually,
they've
been
really
useful
for
developers,
just
as
a
basic
thing.
A
To
be
honest
with
you
forget
about
security
for
a
second
just
to
be
able
to
baseline
understand
the
application
as
they
start
to
do
the
development
that's
been
really
really
useful,
but
the
other
thing
we've
we've
done
with
the
performance
metrics
is.
We
also
use
a
product
called
cystic
and
we
use
cystic
to
monitor
the
entire
cluster.
So
the
metrics
stuff
is
really
useful
for
a
project
administrator
to
be
able
to
look
at
that
particular
project
and
say
I
understand
what
I'm
doing
to
the
CPU
Network
memory.
A
That
sort
of
thing
cystic
from
a
cluster
perspective
allows
us
to
take
a
a
look
at
the
entire
cluster
to
understand
all
the
namespaces
that
are
running
to
understand
disk
I/o,
ops,
there's
a
lot
of
really
good
stuff
in
there
I.
If
you
haven't
spoken
to
those
guys,
I
really
do
recommend
it.
They
it's
a
really
good
product.
We
also
use
their
cystic
secure
product
as
well,
which
I'll
touch
on
briefly
cystic
secure
allows
you
to
get
a
real-time
data
flow
of
all
of
the
activity.
A
I,
probably
a
song
for
this
I'm,
not
a
huge
fan
of
Prometheus.
It's
a
really
nice
tool.
The
problem
with
Prometheus
is
the
in
the
current
incarnation
is
the
role
based
access
security,
or
rather
the
entire
lack
of
role
based
access
security.
It's
very
very
difficult
to
be
able
to
set
it
up,
and
one
of
the
things
we
looked
at
very
early
was
we
could
put
Prometheus
in.
It
would
not
be
a
problem.
We'd
be
able
to
gather
data.
We
could
use
graph
honor.
A
We
could
print
and
praise
lots
of
nice
graphs
and
trending,
but
it
was
really
only
gonna
be
useful
for
the
infrastructure
team
for
anyone
that
really
had
the
right
to
be
able
to
see
the
entire
cluster
I've
also
got
to
be
able
to
deliver
benefit
to
developers.
You
know
the
developers
want
to
be
able
to
know.
What's
going
on
one
of
the
advantages
we
found
with
Cystic
is
we
can
scope
that
data
so
Cystic
has
the
ability
to
understand
Prometheus
export
points,
grab
them
import
them,
and
so
you
can
effectively.
A
We
affect,
can
effectively
use
it
to
add
scope
and
role
based
access
control
to
Prometheus
data,
and
it
also
now
has
a
profile
our
exporter,
so
you
can
even
just
use
it
as
like
a
role
based
Prometheus
plug-in
between
the
two.
So
that
was
actually
one
of
the
other
reasons
we
didn't
go
there.
Prometheus
root
was
because
literally
Prometheus
really
is
is
fantastic
if
you're
a
cluster
administrator,
but
I
also
wanted
to
be
able
to
provide
useful
data
to
the
application
developers
and
the
application
support.
A
That
means
people
who
will
need
to
see
maybe
three
or
four
namespaces.
You
know
maybe
it
doesn't,
but
they
don't
need
to
see
the
cluster.
They
don't
need
to
see
the
queue
bonitas
events,
but
they
do
care
about
exactly
how
much
you
know
what's
what's
going
on,
they
care
about
their
application,
topology
they
care
about
the
eventing,
that's
happening
so
again,
depending
on
your
use
case
and
your
audience.
A
Well,
we
have
a
web
pub
that
talks
to
an
application
pub
the
tools
to
a
database,
pod
and
they're
all
in
the
same
project.
There's
nothing
in
the
normal
VX
land
security.
There
are
things
you
can
do
in
network
policy
now
to
be
fair.
We
didn't
have
network
policy
when
we
put
all
this
in,
but
there's
nothing
at
that
point.
That
would
have
stopped
the
web
pub
being
able
to
reach
around
and
talk
to
the
database
pod.
We
did
look
at.
Could
we
break
these
up
into
separate
projects?
A
You
start
doing
that
you
can
do
it,
but
it
does
start
getting
clunky.
You
hit
scalability
problems,
so
what
we
started
looking
at
was
alternatives
that
would
give
us
effectively
a
software
firewall
that
ran
inside
of
the
cluster,
and
so
we've
been
working
with
new
vector
pretty
early
days
for
Bofors
and
I.
A
Think
for
them
and
they
do
a
layer,
437
firewall
that
actually
runs
on
all
the
nodes
and
it
runs
agnostically
of
whether
it's
on
bare
metal
on
pram
or
whether
it's
running
in
AWS
and
that's
been
really
useful
for
us,
because
it
means
that
we
can
look
at
both
traffic
flow,
but
we
can
also
look
at
doing
more
granular
enforcement
where
we
need
it
inside
of
particular
projects.
So
you
can
literally
say
I
need
my
front-end
pod
to
be
able
to
talk
to.
A
A
The
Sdn
itself
can
be
a
fairly
opaque
place
unless
you
start
getting
quite
advanced
and
plugging
in
and
doing
lots
of
tracing
and
things
this.
This
allowed
us
to
actually
have
something
sitting
there
looking
at
the
traffic
in
real
time
and
identifying
if
it
was
seeing
malformed
packets,
you
know
it's,
it's
almost
moving
to
a
almost
a
light
weight
in
line
laughs,
so
you
know
looking
for
sequel
injections
and
things
as
well.
It's
you
know
not
beyond
the
the
witch
of
possibility
that
actually
has
had
a
couple
of
benefits.
A
One
is
obviously
there's
the
security
element.
We
actually
identify
pretty
early
in
the
development
environment.
We
noticed
there
a
couple
of
applications.
We
continue
to
see
it
said
we're
generating
malformed
packets.
It
actually
turned
out,
it
was
bad
code,
but
we
were
able
to
leave
that
loop
back
for
the
development
team
and
say
you're
pod
over
here
is
not
behaving
the
way
you
think
it
is.
A
A
So
you
know
one
of
things
we've
we've
also
been
looking
at
is,
is
how
do
we
start
to
stitch
together
the
various
pods,
because
it's
alright
again
it
understanding
the
performs
of
the
cluster
and
the
performance
of
the
individual
pods,
but
you've
also
got
an
application
which
is
comprised
of
a
number
of
those
different
pods
and
probably
off
platform
resources.
Because
much
as
I'd
love
it
not.
Everything
is
running
an
open
shift.
A
Quite
yet
you
know
so
there's
going
to
be
databases
that
are
off
platform
web
services
that
you're
calling
you
know
there
are
going
to
be
things
like
that,
so
one
of
the
things
have
been
looking
at
is
you
know
how
start
identifying
those
flows,
understanding
where
the
bottlenecks
are
and
that's
where
you
do
need
to
have
some
form
of
APM
type
tool?
You
know,
there's
still
a
need
for
that
sort
of
thing
that
that
has
been
for
us.
We've
got
a
couple
of
different
tools.
We
sort
of
use
in
that
space.
A
This
dig
I
mentioned
before
that
allows
us
to
do
topology
and
traffic
views.
We've
also
been
working
with
dynaTrace
and
that
enables
us
to
put
together
some
behavioural
maps
of
the
actual
applications
and
those
also
are
invaluable
for
business
stakeholders.
You
know,
because
really
the
rest
of
stuff
we're
talking
about
tends
to
be
the
constituents.
Are
your
storage
teams,
new
development
teams,
the
application
support
teams,
the
APM
view,
that's
the
bit
that
really
gets
the
product
owners
and
the
business
excited,
because
that's
the
piece
where
they
can
actually
contextualise
it.
A
You
know
a
transaction
for
me
is
X
sense.
I
can
understand.
A
drop
transaction
has
a
revenue
impact.
I
can
understand
that
you
know
these
bottlenecks
have
a
particular
financial
impact
and
we
probably
need
to
pay
for
more
resource.
It
really
helps
with
those
sorts
of
conversations.
So
you
know
it's
it's
something
that
for
us
again,
we
we've
we've
been
working
on.
Building
that
capability
in
there
are
a
few
different,
different
application
performance
monitoring
tools
that
can
run
inside
of
sort
of
queue,
benitez
an
open
shift.
A
Now
your
mileage
may
vary
as
to
which
one
you
pick.
One
of
the
things
we
we
tried
to
do
was
to
try
and
avoid
having
the
application
developers
having
to
make
code
changes
to
put
their
product
under
APM,
so
we've
tended
to
look
towards
applications
that
run
privileged
pods
as
a
daemon
set.
So
we
have
a
single
demon
set
APM
that
basically
scans
the
worker
node
and
does
all
of
the
APM.
A
There
are
other
prod
loader
a
sidecar
container
in
that
holds
your
monitoring
agent
and
runs
a
monitoring
agent
inside
the
pod
as
a
separate
container
image.
Some
of
it
depends
on
how
you
prefer
to
set
your
cluster
up.
You
know
we
one
of
the
things
we
did
early
doors
was
said.
We
want
to
try
and
look
at
how
do
we
do
this
to
be
the
most
benefit
and
ease
of
use
of
the
developers,
so
we
don't
want
to
say
to
the
developers.
Congratulations
you've
just
gone
through
the
new
project.
Wizard
and
OpenShift.
A
You've
just
got
everything
configured
now,
what
you
might
just
exporting
you
Yambol
and
then
adding
sike
our
container
to
and
then
deleting
it
and
then
running
the
Sochi
command
to
re-import
it
and
then,
by
the
way,
you'll
have
a
p.m.
because
there's
a
error
related
process
in
there
and
granted,
we
could
have
put
some
stuff
with
the
announcement
error
or
or
something
around
it,
but
you
know
I
think
the
the
metric
we've
used
would
be.
What
would
you
get
if
you
go
into
a
cloud
provider?
You
would
get
this
stuff
that
magically
just
works.
A
So
the
other
thing
to
think
about
is
monitoring
your
kernel
behavior.
So
this
is
again.
This
is
more
the
Cystic
secure
piece.
Actually
so
we've
used
Cystic
in
this
space
containers
offer
an
ability
to
introspect
the
way
your
system
is
behaving
in
a
way
that
you
would
never
be
able
to
do
with
virtual
machines.
So
at
the
moment
I
think
development,
environment.
Last
time,
I
checked,
we
were
running
over
3,000
containers,
and
that
was
first
thing
on
the
Monday
morning
for
anyone
who
started
scaling
stuff
up.
A
If
I
was
trying
to
manage
the
behavior
of
3000
VMs,
that
would
be
3000
agents.
I've
got
to
deploy
and
I've
got
to
manage,
I've
got
to
upgrade
with
cuban
ets.
I've
got
25
worker
nodes,
we
run
big
nodes
and
that's
25
pages.
I've
got
to
maintain
the
support
and
it's
a
lot
easier
in
the
daemon
set
concept.
Makes
that
really
really
simple
for
me
to
do.
A
A
What
is
it
actually
doing
at
a
kernel
level
system
call,
so
your
security
team
can
have
a
list
of
basically
every
command,
that's
being
run
in
every
shell
in
every
container,
so
what's
being
executed
as
a
runtime,
but
you
can
also
starts
at
behavioral
policies,
so
you
can
say
things
like
if
I'm
in
a
production,
environment
and
I
see
something
installing
up
into
the
rpm
database
kill
the
pod,
because
nothing
should
be
installing
an
RPM
inside
of
production.
You
know,
if
you're
in
development,
environment,
great
that's
where
you
build
your
container
at
least
for
us.
A
No
one
builds
in
production.
You
know,
you'll
your
development
image
moves
from
development
into
production
and
you
do
not
rebuild
it
there.
So
you
know
if
we
see
something
trying
to
install
an
RPM
or
something
trying
to
write
to
you
know
files
under
etcd.
Sorry
under
each
et
Cie,
for
example,
we
probably
want
to
terminate
that
pot,
and
you
can
start
to
think
about
and
set
some
some
policies
with
with
tools
like
this
in
a
way
that
is
you
just
wouldn't?
A
Theoretically,
you
could
have
done
it
with
virtual
machines,
but
the
the
pain
of
trying
to
manage
that
many
thousands
of
images
just
meant
it
with
the
problem
of
scale
meant
you
would
never
ever
countenance
doing
it
because
you
just
look
at
it
and
go.
This
is
insane
I've
got
too
many
other
problems
with
my
other
agents
and
then
the
other
thing
for
ROS
is
looking
at
cluster
activity,
so
cluster
activity
there
are
robust
access.
Control
is
really
really
useful.
It's
it's
it's
very
deep.
A
It's
very
good
and
it's
something
I'm
really
glad,
but
open
she's
contributed
back
to
cuban
ets
and
communities
are
sort
of
standardized
on
the
the
role
based
model.
There
are
things,
though,
that,
where
we
found
that
role
based
access,
we'd
prefer
it
doesn't
quite
go
far
enough
yet
for
us,
so
we're
supplementing
it
with
tower.
So
an
example
for
us
would
be
on
our
excellently
facing
production
clusters.
A
If
I
have
a
developer,
who
is
just
provided
an
application
they're
just
test,
they
want
to
test
detailed
I
want
them
to
be
able
to
create
a
route
so
a
URL.
For
those
event,
no
Egypt
is
a
URL
that
basically
applies
to
about
a
particular
application.
Endpoint
I
want
them
to
be
able
to
publish
that
URL
and
as
long
as
that,
urls
internal,
that's
okay,
because
that
can
only
be
accessed
internally.
They
can.
A
They
can
test
the
application,
QA
a
sign-off,
and
it
make
sure
it's
good
I
do
not
want
them
to
be
able
to
publish
an
externally
facing
URL,
because
that
is
something
that
has
brand
reputation
will
impact
that's
got
to
go
through
QA.
It's
got
to
go
through
pen
testing,
there's
a
whole
other
process.
We
have
to
wrap
around.
All
of
that
which
comes
back
to
again
my
point
earlier
about
information,
security
policies
and
best
practice.
A
So
at
the
moment
the
role
based
access
control
doesn't
make
it
well
really
possible
to
be
honest
to
to
differentiate
between
the
two
URL
base
points
and
be
able
to
say,
you're
allowed
to
create
one
type
of
route,
but
not
another.
So
this
is
where
what
we've
done
is
used
ansible
tower
to
wrap
ansible
around
some
of
those
role-based
access,
control,
type
limits
and
we've
basically
written
play
books
that
we've
then
published
as
REST
API
to
the
to
the
developers.
So
what
we've
said
to
them
is:
ok
we'll
do
that
on
your
behalf.
A
You
just
call
this
REST
API.
So
if
it's
an
internal
URL
call,
this
REST
API
will
create
it.
For
you,
it's
fine.
If
it's
external,
you
can
call
this
URL,
and
then
we
will.
Basically,
it
will
automatically
go
through
the
approval
process
and
it
will
kick
things
off
and
I
actually
think
the
way
we're
going
to
do.
That
is
probably
that's
going
to
change
and
it
will
actually
move
underneath
our
service
now
and
we'll
go
through
a
release,
managed
service,
no
release
management
process,
and
that
will
then
call
answer
ball
tower.
A
But
tower
has
been
really
really
useful
for
us
for
that
to
be
able
to
to
take
the
team
skill
sets
in
ansible
and
to
be
very,
very
quickly
be
able
to
turn
those
into
REST
API
endpoints,
to
be
able
to
turn
stuff
around
and
and
enrich
what
we
we
offer
out
for
developers
and
do
it
in
a
way
that's
fairly
scalable.
You
know,
because
for
stuff
that
we
literally
will
just
do
on
their
behalf,
it's
fantastic
because
you
can
literally
just
say
you
know,
call
this!
You
or
you
call
this.
A
This
URL
call
this
rest,
endpoint
and
we'll
we'll
deal
with
it
for
you.
So
another
example
format
froze
might
be
if
I
have
two
projects
and
I
want
to
join
them
together
in
our
development
environment.
They
don't
need
to
call
the
administrator
to
do
it
as
long
as
they
own
both
projects.
We've
got
some
ansible
code
that
basically
checks
their
own
project,
a
and
their
own
project
B
and
we'll
and
join
them
for
them.
A
Because
that's
the
logical
step
we
would
go
through
as
administrators,
so
we've
codified
a
load
of
that
so
some
of
the
stuff
with
cluster
activity.
You
know
think
about
for
a
security
perspective,
where's
the
role
based
access
control,
taking
it
where
the
limitations.
Can
you
extend
it
again?
It
doesn't
have
to
be
answerable.
You
know
for
Rose.
Ansible
was
a
good
fit
because
we're
already
working
with
it
with
the
open
shift
installation.
But
it's
it's
worth
thinking
about.
You
know
the
the
activity
basis.
A
I
actually
think
it's
better
to
have
role-based
api's
that
the
operator
30.
So
if
I've
got
a
database
or
I've
got
a
message,
queue
and
I've
got
an
operator
that
manages
that
queue
I.
You
think
it's
probably
better
to
have
a
separate
command
line
that
that
calls
that
message
key
rather
than
giving
everybody
the
the
control
binary.
You
know.
A
That's
there
are
some
vendors
that
go
one
way,
some
that
go
the
other
for
me
personally,
I
think
it's
better
to
have
that
differentiation
of
tools,
because
if
you
do
ever
get
to
the
point
where
there
is
some
kind
of
exploit
or
some
kind
of
weakness
in
the
OSI
or
to
keep
control
binary,
then
and
that's
rolled
out
really
really
widely.
That's
a
big
patching
effort
to
deal
with,
and
also
you
know,
you're
giving
users
tools.
Then
they
don't
really
need
for
necessarily
for
their
role.
A
So
again,
you
know:
that's
some
of
what
we're
starting
to
think
about
with
operators
is
the
sort
of
activities
that
will
take
place
in
the
cluster
and,
what's
an
appropriate
way,
to
manage
that,
and
so
final
diagram
is
just
really
to
put
all
that
together.
So
you
know
we've
sort
of
think
about
your
image
provenance.
Your
registry
scanning
how
your
images
are
composited.
Obviously
you
know
you're
going
to
need
performance
and
monitoring.
Think
about
the
kernel
monitoring
there
is.
There
is
an
ability
to
do
stuff
in
there.
That
is
really
really
good.
A
These
days,
obviously,
the
the
CNI
the
network
space
think
about
what
you
do
there
again,
depending
on
what
your
current
network
security
looks
like
what
it
is
you
you
want
in
that
space.
There
are
different
tools,
different
ways
of
doing
it,
and
the
the
network
space
is
to
be
on
this
very,
very
pluggable.
What
we've
done
is
what
fits
our
security
needs,
but
you
know
that's
an
area
I
would
I
would
recommend
taking
a
real
look
out.
Obviously
you
know
the
behavioral
side
of
things,
application
performance
monitoring
again
for
those
that
have
used
it
already.
A
It
continues
to
be
a
really
really
useful
tool
and
then,
as
I
mentioned,
the
role
based
access
control.
This
is
a
cluster
activity.
Side
of
things
think
about
things.
You
can
do
there
to
to
limit
your
users
access,
but
also,
if
there
are
things
common
things
that
they
should
be
able
to
do.
How
can
you
do?
A
How
can
you
extend
it
to
grant
them
that
access
and
that
has
pretty
much
brought
us
in
with
a
few
minutes
left
I've
got
a
few
I
used
some
open
source
icons,
so
I've
properly
attributed
the
the
people
that
use
them
from
so
I've
got
time
for
four
questions
for
a
few
minutes.
If
anyone
has
any
questions,
hopefully
that
was
useful.
A
A
That's
been
an
interesting
one
because
we
did
think
at
one
point.
We
might
need
to
starts
regionally
segregating
that,
but
some
of
them
we
have
a
lot
of
globally
distributed
development
teams
and
what
we
started
to
get
was
feedback
from
the
developers
saying
actually
we're
willing
to
take
a
little
bit
of
latency,
because
we
can
all
work
together
in
a
single
environment.
So
for
us,
that's
turned
out
to
be
a
good
fit
beyond
that.
Then,
because
of
the
nature
of
the
data
that
we
use,
we
have
to
get
into
regional
clusters.
A
Basically,
so
we've
got
clusters
in
North
America
and
Brazil
the
UK
in
amia
Thank,
You,
brick,
sir
we've
got
asia-pacific
in
Australia.
It
began
because
of
some
of
the
differences
in
in
in
the
laws
down
there
and
we're
getting
ready
for
Africa
as
well.
So
we've
got
a
plan
over
the
next
year
to
finish
building
a
bunch
of
those
now
and
the
reason
we've
got
all
those
is
basically
down
to
the
the
data,
protection
legislation
or
particular
sometimes
client
contracts
around
the
requirements
for
data
protection,
with
the
the
datasets
that
we
hold
so
yeah.
A
A
For
cystic
for
cystic,
we
are
currently
using
the
SAS
offering
yeah
we,
we
went
through
a
we're
through
quite
a
lot
of
conversations
back
and
forth
with
them,
but
fundamentally,
at
the
end
of
the
day
we
decided
the
quite
honestly
it
was
just
another
kubernetes
cluster
to
manage
into
patch
in
to
maintain
and
sorry
red
hat
folks
in
the
room
here
comes
my
rant
again.
If
only
OpenShift
were
a
damn
sight
easier
to
patch
in
an
upgrade,
then
I
wouldn't
mind,
upgrading
and
managing
another
cluster,
but
it
isn't
it's
too
painful.
A
So
right
so
the
way
we
do
our
clusters
at
the
moment
is
we
have
so
we're
we're
doing.
If
we
do
something
to
say
WS
specific,
we
have
architected
across
three
availability
zones
in
a
region,
so
the
control
plane
is
separated
across
three
and
the
workers
are
separated
across
three,
the
intent
being
that
yeah
we're
we
to
lose
a
zone.
We
can
work
without
that
and
we
can
reconstitute
using
ansible
the
the
missing
nodes
in
a
and
the
availability
zone
for
the
on-prem
clusters.
The
way
we
have
those
configured
is
we
use.
A
We
have
two
data
centers
and
we're
fortunate.
We've
got
private,
dark
fiber
connecting
them
and
they're
fairly
close,
so
the
latency
is
very,
very
low.
So
for
those
we
have,
the
control
plane
runs
across
the
two
data
centers
and
we
have
worker
nodes
in
the
data
centers
and
also
in
three
AWS
availability
zones
and
the
reason
we've
done.
That
is
because,
at
some
point
those
data
centers
over
the
next
10
15
years
will
probably
start
to
go
away.
A
So
we
can
then
start
to
decommission
the
date,
sent,
remove
the
control
plane
out
into
AWS,
and
so
we
sort
of
planned
for
the
the
future
from
an
architecture
perspective
as
well,
but
everything
we.
Those
are
stretched
clusters
at
the
moment.
Ultimately
now
for
I
mean
the
the
communities
Federation
has
started
moving
again
with
version
2
of
the
spec,
so
hopefully
we'll
get
to
a
point
where
we
can
do
more
about
having
contained
clusters
with
Federation
between
them.