►
From YouTube: Evolving Dev and Ops with Kubernetes: Enterprise Adoption for Security, Speed, and Ease
Description
Evolving Dev and Ops with Kubernetes: Enterprise Adoption for Security, Speed, and Ease - Drew Rapenchuk & Pierre Yanni, Bloomberg
Most large enterprise companies have vastly different hurdles to hop over in comparison to small startups and independent tech firms. Bloomberg’s 5 person WebOps team is tasked with performing devops tasks at an ever growing scale- see how we are changing our mentality and leveraging the power of Kubernetes to pave the way for the future of web at Bloomberg. We’ll dive into some key choices we made, and take a look at the tools we are using to improve development and deployment speed while improving security and enforcing standards.
About Drew Rapenchuk
Web Platform Engineer, Bloomberg LP
About Pierre Yanni
Senior DevOps Engineer, Bloomberg LP
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Welcome
to
my
presentation:
I'm
drew
rappin
Chuck
I'm
from
Bloomberg.
Bloomberg
is
a
company
where
a
Data
Services
software
companies,
so
essentially
the
bulk
of
what
we
do
is
we're
taking
publicly
available
data
and
we
transform
it
into
a
product
that
other
people
consume.
Essentially,
our
team,
specifically,
we
do
web
operations
where
the
web,
ops
group
and
specifically
the
web
operations
for
Bloomberg,
is
a
very
different
thing
from
terminal
services.
Whereas
terminal
is
our
core
product.
A
A
So
the
reason
why
we
kind
of
went
our
own
direction
is
because
we
have
unique
needs
in
comparison
to
everyone
else
within
the
company,
so
the
web
ops
team
in
Bloomberg,
we
create
tools
and
adopt
technologies
to
improve
if
interestin
c
for
the
web
teams,
we
manage
software
packages,
we
create
methods
for
deploying
new
software
and
we
create
the
tools
that
are
required
for
software
teams
to
be
able
to
get
their
applications
out
to
the
consumers
so
generally
we're
not
directly
managing
historically
the
deployment
of
the
applications
themselves.
We
create
the
packages
for
the
frameworks.
A
We
create
the
standards
for
deploying
their
configurations,
but
again
these
teams
like
to
maintain
control
over
when
they're
deploying
and
how
they're
deploying
and
why
they're,
deploying
because
we
have
multiple
teams
underneath
us
that
all
have
various
needs.
They
all
do
different.
They
all
use
different
programming
languages.
So
again,
our
team
really
focuses
a
lot
on
setting
standards
and
coming
up
with
best
practices
and
trying
to
help
these
development
teams
reach
their
goals.
A
So
what's
the
problem
with
that
again
kind
of
how
will
you
went
into
kind
of
how
I
describe
that
we
create
these
methods
and
we
create
standards?
A
lot
of
time
is
spent
on
communicating
with
the
teams
we're
trying
to
describe
to
them
how
they
should
go
about
doing
things,
we're
not
necessarily
setting
things
in
stone,
but
the
issue
with
that
is
that
we
spend
a
lot
of
time
on
training
them
a
lot
of
time
on
education.
A
A
big
part
of
that
problem
is
when
people
move
around
within
teams
within
our
development
teams,
underneath
us
that
it
requires
various
developers
to
relearn
methodologies
specific
for
their
team.
There's
a
lot
of
effort
into
changing
the
standards.
When
someone
spends
all
that
time,
trying
to
learn
a
new
process
learn
a
new
method.
A
They
don't
want
to
relearn
something
again:
I'd,
never
want
to
relearn
something
when
I
know
that
the
process
I'm
doing
words
unless
I
absolutely
have
to
so
it
ends
up
being
a
huge
burden
on
the
developers
themselves
and
that
really
ultimately
kind
of
leads
to
a
poor
developer
experience
and
that's
one
of
our
jobs
is
to
create
a
developer
experience.
That's
going
to
be
good
for
our
developers.
A
We
also
spend
a
lot
of
time
on
keeping
the
lights
on.
We
do
a
lot
of
effort
on
maintaining
log
rotations.
We
do
a
lot
of
effort
on
helping
to
troubleshoot
issues
with
old
legacy
deployment
mechanisms,
and
we
spend
a
lot
of
time
revamping
old
solutions
that
probably
aren't
going
to
be
sustainable
in
the
future
and
another
problem
we
run
into-
and
this
is
kind
of
a
recurring
issue
with
us-
is
we
have
very
little
infrastructure
automation.
A
A
lot
of
stuff
that
we
have
historically
is
still
very
ticket
driven
ticket
based
a
lot
of
its
manual
operation.
So
we
can
kind
of
create
some
tools
that
can
interact
with
the
infrastructure
in
the
somewhat
automated
way,
but
again,
a
lot
of
the
things
like
firewall
rules
and
stuff
like
that
are
still
relegated
to
another
team
that
has
to
go
into
a
system
somewhere
and
plug
something
in
or
at
least
we
have
to
wait
for
approval
at
some
point,
so
it
makes
automation
of
our
system
really
difficult
at
a
lot
of
times.
A
So
what
do
we
want?
What's
the
main
overarching
theme
for
you
know
moving
forward
with
our
products.
We
want
the
developers
to
have
a
better
experience.
We
want
them
to
be
able
to
deploy
their
application,
how
they
want
to
when
they
want
to
in
the
method
that
they
want
to
in
a
simple
way,
the
developers
want
to
only
focus
on
their
apps
and
not
the
infrastructure
again.
A
We
as
a
team
one
better
control
of
the
chaos,
so
we
want
to
really
have
a
lot
more
of
a
standardized
method
for
deploying
these
infrastructure
services.
We
want
to
set
up
the
servers
and
we
want
to
kind
of
take
more
control
away
from
the
developers
where
they
don't
necessarily
need
or
want
that
control
a
lot
of
developers
don't
want
to
mess
with
the
infrastructure.
A
A
So
what
specifically,
do
we
want?
Most
of
what
we
want
is
easy
in
concept,
but
perhaps
it's
not
so
easy
in
implementation,
especially
in
our
infrastructure
setup,
where
there
isn't
a
lot
of
automation.
So
we
have
to
create
those
automation
systems
or
we
have
to
come
up
with
a
mechanism.
That's
going
to
work
around
the
automation
or
allow
us
to
take
control
of
automation,
so
either
we'll
virtualized
stuff
or
we
will,
you
know,
implement
a
framework,
that's
going
to
take
control
of
a
lot
of
those
functionalities,
so
we
want
improved
deployment
cadence
for
applications.
A
We
want
improved
availability,
improve
portability,
security,
tighter
control,
simplified
scaling
and
easier
management.
All
of
this
is
conceptually
easy
and
great,
but
in
implementation,
it's
not
necessarily
so
easy
and
the
biggest
thing
that
we're
driving
for
this
is
support
for
12
factor
applications
in
the
Bloomberg
web
world.
We're
really
trying
to
push
stateless
applications.
We
don't
want
to
necessarily
store
the
data
with
the
application.
A
So
what
technology
can
we
use?
And
what
can
it
get
us?
So
we
have
to
spend
a
lot
of
time
on
thinking
about
what
specific
technologies
we
want
to
use.
So
after
we
got
a
concept
of
what
the
solution
is,
that
we
want
we
kind
of
have
to
play
around
with
the
specific
technologies.
Do
we
want
to
use
Linux
containers?
What
are
we
going
to
do
for
logging
for
a
CI,
CD
workflow?
What
are
we
going
to
do?
A
That's
going
to
work
for
the
developers,
but
also
allow
us
to
retain
the
control
that
we
need,
what
security
mechanisms
are
we
going
to
implement,
because
we
also
want
to
improve
our
security
posture?
What
are
we
doing
for
monitoring?
What
are
we
doing
for
scheduling?
These
are
all
the
same
questions
for
the
entire
platform
moving
forward.
We
want
to
take
control
of
all
of
this
stuff
and
we
want
to
come
up
with
a
very
solid
mechanism.
A
That's
going
to
be
more
maintainable
for
us
moving
forward
and
it's
going
to
be
flexible
enough
that
we
don't
have
to
completely
redo
the
entire
system
at
some
point
in
the
near
future.
So
it's
going
to
be
worth
our
effort,
so
we
kind
of
have
to
sit
down
when
we
evaluate
multiple
technologies
for
each
one
of
these
things.
So
are
we
going
to
look
at
mezzos
we're
gonna,
look
at
swarm,
Nomad
kubernetes,
so
back
when
we
started
looking
at
these
technologies.
A
It
was
about
a
year
ago
now,
when
we
started
doing
POCs
and
we
kind
of
spend
a
lot
of
time,
throwing
back
and
forth.
What's
gonna
work
best
for
us
and
of
course
it
doesn't
matter
what
you
really
choose
in
the
concept
and
planning
and
design
phase
when
you
go
to
implement
it,
you're
most
likely
inevitably
going
to
run
into
some
sort
of
issue,
that's
going
to
lead
to
a
change
in
your
plans.
So
what
are
we
trying
to
prevent?
A
We
don't
want
vendor
lock-in
or
tool
lock-in,
so
we're
trying
to
stay
away
from
proprietary
tools,
we're
trying
to
stay
away
from
something
that's
going
to
make
it.
So
we
can
deploy
this
method
somewhere
else.
If
we
want
to
deploy
to
multiple
clouds,
we
can't
use
cloud
provider
specific
functionality
because
that
cloud
provider
specific
functionality
isn't
going
to
exist
on
our
internal
infrastructure,
for
example.
A
So
we
want
to
avoid
general
and
flexibility
again,
we
don't
want
specific
functionality,
that's
not
going
to
be
flexible
and
we
may
end
up
having
to
completely
replace
a
part
of
our
whole
framework
moving
forward
and
we
want
to
avoid
setting
ourselves
up
for
failure
again.
This
is
kind
of
common
sense,
but
when
you
see
the
you
know
shiny
surface
of
a
new
technology,
sometimes
you
don't
really
understand
what
the
possible
negative
impact
could
be
underneath
that
sometime
down
in
the
future.
So
it
requires
a
lot
of
testing
and
testing
and
testing
and
testing.
A
So
what
choices
did
we
make
and
why
so
four
containers
we
went
with
docker.
One
of
the
reasons
is
because
it's
been
around
for
a
while.
It's
already
understood
by
people
there's
lots
of
tools
in
support
around
the
community.
It's
stable
and
secure
if
use
the
right
provider
and
it
can
be
replaced
so
there's
a
potential
to
you,
something
like
rocket
or
other
container
runtimes,
where
you
can
replace
it.
A
Essentially,
drop-in
replacement
for
the
deployment
methods
and
you'll
still
be
able
to
retain
your
development
processes
on
the
inside,
so
the
developers
never
have
to
change
anything
on
their
side
as
well.
So
we
can
take
the
runtime.
We
can
get
rid
of
it
if
it
ends
up
being
too
much
of
a
burden
for
us
to
use
or
if
we
see
that
the
community
as
a
whole
is
shifting
a
different
direction.
A
Another
added
benefit
is
because
when
we
go
up
to
management-
and
we
say
we
want
to
start
doing,
containerized
applications
a
lot
of
times
management
doesn't
like
the
sound
of
that
it's
a
new
technology,
it's
kind
of
scary,
so
you
do
have
to
kind
of
you
know,
meet
at
a
happy
middle
ground
and
say
like
well
we're
going
to
use
docker
it's
something
that
it
has
the
established
name.
It's
an
understood.
A
Technology
that
other
companies
are
using
and
production
perhaps-
and
that
really
is
a
big
leveraging
factor
with
getting
management
and
security
on
board.
Is
they
want
that
fuzzy
warm
feeling
that
other
people
are
still
doing
this?
So
we
don't
necessarily.
We
want
to
jump
to
the
very,
very
cutting
edge
and
use
rocket,
because
that's
kind
of
setting
ourselves
up
for
failure
in
the
future.
Someone
may
not
like
that
in
the
management
change.
That's
because
it
really
is
improving.
So
for
logging
we
went
with
flew
in
D,
Y
flew
in
D.
A
It's
super
simple,
it's
super
flexible
and
it
doesn't
tie
us
to
any
sort
of
a
specific
back-end.
So
we
can
use
various
plugins
in
order
to
pump
it
to
Splunk.
We
can
plump
it
too
or
pump
it
to
logstash.
We
can
pump
it
to
s3,
so
we
can
dump
the
logs
where
every
will
want
to
if
we
need
to
in
the
future.
A
So
if
our
logging
back
and
that
we
have
provided-
or
we
created
changes
at
some
point,
we
don't
have
to
completely
remove
our
you
know
our
logging
mechanism
and
redo
it
we're
not
locking
ourselves
into
a
specific
process.
So
it's
a
simple
configuration
change.
Moving
forward
for
CI
CD
we're
using
Jenkins
a
lot
of
people,
don't
like
Jenkins
personally
I.
Think
it's
a
little
clunky,
but
one
of
the
advantages
is
it's
been
around
for
so
long
developers
are
familiar
with
it.
Management
mostly
understands
it.
A
It's
relatively
simple
to
use
and
allows
for
strict
security
controls
and
job
creation
and
drop
management
and
access
and
authorization
schemes
for
security
open
s
cap.
So
we
want
to
do
security
scanning.
We
want
to
make
sure
that
these
applications
that
we're
putting
into
production
don't
have
Varner
abilities
in
them.
A
They
don't
have
an
8c
v's
at
them
on
the
bottom
level
that
are
going
to
cause
us
to
potentially
be
more
chaotic
in
the
deployment
where
we're
going
to
have
to
remediate
problems
in
the
future,
we're
using
TLS
trusts
because
we
need
security
of
the
api's.
We
don't
want
everyone
accessing
our
api's
and
just
doing
whatever
and
part
of
the
problem
with
potentially
tokens
is.
Someone
could
theoretically
generate
a
token
depending
on
how
complex
it
is
and
still
access
your
API
s.
So
you
need
HTTPS
access.
A
A
We
want
to
maintain
all
of
the
access
to
our
registries,
to
the
api's,
to
everything,
because
when
you
start
distributing
secrets
and
keys
and
stuff
like
that
you're
getting
into
key
management,
that's
another
layer
of
management
that
you
have
to
deal
with
so
you're,
creating
more
work,
you're,
creating
more
Kelo
you're
going
to
have
to
redo
the
system
at
some
point
eventually,
so
everything
that
you
create
you're
going
to
have
to
redo.
At
some
point,
it's
inevitable:
that's
TechNet!
It's
how
it
happens.
A
So
a
lot
of
our
deployment
mechanism
is
Jenkins
based
its
shell
scripts
for
the
most
part
right
now,
it's
easy
to
read
it's
very
simple
and
we're
doing
relatively
basic
stuff,
but
we're
implementing
it
in
a
way.
That's
really
feature-rich
as
far
as
the
deployment
mechanism,
but
it's
secure
and
it's
also
easy
for
the
developers
to
use
so
for
metrics
kind
of
a
custom
dashboard
we're
using
heaps
term
Prometheus.
These
are
already
kind
of
understood
standards
for
container
metrics
in
the
world.
A
For
the
most
part,
we
have
a
little
bit
of
a
custom
dashboard,
so
we
can
display
the
metrics
that
are
going
to
be
useful
to
the
application
teams,
but
we
want
to
abstract
away
a
lot
of
the
fluff.
That's
not
useful
for
them.
They
don't
care
about
necessarily
like
how
much
memory
DNS
is
using
on
their
cluster,
because
why
would
they
care
they
want
to
know
what
their
application
is
doing?
We
theoretically
owned
the
infrastructure,
they
just
owned
the
applications,
so
we
care
about
DNS.
We
care
about
everything
they
don't.
A
Why
show
it
to
them
so
for
orchestration.
Of
course
we
went
with
kubernetes,
we
wouldn't
be
here.
If
we
did
it,
the
reason
why
we
went
with
that
is
because
the
biggest
reason
is
the
community
and
the
development
pace.
Every
few
weeks,
there's
a
new
minor
release,
that's
being
cut.
That
has
more
features.
If
you
need
help
with
something
you
can
always
reach
out
to
slack
channels
or
anything
like
that
or
the
github
issues
and
people
are
always
willing
to
help
you.
A
You
don't
necessarily
need
a
support
contract
with
someone
to
help
maintain
this
or
try
to
figure
out
vague
issues
with
some
weird
miss
type
in
the
code
somewhere.
There's
going
to
be
someone
somewhere
in
one
of
the
special
interest
groups
or
the
standard
user
groups,
or
something
that's
going
to
know,
this
problem
exists
and
why
it
exists
and
it's
going
to
get
fixed
pretty
fast.
A
So
it's
a
very
simple
to
use
tool
once
you
get
it
implemented,
configuration
and
implementation
itself,
maybe
not
so
much,
and
we
had
issues
with
that,
but
once
you
get
it
in
place
and
you
get
it
working,
it's
actually
a
lot
of
it's
kind
of
common
sense
from
a
top-level
perspective,
it's
simple
to
use
and
we
actually
find
it
very
supportable
moving
forward.
So
it's
also
got
self
cleaning
and
self
éand
healing
mechanisms
as
well.
A
So
we
no
longer
have
to
come
up
with
any
sort
of
logic
loop
or
something
like
that
to
fix
an
application
when
an
application
dies.
If
an
application
dives,
you
point
a
health
check
at
it.
Kubernetes
it's
going
to
see
that
it's
dead
and
it's
going
to
try
to
recreate
it
and
try
to
heal
it
itself.
So,
there's
a
lot
of
stuff,
that's
built
into
kubernetes!
Natively!
That's
going
to
try
to
fix
our
problems
for
us,
so
we're
kind
of
killing
two
birds
with
one
stone
there.
A
So
the
six
million
dollar
plan
we
have
the
technology
we
want
to
make
it
better
than
it
was
better
stronger,
faster.
We
hope
so,
as
with
every
implementation
story,
a
lot
of
people
will
tell
you
that,
oh
it
went
great.
You
know
when
you
go
up
to
talk
to
any
of
these
folks
that
are
giving
these
stories
here.
A
recurring
theme
is
everything
is
great
now,
but
this
this
this
and
this
you're
always
going
to
run
into
roadblocks
with
new
technology
and
that's
okay,
that's
understandable!
You
shouldn't
be
discouraged
by
running
into
issues.
A
You
shouldn't
be
discouraged
by
having
to
look
at
documentation
or
maybe
even
having
to
take
a
peek
at
the
code
or
asking
other
people
in
the
community
for
help.
That's
normal,
but
there's
also
stuff.
That's
not
directly
tied
to
the
technology
itself
that
you
have
to
work
around
when
you're
in
a
large
enterprise
company
like
ours,
you
may
have
a
change
in
the
project
scope
you
may
have
to
abstract
terminology
like
we
are.
We
don't
want
our
developers
to
have
to
learn.
A
Kubernetes,
we
learn
kubernetes,
we
give
them
a
deployment
mechanism,
so
we
try
to
retain
a
lot
of
the
standards
and
the
terminology
that
they
already
understand
and
know.
The
only
difference
is
okay.
They'll
have
to
kind
of
learn
what
containers
are
and
how
containers
word,
but
for
the
most
part,
all
of
the
specific
kubernetes
things
such
as
pods
and
stuff,
like
that
they
don't
have
to
learn
that
stuff
we're
going
to
abstract
all
of
that
away
from
them.
We
also
have
to
satisfy
the
security
team.
So
again,
a
big
part
of
that
is
unknowns.
A
Before
we
came
along
and
started
deploying
applications
on
containers.
It
was
very
much
so
untested
by
most
people
in
the
company
to
deploy
containerized
applications.
So
this
whole
thing
was
not
even
just
implementing
a
new
orchestration
framework
that
is
kubernetes,
but
we're
also
adopting
a
new
technology
which
is
Linux
containers.
So
it's
a
lot
of
big
changes
to
a
lot
of
things
for
security
teams
and
management
that
they
don't
necessarily
understand.
A
A
So
again,
we
had
to
spend
a
lot
of
time
working
with
the
teams
that
are
going
to
be
deploying
applications
on
this.
So
we
had
a
proof-of-concept
application
where
we
plan
for
release
and
we
had
to
work
hand-in-hand
with
them
so
they're
comfortable
with
the
deployment
mechanism
and
we're
not
making
it
too
chaotic
for
them
to
adopt
this
workflow
it
if
it's
too
hard
for
them
to
adopt
the
workflow
they're
not
going
to
want
to
use
it
and,
like
I
said,
a
lot
of
effort
is
changing
people's
minds.
A
So
how
did
we
make
it
look
good
in
the
end?
That's
the
key
point
with
a
lot
of
these
presentations
is:
how
did
you
make
it
happen?
So
there's
a
little
bit
of
magic
going
on,
but
it
took
a
lot
of
patience.
It
wasn't
exactly
a
wool
star
playing
with
kubernetes
this
week
and
we're
deploying
applications
to
production
next
week.
No,
that's
definitely
not
what
happened.
A
We
had
original
plans
to
deploy
to
public
cloud
and
then
we
kind
of
changed
in
scope,
so
we're
not
deploying
to
public
cloud
right
now,
but
we
need
to
maintain
that
ability
to
possibly
deploy
the
public
cloud
in
the
future.
So
we
had
to
shift
infrastructure.
We
had
to
change
a
lot
of
stuff
at
a
high
level,
but
we
also
spent
a
lot
of
time
on
education
and
a
lot
of
time
on
tiny
little
small.
A
A
A
We
don't
necessarily
trust
the
images
that
are
there
not
saying
that
everyone
is
untrustworthy,
but
we
have
a
duty
to
our
company
to
make
sure
that
our
applications
are
built
a
very
specific
way,
so
we're
security
scanning
our
containers
were
creating
base
containers
for
the
application
teams
to
use
and
we're
also
making
sure
to
the
security
teams
and
to
the
application
teams
that
these
are
meeting
a
standard
for
deployment
and
again
lots
of
testing.
So
what
do
the
clusters?
Look
like
nothing,
amazing
right,
so
we
have
application
load
balancers.
A
Essentially
we
have
an
internal
load
balancer
for
managing
the
api's.
We
have
workers,
we
have
nodes
and
we
deploy
to
multiple
tiers,
devestating
and
prod.
It's
a
very
understandable
setup.
This
is
what
most
people's
setups
are
going
to.
Look
like.
We
do
that
per
data
center
per
team,
but
why
do
we
do
that?
So
essentially,
we
kind
of
committed
ourselves
to
maintaining
and
managing
anywhere
from
dozens
to
possibly
hundreds
of
small
clusters
across
our
network
in
order
to
deploy
these
applications
and
we
wanted
to
retain
a
physical
separation
of
workloads.
A
Like
I
said
before
we
have
a
lot
of
teams
that
have
many
different
requirements.
They
have
many
different
languages
that
they're
implementing
business
doesn't
one
application
foo
with
application
bar
if
application
bar
crashes
that
might
potentially
cause
problems
to
the
application
foo.
So
we
need
to
provide
some
sort
of
a
separation
that
we
didn't
feel
like
that.
A
Namespaces
alone
would
necessarily
provide
that
so
we're
kind
of
foregoing
namespaces
and
we're
using
physical
kubernetes
clusters
as
our
namespaces,
but
we
found
out
that
it's
really
not
that
difficult
to
manage
when
we're
just
managing
the
deployment
against
those
clusters.
So
the
teams
don't
have
to
remember
these
end
points.
We
have
a
file
that
essentially
says
like
coop
config
file
that
says
here's
our
endpoints.
These
are
the
clusters
we
deploy
to
on
our
CI
CD
pipeline.
So
the
developers
never
have
to
learn
anything
new.
A
They
just
say:
I
want
to
deploy
to
dev
I
want
to
deploy
to
staging
I
want
to
deploy
to
prod
and
on
our
CI
CD
pipeline.
We
take
care
of
everything
on
the
backend.
We
do
a
lot
of
resolution.
We
make
sure
this
team
is
deploying
to
their
cluster,
so
we're
not
doing
cross
cluster
applications,
necessarily
we're
not
doing
cross
team
applications
or
something
like
that.
Every
team
is
sort
of
siloed
in
a
certain
way,
so
they're
managing
their
applications,
and
they
want
that
segregation.
A
So
what
does
our
CI
CD
pipeline?
Look
like
again.
Nothing
is
too
crazy
about
this.
It's
all
Jenkins,
driven
the
developers
pull
their
code
from
github.
They
pull
our
created,
approved,
base
image
from
our
image
registry,
and
then
they
build
their
application
within
that
container.
Nothing
incredible
there.
The
next
step
that
they're
going
to
do
is
they're
going
to
do
a
security
compliance
scan
an
image
push.
So
we
maintain
the
second
registry
that
only
has
approved
images
for
deployment
on
our
side,
so
we
again
security
scanned
it
after
they
put
their
code
into
it.
A
So
we
security
scan
the
base
image
they
put
their
code
into
it.
We
security
scan
their
application
image
and
once
it
meets
our
requirements,
we
verify
that
it
comes
from
at
one
of
our
base
images.
We
verify
that
it
is
secure,
it's
not
vulnerable
to
any
CVEs.
We
do
a
promotion
step
and
we
essentially
push
it
to
our
registry
that
nobody,
but
us
has
access
to.
So
not
even
the
developers
have
access
to
this
registry,
so
we're
segregating
our
images
into
its
own
secure
registry
that
nobody
else
can
touch
except
from
the
web.
Ops
team.
A
After
that,
they
can
simply
trigger
the
deployment
to
dev
staging
in
prod
we're
doing
a
little
bit
of
logic.
So
we're
making
sure
that,
in
order
to
deploy
your
new
application
image
to
production,
it
does
a
check
to
make
sure
that
it's
already
deployed
to
staging
so
we're
kind
of
providing
a
little
bit
of
an
assurance
that
there
were
some
semblance
of
testing
they're
still
in
control
of
their
testing
mechanisms,
because
we
don't
know
specifically
what
their
application
is
doing.
There's
many
applications
that
get
deployed
in
the
bloomberg
web
world.
A
We
don't
know
all
of
the
testing
criteria
forum
and
for
a
team
our
size,
it's
just
what
six
of
us
about.
We
can't
maintain
those
testing
mechanisms
in
a
way
that's
going
to
meet
everyone's
needs,
so
they're
using
different
languages.
They
have
different
types
of
applications
with
different
frameworks.
They
have
different
needs.
They
have
different
acceptance
criteria.
We
can't
control
everything,
so
they
do
their
testing.
We
give
them
a
method
to
deploy
to
dev.
A
They
would
trigger
their
testing
externally
to
our
CI
CD
pipeline
and
then
they
would
deploy
to
staging
they
can
test
again
and
then
they
deploy
to
production
so
we're
giving
an
assurance
that
wasn't
necessarily
there
before,
except
by
a
manager
saying
yes,
this
was
deployed,
or
this
was
testing,
or
this
is
a
good
deploy,
we're
at
least
forcing
them
to
deploy
to
staging
first.
So
they
know
that
it
was
deployed
somewhere
else
and
it
didn't
break
everything
when
they
deployed
it
test
dating
so
the
flow
the
COS
control,
their
builds.
A
We
created
Jenkins
jobs
essentially
and
allow
them
to
access
those
Jenkins
jobs
using
the
remote
job
API.
So
they
don't
necessarily
need
Jenkins
to
do
something.
They
can
pass
a
JSON
blob
to
the
Jenkins
API
that
we
control
that
just
dictates
their
deployment
parameters.
So
we
give
them
the
ability
to
upload
secrets
that
are
going
to
be
loaded
as
environment
variables
in
their
application.
We
give
them
the
ability
to
control
the
scaling
if
they
want
to.
A
We
give
them
the
ability
to
take
advantage
of
a
lot
of
the
functionality
that
kubernetes
has
underneath
if
they
want
to.
But
we
have
same
defaults
also
set
for
all
these
things,
so
if
they
aren't
specifying
like
memory
requirements
or
if
they're
not
specifying
health
check,
URLs
we're
also
setting
standards,
so
they
can
just
comply
with
those
standards
and
they
don't
have
to
worry
about
any
of
this.
A
So
some
of
our
standards
are
somewhat
optional
in
a
way,
but
we're
making
it
simple
for
them
to
implement
them,
so
they
want
to
implement
it
and
they
don't
have
to
worry
about
changing
their
methods.
If
one
of
the
developers
goes
to
another
team,
it's
a
unified
deployment
methodology,
but
not
necessarily
the
same
exact
deployment
pipeline.
Each
team
still
drives
their
cadence.
A
So
why
do
we
provide
the
pipeline
again?
We
need
that
control.
We
don't
want
the
developers
to
have
to
learn
new
tools
every
time
they
go
to
a
new
team.
We
don't
want
them
to
have
to
learn
old,
archaic
legacy,
deployment
mechanisms.
We
want
to
provide
them,
something
that's
feature-rich,
easy
for
them
to
use
and
overall
results
in
a
better
developer
experience
for
them.
So
that's
one
of
the
greatest
reasons
that
we
worked
hand-in-hand
with
that
proof
of
concept
team.
A
They
created
that
application
and
every
time
they
made
a
code
change,
we
essentially
held
their
hand
and
we
wanted
feedback
and
we
required
feedback
from
them
about.
How
was
it
did
it
go
good?
Did
it
meet
all
of
your
expectations?
What
else
do
you
need
so,
every
time
they
did
a
deployment,
they
would
tell
you
oh
well,
maybe
I
want
to
also
do
this
or
I
want
to
think
about
doing
this.
A
A
Here,
Bloomberg
technology
re-release:
this
was
our
proof
of
concept
application.
This
right
here
that
you're
looking
at
right
now
is
on
kubernetes
its
containers.
It's
a
big
deal
for
us.
It
was
the
web
world's
first
application
deployed
to
containers.
It
was
deployed
on
a
new
framework
and
it
was
something
that
the
developers
loved
doing.
We
love
providing
that
platform,
and
this
is
essentially
the
future
of
the
web
world
at
Bloomberg.
A
But
again,
like
I
said
this
is
the
web
world,
so
there's
other
teams
such
as
the
DTP
team,
which
they're
in
another
Ballroom
in
the
next
session,
giving
their
presentation
about
what
they're
doing,
but
this
meets
our
needs
for
the
foreseeable
future.
But
again
we
needed
to
retain
that
flexibility
and
we're
always
going
to
be
requiring
that
feedback
loop
from
the
developers
is
this
working.
Is
this
ideal?
What
do
we
need
to
change?
But
again
we
retained
all
the
flexibility
from
kubernetes.
A
Sure
so
the
question
was:
what
were
the
trade-offs
that
we
had
to
make
in
order
to
implement
one
or
multiple
clusters
per
team?
And
you
know
like
he
mentioned:
you
do
kind
of
lose
a
little
bit
of
flexibility
and
resource
utilization
compactness
with
the
applications,
because
you
do
have
to
manage
scaling
across
multiple
infrastructure
clusters.
You
have
to
manage
application
scaling
at
some
some
level
between
many
applications
across
many
clusters.
A
Really
the
key
to
it
is
creating
a
mechanism
for
spinning
up
these
clusters
in
a
unified
method.
So
there
are
solutions
out
there
like
using
ansible,
we're
also
playing
with
using
salt
to
get
us,
maybe
a
little
bit
more
feature-rich,
orchestration
and
stuff
like
that,
because
we
couldn't
really
implement
the
the
open-source
kubernetes
deployment
mechanisms
that
are
available
right
now,
because
we
did
end
up
having
very
specific
needs
for
our
implementation.
So
we
did
kind
of
have
to
create
a
little
bit
of
our
own
infrastructure
orchestration.
A
But
again
we
also
benefit
from
having
a
platform
where
we
also
right
now
have
API
access
to
create
compute
instances
and
stuff,
like
that.
If
you
don't
have
access
to
those
creation,
api's
and
stuff
like
that,
like
if
you're
using
vmware-
and
you
don't
have
access
to
those
api's,
it
could
potentially
be
an
issue
and
you
might
lose
out
on
infrastructure
of
scaling
itself.
A
If
you
need
to
using
heaps
or
whatever
and
HPA's
so
you
can
still
scale
the
application
and
you
can
put
multiple
applications
on
the
same
set
of
tool
or
the
same
set
of
infrastructure
as
long
as
you
know,
they're
not
going
to
necessarily
clash
with
each
other.
Does
that
make
sense?
Ok,
any
other
questions
back
there.
A
So,
actually,
our
security
pipeline-
again,
it's
all
Jenkins
driven.
So
it's
essentially
when
the
developers
upload
their
image
to
our
public
internal
registry,
it's
internal
to
the
company,
but
everyone
has
access
to
it
when
they
upload
it
to
that
registry.
They
would
trigger
that
promotion
job
and
what
we
do
is
we
pull
the
image
from
the
registry.
We
use
open
s,
cap
docker,
to
stand
up
that
container
locally
and
it's
as
simple
as
dr.
oz,
cap
image,
CVE
and
then
the
image
name.
You
do
a
scan
and
we
collect
the
output.
A
A
Sha
look
up
where
we
look
at
the
hash
value
of
the
image
layers
themselves
and
we
have
an
inventory
of
our
based
provided
images
that
we
approve.
So
we
kind
of
bounce
those
two
against
each
other
to
make
sure
that
they
at
least
use
one
of
our
approved
images.
At
some
point
they
can
build
more
and
more
layers
on
top
of
it,
that's
fine
they're
going
to
if
they
manage
their
own
doctor
file,
but
we're
also
making
sure
that
at
some
point
they're
using
one
of
our
base
images.
A
So
it's
essentially
a
simple
collection
of
shell
scripts,
because
there's
already
command
line
clients
that
do
a
lot
of
that
functionality.
We
just
don't
want
to
do
it
manually
every
time,
so
essentially
they
hit
the
job
API
and
it's
just
a
pipeline
of
jobs
that
do
it
for
them
and
they
don't
even
see
what
it's
doing
on
the
backend.
It
just
happens.
A
So
the
question
was:
how
do
we
maintain
the
clusters
and
keep
them
updated
so
right
now
we
implemented
kubernetes
one
three
five
and
that's
kind
of
the
sweet
spot
for
us.
We
don't
really
necessarily
see
the
need
for
our
use
case
to
implement
further
versions
in
the
near
future,
yet
we're
going
in
an
immutable
infrastructure
route,
so
we're
kind
of
building
base
images
with
the
binaries
on
them
and
the
logging
daemons
on
them
and
stuff
like
that.
So
what
we
do
deploy
the
infrastructure,
it's
all
pretty
configured.
A
So
if
we
do
need
to
do
updates,
we
would
essentially
build
a
new
base
image
and
we
would
slowly
roll
that
out
replacing
members
within
the
cluster.
But
again
that's
going
to
be
after
we
do
stability
testing
and
we
do.
We
need
a
driving
reason
for
us
to
do
that,
because
that's
a
core
infrastructure
change
that
could
have
negative
impacts
on
a
lot
of
people.
So
if
we
don't
need
to
update
for
functionality
or
for
a
dire
bug,
fix
we're
not
going
to
so
we
see
it
as
being
something.
A
That's
we
do
relatively
rarely
does
that
make
sense,
but
generally
it
would
be
handled
at
the
base
image
level
where
we
would
delete
the
virtual
machine
deploy
a
new
one.
Kubernetes
is
going
to
worry
about
balancing
the
applications
within
the
cluster
itself.
We
would
just
kind
of
have
to
trigger
a
delete
and
then
recreate
of
the
infrastructure
make
sense.
Okay,
cool.
Any
more
questions.
A
That's
just
because
we
have
a
team
that
provides
that
as
a
service
inside
the
company,
our
registry,
that
we
use
that
segregated
is
kind
of
maintained
by
us
and
we
don't
know
really.
What's
the
future
of
that,
we
might
change
who
the
provider
is
moving
forward.
We
may
change
the
technology
if
the
problem
that
we
ran
into
is
artifactory
is
in
super
granular
with
the
permissioning
of
docker
containers
when
you
deploy
to
it.
A
So
if
someone
has
delete
access
to
a
container
on
that
registry,
they
can
delete
all
containers
on
the
registry,
so
there's
nothing
stopping
Team
a
from
overwriting
team
B's
container
technically.
So
that's
part
of
why
we
do
that
security
scan
and
that
verification.
So
we
can
almost
take
a
snapshot
of
that
container,
move
it
out
of
that
registry
that
it's
not
insecure,
but
it's
just
maybe
not
quite
as
stable,
and
we
put
it
into
our
registry,
so
we're
also
playing
with
registry
services
and
stuff.
A
Like
that,
we
may
outsource
that
to
someone
else,
we
haven't
really
decided
that
yet,
but
again,
that's
part
of
the
whole.
We
have
to
kind
of
stay
flexible,
because
this
is
still
new
technology
and
we
don't
want
to
lock
ourselves
into
one
solution
too
early
and
it
ends
up
coming
back
and
biting
us
in
the
butt.
A
We
manage
storage
for
that
using
local
storage,
but
we're
not
allowing
the
application
teams
to
utilize
that
storage
at
all,
because
that
starts
running
into
messiness
when
the
app
has
to
move
and
stuff
like
that.
So
it's
something
we
want
to
support
in
the
future,
but
right
now
kind
of
the
state
of
our
setup.
It
just
doesn't
really
allow
that
I
think
I'm
pretty
much
out
of
time.
One
more
real,
quick
question.
A
So
so,
actually,
we
had
very
often
meetings
just
about
weekly
with
our
security
folks,
where
we
showed
them.
These
are
the
configurations
we're
using.
These
are
our
concerns,
so
a
lot
of
the
people
within
our
team
are
within
our
team.
Web
ops
team
is
very
security
minded
regardless,
so
we
already
knew
a
lot
of
the
risks
and
we
kind
of
jumped
at
offering
a
solution
to
mitigate
those
risks
ahead
of
time.
So
we
can
say
that,
yes,
this
is
a
potential
risk,
but
this
is
how
we're
gonna
mitigate
it.