►
Description
ITNW (If This Now What) - Orchestrating an Enterprise - Michael Ward, Pearson
"With growing demand for containers in the enterprise, Pearson chose to lay its foundation with Kubernetes. With over 400 development teams across many different business units, the platform is absolutely critical. Pearson had to move away from traditional large enterprise infrastructure and aggressively pursue the efficiencies only containerization can provide. In this talk we'll demonstrate how we have built in security compliance, performance testing, quality assurance, abstracted away complexity, reduced overhead, aim to recover 10% of developers time, turned build tools into cattle, reduced deployment times and gain efficiencies in many other areas. Throughout the presentation we'll demonstrate all our current use cases with Kubernetes to include automation, tools used, management of external resources from within Kubernetes, SSL cert rotation and ChatOps integrations to name a few. We'll provide our latest Apache 2.0 code around Terraform deployment automation, our custom Jenkins plugin and if time allows our Hubot code as well.
This represents the story to date of an in-flight engineering project to modernize the digital estate of a global enterprise organization and how scale of the operation is leading us to challenge many established beliefs. Attendees will walk away with everything from workflows, to code, stories and an enterprise production use case which they can use to get started in their own endeavors."
About Michael Ward
Michael Ward is the Principal Systems Architect at Pearson responsible for leading technical design around enterprise Platform-as-a-Service based on Kubernetes. Prior to Pearson, Michael has spent many years in the industry in various roles including Chief of Site Reliability at Ping Identity, the Identity Security company. Take him for a beer and pick his brains on anything you like. You might even come away with something valuable. No guarantees of cost:benefit ratio implied or guaranteed. @devoperandi
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
My
name's
Michael
Ward
I'm,
a
part
of
Pearson
technology,
we're
all
about
learning,
so
we
have
been
around
for
a
hundred
and
seventy
two
years.
We
have
40,000
employees,
and-
and
this
is
if
this
now,
what
orchestrating
an
enterprise?
What
we're
going
to
do
is
we're
going
to
talk
about.
How
did
we
take
Pearson,
who
is
a
really
old
company,
has
70
hard
data
centers
and
migrate
to
not
only
the
cloud
but
to
kubernetes
again
I'm
Michael,
Ward
I'm,
the
principal
architect
at
Pearson,
on
a
project
called
bite-sized.
A
Bite-Size
is
our
platform
as
a
service
based
entirely
around
kubernetes.
If
you've
never
read
my
blog
go
to
dev
operandi
comm,
we
talk
a
lot
about
everything.
We're
doing
there.
Our
failures,
our
mistakes,
our
successes,
everything
is
there,
feel
free
to
ask
questions,
feel
free
to
reach
out
Twitter
email.
A
A
Alright,
let's
start
out
with
some
quotes.
These
are
two
of
my
favorites
one
by
Thomas,
Edison
and
one
by
Edwin
land.
Anyone
know
ed
woodland.
Anybody,
Beeler,
Edward
land
is
the
guy
that
started
Polaroid
and
I
happen
to
like
photography,
and
he
learned
how
to
polarize
light,
which
is
why
the
term
Polaroid
came
along.
But
one
of
his
key
sayings
was
don't
undertake
a
project
unless
it's
manifestly
important
and
nearly
impossible.
I
truly
believe
that
that
is
important
and
the
reason
for
that
is
because
really
we
all
know
that
things
aren't
really
impossible.
It's
just.
A
Our
focus
is
on
education
and
really
not
just
for
educating
students
and
people
and
communities
that
have
money,
but
communities
that
don't
have
money
so
oftentimes
you'll
see
us
go
into
countries
or
areas
of
the
world
that
really
have
zero
money
and
zero
funding
in
order
to
provide
proper
education,
and
so
that
really
makes
our
dynamic
deployments
very
important
because
we
have
to
build
applications
that
aren't
necessarily
always
connected
to
the
Internet,
like
we
think
of
when
we
think
of
cloud
operations
and
cloud
applications
right.
Sometimes
these
companies
or
these
students,
these
schools.
A
They
may
only
have
internet
access
for
a
couple
hours
a
day,
so
really
important
that
that
we
hone
in
on
that,
because
that
changes,
the
whole
game
and
the
whole
gambit
about
how
we
work
and
how
we
deploy
systems
all
right.
So
we've
got
36,000
servers,
2,000
applications,
400
development
teams,
each
application,
the
ones
that
are
micro
services,
aren't
it
can
have
anywhere
between
3
and
30
micro
services
per
application
stacks.
So
when
I
talk
about
an
application,
I'm
actually
talking
about
an
entire
stack
so
that
2,000
applications
there
are
entire
application
stacks.
A
These
are
a
couple
of
the
projects
that
we've
worked
on
or
open
source
code,
for
we
created
an
Aussie
web
hook,
which
was
one
of
the
first
of
its
kind,
with
kubernetes
we're
working,
hopefully
to
get
that
in
to
contribute,
have
availability
to
it
stack
storm
is,
is
our
event-driven
automation
platform
stack
storm
itself
is
open
source
we've
added
to
it
too,
to
work
with
the
kubernetes
api
you're
actually
going
to
get
to
see
that
today,
pretty
exciting
and
kubernetes
test
kübra
is
test.
Is
all
about
testing
your
infrastructure
once
it's
come
up.
A
So
how
do
you
know
when
you
deployed
that
sucker
that
it
actually
came
up,
is
usable,
available
and
and
can
be
counted
on
to
move
forward?
So
I
spoke
at
Q
Cana
me
you
and
I
just
want
to
give
you
a
little
bit
of
a
status
update
now
I'm
going
to
talk
about
business
justification
today,
because
we
already
did
that
you
can
go.
Look
at
the
cube
con
video
check
it
out.
So
where
are
we
at?
A
There
were
a
bunch
of
requirements
that
we
had
when
we
decided
to
build
this
platform
as
a
service
on
kubernetes
geographically
distributed
mean
we
need
to
be
able
to
be
cloud
agnostic,
deploy
anywhere
multi-region.
We
need
to
have
cattle
jenkins.
Anybody
tried
to
upgrade
jenkins
before
muller
yeah
pain
in
the
ass
all
right,
and
we
want
it
to
be
fast.
We
want
to
have
automated
upgrades,
so
that's
automated
a
B
deployments
of
our
our
entire
infrastructure.
Obviously
you
want
to
automate
containers
as
well.
Our
objective
is
to
automate
our
infrastructure
upgrade
process
feedback
loops.
A
We
want
to
provide
clear
communication
across
our
teams.
We
want
them
to
know
because,
in
some
of
our
application,
stacks
we've
got
maybe
a
hundred
and
fifty
micro
services
in
an
application
or
multiple
across
an
umbrella
stack,
and
we
want
to
make
sure
that
that
when
someone
else
is
having
a
problem
hitting
your
API
that
you
know
about
it
right
scalability.
Obviously
we
need
to
be
able
to
scale.
A
We
need
to
be
able
to
reduce
costs
and-
and
we
wanted
to
have
automated-
deploys-
not
just
upgrades
so
far-
we're
seeing
about
a
45
percent
cost
reduction
by
moving
to
kubernetes
currently
and
our
our
sample
size
is
relatively
small,
but
currently
it's
about
45
percent.
That
does
not
include
databases.
We
do
not
have
other
than
a
couple
persistent
storage
devices.
We
do
not
have
databases
in
this
equation.
We're
currently
working
very
hard
on
that.
Our
last
performance
test
you
can
see
under
fast.
A
It
says
5
to
15%
faster
at
a
quarter
of
the
resource
consumption.
What
we
found
was
we
had
these
developers
and
they
would
pull
on
these
massive
AWS
machines
because
they
wanted
the
enhanced
Network
right,
the
only
they
wanted
a
fast
network,
and
they
could
only
get
that
if
they
built.
You
know
like
a
c4x
large,
so
when
they
moved
to
pass,
they
were
actually
able
to
reduce
their
application
sites.
A
Down
to
you
know
a
tenth
of
a
CPU
and
and
one
gig
of
ram,
maybe
especially
for
Java
machines
and-
and
they
were
actually
faster
than
they
were
in
their
AWS
environment-
pretty
exciting
stuff
I
honestly
was
hoping
it
would
be
fast,
but
I
would
sure
that
it
actually
would
be
Dinkins.
This
cattle,
so
Jenkins
in
our
environment,
can
go
away
and
give
him
a
point
in
time.
If
one
of
you
guys
asked
me
to
do
it,
I'll
be
glad
to
blow
it
away
during
this
demonstration.
Today,
all
right
standardization
same
build
process.
A
We've
got
400,
freaking
development
teams.
We
want
them
to
all,
have
a
consistent,
build
process.
So
when
they
migrate
between
teams,
they
recognize
the
same
sort
of
requirements:
compliance
security,
visibility,
ease
of
use
less
than
5%
time
management
required.
So
one
of
our
business
requirements
was
that
our
developers
had
to
spend
5%
or
less
of
their
time
with
deployment
pipelines
into
their
environments,
quality,
testing
infrastructure,
kubernetes,
kubernetes
tests,
I
mention
earlier,
we'll
go
into
it
more
demo,
I've
seen
out
it
heads,
but
it's
like
10
people
going
yeah
come
on.
Give
me
some
alright.
A
A
A
A
So,
as
you
can
see,
we
create
a
namespace,
we're
gonna,
add
some
labels
to
that
namespace.
So
we're
going
to
save
this
repository
off
locally
so
that
we
can
then
go
ahead
and
and
use
that
information
for
later
now
in
in
our
environment,
we
have
to
be
able
to
approve
namespaces,
because
we
allow
our
developers
to
go
out
and
create
these
namespaces
on
their
own,
but
we
don't
want
them
to
be
able
to
just
create
a
hundred
of
them
just
because
they
feel
like
it.
A
A
All
right,
hopefully,
I
proved
the
correct
namespace.
So
now
what
we're
going
to
do
is
we're
going
to
take
in
Jenkins
I
mean
we're
going
to
pull
in
a
set
of
template
files,
we're
going
to
deploy
Jenkins
into
the
namespace,
we're
going
to
delete
the
local
files
and
we're
gonna
set
up
some
quotas.
So
essentially
we
don't
really
care
much
about
you
know.
Let
me
make
this
bigger.
We
don't
care
much
about
quotas,
meaning
particular
to
CPU
and
RAM.
A
A
A
Cool
so
now
we've
got
Jenkins
running.
We
haven't
actually
deployed
anything
through
the
command
line.
All
we've
done
is
click
a
couple
buttons
at
this
point
now.
I
didn't
tell
you
this
earlier,
but
my
objective
for
this
is
to
actually
be
able
to
deploy
this
entire
application
in
under
10
minutes
and
hopefully
today
we'll
be
able
to
demonstrate
that.
A
All
right,
so
what
we're
going
to
do
is
we're
going
to
let
that
deploy
and
then
I'm
gonna
go
ahead
and
continue
on
with
some
of
our
information,
we'll
be
able
to
check
back
at
a
few
minutes
on
whether
or
not
everything
deployed
properly
and
if
we
actually
have
a
website
up
and
running.
So
what
do
we
just
do?
Well,
we
did
quite
a
bit
as
it
turns
out
stack
Storm.
We
hit
the
UI,
we
deployed
a
namespace
through
the
kubernetes
api.
A
We
deployed
Jenkins
Jenkins
then
connected
back
to
this
config
repository
for
a
set
of
what
we
call
bite-size
configuration
files.
These
configuration
files
are
just
the
animal
based
configuration
that
you
can
take
advantage
of,
and
it
automatically
gives
you
all
the
guidelines
you
need
to
deploy.
Your
application
Jenkins
then
reached
out
to
our
code,
repo,
where
our
site
is
living
our
test
repo
and
then
it's
going
to
create
a
series
of
slaves.
A
So
we
have
a
built
parallel
build
process,
meaning
Jenkins
is
not
just
one
massive
server
deploying
this
stuff,
Jenkins,
actually
a
spin-off
slaves
and
tell
these
slaves
to
do
stuff
for
it,
which
is
great,
then
what
it's
going
to
do
is
deploy
it,
deploy
our
site
and
run
a
series
of
tests
against
it
and
yeah.
Let's
get
into
details,
let's
see
some
code
alright,
but
before
we
talk
about
code,
let's
look
at
the
features
of
this
deployment
pipeline,
so
it's
yellow,
based
configuration.
We've
got
blue
green
deploys
in
place
its
beta
still
working
through
it.
A
It's
really
fantastic
because
our
developers
don't
have
to
worry
about
managing
this
stuff
right.
You
can
set
it
to
continuous
or
manual.
Obviously
we
want
to
get
to
a
continuous
deployment
environment,
but
we
do
allow
our
developers
to
do
this
in
a
manual
process
so
that
they're
able
to
test
this
or
test
out
their
deployment
process,
their
testing
process
and
what
before
they
it
allows
them
to
set
those
gates
until
they're,
truly
comfortable
with
a
continuous
deployment
scenario.
A
Common
build
tools
are
just
dependencies,
so
maven
or
Java
nodejs
building
of
those
applications.
It's
just
a
build
tool,
dependency
to
us.
We
allow
our
developers
to
specify
how
they
want
to
build
it
out.
We
don't
really
care
Dinkins,
it's
cattle.
We
can
deploy
Jenkins,
we
can
blow
it
away.
It'll
come
right
back
up
and
continue
on
its
business.
We've
got
versioning
build
once
run
anywhere,
which
is
popular
of
docker
docker
image
management.
So
it's
key
to
know
that
our
developers
don't
actually
have
to
know
how
to
deploy
a
docker
image.
A
A
Alright,
some
code
so
Jenkins
our
configuration
files,
there's
only
three
of
them:
environments,
top
bite-size
application,
dot,
bite-size
and
build
up
bite-size.
Now
I
can
pretty
much
guess.
You
already
know
what
these
things
are
going
to
do
right,
so
application
dot
bite-size
the
yamo
file
is
basically
going
to
specify
everything
that
is
needed,
nor
to
build
an
actual
application.
So
if
you
need
multiple
components
to
build
into
one
docker
image,
this
is
where
you
do
that.
A
If
you
need
to
specify
or
want
to
specify
versions,
you
can
do
that
here
and
then
you
can
specify
what
type
of
runtime,
if
you
want
so
base
image.
In
this
case,
it's
no
js'
4.23
every
project
and
how
we
kind
of
coagulate.
All
these
bite-sized
files
together
is
through
the
project
name.
So
in
this
case,
that's
head
console
and
then
we
establish
a
runtime
command
here
and
basically
all
that
saying
is
it's
the
same
thing
as
a
command
within
your.
A
If
you're
going
to
run
it
inside
of
a
docker
image,
all
we've
done,
it
is
abstracted
it
out
for
you
all
right
so
build
that
byte
size
or
build
that
yellow
again,
you
notice,
we've
got
the
same
project
name
there,
we're
specifying
OS
linux
up
here
in
the
future.
We
do
plan
to
offer
windows
containers
if
people
want
to
use
that,
but
we
don't
currently
have
that
capability.
We've
just
added
that
in
there
and
then
you
specify
some
core
dependencies.
A
These
dependencies
tell
Jenkins
everything
it
needs
to
have
installed
on
itself
in
order
to
build
these
applications.
So
this
is
entirely
flexible.
Gim
packages,
Debian
packages-
you
can
reach
out
to
your
own
specific
repo
for
a
build,
it's
very
flexible
in
the
way
you
do
that
all
you're
doing
is
specifying
the
dependencies
here,
which
is
great
because
once
it's
set
its
forget
Jenkins
if
it
bombs,
it
comes
back
up
reasons
what
it's
supposed
to
have
and
it
keeps
moving
forward.
We
then
specify
its
initial
repository
here.
A
So
that's
the
same
repository
that
I
put
inside
of
stack
storm
just
a
few
minutes
ago,
and
then
we
specify
some
build
commands
in
shell.
So
we're
saying:
okay,
we're
gonna
run
build
out.
Sh
we
run
F
p.m.
and
our
artifact
is
going
to
be
a
Deb
package
and
then
and
then,
of
course,
we
can
keep
on
doing
this-
continuing
to
add
other
piece
of
code
and
and
additional
applications.
But
oh,
no,
that's
really
not
that
much
code
to
be
able
to
continuously
deploy
your
application.
It's
important
to
note
once
this
is
done.
A
You
really
don't
have
to
touch
it
again.
We
offer
versioning
as
a
capability,
but
versioning
is
not
a
requirement.
We
can
do
versioning
for
you
as
well.
Alright,
so
environment
stop
bite-size,
as
you
might
guess,
all
the
environments
files
doing
is
saying:
okay,
I'm
gonna,
deploy
to
a
test
environment,
I'm
gonna,
execute
these
actions
I'm
going
to
deploy
these
containers
I'm
going
to
deploy
these
tests
inside
of
each
given
environment.
So,
as
you
can
see,
we've
got
a
concept
called
configuring
service
and
ingress.
A
There
we're
actually
doing
three
things
when
what
is
that
six
lines
of
code
we're
giving
it
a
service
name,
ie,
a
kubernetes
service,
name,
we're
telling
you
what
port
we're
going
to
expose
it
on
for
each
container,
we're
setting
some
environment
variables
and
we're
giving
it
this
URL.
So
if
you
didn't
know
this
automatically
configures
or
your
ingress
for
you,
so
all
you
don't
have
to
do
anything.
You
can
add
SSL
equals
true,
you
can
do
all
kinds
of
stuff
in
here.
A
This
is
just
really
a
basic
example,
all
right
so
tests
notice
that
we
have
this
test
window.
What
this
is
for
is
currently
our
tests
only
deploy
out
once
an
application
has
been
built
and
deployed,
so
these
are
kind
of
into
in
tests.
If
you
will
we're
currently
working
to
get
tests
inside
of
each
build
for
unit
testing
and
things
of
that
nature,
alright,
so
I've
been
talking
long
enough.
Let's
see
if
it
actually
worked,
keeping
in
mind.
I
do
have.
A
Caching
problem
all
right,
so
there's
Jenkins
you
guys
saw
have
been
talking
out
throughout
this
entire
time.
Jenkins
went
through
the
build
process.
It
created,
ran
a
seed
job
connected
to
the
repo.
We
have
a
thing
called
service
manage
here.
What
that
means
is,
if
you
do
benign
changes
like
you
want
to
scale
the
number
of
replicas
in
your
application.
All
you
have
to
do
is
set
it
and
it
doesn't
do
a
complete
redeploy.
A
All
it
does
is
as
managed
as
the
services
and
scales
it
up
for
you,
so
we've
then
ran
our
at
there
at
the
very
top,
and
what
that
does
is
that's
saying.
Okay,
these
are
the
actions
I
need
to
perform
to
build
this
website.
Then
we
went
ahead
and
built
the
docker
image.
We
ran
through
the
environments
pipeline
to
determine
which
environments
we're
going
to
deploy
to,
in
this
case
we're
only
deploying
to
production.
Then
you
see
something
called
doc
site
tests.
A
Those
are
the
tests
that
we
wanted
to
run
against
the
website
to
make
sure
it
was
up
and
running
right
all
right.
So
let's
look
at
those
tests
real
quick
now.
This
is
obviously
a
very,
very
simple
example,
but
we
just
want
to
make
sure
that
it's
not
going
to
404
out
on
us
when
it
runs
we're
saying.
Is
it
good
if
no
exit
one
that's
another
thing,
that's
important
about
our
build
pipeline
in
order
to
fail
a
given
build
within
the
process,
all
you
have
to
do
is
exit
something
other
than
zero
right.
A
A
A
A
A
So
it'll
check
against
our
repositories,
make
sure
nothing's
change
if
something
has
changed,
it'll
execute
that
I
also
mentioned
that
we
don't
deploy
databases
inside
of
kubernetes,
yet
we
do
have
some
statically
or
clustered
technologies
and
inside
of
kubernetes
console
as
an
example,
both
as
an
example
elasticsearch
as
an
example,
but
those
are
all
global
services
that
we
manage
in
a
slightly
different
way.
This
is
only
for
our
applications
that
or
for
our
development
teams,
are
actively
deploying
applications.
A
A
Now
we
to
our
developers
our
developers,
actually
don't
care
what
type
of
database
they
are,
what
database
they
get.
They
don't
care
if
their
databases
are
deployed
inside
of
virtual
machines
if
their
AWS
RDS,
if
their
elasticsearch
in
AWS
or,
if
they're
an
elastic
search
cluster
deployed
on
virtual
machines,
they
don't
care,
they
don't
have
to
know
about
that.
It's
all
distracted
for
them
all
they
have
to
do
is
say,
I
need
my
sequel
5.6
and
be
good
to
go.
Don't
fail
me
now!
Oh.
A
A
Databases
are
launching
notice
that
this
is
for
servers
right.
We
can
expand
that
out
as
large
as
we
want
as
small
as
we
want.
It's
all
entirely
dynamic
to
our
customers.
What's
really
really
cool
about
this?
Is
our
customers
don't
have
to
know
where
these
databases
exist?
They
don't
have
to
know
the
username
and
password
they
don't
have
to
know
anything.
All
they
need
to
know
is
the
environment
variables
that
are
stuck
in
their
containers
at
runtime
in
order
to
access
a
given
database.
A
That's
pretty
exciting,
because
now,
although
I
disagree
this
from
a
technical
direction,
because
I
have
a
database
background,
I've
heard
a
lot
of
people
say
this
is
database
as
a
service.
No,
it's
really
not
we're
not
providing
all
the
configuration
values
in
here.
Currently
we're
not
providing
a
lot
of
things
that
would
really
truly
equate
to
a
database
as
a
service,
but
it's
pretty
kick-ass
functionality.
A
A
Yes,
absolutely
so,
within
the
Dinkins
bill
pipeline,
you
can
specify
whether
or
not
you
want
to
run
databases
as
containers
or,
if
you
want
to
run
them
as
as
virtual
machines.
In
my
sequel's
case,
it's
just
RDS
at
the
moment,
but
it's
just
a
matter
of
you
having
a
container
image
that'll
build
these
databases
for
you
at
the
current
we
basically
to
build
and
develop
this
based
off
of
our
current
key
teams
demand,
and
so
we
work
from
that
standpoint.
But
we
got
a
cluster.
We
got
an
RDS
cluster.
A
We
got
a
full
application
stack
in
15
minutes.
Maybe
that's
not
too
shabby.
So,
let's
get
back
to
the
present.
Let's
see
how
it
works
all
right,
so
what
we
did
here
was
you
can
see
at
the
top
left
hand,
corner
cube
version.
2.6
jenkins
picked
that
up
called
out
to
the
api
created
a
third-party
resource.
Everybody
know
what
a
third-party
resources
I
hope
so
you're
in
the
advanced
class
right
now,
all
right
so
kicks
out
a
third-party
resource,
which
is
just
an
api
json
hook.
A
A
All
right,
so,
as
you
can
see,
stacks
tore
us
win
stack
storm
execute
against
a
third-party
resource.
It
also
we
we
execute
some
cloud
formation
type
templates
and
we
went
ahead
and
populated
the
instance
ip's
of
the
cluster
into
console,
so
the
applications
can
consume
that
data
at
runtime
through
environment
variables.
So
again
they
don't
have
to
care.
What
they
actually
are.
A
A
All
right
we're
officially
open
source
now
I
should
make
note
here,
as
you
guys
saw
in
the
demo
today,
we're
utilizing
stack
storm.
You
can
also
go
to
the
Pearson
Technology
github
page,
and
you
can
see
and
utilize
all
of
our
tools,
all
of
our
event-driven
automation
in
stock
storm,
to
combine
with
some
of
your
third-party
resources
in
order
to
do
a
lot
of
same
things,
you've
seen
today.
It's
also
know
we're
only
in
AWS.
At
the
moment.
Okay,
we
are
working
to
be
able
to
go
to
a
sure
to
Rackspace
to
wherever
the
hell.
A
We
want.
We've
already
done,
POCs
and
OpenStack
to
to
make
all
this
work.
So
that's
it
for
that,
but
I
wanted
to
make
sure
that
we
covered
a
couple
more
things.
What
would
be
that'd
be
kubernetes
tests.
I
talked
about
kubernetes
tests
a
little
bit
earlier
and
I
just
want
you
to
know
that
that
is
also
open
source
available
for
you
to
consume.
There's
already
people
taking
advantage
of
this,
even
hopefully
contributing
back,
and
then
let's
talk
about
our
web
hook.
A
So
what
we
do
with
our
kubernetes
tests
is
we
actually
run
Travis
CI
to
kick
off
a
build
on
every
PR
for
our
we
run
terraform
and
ansible
for
building
our
entire
infrastructure.
What's
really
cool
about
this?
Is
we
actually
deployed
two
different
platforms?
Paas
a
and
pass
B,
then
we
use
B
to
test
pass
a
so
every
all
of
our
tests
are
built
into
container.
Well,
not
all
of
them,
but
most
of
them
are
built
into
containers,
and
we
have.
This
also
allows
for
a
B
deployments
of
our
entire
infrastructure.
A
As
you
might
have
noted
earlier,
didn't
have
a
big,
pretty
green
check
box,
we're
currently
working
to
make
sure
it's
fully
automated,
but
this
is
how
we
do
testing
of
our
infrastructure.
We
deployed
I
think
three
to
six
times
in
the
last
two
weeks:
the
entire
infrastructure.
What
we
do
is
we
migrate
applications
from
A
to
B
after
this
testing
is
done
and
we
verified
that
our
deployment
is
ready,
all
C
webhook
so
as
far
as
I
know
we're
one
first
teams
to
actually
create
a
web
hook.
A
Besides
the
standard
kubernetes
are
back
web
hook,
it
offers
HCL
level
compatibility
and
configuration,
you
can
treat
service
counts
and
users
the
same,
and
it's
it's
pretty
cool,
so
feel
free
to
go
check
it
out.
That
is
also
open
source
and
here's
a
team,
so
you'll
notice
that
we
have
some
cloud
ops,
guys
in
there
scrum
master
really
up
until
recently.
We
only
had
five
engineers
to
do
all
this.
We've
been
working
on
this
for
about
a
year
now
we're
in
production
in
two
different
regions
at
the
moment.
A
So
we
have
three
minutes
left
and
I've
cut
you
guys
off.
I
want
to
make
sure
that
you
guys
have
questions
and
can
can
can
ask
some
questions
on
the
slide
deck.
You
can
see.
I've
got
links
here
for
all
the
different
things
we've
contributed
back,
including
our
vault
SSL
integration
by
Martin,
Devlin
questions.
A
A
Yes,
sir
yeah,
so
when
we're
talking
about
versioning
we're
talking
about
versioning
of
each
individual
application
component,
and
so
it
basically
allows
you
to
go,
you
know
minor
or
major
versions,
and
that
is
record
as
a
part
of
the
metadata
framework
in
the
kubernetes
api.
That
way,
we
can
then
pull
in
all
that
data,
governance,
metrics
and
that
sort
of
thing
so
that
you
can
so
that
our
business
units
can
see
what's
deployed
in
a
given
point
in
time.
Yep
thing
else,
sir:.
A
A
A
A
Yep
yep,
it's
it's
so
Jenkins
has
the
ability
to
manage
multiple
environments.
So
the
question
for
everybody
was
is:
is
that
Jenkins
running
inside
the
the
environment?
The
answer
is,
yes,
it
deploys
into
a
namespace,
say,
slash,
dev
or
slash
stage
whatever,
and
then
it's
able
to
manage
multiple
namespaces
with
the
same
standard
naming
convention,
yep
yeah.
A
So
we're
doing
all
kinds
of
tests
and
cabarets
tests
to
include
to
make
sure,
for
example,
as
a
part
of
our
base
deploy.
We
deploy
an
elastic
search
cluster
of
consisting
of
three
masters,
three
clients
and
three
data
nodes.
So
we
do
test
to
make
sure
that
that
came
up,
and
then
we
run
tests
against
the
endpoint
to
make
sure
that
it's
available
and
running,
and
then
we
run
more
tests
to
make
sure
like
Griffon,
a--,
our
Griffon
or
front-end
is
running,
and
we
do
that
for
the
number
of
minions.