►
Description
This is the start of a series of “Under the Hood” briefings.
In this briefing, we will focus on integrating OpenShift with external routers. With that in mind, we will start with a walk through the high level overview of OpenShift as it is related to this topic and then explain our new F5 integration.
For the latest information on OpenShift 3 and available briefings, please visit http://commons.openshift.org or subscribe to the OpenShift Blog (https://blog.openshift.com).
The OpenShift Commons exists to provide a platform for customers, partners, developers and other open source technology initiatives to collaborate, share and accelerate the pace of innovation and adoption of OpenShift globally. The OpenShift Commons represents a new open collaborative community model designed to facilitate communication and sharing of best practices, feedback and development across the many open source initiatives that integrate with OpenShift. The best way to get involved is to join the conversation today at http://commons.openshift.org
A
Welcome
everybody
to
this
openshift
Commons
briefing.
We're
really
pleased
to
have
applying
masters
and
mike
barrett
here
with
us
today
to
talk
about
external
routers
and
everything
under
the
hood
that
it
takes
and
took
to
do
our
new
f5
integration
that
I'm
going
to
let
micaiah
and
Mike
post
abuse
themselves
and
kick
it
off
great.
B
Thanks
I
an
as
you
guys
know,
the
under
the
hood
is
a
series
that
we've
come
back
to
the
commons
periodically.
We
try
to
take
a
specific
piece
of
the
puzzle,
pieces,
the
technology
and
explain
it
a
bit
further.
The
first
ones
we
went
over
the
entire
platform
just
give
everybody
a
an
introduction
to
it.
So
now
moving
forward,
we'll
just
spend
the
time
on
a
very
specific
part
of
it
and
answer
any
questions
that
you
might
have
on
it.
I'm,
as
I
am
mention.
B
I'm
mike
barratt
I'm,
one
of
the
product
managers
over
here
on
the
open
shift
product
and
I've
invited
micaiah
masters
with
mila
kya
is
kind
enough
to
join
me
at
customer
sites
and
around
the
world
of
different
locations.
He's
one
of
our
star
engineers,
makhaya,
wanted
to
say
a
little
bit
about
yourself.
All.
C
Right
I'm
well,
I'm
micaiah,
I
joined
Red
Hat
about
four
years
ago,
so
I
was
working
on
openshift,
v1
and
v2
and
I
was,
among
other
things,
other
things
responsible
for
the
routing,
Damon
and
v2.
So
with
v3
we
have
a
similar
approach.
I
implemented
be
about
the
initial
implementation
of
the
f5
integration
for
v3,
and
we
we
now
that
is
now
under
the
responsibility
of
the
network.
C
B
B
It's
a
pretty
long
link
or
just
google
open
chef,
Roadshow
and
it'll,
be
the
top
two
hits
that
you
get
I
definitely
register
and
show
up
it's
a
great
place
to
see
the
community
face
to
face
in
your
local
region,
so
I
as
these
partners,
customers,
people
who
are
hobbyists
everybody's,
come
into
these
things
and
it's
free
food
and
beer
at
the
same
time.
So
before
we
get
in
just
a
fresh
reminder,
this
is
sort
of
the
marketing
architecture.
B
B
On
that
network
to
be
able
to
go
into
your
firewalls
into
your
reverse,
proxies
until
all
the
layers
that
you
sort
of
surround
applications
with
you
have
to
provide
some
IP
addresses
and
there's
a
couple
ways
to
do
that
either
you
you
manually,
you
disengage
the
software-defined
Network
in
open
shift
and
you
take
o
on
that
owners
yourself,
which
is
not
the
popular
road
to
go.
Where
you
establish
an
overlay,
Network
and
open
chef
is
using
open,
V
switch,
and
so
these
pods
are
getting
a
network
from
that
open.
B
The
switch
solution
that
open
flow
solution,
which
means
it's
on
almost
that
that
private
network
that
is
not
routing
itself.
Those
specific
tend
on
IP
addresses
out
to
the
world,
that's
flowing
through
a
routing
tier
and
that
routing
tears
connected
to
your
service
declaration,
and
it
involves
an
H,
a
proxy.
That
is
the
default
implementation.
B
But
we
understand
that
you'll
want
to
change
that
if
you
are
heavily
invested
in
hardware
routers
or
if
you
have
different
needs
and
requirements
on
getting
traffic
in
and
out
of
your
environment,
you're
going
to
be
changing
that
layer,
you're
going
to
be
messing
around
in
changing
configuration
files
and
understanding
the
flow
and
understanding
how
we
do
that
really
helps
you.
The
perfect
example
of
this
is
the
f5
router
it
shipped
last
night.
B
C
C
C
Secondarily,
we
wanted
to
provide
an
alternative
to
the
template,
hahe
proxy
router
that
could
be
used
as
a
reference
when
implementing
integration
with
other
technologies
that
rely
on
a
crowd.
A
create
update,
delete
based
api
citrix
netscaler
would
be
one
example.
Openstack
Elvis
tentatively
would
be
another
example
of
a
crowd
based
design
where
that
five
integration
may
serve
as
a
good
reference
to
implementing
a
similar
solution,
probably
a
better
reference
than
the
template
achieved
proxy
router
next
slide.
Please.
C
So
I
need
to
mention
one
thing
about
terminology
initially,
because
we
had
the
template:
router
the
H,
a
proxy
router.
We
were
calling
the
f5
solution,
the
f5
router.
More
recently,
we've
been
calling
it
the
f5
route
synchronizer
to
better
reflect
what
it
actually
does.
It's
not
routing
connections.
It's
synchronizing
route
configuration
and
open
shift
with
an
f5
big-ip
system
which
an
f5
does
the
actual
routing
the
route.
Synchronizer.
Doesn't
route
connections
doesn't
route
pack?
C
Is
it
just
configures
f5
throughout
connections
and
packets,
into
open
shift
the
route
synchronizer
uses
the
open
shift
master
as
its
source
of
truth?
It
gets
all
its
information
from
there.
The
route
synchronizer
itself
carries
minimal
state
just
credentials
to
connect
two
f5s
and
some
basic
cashing.
It
doesn't
store
any
configuration.
It
does
not
rely
on
a
five
big-ip
for
state
it.
Although
will
query
big-ip
when
it's
doing.
A
C
During
initial
sinks,
it
it
doesn't
rely
on
big-ip
or
on
f5,
it's
getting
the
data
from
the
open
shift
master
and
using
that
to
configure
the
f5
big-ip
host.
So
it's
the
route
synchronizer
itself
is
largely
stateless.
It
configures
the
big-ip
host
using
the
f5
I
control,
REST
API,
with
a
couple
exceptions.
I'll
note
later
on
door
was
I
yeah.
C
Development
I
used
f5
big-ip,
11
dot,
six
virtual
edition
on
in
Amazon
ec2,
because
we're
using
the
fi
fi
control
rest
api.
We
do
require
f5,
big-ip,
11.4
or
later,
which
11.4
is
when
they
arrest
date.
That
was
was
introduced
in
f5.
We
also
rely
on
ssh
access
to
f5
big-ip
to
upload
keys
and
certificates
for
outs,
because,
unfortunately,
the
REST
API
does
not
provide
a
means
to
do
so.
C
You
need
to
upload
the
key
and
then
trigger
a
REST
API
call
to
to
incorporate
that
key
into
an
SSL
profile
in
the
f5
configuration
mostly
it's
rest.
There's
that
one
aspect
where
we
need
SSH
access
to
the
FI
post
next
slide.
Please,
ok!
So
the
route
synchronizer
itself
runs
an
apollon
open
shift.
Typically
on
a
note
labeled
as
infrastructure,
it's
not
a
hard
requirement,
but
usually
you
would
want
to
have
dedicated
notes
labeled
for
infrastructure
components
such
as
this,
the
pod.
C
It's
it's
just
a
regular
open
shift,
pod
managed
by
replication
controller,
which
monitors
health
make
sure
the
the
pods
running
if
it
dies
the
replication
solar
restarts
it.
The
note
goes
away.
The
master
will
reschedule
the
pod
on
another
note,
though,
and
remember
the
router
synchronizer
itself
maintains
minimal
state,
so
we
can
do
that
if
something
bad
happens,
if
it
dies.
If
the
note
goes
away,
we
just
restarted
on
another
note
to
get
the
information
from
the
master.
C
The
route
synchronizer
uses
kuber
Nettie's
reflector,
which
is
built
on
the
list
or
walk
with
Lister
watcher
interface
to
watch
for
updates.
This
enables,
with
synchronizer
to
perform
an
initial
synchronization
of
configuration
when
it
starts
up.
It
gets
that
configuration
from
open
shift
and
then,
as
routes
are
modified
or
routes
and
services
are
created
deleted
using
this
interface,
the
route
synchronizer
gets
these
updates
and
updates
fi
big-ip
dynamically,
using
the
REST
API
interface.
C
So
it
updates
the
f5
big-ip.
It
creates,
updates
and
deletes
f5
pools,
corresponding
to
the
open
shift
services
and
creates,
updates
and
deletes
f5
local
traffic
policy
rules,
corresponding
to
open
shift
routes
to
route
incoming
requests
coming
into
f5
to
the
appropriate
pool
and
then
to
the
appropriate
pods
on
openshift.
C
So,
while
I'm
here,
I
want
to
make
a
couple
of
technical
side
notes
just
for
completeness
and
and
correctness
so,
like
I
mentioned,
we
are,
we
are
using
SSH
and
SCP
for
a
couple
things
here,
we're
not
strictly
using
the
rest
api
because
it
because
of
limitations
and
rest
api,
and
second,
I
want
to
mention.
While
we
use
policy
rules,
local
traffic
policy
rules
in
f,
54,
plane
and
encrypted,
that
is
edge,
terminated
where
f5
terminates
encryption
or
rhian
cryptid,
where
five
decrypt
sandri
encrypts
connections.
C
C
The
f5
local
traffic
policy
feature
was
implemented
before
sni
was
popular,
so
it's
not
supported
by
by
the
local
traffic
policies
and
we
have
to
have
a
custom
I
rule
the
eyeroll
parses,
the
TLS
client,
hello,
handshake
packets,
for
the
SMI
server
name,
looks
up
the
server
name
and
a
data
group
and
routes
that
way
so
pass
through
routes
are
handled
a
little
especially.
I
just
want
to
mention
that
for
completeness.
C
Graham
shows
what
I
just
described
upper
left.
You
have
the
master
where
we're
connecting
actually
to
the
API
server,
the
FI
routers
running
on
an
infrastructure
no
connecting
to
the
API
server,
which
is
part
of
the
open
shift
master,
that's
how
it
gets
route
and
service
information,
and
then
it's
connecting
to
f5
big-ip
host.
So
the
route
synchronizer
is
acting
as
this
intermediary
between
the
open
shift,
master
and
f5
big-ip.
C
Again,
the
master
is
a
source
of
truth.
The
wrath
synchronizer
itself
does
not
store
any
state
outside
of
its
own
configuration
and
some
caching
to
avoid
repeating
too
many
rest
calls
next
slide.
Please.
So
once
the
route
synchronizer
is
configured
f5
big-ip
next,
we
need
I
want
to
talk
about
how
connections
and
packets
get
routed
to
applications
or
services
on
openshift.
So
a
client
initiates
a
connection
with
f5
big-ip
to
the
virtual
IP,
the
VIP
of
the
virtual
server
configured
in
f5,
the
f5
local
traffic
policy
rules
or
custom.
C
The
custom
I
rule
use
the
HTTP
host
header
using
virtual
hosts
or
the
TLS
sni
server
name
to
map
the
connection
to
a
pool
corresponding
to
the
open
shift
service.
The
pool
contains
members
corresponding
to
pod,
endpoints
and
I
want
to
be
clear
about
this,
because
it
is
a
little
confusing.
We
do
not
route
through
services.
We
do
not
route
through
cube
proxy
with
this
solution,
we're
relying
on
service
definitions
at
configuration
time
in
order
to
map
to
figure
out.
How
do
we
wrap?
How
do
we
map
this
virtual
server
to
this
route?
C
Where
does
it?
How
does
it
map
to
the
pod
ports
or
the
pod
IP
address
services
help
us
in
figuring
out
that
configuration,
but
once
the
route
synchronizer
has
configured
f5
big-ip
we're
not
going
through
the
services
not
going
to
Q
proxy,
where
the
pool
level
pool
members
will
be
pointing
directly
to
the
pods,
sdn
IP,
address
and
port,
and
that
means
in
order
to
route
to
these
pods.
We
need
to
go
through
the
open
shift
sdn,
which
takes
some
extra
configuration.
You
have
to
set
up
an
IP
over
IP
tunnel
to
a
ramp
node.
C
C
I'll
talk
a
little
bit
more
about
this.
On
the
next
slide
for
high-availability,
we
would
recommend
creating
multiple
ramp
nodes
using
a
VIP
configured
in
open
shift
and
open
shifts,
IP
failover,
to
provide
to
make
sure
that
even
if
one
ramp
node
dies,
another
one
will
come
in
place,
and
so
this
tunnels
at
f5
is
using
to
get
to
the
SDN.
C
C
C
Okay,
so
looking
at
this
diagram
and
starting
from
the
box
in
the
upper
left,
we
go
clockwise.
We
have
the
client
connecting
to
a
VIP
on
f5
big-ip,
and
this
is
the
normal
tcp
connection.
Http
TLS,
f5
big-ip
will
pick
the
appropriate
pool
and
some
member
from
that
pool,
which
corresponds
to
a
pod
on
the
open
shift
sdn
to
route
the
packet
to
the
pod.
F5
big-ip
needs
to
route
through
its
tunnel
to
a
ramp
node,
and
so
at
this
point
this
is
a
nip
tunnel.
C
Let's
see
yes,
the
pod
IP,
endpoints
and
routes,
we
have
routes
and
services
that
information
is
all
stored
by
the
master
and
at
CD
it
is
it's
not
going
directly
to
at
TD
it's
it's
doing.
A
watch
on
on
the
resource
in
the
open
shift
master.
So
we
do
have
a
nice
interface
into
the
same
interface
to
the
template
and
HD
poxy
routers
use
to
get
updates
to
routes
and
services.
C
C
Okay,
so
to
use
the
f5
route
synchronizer,
you
need
to
have
an
f5
big-ip
deployment
configured
with
virtual
servers
for
HTTP
and
https
quickly.
Speaking
you
can.
We
don't
require
that
you
have
both
enabled,
although
typically
you
would
have
virtual
server
for
each.
You
would
want
to
support
both
types,
cryptid
or
unencrypted.
You
need
to
have
a
VIP
or,
if
you
prefer,
multiple
VIPs
for
one
for
each
or
even
multiple
for
each
virtual
server.
C
C
Once
that's
all
in
place,
you
can
use
the
oh
ADM
command
the
sink
man
or
similar
to
what
you
would
use
to
start
the
template
or
H
a
proxy
router.
We've
used
a
similar
080
admin
command
to
start
the
f5
route
synchronizer,
and
you
need
to
specify
specify
the
address
and
credentials
for
the
f5
big-ip
hose
the
location
of
the
private
ssh
key
with
which
the
route
synchronizer
will
connect
to
that
five
big-ip
hose
to
upload
certain
keys
for
routes.
You'll
need
your
open
shift
credentials
because
you're
creating
this
open
shift
objects.
C
C
This,
the
five
integration-
the
first
was
at
the
f5
I
controller.
St
api
is
relatively
new.
It's
much
newer
than
the
than
the
f5
soap
api
or
five
itself
and,
as
one
might
expect,
the
the
documentation
has
a
lot
of
gaps.
Still
there
a
lot
of
useful
resources.
Online
people
in
the
f5
IRC
channel
are
really
helpful,
but
it
still
took
a
lot
of
trial
and
error
to
determine
the
correct
rest.
Api
calls
required
parameters
and
responses.
C
Another
challenge
was
handling
pls
pass
through
and
again.
This
is
routes
where
the
route
itself,
the
rather
self,
does
not
specify
TLS
certificate.
So
we're
just
passing
traffic
through
encrypted
coming
into
the
f5,
pass
it
through
to
the
pod
without
decrypting
it
and
we
do
support
it,
but
it
did
take
some
extra
effort,
so
we
had
to
create
a
custom
I
rule
that
parses
the
TLS
client
hello
handshake.
So
it's
doing.
C
Its
kind
of
parsing
the
package
doing
some
low-level
work
there
to
figure
out
how
to
route
connections
of
the
tail
has
passed
through
type,
so
that
required
some
custom
TCL
code.
The
rap
synchronizer
will
add
that
custom
I
will
automatically
when
it
first
starts
up.
If
it
doesn't
find
that
rule
on
that
five
big
back,
because
so
it
will
create
that
rule,
but
that
was
a
challenge
during
development
to
figure
out
how
to
deal
with
that
type
of
connection.
C
The
biggest
challenge
was
in
figuring
out
how
to
route
connections
to
pods
on
the
SDN
routing
through
to
proxy
would
have
been
an
efficient
and
practical
for
several
reasons
and
without
a
way
to
route
route
to
pods.
Of
course,
we
can't
really
do
anything,
so
I
want
to
say
many
things
to
ram
ranganatha
or
amar
and
result
oprah
who
helped
in
who
came
up
with
the
ramp,
node
and
tunnel
approach
to
providing
connectivity
between
a
5
and
pods
on
the
STM.
Ok,
these
are
a
few
general
techniques
that
I
use
during
development.
These
aren't.
C
These
are
not
things
you
do
in
production.
These
are
just
some
some
things
that
come
to
mind
when
thinking
about
the
development
process.
So
one
thing
is
first
thing
is
rather
than
launching
and
deleting
the
pod
replication
controller
deployment
config
the
secret
containing
the
ssh
key.
You
know
you
know
creating
that,
and
then
you
can
even
do
that
with
odm,
0
ADM
command,
but
then
you,
then
you
have
to
clean
it
up.
That's
a
little
much
to
do.
C
Every
iteration,
sound
of
was
a
lot
easier
to
troubleshoot,
as
I
was
developing
and
iterate
quickly,
just
running
the
binary
directly
on
an
open
shift
host
in
my
development
environment.
What
with
log
output
going
to
my
terminal
and
as
I
mentioned
it
took
a
lot
of
trial
and
error
with
curl
commands
against
the
AI
control
rest
api
to
work
out
the
needed
api
calls.
I
also
use
girl
against
a
five
VIP
to
test
the
configured
routes.
To
make
sure
policy
rules
were
correct,
that
the
correct
the
search
the
the
f5
was
using
the
correct
search.
C
C
First,
I
was
using
that
to
make
it
easier
to
read
through
api
responses
and
later
I
I
realized
is
really
nifty,
because
it
validates
your
JSON
and
can
find
errors
in
the
JSON
I
ride.
Without
my
tools,
creating
I
did
find
that
pythons
Jason
acceptor
accept
some
JSON.
That
f5s
parser
will
not
turns
out.
Golden
can
generate
a
salon
that
Python
except
sleep
f5
rejects
which
necessitated
a
work
work
around,
and
that
was
one
of.
C
A
third
challenge:
well,
this
is
not
so
much
a
third
big
thing
was
well
in
open
shift
and
goober
Nettie's
generally.
Testing
is
really
heavily
emphasized.
Having
a
ready
set
of
example,
services
and
routes
with
example.
Search
was
really
helpful,
so
kudos
to
Paul,
while
who
developed
those
the
the
hello
and
Jeanette
stalker
examples
for
his
work
with
the
template
router
as
development
progresses,
as
the
code
base
becomes
more
complex
having
good
text
pests,
colored
unit
tests,
integration
tests
is
really
helpful
in
identifying
it
mistakes
quickly.
We.
B
C
Roughly
seventy-five
percent
test
called
coverage
in
the
standard
tests
that
run
with
every
every
build,
and
these
tests
essentially
create
a
maka
five
big-ip
unfermented
as
an
HTTP
test
server
recognizes
the
same
routes
spits
out
the
same
JSON
sf5.
We
also
have
integration
tests
for
testing
against
the
real
f5
big-ip
host.
C
One
thing
I,
maybe
could
have
done
better,
would
would
have
been
to
be
a
little
more
proactive
in
writing
tests
writing
test
cases
before
or
at
least
while
writing
the
code
that
was
being
tested,
first
of
all,
to
reap
benefits
of
testing
sooner
and
second,
to
reduce
the
feeling
of
thread.
When
you
have
this
big
pile
of
code,
I
have
to
write
a
big
pile
of
test
cases
for
it,
but
these
are
pretty
standard
recommendations
for
for
software.
Engineering
always
bear
repeating.
Those
tests
are
good.
C
The
v3
work
is
actually
very
similar
to
the
v2
routing
Dame
conceptually,
the
v2
is
a
little
more
complex.
We
had
the
broker
plugin
and
the
separate
routing
Damon.
It
was
a
little
more
cumbersome,
but
just
that,
having
that
knowledge
of
how
we
can
integrate
with
f5
and
v2
is
very
helpful.
Having
some
experience
with
the
REST
API
interface,
we
have
a
similar
approach.
C
I,
the
SSH
CP
thing
was
kind
of
something
it
would
have.
It
would
be
nice
if
their
way
to
do
it
in
the
REST
API
that
using
SSH
feels
almost
kind
of
like
a
workaround
for
this
gap
in
the
the
rest
api,
and
that
was
something
we
had
to
figure
out
in
v2.
So
that's
something
else
we
carried
over
into
v3
knows
that
fives
the
same
so
we're
using
us
the
same
technique
in
v3
I
can't
really
think
of
more
more
specific
lessons
learned
is
that
that
comment
to
your
question:
Nicholas,
okay,.
C
And
I
and
we
we
have
documentation,
it
should
be
being
pushed
out
with
30
to
others,
documentation
the
official
openshift
enterprise,
oblivious
also
in
the
origin,
documentation
for
setting
up
the
ramp.
Now
an
f5
tunnel.
We
have
documentation
on
launching
the
router,
we're
figuring
or
the
route
synchronizer,
and
that's
all
on
Oak
docs
on
openshift
com
I
can
try
to
I.
Can
give
you
a
more
specific
link.
If
you
like
you,
alright,
are
there
any
other
questions
all
right.
C
Okay,
using
different
load,
balancing
methods,
algorithms,
okay,
f5,
provides
that
functionality
right
now
the
route
synchronizer
doesn't
make
use
of
it.
It
hard
code,
the
I
believe,
the
round
robin
load,
balancer
method,
but
that's
something
we
could
definitely
add.
If
you
open
an
RF
e,
then
that
should
be
that
that's
definitely
something
that
we
feasible
to
implement
in
a
future
release.
C
Feel
free
to
suggest-
and
we
can
look
into
implementing
more
some
more
features
to
make
use
of
more
features.
Nf5,
okay,
Nicholas
shall
toots
the
configuration
Singh
can
commit.
That's
not
in
the
current
release.
It's
something
it
shouldn't
be
difficult
to
add,
but
that
is
missing
with
the
version
in
302.
Right
now,
sorry,
we
follow
the
number
of
connections,
specific
specific
applications.
B
C
Yes,
I
wouldn't
phrase
it
quite
like
that:
the
ramp
that
is
really
a
regular
openshift
node.
It's
typically,
you
would
mark
it
on
schedule,
so
you
won't
be
putting
pods
on
it.
It's
only
there
to
act
as
a
gateway
into
the
SDN
I
wouldn't
have
I,
wouldn't
think
of
it
as
a
component.
I
guess
you
can
think
of
it
that
way,
but
but
but
yeah
that
the
ramp
nodes
ramp
no
to
ramp
nodes
and
the
rot
synchronizer.
C
C
Okay,
just
throughout
synchronizer
running
on
a
ramp
note
itself.
Well,
you
could
run
it
on
the
ramp
node
we're
recommending
running
it
on
a
regular
infra
node
and
having
the
ramp
node
be
dedicated
just
to
acting
as
basically
a
gateway,
so
that
doesn't
have
won't
have
high
resource
requirements.
It's
just
shuttling
packets
between
f5
and
pots
on
other
nodes,
I'll
have
to
check
with
ROM
or
or
Azad
I.
Think
there
were.
There
were
some
reasons.
We
thought
it
would
be
better
not
to
put
the
rap
synchronizer
on
the
ramp
net
itself.
C
Ok,
atomic,
I'm
not
sure
exactly
how
how
we
implement
open
chef
sdn,
I
would
think
it
would
be.
It
would
work
the
same
way
with
atomic
all
we
all
we
need
is
a
host.
That's
on
the
SDN,
where
you
can
configure
that
you
configure
has
an
endpoint
for
the
tunnel
and
that
you
can
assign
a
VIP
in
open
shift.
I,
don't
see
any
challenges
atomic
would
pose,
but
I'm
not
certain
right
now.
A
So
thanks
again,
you'll
send
out
a
copy
of
the
presentation
and
we'll
hook
it
up
to
the
website
at
openshift
commons
org,
slash
briefings
with
along
with
this
video.
Probably
tomorrow
morning,
then
the
site
gets
rebuilt
overnight
each
night.
So
thanks
makhaya
and
thanks
Mike
for
bringing
this
topic
and
for
doing
another
under
the
hood.
We
always
look
forward
to
these
and
we
really
appreciate
you
taking
the
time
out
from
your
day
today,.