►
Description
Distribution team demos Orchestrator deploying a complete 2k reference architecture. Discussed some of the challenges resolved during the final steps of this functionality, documentation additions to be made, and features that should be on by default going forward.
A
A
A
All
right
so
some
things
that
have
changed
we
now
thanks
to
dustin.
We
will,
I
definitely
install
licenses
so
one
of
the
things
with
get
lab
licenses.
Typically,
you
put
your
license
at
the
beginning
and
then,
if
you
want
to
upgrade
or
do
more
later,
you
have
to
go
into
the
web
ui
and
update
if
you're,
trying
to
do
just
regular
unintended
installations.
A
lot
of
folks
seem
to
have
the
pattern
of
when
we
do
a
release.
They
do
an
update
right.
A
A
A
Let
me
stop
sharing
my
screen
for
a
second,
so
they're
in
random
issues
in
a
pipeline
where
you
run
a
pipeline
and
you
do
it
the
first
time
and
you
stand
up,
say
geo,
secondaries
and
you'll
get
this
message
saying
hey.
I
can't
do
that
and
the
number
of
wall
senders
is.
Actually
you
know
you
increase
the
number
of
wall
senders
and
you
keep
increasing
it
until
it
seems
to
work,
but
the
documentation
all
says
have
your
number
of
nodes
plus
one
well,
the
answer
is
it's
not
quite
right
and
what
goes
on?
A
Is
it
when
you
do
that
initial
replication,
there's
a
pg
backup
upgrade
and
that
doesn't
take
one
slot.
It
takes
two
so
for
as
many
geosites
as
you
have
an
additional
to
your
primary
site,
you
need
plus
two
extra
slots
to
handle
that
first
initial
application,
because
when
they
all
start
in
parallel,
you'll
have
your
number
of
replicas
in
the
primary
site,
plus
the
number
of
geo
sites
you
have
by
two
plus
one
slot
required
which
is
not
transparent
in
either
the
postgres
documentation.
When
you
go
through
it,
the
first
time
or
in
our
documentation.
A
A
So
if
it's
not
application,
one,
which
is
the
we're
going
to
run
all
of
the
database
migrations
everything
fails
from
then
on,
because
the
encryption
key
to
the
database
has
changed
because
it
puts
the
encryption
key
for
the
wrong
node
in
before
it
tries
to
run
the
actual
database
migrations.
So
this
is
a
thing
that
we've
learned.
Stan
is
going
to
be
looking
into
that
more
and
we're
going
to
see
if
we
can
get
rid
of
that
whole
set
application.
A
A
Screen
now,
as
you
can
see,
we're
doing
the
load
balancer
and
what
I
wanted
to
point
out
now,
there's
an
interesting
thing
when
you
do
a
load,
balancer,
okay,
the
machine
comes
up.
It's
the
vault
port
is
port
22
for
ssh,
but
when
you
put
a
load
balancer
in
front
of
application
nodes,
one
of
the
things
that
you
have
to
do
is
reroute
ssh
traffic,
so
that
when
you
do
a
git
clone
or
a
push
that
traffic
actually
hits
one
of
the
application
servers
behind
the
proxy.
A
So
that's
just
kind
of
a
nice
baked
in
features
that
you
don't
have
to
worry
about.
You
know
which
one
is
it
when
I
set
it
up
and
also
keeps
it
identical,
so
that
job
can
just
run
over
and
over
and
over
again
from
the
first
time,
all
the
way
to
like
the
30th
time
so,
and
that
is
one
of
the
things
has
been
added.
So
since
the
last
time
we
showed
this
last
time
we
did
ticket
architecture.
We
could
just
really
provision
the
nodes.
A
So
we've
added
a
couple:
we've
added
everything,
except
for
buckets,
which
means
that
we
can
now
set
up
giddily
behind
the
system.
Both
applications
nodes
set
up
instead
of
successfully
the
database
sets
up
correctly.
The
redis
sets
up
correctly
and
starts
using
what
is
caching
and
the
load.
Balancer
properly
stands
up
in
front
of
the
two
application
nodes.
B
A
Bottom,
all
right,
so
one
of
the
other
cool
things,
thanks
to
gerard
that
I
found
out
with
h
proxy-
is
that
you
have
this
interesting
interface
at
port
1936.
If
you
expose
it,
it
shows
you
what's
going
on
for
hi
proxy
on
the
load
balancer.
So
I
found
this
invaluable
because
this
tells
us
what
all
listeners
are.
A
A
Now
it's
not
done
configuring,
but
we
can
see
that
I'm
sorry
the
ssh
on
the
endnote,
so
we
can
see
that
it
can
now
connect
on
ssh,
which
is
good
and
we're
waiting
on
the
actual
rails,
app,
which
is
going
to
be
more
configuration,
but
this
gives
us
a
great
overview
to
know
if
our
cluster
is
healthy
or
not.
So
I
just
wanted
to
share
that
because
I
thought
that'd
be
a
really
awesome
tool
again.
Thank
you
to
gerard
for
for
sharing
that,
because
I
hadn't
seen
that
so.
A
C
B
A
A
I
go
back
to
the
reference
architectures
and
the
geo
pages,
and
I
go
back-
and
I
add
like
for
here
one
of
the
things
that
wasn't
clear
is
that
the
2k
architecture
doesn't
talk
about
much
about
why
you
do
nfs
and
it
says
nfs
is
optional,
but
what
it
doesn't
tell
you
is
that
if
you
don't
do
nfs,
you
need
to
set
up
fast
keys,
because
nfs
is
how
authorized
keys
for
your
s.
When
you
set
your
key
to
the
ui.
A
That's
how
all
the
different
application
nodes
know
how
to
connect,
and
so
you'll
get
random,
doesn't
work
or
does
work,
because
the
load
balancer
will
either
send
you
to
the
one
where
your
key
went
or
to
the
application
node
that
doesn't
so.
You
have
to
set
up
fast
keys
if
you
don't
set
up
nfs
and
since
we're
deprecating,
nfs
bas
keys
is
becoming
default.
So
this
is
an
mr
that
adds
that
information.
A
A
So,
if
you
run
a
license
check
on
a
geo,
secondary
you'll,
say
geo
won't
work
because
there's
no
license
but
geo
doesn't
need
a
license
on
the
secondary,
only
needs
to
be
on
the
primary
site.
So
this
I
have
updated
the
rate
task
at
this
point
and
it's
still
going
to
review
but
updated
the
rate
test
to
actually
check
that
correctly.
A
C
Right
and
and
for
reference
of
other
people,
we've
we've
been
turning
on
the
fast
keys
or
the
fast
ssh
key
lookup
by
default
as
part
of
the
cloud
native
pretty
much
since
inception.
For
this
particular
reason
and
in
reality
there's
pretty
much,
no
reason
not
to
do
that.
Anyways
early
on
we
were
concerned
about.
You,
know,
performance,
difference
and
stuff
like
that.
But
there's
just
no
point
the
fast
keys
look
up
is
so
performant
that
it's
not
impactful
and
actually
at
scale
it's
more
performant
than
nfs
itself.
C
A
A
You
could
get
an
error
state
after
because
the
service
could
be
there,
but
not
listening
like
there's
this
there's
this
gray
zone
between
when
the
service
for
bouncer
pg
bouncer
starts
and
when
it's
actually
properly
listening
and
the
check
was
saying,
are
you
started,
but
it
did
not
necessarily
imply
and
you're
listening
to
so
so
because
of
that
we
now
pg
bouncer
running
now.
Checks
does
the
check
in
such
a
way
that
it
checks
for
both
right.
So
we
never.
We
never
let
this
run
where
you're
not
actually
listening
right,
because
that's
frustrating.
C
Right,
so
that's
basically
implementing
in
for
context
of
others
watching
that
effectively
implementing
the
readiness
probes
that
you
would
find
in
kubernetes
or
the
various
container
orchestration
platforms.
We
were
doing
a
liveness
probe
by
just
going.
Is
it
running,
but
we
weren't
actually
doing
a
readiness
probe
by
making
sure
that
it's
listening
and
actually
able
to
respond.
A
A
To
that
point,
while
that
still
installs,
the
other
thing
that
we
are
working
on
right
now
is
the
concept
of
putting
in
rate
tasks
to
accomplish
some
other
things.
So
this
is
a
small
proof
of
concept
that
I
worked
on.
It
got
done
last
night,
but
the
idea
is
that
when
you
migrate
currently
the
orchestrator
when
it
runs
on
a
multi
database
when
it
or
in
any
database,
you
can
run
it
once
and
it'll
configure,
but
then
the
second
time
it'll
break.
So
it's
not
idempotent
with
this.
A
This
is
a
rate
test
that
will
detect.
Have
I
actually
run
or
not?
Should
I
load
the
database?
Should
I
run
migrations
and
just
give
us
back
a
simple
change
or
false
still
going
through
the
the
process
of
talking
it
through
with
the
database
for
team
reviewers,
but
this
is
kind
of
where
we're
heading
and
this
rank
task
will
be
available
to
anyone
and
we're
going
to
take
what
we
do
with
licenses
that
dustin
did
for
licenses
and
we
have
a
similar
asking
charts.
So
we're
going
to
take
that
logic.
A
We
have
for
licenses
turn
it
into
a
rank
task
and
be
able
to
share
that.
That's
just
a
couple
of
things
that
are
kind
of
things
that
are
popping
out
in
orchestrator
and
they're,
going
to
keep
getting
folded
back
because
really
orchestrator
and
charts
have
a
lot
in
common.
As
far
as
some
of
those
the
problems
we're
solving,
so
we're
going
to
try
and
double
that
double
up
wherever
we
can
so
that
we
everybody
benefits
here.
A
Okay,
now
here's
another
thing
to
talk
about.
We
still
have
to
do
this
thing
where
we
copy
the
get
lab
secrets
json
file
around,
and
that
is
because
there
are
certain
secrets
to
get
made
in
that
file,
but
they
have
to
be
on
every
node.
It
would
be
better
to
do
that
declaratively
and
there's
a
just
a
couple
of
things
that
if
you
do
it
declaratively,
then
those
secrets
live
in
the
gitlab
rv.
A
A
B
B
B
A
A
C
C
If
you
go
to
jobs
in
the
navigation
menu
just
above
getaway
servers
that
actually
takes
you,
I
wonder
if
they
moved
it,
maybe
they
did
and
at
one
point
and
it's
probably
still
floating
around
somewhere,
you
actually
had
the
ability
to
get
to
the
sidekick
administration
status.
Page,
I
think
that's
under
monitoring
is
that
our
monitoring
now
yeah
so
go
to
monitoring
and
then
background
jobs.
C
There
you
go
just
this
is
the
actual
like
we're
doing
an
authorized
pass
through
directly
to
sidekick
through
the
rails
console,
I
should
say
directly
through
rails
versus
through
workhorse,
and
we're
actually
checking
the
status
of
sidekick
and
its
integration
with
redis.
So
we
can
look
at
it
how
busy
they
are,
how
many
clients
there
are,
how
many
things
have
failed
or
things
like
that.
It's
one
of
those
points
that
a
lot
of
people
don't
realize
is
actually
a
way
to
see
how
that
is
behaving.