►
From YouTube: Consul discussion gitlab-org/gitlab!271575
Description
Skarbek and Jason Plum investigate a target event involved in consul misbehaving with some queries, and target some options for moving forward to continue the investigation or possible solutions to the issue.
Reference: https://gitlab.com/gitlab-org/gitlab/-/issues/271575
B
B
A
B
A
You
have
a
series
of
error
messages.
You
can
get
an
actual
fail
to
connection
right,
so
you
connect
to
no
route
to
host
or
connect
timeout
right.
Then
you
get
the
no
response,
which
is
where
there's
literally
nothing
came
back
across
the
pipe
and
then
there's
effectively
a
which
is
a
no
answer
was
given.
B
B
A
A
B
B
A
So
in
the
message
I
have
from
10
a.m.
Eastern
this
morning
points
out
that
there's
the
two
types,
the
ones
that
we
really
care
about,
which
is
the
no
response
from
name
servers
list.
There
is
an
instance
where
we
were
getting
more
than
one
actual
error,
but
we're
swallowing
it.
We
fixed
that
since
so
now
we
get
a
new
response
from
dns
servers
when
we
try
to
look
up
the
dns
server
that
we're
supposed
to
be
asking,
so
that
problem
is
out
of
the
way
we
can
identify
that
now.
B
A
There
are
very
rare,
no
route
to
hosts
those
up
here
based
on
their
sparsity
that
it's
either
pod
scaling
or
node
scaling
yeah
right,
almost
always
what
I'm
seeing
are
the
no
route
to
hosts
are
trying
to
reach
out
and
hit
the
actual
dns
server
like
the
tube
dns,
and
it
goes
poof.
It
goes
away.
I
need
to
verify
this,
but
it
appears
that
that's
what
I'm
seeing,
I'm
not
seeing
it
getting
a
no
route
to
host
to
connect
to
a
service
ip
I'm
seeing
it
get
a
pod
ip
yeah
and
then.
A
A
A
Okay,
so
almost
all
the
time,
it's
like
proof
something
went
away
and,
as
a
general
rule
of
thumb,
a
service
so
long
as
the
service
is
present
will
have
a
service
id
yep
no
route
to
host
you'll
get
a
route
to
that
host.
What
you
won't
get
is
a
connection.
The
connection
will
time
out
trying
to
connect
you'll
still
get
a
connect
to
error,
but
it
won't
be
no
route
to
host.
B
A
So
on
my
hunch
of
the
no
response,
meaning
that
there
was
actually
a
timeout
somewhere,
I
went
looking
in
the
logs
and
just
took
the
same
sector,
the
same
base,
selector
and
time
zone
and
looked
for
anything
with
timeout
and
then
realized.
Oh
look:
there's
a
very
nice
clean
message
from
the
warren
long
out
of
standard
out
that
not
responding
within
tcp
timeout,
I'm
going.
A
You
know
the
last
day
with
this
message
or
this
message
and
I
just
kept
cutting
it
down
to
the
smallest
window,
to
show
replication
of
the
events
and
what
we
end
up.
Seeing
is
every
time
that
we
get
the
message,
no
response
from
name
server
list.
You
can
see
the
underlying
net
dns
trying
to
connect
timeout
expires.
Try
it
again,
try
it
again
and
I'm
like
bingo
yeah.
Now
we
have
a
at
least
we
have
a
point
in
correlation
that
we
can
try
to
use.
A
A
There
are
ways
with
cube
dns
to
make
a
request
and
get
all
of
the
ip
addresses
of
all
the
services
currently
backing
the
pod,
and
then
you
can
walk
down
it.
We've
seen
this
in
some
experience
in
dealing
with
nginx
yeah
right,
you
get
the
end
points
back
instead
of
a
particular
individual,
ip
or
the
service.
I'd
be
doing
the
proxy
result.
A
B
B
A
A
A
Now
one
of
the
things
that's
more
expected
from
their
documentation
and
their
chart
from
what
I
can
see
is
they
expect
that
you
shim
or
stub
the
console
domain
into
the
actual
cube
dns
or
the
core
dns
that's
implemented
in
your
cluster.
You
basically
convince
it
to
send
anything.
That's
you
know,
dot
console
to
the
console
service
ip
address
now,
there's
an
upside
in
that
regard
in
that,
even
if
the
ttl
is
set
to
one.
A
Second,
if
you
get
10
requests
in
one
second,
console's,
not
trying
to
answer
that
request
for
all
ten
right,
let's
just
say
as
an
example
of
should
we
scale
plus
two
nodes
and
both
of
those
nodes
will
happily
fit
20
web
service
pods.
They
won't
at
our
scale
the
nodes,
but
let's
give
it
an
example:
there's
a
sudden
scaling
event,
the
pods
will
come
up
and
they
will
probably
come
up
before
cut
the
local
daemon
set.
Console
has
actually
joined
two
console
cluster
right.
Neither
of
them
are
members
yet.
A
The
10
pods
come
up
and
you
know
their
dependencies
run
pretty
quick
and
then
they
all
try
to
fire
at
once
and
they're
going
to
be
within
a
second
or
two
generally,
then
you've
got
the
problem
of
boom.
Who
do
I
get?
Do
I
actually
end
up
spreading
the
service
requests
like
to
the
service?
Ip?
Are
those
requests
being
spread
across
the
other
eight
members
of
the
cluster
that
are
active
and
members
or
is
one
of
them
just
getting
hammered
all
of
a
sudden
right.
A
A
A
A
B
A
B
B
B
B
B
B
B
B
Alright,
so
using
the
query
that
you
were
that
you
had
posted
in
the
issue,
clusters
c
and
d
haven't
suffered
anything
in
the
last
five
hours.
Cluster
b
has
a
sporadic
set
of
events
that
happened
at
1400
utc
again
at
14,
30
and
then
16
30.
So
like
10
minutes
ago,
roughly
I'm
tempted
to
make
this
slightly
easier
and
just
target
on
one
of
those
time
zones.
B
B
B
B
A
A
A
B
B
B
Its
period
seconds
is
set
to
ten,
that's
every
so
it
runs
every
ten
seconds
and
its
timeout
is
one
second.
So
if
this
command
fails
to
run
within
a
period
of
one
second,
we'll
fill
the
readiness
probe.
We
only
failed
the
probe
once
and
assuming
our
timestamps
are
all
in
a
kosher
standpoint.
Here
it
looks
like
we
could
run,
we
will
have
failed,
we
could
have
a
pod
in
a
failing
state,
receive
a
request.
The
request
is
not
met,
and
then
the
next
rareness
probe
is
set
to
the
pod
and
fails.
B
B
B
B
A
A
Because
if
a
message
refutation
comes
in
and
then
blip
it
goes
off,
is
it
is
that
resulting
in
whatever
is
causing
the
one
second
curl
timeout
to
not
pass,
or
are
we
seeing
a
cpu
spike
while
it's
processing,
some
sort
of
event
or
other
network
traffic?
Is
there
like?
There's
got
to
be
something
correlative
that
we
can
bind
to
why
this
is
happening?
I
know
that
we
see
these.
B
Refuting
so
still
excluding-
I
don't
know
what's
in
info
logs,
but
excluding
info
logs
and
counting
the
word
refuting
5600
times
wait
is
that
the
right
line,
number
yeah,
5625
times
refuted
a
suspect
message
from
some
machine,
and
this
varies
like
around
that
time
was
specifically
a
vm.
But
I
also
see
this
for
gke
nodes
as
well,
that
are
participating
with.
B
A
B
A
B
B
A
B
We
know
the
pod
has
not
scaled,
so
maybe
the
next
thing
to
look
at
is
metrics
for
that
node
to
see
if
we
are
suffering,
maybe
a
high
usage,
just
cpu
or
memory
for
that
node
that
this
is
operating
on
and
maybe
also
for
this
pod,
like
maybe
this
pod
was
suffering
from
unfortunate
event,
so
where
in
our
logs
is
the
node
that
we're
running
on?
So
this
is
the
target
node
that
we
want
to
look.
B
B
A
A
A
A
A
B
B
B
B
B
B
B
A
B
B
A
B
A
B
B
A
A
B
A
B
B
A
A
A
B
A
A
Yeah,
I
definitely
definitely
want
to
do
some
investigation
to
watch
generating
these
massive
spikes
on
average
per
chord
load
like
it's
good
that
we're
using
it
right.
Don't
get
me
wrong,
but
when
you
can
just
kind
of
trace
the
line
average
and
you're
below
75,
but
you're
regularly
spiking
over
a
hundred
yep
like
there's
a
problem
here
and.
B
A
A
B
B
A
B
B
B
B
A
A
B
B
A
A
B
B
A
B
A
B
B
B
B
B
B
B
B
A
A
A
B
A
B
A
A
Honestly,
my
expectation
is,
we
we
won't
need
more
than
a
couple
of
hundred
meg
like
100,
like
200
megabyte
of
memory,
seems
probably
not
needed.
That's
an
easy
one
to
find
out
just
like
grab
all
the
console
pods
and
see
what
their
lifetime
memory
is
boom
and
I
would
say
not.
We
probably
don't
need
to
guarantee
more
than
say,
300
400,
like
300
m
on
cpu.
A
A
A
And
then
you
start
having
long
entries
that
like
say,
I
tried
that
one.
Then
I
tried
that
one
and
then
I
tried
that
one
that
would
be
somewhat
feasible
if
we
could
ensure
that
not
all
traffic
is
being
directed
to
a
given
top
of
the
list
and
two.
If
our
gitlab
database
load,
balancing
resolver
actually
returned
more
than
one
entry
because
it
doesn't.
A
B
This
is
a
bad
idea,
but
instead
of
having
our
pods
reach
out
of
that
node
in
order
to
reach
console,
is
there
a
host
port
configuration
that
we
could
leverage
and
then
we
could
somehow
configure
all
of
our
pods
to
talk
to
local
hosts
and
some
port
that
we
know
that
console
should
be
running
on
since
we
run
console
in
all
of
our
nodes
anyways.
Currently,
that
would
eliminate
network
copy.
B
However,
if
we
set
the
priority
class
to
a
certain
value
and
we
set
the
resources
as
we
should,
because
currently
we
don't
theoretically
console
should
bounce
back
and
since
we
pull
for
console
with
every
60
seconds,
I
believe
it
is
we'll
put
pressure
for
that.
One
node
onto
the
primary
database
for
that
one
minute
in
which
all
of
those
pods
failed
the
request
for
so
hopefully,
if
something
like
that
happened
it
shouldn't
last,
but
so
long
assuming
console
was
able
to
spawn
back
up
correctly.
B
A
A
B
A
A
A
B
B
A
A
B
B
B
A
B
A
Instead
of
having
figuring,
I
should
say
I
want
to
try
to
figure
out
how,
instead
of
having
us
ask
for
where
is
console
and
then
ask
console,
is:
does
it
make
sense
to
route
the
request
through
the
system
dns
through
cube
dns
through
core
dns
in
a
stub
domain,
because
if
anything
else
at
least
then
we
would
have
when
you
get
10
requests
all
of
a
sudden?
It's
not
hey!
Here's
ten
request
console.
I
hope
you
return
right
like
yeah.
A
A
It
so
I'd
have
to
go
digging
into
the
load.
Balancer.
The
gitlab
database
load
balancer
code
base
again
to
see
what
parameters
we're
passing
to
search
and
if
the
timeout
configurability
is
there.
I
know
that
the
gems
default
is
fine.
I
have
to
see
if
our
default
is
that
or
there's
any
timeout
configurability
that's
beyond.
What's
there
we
can
pass
it
a
different
timeout.
We
just
have
to
make
it
possible.