►
From YouTube: 2023-06-05 Delivery team weekly EMEA/AMER
Description
No description was provided for this meeting.
If this is YOUR meeting, an easy way to fix this is to add a description to your video, wherever mtngs.io found it (probably YouTube).
D
So
welcome
everybody.
This
is
the
5th
of
June
email,
America
time
zone.
We
have
a
small
announcement
like
last
time,
just
a
reminder,
since
we
have
the
last
hours
for
the
engagement
survey
and
we
start
to
have
a
discussion
topic
that
I
didn't
finish
to
write.
Maybe
we
can
start
to
I'm
gonna
verbalize
it
and
then
I'm
gonna.
Add
my
notes
on
that
during
the
16.0
release,
with
a
problem
with
the
release.gitlab.net
and
with
the
QA
jobs.
So
this
is
actually
something
that
probably
you
know,
especially
in
a
major
release.
D
D
What
do
you
think
about
that?
And
in
particular,
what
do
you
think
needs
to
be
done
to
be
able
to
not
incur
in
this
problem
in
any
release,
not
even
a
major
one,
so
we
can
Define,
maybe
which
are
like
the
main
working
items
on
that,
and
maybe
we
can
stop
them
to
prioritize
some
of
this
work
and
solve
this
problem
once
for
all.
E
So
I
would
like
to
add
something
to
this,
because
I'm
actually
working
for
the
past
few
hours
on
this
with
sres
and
QA.
So
on
this
specific
issue,
it
seems
like
that
something
changed
in
16.0
so
that
when
we
process
outgoing
emails
and
it
failed
something
changed,
and
so
three
years
ago
we
misconfigured
release,
so
it
never
sent
outgoing
email
because
of
the
wrong
part
in
the
SMTP
server
and
from
with
16.0.
It
started
failing
badly
because
we
have
a
lot
of
errors
and
what
we
are
thinking
is
happening.
E
There
is
that
every
QA
job
that
is
generating
an
outgoing
email
is
basically
failing
and
keep
failing
and
slowing
down
the
sidekick,
the
sidekick
on
that
single
node
and
so
all
of
all
the
QA
failures,
and
they
happened
when
things
run
in
parallel,
because
it
can't
keep
up
with
the
amount
of
SMTP
fails,
we're
talking
about
30
seconds
timeout
for
each
one,
so
very,
very
likely.
Those
jobs
are
sitting
there
for
30
seconds,
doing
nothing
and
then
retry
right.
So
this
is
what
is
happening.
E
What
was
what
happened
to
me?
Did
I
realize
today
when
I
was
doing.
This
is
that
we
don't
really
have
no
one
has
really
good
understanding
of
how
release
is
configured
who
owns
it.
We
own
it,
but
basically
so
things
like
we
are
slow,
slow
it
down.
Where
should
we
take
a
look
at
and
we've
been
left
with
asking
the
the
on-call
engineer,
the
Australian
call
to
just
SSH
into
the
box
and
try
to
do
some
manual
checks,
because
we
had
no
dashboard.
E
No
anything
and
so
being
this
vital
component
in
in
the
release
cycle
probably
should
just
improve
the
monitoring
in
general,
and
maybe
also
our
level
of
troubleshooting
as
release
managers
right,
so
that
everyone
in
the
team
can
do
some
basic
understanding
of
what,
if
that,
it's
a
single
instance
machine,
so
it
should
be
easier
than
debugging
system,
but
I
had
no
idea
what
to
do
right.
So.
D
Okay,
so
having
better
obserability
around,
this
should
be
probably
best
requirement
that
we
need
to
improve
as
far
as
understood
that
as
soon
so.
This
is
a
single
node
right
release
do
even
when
we
have
more
observability.
Do
you
think
having
this
as
a
I
mean
I
has
to
be
a
VM
this
one
or
it
has
to
be,
or
it
can
be
on
kubernetes
we
want
to
I
mean
the
question
is:
do
we
want
to
keep
testing
Omnibus
on
this
yeah.
F
C
Yeah,
let's
ask
to
the
work
that
Graham's
doing
with
the
release.
Well,
the
work
Graham
was
doing
with
the
release
environments
before
we
pulled
him
off
kind
of
has
an
assumption
that
we'll
still
have
release
at
the
end
of
that
process
to
have
the
Omnibus
test
so
yeah.
This
needs
to
stay
as
VMS
single
note.
Sorry,
something
that
runs
omnibus.
D
Do
would
be
any
need
to
add
these,
not
a
single
node,
but
the
multimodal
node,
even
if
zomb
on
this.
E
But
the
downside
of
this
is
that
it
will
take
more
time
to
deploy
okay
and
More
in
general
when
we're
talking
with
the
engineers
from
quality,
they
basically
told
we're
not
count,
so
those
those
instance
tends
to
be
downscaled
compared
to
our
reference
environment.
D
E
But
the
the
QA
test
is
not
that
big
of
a
load
right,
so
the
the
load
that
QA
is
putting
on
the
system
is
not
even
compatible
with
the
smallest
reference
Target
architecture,
which.
B
G
E
With
us,
because
we
there
are
many
shops
out
there-
maybe
they're
three
four
five
Engineers
working
on
software
company
and
they
have
GitHub
running
on
a
small
machine
and
it
should
work
so
I
mean
I
think
that
it
is
important
that
this
fail.
It
is
important
that
it
failed.
It
is
the
things
that
is
not
good
is
that
it
is
when
it
fails,
is
to
is
to
late
for
us
right.
So
it's
on
the
critical
part.
A
C
E
Safe
it
fails.
We
had
our
release,
blocker
issued
and
Reuben
said
this.
Is
this
already
happened?
So
we
know
what
has
happened?
No,
no.
We
know
what
is
happening.
It
already
happened,
so
we
say
okay.
We
should
bump
priority
here
because
we
found
a
problem
three
two
weeks
ago
and
this
was
before
and
no
not
in
any
board
and
no
one
knew
about
it,
and
that
was
why
this
should
be
P1.
E
In
the
meantime,
we
were
actually
working
on
this,
and
so
as
initially
that
was
frame
as
we
need
to
upgrade
the
instance
or
I
don't
know
doing
what
we
it
looks
like
it
seems
to
be
just
a
misconfiguration,
plus
Behavior
change
in
the
application,
with
the
major
release.
E
C
C
C
E
So
basically,
what
we're
trying
to
do
right
now
is
to
see
if
this
was
just.
It
was
the
cause
of
this
failure,
so
the
misconfiguration
and
knowing
that
the
things
has
been
always
misconfigured.
E
F
A
Thank
you
should
be
awesome,
so
deploy
to
release
instance
more
often
so
that
we
get
problems
like
this
before
they
are
critical.
E
C
D
E
Yeah,
we
also
had
a
big
moment
of
uncertainty
when
no
one
knew
what
to
do
right.
So
should
we
rise
an
incident
call
for
releases
who
owns
releases?
Will
the
EOC
be
able
to
do
something
if
releases
isn't
working
so
and
oh
I
mean
we
already.
We
were
already
trying
to
get
the
release
out
and
we
had
other
problems
and
then
and
then
we
had
this
layer
of
complexity.
That
would
not
even
know
what
knowing
what
to
do
right.
So
quality
had
no
ideas.
The
engineering
code
had
no
idea
as
well.
E
I
was,
let
me
create
an
issue
that
we
can
start
work
on
that,
but
I
had
no
idea
as
well
right,
so
so.
B
C
Point
of
an
incident
is
it's
a
way
for
us
to
escalate,
so
you
know,
even
if
it's
something
that
we
fully
own.
If,
if
you
need
help-
and
you
need
to
help
pull
people
in
then
raise
an
incident
and
we
can
like,
we
can
coordinate
there
if
we
need
to.
C
Would
it
be
worth?
Is
it
someone
who
could
actually
take
on
the
action
of
create
some
issues
for
improving
observability
on
release
because
I'm,
assuming
those
are
relatively
standard,
sort
of
like
tasks?
It
doesn't
particularly
depend
on
the
outcome
of
this
released
investigation
right,
that's
just
environment
observability,.
D
Okay,
thank
you
for
this.
Unless
you
please
keep
me
posted
if
you
have
any
discoveries,
so
we
can
just
I
can
just
look
at
it
immediately.
A
E
G
Let's,
let
me
just
have
things
open,
first
of
all,
say
deployment,
frequency
and
I
think
that's
about
it.
Okay.
So
let's
take
a
look
at
first
of
all
sharing
my
screen,
I
guessing
everyone
can
see
my
Chrome
tabs
okay,
so
this
was
last
week.
We
didn't
have
that
many
Brokers
as
compared
to
the
other
previous
weeks.
G
G
Let
me
let
me
know
if
I'm
missing
anything
to
note
about
this
graph
other
than
those
and
deployment
frequency
over
the
last
month
or
we
peaked
that
five
deploys
on
June
1st.
That's
Thursday,
I'm,
not
sure
what
happened
in
that
dip.
May
31st.
G
There
seem
to
have
been
something
on
May
31st,
but
can't
really
tell
what
that
was
right
now.
Does
anyone
know.
G
G
Yeah,
we
should
definitely
capture
that
I
totally
didn't
I
forgot
about
that
when
I
was
filling
out
this
graph,
so
I'll
just
I'll
just
redo
that
after
this
call
cool
nice
catch
lead
time
last
month,
nothing
too
abnormal.
What
we
expect
out
of
what
we
just
saw,
what
else
we
went
over
the
deployment
blockers
and
yeah?
That's
that's
about
it.
E
I
have
a
consideration,
or
maybe
more
of
a
ask
for
who
have
been
in
this
shift
recently.
So
today,
Dev
is
taking
five
hours
to
process
each
tag
request
so
building
packages.
E
I
do
remember
in
old
issues.
We
had
something
like,
and
now
you
take
some
time
off,
because
it
takes
18
minutes
to
to
build
the
package.
80
minutes
is
not
five
hours
so
and
I.
Remember
I
was
complaining
about
this
probably
last
week
last
week,
because
we
did
a
lot
of
release
last
week
as
well
and
I
think
that
Steve
told
me
now
now
it
usually
takes
around
three
hours.
Again.
Three
hours
is
not
five
hours,
so
anyone
has
any
idea
why
tagging
a
security
release
for
each
tag
is
taking
so
long.
F
I
noticed
that
it
takes
a
while
for
jobs
to
be
picked
up
by
Runner.
They
are
using
these
pending
States
and
when
I
was
watching
the
security
release
or
a
pass
release.
The
other
day,
I
noticed
that
it
took
at
least
20
minutes
to
pick
up
a
job.
So
if
you
put
security
packages
that
time
could
pile
up
and
could
explain
why
it's
taking
longer.
F
E
This
is
the
runner,
so
it
stays
in
pending
State
until
a
valid
Runner
that
is
designated
for.
That
pipeline
can
actually
pick
up
the
job.
So
this
either
means
that
I
know.
I
mean
very
likely
means
that
we
are
at
capacity
in
terms
of
the
ability
to
build
packages,
and
this
reflects
on
Ocean
deploy
as
well
right,
because
then
I
noticed
this
still
today,
I
had
three
package
for
the
security
release
that
we're
building
I
tried
to
stagger
them
a
bit.
E
E
So
it
gave
me
that
error
about
the
is
still
running
and
then
I
was
checking
the
status
and
was
still
pending
the
generation
of
the
build,
and
usually
this
is
because
the
the
runner
doesn't
have
capacity
to
pick
up
another
job,
so
either
we
split
the
those
run
so
that
we
have
say
yeah
and
also
deploy
dedicated
runner.
That
can
pick
up
all
the
auto
supply
package
generation
and
then
we
have
another
one
for
the
releases.
E
B
E
Yeah,
that's
that's
the
thing
right,
so
one
question:
so
if
it's
taking
five
hours
because
in
general
generally
spending
one
hour
to
pick
up
a
job,
then
yeah,
maybe
just
removing,
because
the
amount
of
packages
we
are
building
with
a
security
release
is
huge.
There
are
really
a
lot
of
packages
there.
So
maybe
that's
enough,
but
yeah
I,
don't
I
mean
it
really
depends
on
how
how
many
jobs
those
are
running.
So
maybe
they're
just
run
every
type
of
build
that
happens
on
dev
and
it's
this
same
set
of
Runners.
C
Yeah,
when
you're
through
this
release,
Alessia,
would
you
mind
just
sort
of
seeing
if
you
can
break
that
down
a
little
bit
into
an
issue
or
or
even
if
it's
an
investigation
issue,
and
then
we
can
start
trying
to
prioritize
that
so.
C
It's
technically
owned
by
reliability.
Technically,
the
runners
Runners
are
not
no,
actually,
maybe
maybe
they
are.
Reliability
have
stable
counterparts
that.
E
D
Hey
anything
else
on
recording.