►
From YouTube: Releasing Ceph - Deep Dive Into Build Infrastructure - Alfredo Deza & Ken Dreyer, RedHat
Description
Releasing Ceph - Deep Dive Into Build Infrastructure - Alfredo Deza & Ken Dreyer, RedHat
Building packages for Ceph is a non-trivial task, it involves various steps and several pieces of infrastructure, all acting in unison. It has evolved in the past years into a scalable system that can handle load elastically. From ephemeral build nodes to load balancing repositories, both the development and release packages are able to benefit from this system. This presentation will go into some of the details that makes it robust, extensibility, and some of the difficult problems (some of which are still unsolved!)
About Ken Dreyer
Senior Software Engineer, Red Hat, Inc.
Ken Dreyer is a software engineer working for Red Hat on the Ceph distributed storage system. He handles the release process to ship Ceph in Red Hat's product line: bug triage, build pipelines, and continuously improving the tooling for a smooth process.
About Alfredo Deza
Redhat
Principal Software Engineer
Alfredo Deza is a principal software engineer working for RedHat on the Ceph distributed
storage system, avid open source developer, unit test enthusiast, Vim plugin
author, photographer and former athlete.
As a passionate knowledge-craving developer he can be found giving
presentations in local groups about Python, file systems and storage, and
system administration.
Currently leading the development of ceph-volume, ceph-medic, and the build and release infrastructure for Ceph.
A
My
name
is
Fred
Odessa
and
I'm,
going
I'm
B
I've
been
working
with
the
safe
team
for
a
while
now
five
ish
years,
and
you
know
doing
a
little
bit
of
everything-
I've
been
a
release
manager
for
a
few
years
now.
I
share
that
responsibility
with
a
few
guys
from
Red
Hat
and
a
few
from
sousei
who
helped
us
out
every
now
and
then
for
cutting
releases
and
yeah.
So
these
presentations
split
between
me
and
Ken
dryer.
B
So
in
2015
it
was
my
first
year
at
Red,
Hat
I
hadn't
completed
a
year
there
and
I
got
invited
into
a
meeting
and
the
invite
said
we
need
to
talk
and
I
I
went
into
the
meeting
and
there
were
a
couple.
People
there
from
Red
Hat
product
security,
including
mark
Cox
and
mark,
is
the
lead
of
product
security
at
Red
Hat.
Something
had
gone
wrong
so
to
talk
about
what
went
wrong.
I'm
gonna
back
up
a
few
more
years
to
talk
about
how
we
were
initially
building
staff.
B
Initially
we
had
the
code
in
github
and
we
were
building
Debian
packages
and
a
blend
two
packages
and
we
were
pushing
those
up
to
the
Ceph
comm
web
server.
So
we
would
index
those
and
then
GPG
signed
those
and
published
them
and
if
you
know
about
apt
repository
signing
basically
that
GPG
key
is
what
authenticates
that
the
packages
are
coming
from
the
Ceph
project.
So
if
you
have
that
key,
what
that
means
is
you
are
able
to
forge
packages
to
make
it
look
like
they
came
from
from
the
project.
B
So,
basically
when
in
those
early
days
we
were
building
a
lot
of
things
and
we
were
signing
a
lot
of
things
for
more
and
more
distributions,
and
eventually
we
added
CentOS
6
and
CentOS
7
in
there
and,
along
with
those
platforms,
came
a
lot
more
packages
and
a
lot
more
dependencies,
a
lot
more
build
scripts
and
a
lot
more
processes
that,
unfortunately,
were
not
quite
written
down.
So
we
would
end
up
building
more
and
more
things,
and
there
were
many
many
bugs
and
problems
with
this
process.
B
So
sometimes
there
would
be
bugs,
and
sometimes
there
would
be
problems
with
the
build
environment,
different
things
and
we
would
just
continually
have
to
be
building
pushing
signing
over
and
over
and
over
again.
At
one
point,
someone
got
the
idea
that,
in
order
to
make
this
a
lot
faster,
we
would
copy
the
GPG
key
that
we
use
for
signing
on
to
the
web
server.
That
was
also
running
WordPress.
B
So,
yes,
it
got.
It
got
a
little
bit.
Worse,
unfortunately,
because
we
also
copied
the
GPG
key
onto
every
build
system
that
was
building
Ceph,
so
in
order
to
build
it
at
the
end,
the
job
would
sign
the
release
as
well.
Then
we'd
collect
that
and
push
it.
So
by
the
end,
we
weren't
quite
I'm
not
sure
that
anybody
had
a
full
picture
of
where
the
GPG
key
had
been
copied.
And
then
a
member
of
the
public
said:
hey
I
found
your
GPG
key,
it's
on
I
forget.
B
Was
it
the
mirror
or
on
the
website,
or
something
like
that?
Basically,
it
was
exposed
to
the
public,
and
you
know
so.
We
had
basically
long
story
short.
We
had
to
go
through
this
exercise
of
generating
a
new,
stronger
key
keeping
that
on
a
very
restricted
system
and
improving
a
lot
more
of
the
build
system,
the
buildin
really
process.
So
that's
what
we're
going
to
talk
about
today,
Alfredo.
A
A
What's
the
next
thing
on
our
on
our
to-do
list
and
and
and
get
there
so
a
bunch
of
other
things
related
to
the
specific
way
that
we
cut
releases
and
we
build
so
I
inherited
a
Jenkins
server
that
basically
wasn't
using.
You
know
the
things
that
compose
a
a
nicely
automation
and
continuous
delivery
system
it,
which
is
basically
one.
No,
they
had
a
bunch
of
scripts.
A
bunch
of
the
problems
are
like
right
here.
You
know
it
was
a
lot
of
bash.
It
was
monolithic,
hard-coded
scripts,
not
controlled,
operation
control.
A
It
was
completely
undocumented,
so
you
had
to
go
and
go
in,
try
to
figure
out
things
and
and
see
how
they
were
built
and
you
it
wasn't
clear
how
things
were
put
together.
So
in
the
case
of
self,
some
distributions
needed
certain
dependencies
and
certain
packages
to
be
available,
and
somehow
these
packages
would
end
up
in
the
repo,
because
part
of
the
scripts
depended
on
these
packages
say,
for
example,
an
RPM
and
RPM
would
have
to
exist
on
these
directory
and
then
will
get
pulled
into
the
repo.
But
where
did
these
RPM
come
from?
A
A
Basically,
so
a
lot
of
a
lot
of
these
was
very
frustrating
and,
and
we
tried
to
stop
and
think
how
can
we
go
and
and
try
to
take
these
pieces
out
and
in
and
make
it
you
know
more
more
consistent
and
more,
you
know,
basically
better
and
in
one
of
the
one
of
the
things
that
I
want
to
touch
here.
Is
that
a
lot
of
in
a
lot
of
these
situations?
There's
there
is
this
thing
called
s
mo
P,
which
you
say
is
a
simple
matter
of
programming.
A
I,
don't
know
if
you
heard
the
term
when
I
first
heard
it
I
was
like.
Oh
man,
that's
happened
to
me
a
lot
of
times
where
you
say
well,
you
know
like
this
is
a
pretty
complicated
system
and
releasing
stuff
is
you
know
not
an
easy
thing?
It's
like
and
someone
will
come
it's
like.
What
do
you
mean?
How
about
you
know
in
the
case
of
the
GP
gig
is
like
well,
you
know,
GPG
sonny
is
not
difficult
if
you
just
copy
the
GP
geeky
everywhere.
A
It's
like
well,
okay,
you
know
it's
like
well,
but
what
about
the
passphrase?
Well,
you
know
it's
just
a
C
seon-ah
as
not
requiring
the
passphrase,
so
the
simple
matter
of
programming
thing
applies
keeps
applying
to
all
of
all
of
these
problems.
It's
like
whenever
we
face
a
difficulty
in
the
road
in
the
path
of
like
cutting
a
release
or
whatever
problem
we're
trying
to
solve
it's
inevitable,
to
receive
like
an
email
or
a
message,
or
a
comments
like
well.
You
know
like
this
could
be
really
easy.
We
just
just
make
this
change
here.
A
It's
like!
Well,
you
know
things.
Can
you
know,
get
a
little
bit
more
complicated,
so
here's
the
old
overview
of
how
things
used
to
work,
so
a
set
branch
would
trigger
a
change.
A
set
branch
on
github
would
trigger
this
continuous
integration
system.
We
had
one
server
per
distro,
the
same
server
would
build
and
the
same
server
would
make
the
repos
and
the
same
server
would
host
the
repos.
A
So
that
meant
that
you
had
one
server,
$0.04,
OS,
6,
1
9,
1
%,
OS
7
and
r14
for
debian,
in
an
r1
for
ubuntu,
and
whenever
one
of
these
things
was
down
like
the
whole
thing
was
down
like
just
like
nobody
could
like.
Oh
man,
CentOS
7
is
down
like
oh,
nobody
can
just
test
again
sent
to
us
and
we're
done
so.
A
It
got
to
a
point
where
you
could
wait
for
a
day
and
a
half
for
getting
a
repository
after
you
push
the
change
to
github,
which
was
you
know,
completely
unacceptable
now,
so
what
when
ended?
Wouldn't
it
happening
was
also
that
sometimes
these
servers
would
not
work.
They
would
have
like.
Oh
man,
we're
missing
a
dependency.
So
what
do
we
do?
Yeah?
Well,
let
me
ssh
into
this
server
over
here.
Add
the
dependency
problem
fix.
We
can
continue
the
bills.
Okay,
so,
but
who'd
get
notified
about
these
changes
in
dependencies.
A
Is
the
one-cent
OS
7
machine
that
nobody
can
touch
so
it
can
go
away,
can
go
you
can
go
up
in
flames,
will
bring
an
r1
and
we'll
continue
our
process.
Put
everything.
Immersion
control
highly
documented,
remove
like
the
tons
of
scripts
and
make
it
more
concise
a
lot
of
configuration
management.
So
we
we
can.
We
can
have
these
repeatable
deployments
and
and
move
forward.
A
We
use
Jenkins
job
builder
with
which
I'll
talk
in
in
a
second,
but
in
reviews
the
the
interaction
of
developers
and
and
and
the
community
in
into
Jenkins,
because
in
Jenkins
is
you
know,
the
interface
can
be
a
little
bit
rough
and
nobody,
nobody
really
likes
to
know
and
understand
and
figure
out
how
to
get
to
to
what
they
really
want,
which
is
just
the
logs.
You
show
me
the
logs
if
these
failed
and
how
to
get
there,
we
made
everything
really
really
push
button.
A
Elastic
so
when
there
is,
whenever
there's
a
lot
of
demand,
we
start
firing
up
servers
on
on
the
back
and
to
cope
with
the
demand
of
building
servers
and
building
packages,
and
this
all
happens
like
automatically.
We
have
some
services
that
are
checking
how
much
how
many,
how
many
nodes
he
needs
to
like
produce
the
packages
and
and
be
able
to
pull
through
in
in
all
of
these
scripts,
and
all
of
these
configuration
for
all
these
jobs
are
are
ephemeral.
So,
like
I
said
before
it
was
just.
A
You
know:
I'm
gonna
SSH,
on
to
these
one
server
and
fix
the
script
that
has
these.
You
know
syntax
error
or
we
want
to
do.
We
want
to
add
an
RF
condition,
because
we
want
to
release
a
new
version
and
he
has
a
new
dependency
that
we
didn't
have
before.
So
we
need
an
if
conditions,
the
SSH
here
and
fix
it.
So
we
made
it
ephemeral.
So
all
of
these
things
are
just
going
away
after
fifteen
minutes,
you
don't
have
a
server.
A
So
it
was
a
shift
in
thinking.
How
can
we
manage
infrastructure
that
that
that
scales
up
so
the
other
portion
of
these
is
that
we
have
these
huge
dashboard,
if
you,
if
you're,
if
you've
ever
contributed
to
save
you
might
have
looked
at
this,
is
called
shaman
that
says
calm
and
it's
an
HTTP
API
and
a
dashboard
and
a
load
balancer.
So
all
of
our
systems
are
pushing
information
here.
So
we
know
how
many
builds.
A
A
So
the
other
thing
was
that
we
did
automatic
server
pools,
and
what
for
for
these,
specifically,
what
it
means
is
that
we
were
able
to
fire
up
all
of
these
bill
notes
and
automatically
join
yankin.
So
there
was
zero
interaction
before
he
would
take
us
like
between
15
and
20
minutes.
So
manually,
like
oh
man
like
we're
dragging
here.
A
The
most
we've
seen
is
up
to
like
500
posts
per
day,
so
we
are
able
to
to
produce
massive
amount
of
amounts
of
repositories
is
not
only
safe.
We
have
a
lot
of
other
projects
that
started
like
joining
after
self
that
we
are
also
producing
and
we're
also
interested
in
producing
repos
for
NFS
Ganesha
is
one
of
them.
For
example,
I
thinks
if
I
think's
offense
always
it's
here,
but
there's
not
a
lot
of
other
projects,
and
the
good
thing
is
again:
it's
the
the
normalized
API.
A
So
if
you
want
to
go
to
save
the
master
branch,
the
latest
shot
one,
you
can
go
there
in
the
same
way
that
you
go
for
for
other.
For
other
projects,
so
we
made
it
go
faster.
We
went
from
from
from
up
to
a
day
and
a
half
to
initially
45
minutes
with
the
new
dependencies
and
extra
stuff
that
the
Ceph
project
has
been
adding
we're
now
up
to
an
hour
and
a
half
well
think
about
like
the
the
exponential
shifting
in
in
speed.
A
So
we
went
from
like
almost
a
day
waiting
for
repo
to
just
45
minutes
and
most
of
that
time
was
basically
spent
in
creating
the
the
binaries
and
we're
able
to
scale
and
make
it
fast
and
in
and
by
scaling
up
and
down
well
they're,
definitely
cheaper
and
more
efficient.
So
and
there
was
order
and
structure,
and
when
things
are,
you
know
following
a
certain
pattern,
it's
it's
it's
easier
to
scale
because
someone
else
comes
in
into
the
team
and
he's
new
and
like
well.
A
A
A
A
If,
if
I
was
doing
everything
correctly
and
in
the
worst
case
scenario,
it
was
like
a
week
to
cut
a
release
which
was
extremely
extremely
stressing,
for
example,
imagine
releasing
luminous
and
we're
trying
for
a
week
new
things
are
getting
into
the
repo
and
we're
just
trying
to
get
it
out
and
like
wow,
so
we
already
passed.
It
were
so
so
behind
that
things
are
getting
in
there
and
we're
gonna
merge
conflicts
when
we're
trying
to
polish
the
tag
so
so
I
think
we're
we're.
We've
made
a
lot
of
progress.
B
B
You
know
this.
Basically,
this
universe
is
expanding
of
the
ecosystem,
that's
around
seth
and
so
keeping
track
of
all
of
that
and
making
sure
that
those
things
are
building
and
the
changes
that
we
make
don't
break
those
things
and
the
changes
that
all
of
those
projects
make
don't
break
us.
You
know
we
want
that
user
experience,
the
whole
user
experience
to
be
as
good
as
it
can
be,
as
well-documented
as
it
can
be,
and
so
on,
but
tracking.
B
All
of
that
is
a
challenge
like
Alfredo
is
saying
you
know
we
want
to
add
more
and
more
of
those
things
like
rook
and
the
CSI
projects,
and
these
kind
of
things
and
and
so
that
it's
basically
these
things
are
expanding
in
several
dimensions.
The
the
list
of
projects
is
just
one
of
them,
and
the
architectures
is
the
is
another
one.
So
Alfredo
you
can
speak
real
briefly
about
power.
Yeah.
A
So
we
we
had
issues
with
the
arm
builds.
We
we
used
virtualization,
we
were
using
OVH
I,
think
at
some
point
to
produce
the
nodes
to
build
and
we
wanted
to
get
arm
builds
out.
So
the
first
initial
strategy
was
like:
oh
let's,
let's,
let's
use
OpenStack
to
get
some
arm
VMs
and
it
turns
out
you
can
because
it's
tied
to
the
architecture
of
the
hosts.
If
there
are
no
holes
with
arm
that
can
provide
virtualization,
you
can
virtualized
and
have
virtualized
VMs
on
arm.
A
So
we
were,
we
were
I
mean
we're
like
we
can't
and
then
I
was
trying
to
find
and
the
name
of
the
company
that
helped
us
out
and
donate
it
armed
servers.
So
we
have
a
couple
of
farm
servers
that
were
donated
to
produce
builds
and
we
use
those
and
we
were
able
to
finally
produce
Bruce
bills
based
on
arm,
but
these
are
actual
physical
servers.
A
Now
it's
again
it's
a
challenge
because
sometimes
like
these
bass,
basically
they
are
very
slow
because
of
the
ARM
architecture
and
building
self
takes
a
while
and
sometimes
they
fail
and
it's
hard
to
keep
up
with.
But
you
know
like
this
is
another
like
Ken
was
saying:
this
is
another
thing
in
the
matrix
of
building
all
set.
That
makes
makes
it
really
really
tricky.
B
So
another
thing
that
is
also
expanding
is
just
the
number
of
artifacts
and
the
type
of
artifacts
that
we're
trying
to
push
and
support.
So
you
know
when,
with
the
older
releases
it
could
be
older
versions
of
Ubuntu
or
sent
us
all
the
way
into
like
newer
types
of
content
like
containers
and
not
just
a
single
safe
container,
but
also
you
know
the
rook
container
and
the
CSI
containers.
You
know,
that's
just
it's.
It's
basically
going
to
expand
further
and
further,
but
basically
you
know
what
what
the
the
point
of
my?
B
What
I'm
saying
today
really
is
that-
and
this
is
a
you
know,
we're
getting
into
like
the
forward-looking
statements
here,
but
basically
I
want.
The
this
build
system
has
been
able
to
scale
really
well
like
we
we've
had.
You
know
up
to
500
repos
a
day
that
type
of
load
we've
tried,
like
you
know,
integrating
with
different
OpenStack
environments
or
different
things
and
oftentimes.
B
So
you
know
the
old
way
was
really
like,
like
you
were,
describing
like
SSH
into
the
system
or
just
give
route
access
to
whoever
to
go
like
login
and
fix
whatever
you
need.
Ninja-Style,
and
the
new
way
is
really
like
where
we're
headed
is
with
well-documented
api's
and
integration.
Points
for
building
SEF
on
the
platforms
that
you
care
about
so
really
great
example
of
this,
so
that
I
want
to
extend
even
further
is
with
the
FreeBSD
stuff,
where
we
have
a
member
of
the
community.
B
That
really
cares
about
making
Ceph
work
on
FreeBSD,
and
so
he
runs
his
own
Jenkins
instance.
In
order
to
build
FreeBSD,
artifacts
and
I
think
that
that
is
a
great
model,
because
he
is,
he
is
so
close
to
the
platform
that
he's
able
to
solve
the
integration
issues
a
lot
more
quickly
than
than
then
our
team
can
so
really
practically.
You
know
that,
currently,
what
that
would
involve
is
like
setting
up
polling,
so
I
think
that
his
setup
in
this
example
uses
polling
for
github
and
and
polling.
B
The
world
is
very
inefficient,
especially
when
it
comes
to
poor
requests
and
these
type
of
things
so
basically
I.
There
is
the
Fedora
project,
which
is
in
a
project
that
builds
the
Fedora
Linux
distribution
and
they
have
a
message:
bus,
the
Fedora
message,
bus
and
this
one
of
the
integration
points,
for
this
allows
you
to
send
all
github
events
into
this
message.
Bus.
So
that
way,
any
consumer
can
can
connect
to
this
bus
and
receive
events
so
I
like
when
we
create
pull
requests
and
stuff.
B
When
somebody
comments
on
a
pull
request,
when
they,
you
know,
CIA
passes
or
whatever
these
events
get
sent
on
the
bus
and
then
there
is
also
a
Jenkins
plugin
for
this.
That
a
lot
of
teams
in
Red
Hat
use
that
allows
us
to
listen
for
these
type
of
messages,
filter
them
and
and
trigger
events
on
the
ones
that
we
care
about.
So
then
that
is
actually
live
today.
B
We're
sending
all
the
Ceph
messages
to
the
forum
message,
bus
and
so
basically
I
I
really
see
the
next
level
of
scaling
this,
instead
of
just
having
like
a
single
Jenkins
or
very
you
know,
restricted
Jenkins
to
having
like
a
federated
system
of
or
listeners
that
we'll
be
able
to
build
even
more
things.
And
then
the
second
thing
that
I
that
I
see
is
an
opportunity
for
integration
is
really
the
community
build
systems
and,
like
I
said
you
know,
we
we
are
building
like
a
ton
of
things,
all
the
time
and
SEF.
B
You
know
it
takes
a
very
long
time
and
it's
very
CPU
and
since
it's
built-
and
you
know
it's
it's
getting
longer
and
longer
as
the
new
releases
go
on.
But
you
know
and
I
think
traditionally,
when
we've
gone
to
these
different
community,
build
systems
and
said
hey,
can
we
just
like
you
know,
build
400
repos
a
day?
They
either
say
please
go
away
or
you
know,
or
maybe
we've
tried
it
and
and
things
are
breaking
already
and
they
say
please
stop.
Please
turn
that
off.
What
are
you
doing?
B
B
Basically,
one
of
the
ways
in
particular
that
I
see
that
working
out
is
with
the
the
CentOS
community,
has
a
community
build
system,
that's
available,
for
you
know
any
open
source
project,
and
there
have
been
some
members
in
this
in-house
community
that
I
built
stuff
in
this
for
a
very
long
time.
I
think
that
we
could
integrate
even
more
deeply
with
this
and
there's
more
information
about
this
on
the
wiki.
B
A
I
think
there
are
a
bunch
of
lessons
learned,
I,
think
I
went
through
a
couple,
I
I
think
it
was.
We
were
able
to
pull
through
trying
to
build
some
that
scale
for
our
team.
I
can
mention
the
the
the
next
milestone
is
how
to
feather
ATS,
expanded
and
open
our
API,
so
that
we
can
we
can.
We
can
receive
bills
from
others.
The
freebie
is
the
example,
is
perfect
for
these
I
think
that
we,
you
know
I
kind
of
failed,
not
seen
that
as
oh,
like
you
know.
Why?
A
Don't
we
think
about
these
from
the
start
and
put
these
things
into
place
and
make
that
happen,
but
part
of
the
I
think
lessons
learn
is
like
well,
you
know
thinking
about
the
community.
First,
you
know
in
in
an
upstream
kind
of
fashion.
I
think
it
was.
It
was
one
of
the
things
that
made
me
think
about.
A
A
A
When
you
know
you're
putting
a
lot
of
good
work
on
these
and
and
someone
is
being
able
to
install
safe
and
have
a
good
experience,
that's
great
and
when
they
don't,
because
you
know
our
signing
process
missed
an
RPM
or
a
Deb
and
the
repo
is
kind
of
incomplete
and
he
crashes
or
it
just
fails.
You
know
that's
that
that's
really
hard,
so
we
always
trying
to
improve
they're.
B
B
Sure
I
can
yeah
I'm
gonna
go
ahead
and
repeat
widows
question
here
was
basically
it's
getting
more
and
more
tricky
to
build
SEF
on
your
own
outside
of
some
of
the
infrastructure
that
we've
described
here
today.
Does
that
reflect
what
you
said?
Okay
sure
I
agree
with
you.
It
is
very
tricky
to
build
stuff.
It
typically
like
melts
down
my
laptop
every
time.
B
I
try
to
do
it,
but,
apart
from
that,
in
terms
of
just
the
the
step
by
step
instructions
yeah,
we
could
definitely
do
a
better
job
of
highlighting
that
and
the
readme
for
staff
as
well
as
I,
think
there
might
be
some
like
a
researcher
tech
talks
about
this
thing.
Yeah.
We
need
to
do
a
better
job
of
that.
I.
B
Think
that
you
know
Alfredo
pointed
out
this
F
build
repository
is
the
canonical
location
for
all
of
the
stuff
that's
running
in
Jenkins
like
if
you
want
to
see
how
we're
building
stuff
you
can
just
look
in
github
and
see
that's
all
of
our
Jenkins
configuration
is
in
the
open.
Does
that
answer
the
question
sure
yeah
suet.
A
A
A
That's
that's
our
systems,
our
nodes,
building
for
the
distros
and
architectures
that
we
want
in
a
specific
way,
so
I
think
one
way
that
we
see
that
is
for
in
again
using
the
freebsd
example
is
we
we
would
allow,
with
with
with
certain
specific
permissions
the
the
freebsd
maintainer
to
push
updates
to
when
he
can
cut
releases
and
he
can
or
he
can
have.
Development
builds
push
notifications
into
the
into
the
dashboard
so
that
other
members
in
the
community
can
can
can
can
benefit
from
that.
A
B
B
I
think
Alfredo
you're
gonna,
stop
writing
bug-free
code
any
day
now,
right,
I
I
think
that's
a
really
tricky
problem
to
solve.
Once
you
push
something
to
unpause,
it
I
think
that
I
would
I
can
answer
for
myself.
I
would
like
to
make
the
release
process
a
lot
more
lightweight
so
that
it's
it's
simply
a
matter
of
yeah.