►
From YouTube: From PR to Artifactory in 60 Secs! Adobe Service Runtime Java Libraries Release Pi... Anirudh Mathad
Description
From PR to Artifactory in 60 Secs! Adobe Service Runtime Java Libraries Release Pipelines Using Argo Workflows - Anirudh Mathad, Adobe
- Adobe Service Runtime (ASR) Libraries are used by more than 60% of Adobe's Java Services. A strong community of contributors maintain the libraries across the company. Instant availability of a release once a PR is merged is mission critical for teams across Adobe. - Our legacy process used Jenkins jobs with Maven Artifactory plugins. The cross dependency of libraries, stale libraries in Jenkins VMs, lack of Graph/DAG view, scaling issues and container upgrades were some of the many problems which slowed our release process. - In this talk we discuss how we migrated our manual Jenkins release pipelines to Argo Workflows. How the visual representation of release pipelines through DAGs helped us troubleshoot build issues. Embracing Gitops has dramatically reduced our maintenance efforts and updates to pipelines have become seamless.
A
A
This
is
more
on
the
application
Level.
So
what
happens
in
ASR?
Is
we
bundle
a
lot
of
cross-cutting
concerns
that
you
would
want
for
a
production
grade
application
and
we
we
kind
of
ship
it
as
a
pawn?
So
if
you
are
an
adobe
developer,
who
wants
to
go
from
concept
to
Cloud
using
ASR?
Will
you
know
give
you
all
the
checklists
that
any
respectable
production
grade
software
will
require?
We
have
six
major
libraries
that
I
will
use
today
to
kind
of
demonstrate
how
the
release
pipeline
looks
like
and
I'll
talk
more
about
these
pythons.
A
The
next
slide
are
these
libraries
in
the
next
slide
and
almost
like.
We
have
a
wide
amount
of
services
at
Adobe
which
use
ASR,
so
the
big
deal
for
us
is
security
and
then
reliability
when
we
deliver
this
software
and
also
speed-
and
that's
where,
like
Argo
workflow
comes,
comes
into
play,
which
is
helping
us
like
push
these
artifacts
super
fast,
and
these
libraries
are
Community
Driven
inside
Adobe.
So
we
get
a
lot
of
community
contributors.
So
we
have
a
lot
of
PR
traffic
that
we
kind
of
merge
and
release.
A
So
a
sneak
peek
into
this
is
a
very
simplified
version
of
how
ASR
libraries
look
like.
So
we
draw
a
lot
of
Open
Source,
you
know
spring
Boot
and
Jersey,
and
all
that
libraries
in
a
dependency
bomb
and
that's
the
root
kind
of
the
library
that
ASR
shared
modules
that
you'll
see
there,
and
then
we
have
like
an
imperative
flavor
and
a
reactive
flavor,
and
the
imperative
flavor
is
the
very
classic.
A
So,
if
I
have
to,
if
I
made
a
change
like
a
spring,
boot
update
I'll
have
to
end
up
releasing
every
single
library
right
here.
So
that's
like
on
a
very
simplistic
way
of
looking
at
how
our
workflow
looks
like
the
development
process,
for
these
libraries
is
pretty
much
like
any
open
source
Library
the
maintainers
us
we
focus
on.
You
know
security
patches
and
then
we
have
our
own
roadmap,
which
we
think
are
like.
A
Moon
charts
are
like
big,
big
mountains
that
we
deliver,
and
then
we
rely
on
the
community
for
any
feature,
requests
that
and
then
we
kind
of
kind
of
crystallize
that
into
the
roadmap,
and
we
also
rely
on
the
community
for
enhancements
if
they
think
there's
something
minor,
they
can
just
raise
a
PR
and
that's
where
most
of
the
development
is
have
happening
these
days
right
now,
it's
completely
Community
Driven.
So
then
there's
a
concept
of
connectors
in
ASR
connectors
are
like
sdks
for
internal
Adobe
Services.
A
If
one
team
has
to
have
has
to
talk
to
a
service,
they
end
up,
writing
a
connector
and
then
ASR
is
the
place
they
come
to
share
it.
So
it's
like
kind
of
that
internal
cncf.
If
you
want
to
say
that
now,
since
we
have
a
lot
of
PRS
that
are
coming
into
these
libraries,
naturally
we
have
to
handle
like
a
lot
of
PR
traffic
in
terms
of
merge
conflicts
and
making
sure
with
the
releases
go
well
and
all
that
our
previous
solution
was
two
weeks
release
kind
of
ceremony
and
we
used
Jenkins
for
that.
A
A
So
we
had
one
Jenkins
job
for
per
Library,
so
that
means
I
have
to
babysit
every
single
release
Library
until
it
finishes
and
Jenkins
kind
of
works
out
of
that
plug-in
model.
Everything
is
a
plug-in.
You
want
to
pull
a
repo,
then
you
have
to
use
the
GitHub
plugin.
You
want
to
publish
to
an
artifactory.
You
have
to
use
the
artifactory
plugin.
What
this
does
is
it
kind
of
makes
your
Kickstart
like
the
hello
world
is
easy,
but
when
you're
stuck
in
something
deep,
some
problem-
debugging
is
hard.
A
One
classic
example
is
artifactory,
did
an
upgrade,
and
then
we
used
to
get
a
data
dump
on
the
kind
of
a
metrics
that
we
use
for
dashboards.
The
whole
thing
broke
because
the
new
upgrade
of
the
plugin
kind
of
stopped
doing
that.
Metrics
dump
things
like
that.
So
we
want
visibility
on
what
we
are
doing,
which
was
kind
of
hard
in
Jenkins
and
also
Jenkins
relies
on
the
Jenkins
agents,
which
is
kind
of
VM
based.
A
So
every
time
we
need
to
do
a
jdk
upgrade,
or
sometimes
we
swap
out
vendors
based
on
the
contract.
It
was
super
painful
to
do
that
in
a
VM.
So
a
little
bit
about
the
release
steps,
the
release
itself
is
pretty
straightforward.
We
pull
a
repo,
do
a
maven
install
like
you
know,
make
sure
the
build
works,
and
then
we
publish
to
an
internal
Adobe,
vulnerability
kind
of
a
dashboard
where,
based
on
the
palm
and
the
dependency
we
get
ticketed.
A
If
you
have
a
vulnerable
library
and
then
we
go
back
and
fix
it,
and
then
we
deploy
to
the
artifactory,
and
we
also
do
the
basic
release,
notes
and
all
the
stuff,
but
any
open
source
Library
does
so
what
were
the
expectations
from
this
Library,
the
maintainers
as
us
and
the
community?
We
have
two
different
and
we're
trying
to
merge
them,
so
the
community
wanted
everything
fast.
They
wanted
everything.
Yesterday
they
raise
a
PR
hey.
Where
is
my
release?
I
want
to
use
it
today
and
also
they
want
to
use
the
latest
dependencies.
A
Okay,
I.
Have
this
cool
feature
I
want
to
use
I,
don't
want
to
wait
for
your
two
week:
deadline
to
you
know,
release
and
us
as
maintainers.
We
wanted
to
work
on
cool
things
and
not
worry
about
babysitting
early,
so
we
wanted
it
to
be
self-service
and
again
the
pipeline
itself
was
too
hard
to
troubleshoot
I'll,
not
go
into.
You
know
Jenkins.
Why
is
it
hard
to
debug
and
looking
at
all
the
logs
and
all
that
stuff
and
I
think
the
key
thing
was
also
documentation.
A
If
any
maintainer
wants
to
do
a
release,
it
was
hard
with
the
Jenkins
setup,
because
you
don't
have
a
graphical
view
of
the
process
and
of
course,
controlling
cost
is
always
a
good
thing
in
any
organization.
So
thankfully
Argo
had
everything
that
we
were
looking
for.
It
has
dags,
it
is
github's
based.
It
gives
you
all
the
benefits
of
kubernetes
and
it
uses
workflow
s
code.
So
let
me
jump
into
what
exactly
happens
with
the
Adobe
IDP
before
actually
show
you
the
solutions.
So
Adobe
IDP
is
a
Adobe
internal
developer
platform.
A
Here,
any
developer
at
Adobe
can
come
in
with
a
concept
we
have
portal
which
will
help
you
generate
code.
We
have
different
flavors,
just
like
Sr
the
starters
for
a
spring
application.
Has
it
we
get
if
we
give
you
starters
like
Java,
node.js
and
Python,
and
then
we
have
something
called
blessed
based
containers
which
are
like
secure,
Adobe
wetted
containers,
which
every
single
application
at
Adobe
uses
are
more
or
less
and
then
then
comes
your
next
step.
Is
your
deployment
configuration?
So
if
you
that's,
where
you
decide
like?
A
Oh,
what's
how
many
environments
do
you
do
I
want
to
have
and
any
like,
submit
all
the
you
know,
PR
the
deployment
statuses
to
stack
channels,
things
like
that.
The
next
step
is
we
provision
your
based
on
this
deployment
configuration
whatever
you
mentioned
here
and
then
the
end
result
is
a
CI
CD
pipeline
again,
we
have
I
won't
go
in
through
the
more
details.
A
So
the
first
step
is:
we
need
an
Argo
git
repo
in
our
world.
That
is
this
is
where
all
the
configurations
are
kind
of
maintained.
So
we
make
you
fill
a
form
and
then
everything
is
stored
in
a
git
repo
and
we
have
a
service
called
provisioner
which
in
itself
is
an
Argo,
CD
kind
of
a
workflow
Runner.
What
it
does
is
it
it
follows
the
kubernetes
apply
principle
like
it.
It
takes
your
configuration
and
manifests
an
Argo
CD
application
based
on
what
you
enter
there.
A
So
the
big
picture
is:
if
we,
you
have
an
Argo,
CD
application
and
what
it
is
doing
is
it's.
It
contains
all
the
definitions
of
the
workflows
that
you're
using
to
release
so
every
ASR
git
repo.
Whenever
there's
a
merge,
that's
happening
in
the
repo
it's
going
to
send
events
to
the
Argo
CD
app,
which
is
going
to
trigger
a
workflow.
A
So
what
does
the
journey?
Look
like
from
a
PR
to
Rd
Factory,
so
the
step
one
you
raise
a
PR
and
then,
if
the
maintainers
and
everyone
is
happy,
you
merge
it
and
then
that's
where
the
events
start
flowing
to
the
Argo
CD
application
and
Argo
CD
application
has
I,
mean
I've
kind
of
or
simplified
it.
It
has
two
main
components:
there's
a
sensor
and
a
workflow
template
which
kind
of
intercept
the
GitHub
events
and
then
trigger
an
appropriate
workflow
which
goes
to
the
artifactory.
A
So
the
the
sensor
that
I
was
talking
about
it
looks
like
this.
So
in
very
simple
terms,
what
it's
doing
is,
is
this
event
coming
from
XYZ
PR
or
like
repo
number
one,
and
then
does
it
belong
to
the
main
branch
then
submit
a
workflow
because
you'll
be
getting
a
lot
of
traffic
to
that
those
events
and
the
reason
why
I
put
the
snippet
is
it's
super
simple
to
configure
this
workflow.
A
Sorry
the
sensor
and
that's
where
the
magic
of
whole
Argo
ecosystem
is
everything
is
describable
as
a
yaml
and
it's
kind
of
like
for
me.
The
aha
moment
was
when,
when
it's
what's
algebra
versus
geometry,
you
describe
an
equation
and
that
becomes
a
graph.
Something
similar
like
you
have
your
yaml
here
and
then
that
becomes
a
graph
on
the
right
and
the
next
step
is
submitting.
The
workflow.
So
what's
happening
is
once
your
hap
once
the
filtering
is
done.
A
There
is
one
workflow
for
one
repo
and
I'll
be
covering
about
how
each
and
every
single
repo
looks
like
the
workflow
for
each
and
every
repo,
and
then
we
pass
some
parameters
that
are
required
for
that
workflow.
So
if
repo
one
is
being
deployed,
then
I
need
to
know
what
are
the?
What
was
the
repo
and
you
know
what
was
the
GitHub
commit
and
who
was
the
person
who
merged
it?
A
Things
like
that
now
the
release
process
now
I
really
got
to
use
the
power
of
Argo
workflows,
because
I
could
paralyze
a
lot
of
tasks
before
I
had
to
babysit
each
and
every
step
and
then
go
manually
right
now.
I
have
this
kind
of
complex
workflow,
where
I
can
paralyze
certain
things
which
was
kind
of
linear
in
before
and
also
Argo
step.
Some
of
the
steps
are
impotent
and
retryable.
A
So
this
this
can
happen
in
your
workflow,
like
if
you're
talking
to
GitHub
your
Enterprise
GitHub
might
be
down,
and
then
you
don't
want
any
long-running
processes,
or
sometimes
our
Adobe
security
dashboard
gets
too
much
load.
Their
SLA
is
not
the
same
as
rsla,
so
you
might
want
to
throw
in
some
retries.
A
A
So
my
favorite
part
is
how
we
can
control
the
characteristics
of
an
atomic
step.
I
mean
that's
how
I
look
at
it.
So
an
atomic
step
is
that
one
job
that
you
want
to
split
across
if
you
have
many
jobs
so
here
like,
for
example,
like
one
step
which
says
verify
git
commit
on
my
right,
I'm
running
a
bash
script
for
this
particular
step.
I
get
to
choose
my
image
that
I
want.
A
So
this
is
where,
if
I
even
I
can
go
Alpine
Ubuntu
based
on
any
any
different
step
and
kind
of
what
what
meets
the
needs?
And
then
sizing
is
a
very
cool
thing
here,
because
not
every
step
requires
the
maximum
amount
of
CPU
or
memory,
so
you
can
kind
of
right,
size
them
and
resiliency,
which
I
talked
about.
If
you
want
to
retry
certain
steps,
you
can
throw
in
a
retry
kind
of
constraint
there.
A
So
this
is
probably
the
most
important
slide
in
this
talk.
I
would
say
because
what's
Happening
Here
is
this
is
the
comprehensive.
This
is
how
a
comprehensive
release
would
look
like
if
I
do
a
release
or
if
I,
do
a
PR
in
the
root
node
or
even
if
I
have
a
change
of
a
springboard
version.
I
have
to
end
up
deploying
every
single
repo
right
here
and
on
my
right
is
how
the
dags
would
look
like.
So
it's
a
dag
of
dags.
A
So,
even
though
one
repo
is
like
one
box
that
you
see
on
the
right
when
you
have
to
chain
all
of
them,
this
becomes
super
powerful.
Now
I
have
a
complex
workflow,
calling
another
complex,
workflow
kind
of
a
sort
of
inception,
if
you
might
say,
and
the
the
beauty
of
this
is
It's
kind
of
self-documenting.
If
someone
asks
me
hey,
how
does
your
release
pipeline
look
like
you
just
come
in
and
show
that
person
this
workflow
and
it's
kind
of
self
describing
not
self-destructive,
so
the
other
beauty
of
the
dags
is
these
are
dispensable.
A
So
what
I
mean
by
that
is
the
previous
slide
that
has
so
many
workflows
if
I
want
to
raise
a
PR
in
many
of
the
child.
Node
I
just
want
to
release
the
bottom
two
after
this,
so
I
don't
have
to
release
the
whole
comprehensive
release
right,
so
you
can
chop
up
the
workflows
and
maintain
them
as
independent
workflows,
which
do
its
own
releases
so
right
now,
what
I
have
is
every
PR
raised
to
any
anywhere
in
the
node
gets
released
and
each
of
them
are
self-living
workflows.
A
That
kind
of
you
know
our
sub
trees
of
the
same
parent
and,
of
course,
these
are
kind
of
written
in
a
Library
fashion.
So
when
we
do
these
workflows,
These
are
actually
Library
libraries
being
used
from
here,
but
but
really
the
efficiency
that
we
gained
is
in
this
chopping
up
of
these
sub
trees
and
then
making
them
kind
of
their
own
organisms
like
you
know,
deploying
that
workflow.
A
And
troubleshooting
is
a
big
thing
in
in
Jenkins
World,
which
we
Face
a
lot
of
problem,
because
something
breaks
I
have
to
start
from
the
line
one
of
the
logs,
because
it's
not
really,
you
know
transparent,
to
know
what's
happening.
But
here
since
my
steps
are
broken
down
in
a
graphical
manner.
Now
I
know
where
which
box
that
I
need
to
focus
on
and
that
I
just
need
to
focus
there
and
then
I
can
click
on
the
logs
and
I
know
what's
happening.
A
So,
overall,
the
workflows
like
what
you,
what
it
gives
you
behind
what
a
traditional
pipeline
would
give.
You
is
the
configurations,
so
you
can
do
retries,
which
you
talk
about.
Sometimes
our
tests
are,
since
these
are
Community
contributed
tests.
Some
tests
practically
fail
in
some
library,
then
it
it
works
on
a
retry,
mainly
because
someone
had
to
tune
their.
You
know
timeouts
in
their
test.
A
This
is
largely
seen
in
a
reactive
world
where,
when
they
write
not
everyone
follows
the
best
practices
and
your
test,
might
you
know
just
run
the
next
time,
and
also
it
kind
of
depends
on
how
you
loaded
the
CPU
is
so
the
retries
are
super
helpful
in
retrying
those
reactive
tests
and
timeouts.
Again,
these
are
any
standard.
A
Resiliency
feature
like
if
you
have
a
long
running
process,
because
you're
waiting
on
some
remote
to
do
something
and
that
their
times
times
you
are
you
can
time
out,
based
on
some
configurations
there
and
I
guess
the
cost
saving
is
something
which
is
we
never
really
thought
we
could
achieve
it
because
of
the
right
sizing
of
the
CPUs.
Now
a
reactive
Library
which
takes
less
CPU
can
be
allocated
less
CPU
and
similarly
the
imperative
ones
is.
A
You
know
you,
you
throw
higher
CPU
and
you
know
memory
over
there
again
unit
tests,
also
based
on
how
you
know
massive.
The
unit
tests
are
like.
We
have
unit
tests
which
runs
springboard
application
for
every
single
test
out
there
and
you
you
might
want
to
size
them
appropriately
for
that
and
the
compliance
and
usually,
which
is
the
boring
part
for
for
developers.
You
know
which
is
kind
of
made
it
easier
because
everything
resides
as
workflow
as
a
code.
A
Now,
if
some
security,
you
know
review,
happens
and
you're
asked
for
hey,
what's
the
version
you're
using
you
just,
you
can
just
hand
over
that
particular
code
and
you
know
what
container
you're
using
and
where
it's
deployed
and
all
that,
so
the
audits
are
easier
and
similarly,
our
the
swapping
out
of
jdk
vendors
was
one
pain
point
for
us.
Now,
it's
just
a
container
swap
out
and
you're
done.
A
I
guess
other
other
problem
that
we
kind
of
Overlook
in
Jenkins
is
the
my
namespace
is
my
workflow.
So
once
what
happened?
Is
somebody
accidentally
deleted
my
container
and
my
whole
workflow
stopped
instamp
functions
like
in
the
AWS
console,
but
in
this
Argo
World
your
workspace
have
your
own
workflow.
So
nobody
is
going
to
mess
with
that
area
and
then
you,
you,
don't
end
up
stepping
on
each
other's
foot.
A
So
the
the
workflows
are
not
I
mean,
as
you
can
see,
it's
very
generic.
It
can
be
used
for
anything
beyond
a
Java
release
pipeline.
You
can
have
a
serverless
cicd
pipeline
if
you
have
an
S3
bucket
and
also
ETL
pipelines,
if
you
may
so
to
conclude
most
of
our
gains
gain
from
git
opsification
and
using
k8s
to
its
fullest.
A
A
A
Hi
and
your
Maven
step
did
you
do
any
sort
of
optimizations
to
make
that
any
faster,
oh
yeah,
so
the
maven
steps
we
have
the
concept
of
threading
right,
that's
something
where
a
Jenkins
jobs
wouldn't
do
it,
because
you
couldn't
justify
the
CPU
allocations
there.
We
could
multi-thread
some
of
the
maven.
You
know
jobs
by
just
increasing
the
threats
there.
That's
probably
it
and
then
the
value
I
think
our
main
problem,
the
slowness,
the
build
itself
for
fast.