►
From YouTube: How Salesforce is using ArgoWorkflows for provisioning add-ons - Mayank Kumar & Andy Chen
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from April 17-21, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
How Salesforce Is Moving From Spinnaker To Argo Workflows For Provisioning Add-Ons - Mayank Kumar & Andy Chen, Salesforce
Speakers: Mayank Kumar, Andy Chen
Many companies are thinking about automating application provisioning on Kubernetes but are stuck with old tools which cannot natively leverage the K8s apis and offer little scalability and extensibility. Mayank will show you how the Salesforce Hyper compute team that manages thousands of EKS clusters, is migrating Spinnaker pipelines that deploy K8s integrations like logging, monitoring and certificates to every EKS cluster on Hyperforce, to ArgoWorkflows. Mayank will talk through the existing problems and walk you through a step by step process, their team is using to convert all Spinnaker pipelines to Argo Workflow templates to improve reliability using custom retries, deployment velocity using memoization and using K8s features to gain more control over the various continuous deployment stages. He will also talk about a reusable library of ArgoWorkflow stages that is helping speed up the conversion process and making the developer experience around maintaining these deployment pipelines a true joy.
B
B
We
believe
that
business
is
the
greatest
platform
for
Change
and
one
of
the
ways
we
use
our
platform
for
change
is
through
our
111
philanthropy
model,
where
we
pledge
one
percent
of
our
time,
equity
and
product
to
giving
back
to
our
communities
a
little
about
Salesforce
and
hyperforce.
You'll
hear
this
word
a
lot
in
the
next
in
the
lot
in
the
rest
of
the
slides.
Hyperforces
are
trusted
Cloud
platform.
B
B
B
B
B
As
a
result,
we
need
an
efficient,
scalable
and
resilient
system
for
managing
our
clusters
and
cluster
components
to
achieve
our
mission.
So
what
are
add-ons
add-ons
or
kubernetes
Integrations?
In
our
internal
terminology
are
the
glue
that
makes
these
Integrations
possible
their
sales
for
specific
services
that
are
running
in
every
hyperforce
cluster.
B
B
B
B
How
do
you
do
that
efficiently
by
making
the
add-on
owners
Happy
by
improving
the
developer
experience
of
these
add-on
owners
by
giving
them
a
faster
Inner
Loop
and
a
faster
and
more
reliable
deployment?
Pipelines
in
our
industry?
We
talk
about
service
ownership
and
production
workloads,
but
we
rarely
talk
about
the
CD
pipelines
that
deliver
these
workloads
to
production.
B
B
Lastly,
as
the
number
of
clusters
keep
increasing
with
business
demand,
Central
CD
solution
is
not
easy
to
scale,
so
we
need
to
offload
the
last
mile
delivery
of
these
per
cluster
add-ons
to
a
more
scalable
model.
That
scales
horizontally
with
the
business
demands
we'll
be
looking
at
our
current
solution
from
these
angles
for
deploying
add-ons
and
then
describe
the
solution
we
are
embarking
upon.
B
This
is
our
current
system
for
managing
clusters
as
pets.
If
you
took
a
hundred
thousand
foot
view
of
our
deployment
pipelines,
it
would
look
something
like
that
infrastructure
in
terms
of
cluster
and
add-on
version
is
deployed
in
git.
Repos
is
sorry
declared
in
git
repos,
as
shown
on
the
left.
Spinnaker
pipelines
run
across
thousands
of
eks
clusters
to
deploy
these
add-ons
in
a
staggered
and
health
mediated
way,
as
shown
on
the
right
in
practice.
This
has
many
dimensions
and
look
something
like
this.
B
B
B
B
The
developer
experience
around
managing
add-ons
is
lacking
in
various
areas
like
no
inner
loop,
hard
to
learn
and
maintain
templating,
complex,
UI
and
and
slow
and
complex
and
debugging
is
harder
due
to
the
sheer
number
of
pipelines
and
nesting.
We
saw
in
the
previous
slides
we
need
to
make
our
CD
pipelines
faster
and
the
UI
more
responsive,
and
we
also
need
to
make
be
able
to
process
more
number
of
workflows
at
the
same
time
to
handle
more
clusters.
B
A
Thanks
Mark,
our
solution
is
to
offload
the
last
mile
delivery
of
our
add-on
provisioning
process
to
Argo
workflows.
Argo
workflows
is
a
container
native
cncf
incubating
workflow
engine
that
you
may
have
heard
of
that.
Some
of
the
other
talks
here
at
kubecon
as
a
reminder:
inaugle
workflows
each
workload,
you're
executing
where
each
execution
instance
is
defined
in
a
crd
instance,
called
a
workflow
and
usually
you
define
the
reusable
templates
or
stages
as
I
like
to
say
for
use
in
your
workflow.
In
other
crd
instances,
called
workflow
templates.
So
essentially
here
are
Spinnaker
pipelines
for
provisioning.
A
Each
add-on
become
our
goal
workflows,
as
you
can
see
right
now,
Spinnaker
will
essentially
handle
the
add-on
provisioning
for
each
of
our
add-ons
in
each
of
the
environments
for
the
high
performance
managed
clusters
on
the
bottom,
it'll
promote
it
through
the
dev
environment
and
then
through
tests,
and
then
for
higher
environments.
What
we
are
doing
now
is
we
have
Argo
workflows
in
the
mix.
A
We
still
use
Spinnaker
to
trigger
the
provisioning
process
because
it's
integrated
with
our
internal
infrastructure
registry
but
Spinnaker
will
hand
off
the
workload
to
a
Argo,
workflows,
controller
running
in
a
control
plane,
cluster
per
environment.
This
has
several
implications.
The
first
is
that
we
can
scale
across
our
environments
more
effectively.
We
even
have
the
option
to
Shard
up
the
number
of
control
plane
clusters
if
we
need
to
handle
more
pods
for
all
the
workflows
in
the
environment
in
workflows,
stages
are
usually
run
under
the
hood
by
pods.
A
So,
let's
take
a
closer
look
at
this
slide,
which
shows
the
setup
in
the
dev
environment,
although
it
also
applies
more
broadly
to
our
other
environments.
We
have
an
Argo
workflows,
controller
running
in
a
ETS
cluster
or
shards
if
we
need
them
and
it's
tied
to
a
Amazon,
RDS
or
Aurora
DB
database
and
S3
bucket.
This
is
because
our
workflow
stores
the
status
of
each
workflow
execution
in
the
status
field
of
the
crd
instance.
A
So
when
you
have
a
lot
of
particular
stages
or
clusters
and
add-ons,
they
can
get
rather
large
and
threatened
to
exceed
the
SCD
database
size
limit
in
the
cluster.
So
we
use
Argos
in
build
functionality
to
offload
workflow
statuses
and
archival
workloads
in
the
SQL
database
and
S3
bucket
is
just
there
for
any
artifacts
that
we
may
need
to
store
in
the
process
of
running
a
workflow.
A
A
So
on
the
next
slide,
we
see
that
the
use
of
workflow
templates
does
help
us
with
versioning
our
add-on
provisioning
processes
and
releases.
So
because
workflow
templates
are
yaml
objects
in
the
cluster.
Like
any
other
kubernetes
object,
you
can
store
mobile
versions
of
them
and
differentiate
them
by
name,
for
example,
you
can
suffix
them
with
release
one
or
release
three.
When
you
submit
a
provisioning
request
through
Spinnaker,
you
can
choose
which
release
is
being
referenced.
A
So
if
we
usually
use
release
three,
but
we
find
out,
there's
a
bug,
we
can
roll
back
to
release
two
very
easily
by
changing
the
reference.
It
also
allows
our
add-on
owners
to
more
easily
make
changes
to
their
provisioning
process,
as
defined
in
their
workflow
templates.
We
have
a
git
repository
where
the
workflow
templates
are
stored
and
add-on
owners
can
check
out
from
that
get
repository.
Make
changes
to
the
definition,
which
is
just
a
yaml
object,
as
workflow
templates
are
defined
and
then
Cube
CTL
apply
the
change
name.
A
A
So
this
is
improved
over
our
previous
process,
because
previously
add-on
owners
would
need
to
either
update
a
pipeline
template
repository
for
Spinnaker
pipelines
and
then
wait
for
the
changes
to
be
generated
before
testing
or
they
would
need
to
update
it
in
the
UI
and
copy
their
changes
over
and
make
sure
nothing
goes
wrong
in
the
process.
So
now
they
can
just
directly
edit
in
this
repository
and
keep
detail,
apply
and
create,
and
don't
worry,
we
have
controls
in
place
to
make
sure
that
this
testing
is
just
in
the
dev
environment.
For
these
test
clusters.
A
It
doesn't
go
past
that
and
have
any
security
issues
from
that.
As
for
why
we
chose
Argo
workflow,
specifically,
there
are
a
number
of
features,
some
of
which
I'll
just
go
over
in
the
slide.
The
first
one
is
that
there
is
overall
feature
parity
with
our
current
implementation
of
the
Spinnaker
pipelines,
which
is
important
for
allowing
us
to
migrate
over
more
smoothly
and
more
easily.
Are
the
workflows,
for
example,
has
support
for
different
stage
execution
ordering
based
on
your
directed
acyclic
graph,
trees
or
step
lists.
A
A
Compared
to
our
pipelines,
you
can
do
things
like
Define,
a
script
that
you
want
to
run
in
the
container
directly
in
the
workflow,
which
can
help
us
with
refactoring
anything
we
need
for
better
reliability
and
performance
and,
of
course,
Argo
workflows
is
also
a
kubernetes
native
which
helps
with
the
versioning
and
the
testing.
As
mentioned
earlier,
it's
more
familiar,
kubernetes,
tooling,
for
teams
like
us
and
for
add-on
owners
who
are
familiar
with
kubernetes,
but
it
also
means
that
all
the
workflows
can
natively
support
things
like
emitting
Prometheus
metrics.
A
So
we
can
get
better
visibility
into
workflow
execution
time
or
workflow
error
accounts.
It
can
do
things
like
set
pod
disruption,
budgets
on
particular
stages
to
help
guard
against
evictions.
It
can
do
things
like
the
next
important
feature
for
us.
Retry
failed
steps
in
case
of
transient
errors
in
our
Spinnaker
pipelines.
We
do
have
a
few
of
those
and
you
can
even
extend
it.
A
Are
the
workflows
also
has
memoization
and
caching
support
which
we're
working
on
implementing
right
now
to
help
improve
our
performance
time,
which,
as
you
saw,
continues
a
little
bit
of
improving
foreign?
Are?
The
workflows
also
has
been
immutable
and
has
simplified
nesting
of
all
the
nodes
in
your
workflow,
which,
as
you
saw,
was
an
issue
with
our
Spinnaker
pipelines.
When
you
submit
a
workflow
to
the
cluster,
Argo
will
look
at
the
template
store
inside
the
cluster
and
store
them
in
the
status
field
of
your
workflow.
So
there'll
be
immutable.
A
What
you
expect
to
run
when
you
submit
the
workflow
will
be
what's
actually
run
as
opposed
to
the
pipelines
where
a
change
in
the
middle
might
be
reflected
in
your
current
execution,
and
the
simplified
nesting
is
great
for
visibility
for
us
by
simplifying
down
all
those
child
pipelines
into
one
easily
viewable
object.
This
is
combined
with
the
UI
for
us
to.
Let
us
see
all
the
stages
and
any
failures
for
a
particular
add-on
in
one
place,
and
the
UI
also
has
other
features
that
want
to
use
like
filtering
out
executions
by
name.
A
But
these
are
the
preliminary
results
we
have
so
far.
Four
of
our
19
add-on
provisioning
pipelines
convert
to
workflows
right
now
and
it's
resulted
in
simplifying
35
Spinnaker
pipeline
executions,
which
are
the
parents
and
children
into
four
article
workflow
executions,
one
for
each
add-on,
which
helps
with
performance
and
with
visibility.
A
lot
no
need
to
look
through
child
pipelines
to
see
the
entire
add-on
provisioning
process,
there's
about
a
20
Improvement
in
add-on
provisioning
time.
A
B
B
B
At
this
point,
we
were
ready
to
start
asking
our
add-on
owners
to
begin
converting
the
Spinnaker
pipelines
to
workflows
templates
in
order
to
provide
an
example
and
a
conversion
guide.
We
even
took
the
work
for
one
add-on
owners
on
ourselves
and
finished
the
conversion
and
provided
that
as
an
example
for
other
add-on
owners
to
follow.
B
One
key
thing
that
required
our
Spinnaker
team's
help
was
to
integrate
Argo
workflow
processing
into
Spinnaker.
Spinnaker,
already
knows
how
to
deploy.
Health
charts
and
Argo.
Workflow
templates
are
just
another
hand
chart,
but
Spinnaker
also
needs
to
know
when
the
Argo
workflow
template
processing
is
complete
so
that
it
and
whether
it
failed
or
succeeded
so
that
it
can
decide
whether
to
move
on
to
the
next
Pinnacle
stage
or
or
halt.
B
The
last
two
are
very
much
work
in
early
stages
of
the
project,
but
we
we
demands
opportunity
to
optimize
and
scale
each
workflows
since
each
workplace,
essentially
just
a
kubernetes
powered.
So,
for
example,
you
can
give
more
resources
to
a
pipeline
or
a
workflow
executing
an
Argo
or
execute
them
with
higher
priority
or
not
execute
already
evaluated
executions,
or
if
you
have
identified
there
are
certain
special
pipelines
that
need
special
processing.
You
can
isolate
the
execution
of
these
workflows
to
special
note,
pools,
or
instance,
types.
B
B
The
first
one
is
to
focus
on
your
organization's
development
experience
measure
and
improve
your
service
owners
experience
in
maintaining
their
service
and
make
it
really
easy
to
deploy
them
to
production
in
a
fast
and
reliable
way.
At
the
same
time,
make
it
really
hard
using
tooling
to
make
mistakes
for
these
service
owners.
B
Our
service
owners
in
this
case
are
our
add-on
owners.
That's
what
we're
doing
for
them.
Second,
one
is
treat
your
CD
pipelines
that
deliver
your
services
to
different
environments,
as
Production
Services
itself
measure
their
slis,
publish
their
slos
and
alert
on
them
when
they
degrade
or
go
down,
they're
equally
important
assets
of
your
organization.
B
As
you
saw
in
our
previous
slides,
we
focus
on
the
CD
pipelines
that
deploy
add-ons
and
are.
We
are
actively
working
on
improving
their
slis
third
adopting
container
native
workflows
engines
like
Argo
and
tecton
into
your
existing
developer
tool
chains
will
make
it
easier
and
faster
to
deploy
your
services
with
confidence.
B
You
can
think
of
these
container
native
workflow
engines
as
many
developer,
Work
Platforms
that
can
help
unlock
a
lot
of
productivity
for
your
teams,
as
Andy
showed
in
the
previous
slides.
We
are
already
seeing
better
execution
times
faster,
Inner
Loop
and
a
lot
of
additional
benefits
that
we
could
not
even
imagine
reaping
from
our
old
system
and
the
last
one
is.
The
service
owners
have
widely
adopted
the
kubernetes
resource
model
in
and
various
other
Cloud
native,
open
source
tools
for
managing
their
workloads
in
production
on
kubernetes.
B
Their
existing
development
tools,
like
cube
cutter,
Opa
Helm
charts,
are
now
being
leveraged
for
various
newer
scenarios
like
managing
their
CD
pipelines,
managing
different
tasks
that
are
exposed
by
various
kubernetes
operators
doing
inner
loop.
So
essentially,
what
you're
doing
is
meeting
these
developers
where
they
already
are,
instead
of
giving
them
new
tools
and
forcing
them
to
move
to
proprietary
Solutions.
B
We
believe
that
kubernetes
resource
model
is
the
language
that
will
bind
the
next
generation
of
Industry
standard
tooling,
with
that
I
wanted
to
say.
Thank
we
wanted
to
say
thank
you
for
coming
to
our
talk
and
giving
us
a
chance
to
share
our
journey
towards
a
container
native
CD
solution
using
Argo
workflows.
A
C
B
B
It
has
a
lot
of
Spinnaker
that
is
already
very
deeply
integrated
into
the
rest
of
the
infrastructure,
and
a
simpler
answer
might
be
that
our
environments
do
not
have
access
to
git,
and
there
is
a
intermediate
step
that
happens,
so
it
is
not
easy
to
integrate.
Argo
CD
and
Argo
workflow
just
fitted
the
bill
in
terms
of
like
being
a
generic
pipeline
that
can
model
whatever
we
were
doing
in
Spinnaker.
D
B
D
B
So
I
think
we
are
very
early
in
that
stage.
We
haven't
made
it
to
production,
yet
we
are
in
Dev
environments.
We
are
like
you
said,
like
we
said
we
are
in
the
process.
We
are
providing
tools,
conversion
guides
and
like
how
Andy
showed
we
have
converted
four
add-ons
to
Spinnaker
Pipelines
I
think
what
would
you
say
is
a
like.
After
everything
has
been
identified,
the
amount
of
time
to
just
convert
would
probably
not
be
more
than
maybe
two
sprints
or
one
Sprint.
A
Yeah,
maybe
one
to
two
weeks,
but
I
speak
as
someone
who's
working
with
this
a
lot
I
think
a
lot
of
it
would
probably
be
just
from
getting
a
bit
more
used
to
syntax
versus
when
you've
been
working
with
Spinnaker
pipelines
for,
however
many
months
or
years,
you're
more
familiar
with
that.
A
lot
of
it
does
translate
over
pretty
like
smoothly,
though
so
I
wouldn't
say.
It's
that
much
effort
to
get
used
to
the
syntax,
I.