►
Description
Lightning Talk: CI/CD for Data Pipelines with Argo Workflows - J.P. Zivalich, Pipekit
Making sure that data pipelines don't break in production is no small feat. To combat this, we are seeing a new paradigm emerging in the data space influenced by modern software engineering discipline. This paradigm borrows heavily from CI/CD concepts, but applies them to data pipelines. In this talk, we'll go over how to implement CI/CD for data pipelines with Argo Workflows as the data pipeline orchestrator. We'll cover storing Workflows and WorkflowTemplates in git, validating them on pull requests, and syncing them to a cluster or clusters when releasing new versions of the data pipeline. Additionally we will cover how to test WorkflowTemplates using sample data, allowing teams to be sure that all base, corner, and edge cases are validated every time that they make a change to their production workflows.
A
All
right,
so
my
name
is
JP
zivilich.
Today
we're
going
to
be
talking
about
CI
CD
for
data
Pipelines
using
Argo,
workflows,
cool,
so
a
quick
little
agenda.
I'm
gonna
have
to
speed
through
this
just
so
that
we
like
stay
on
time.
But
here's
what
we're
going
to
talk
about
like
what
is
CI
CD
for
data
pipelines,
what
it
solves
and
then
we're
going
to
do
a
quick
like
demo
cool
all
right.
A
A
So
real
quick
about
me,
my
name
is
JP
zivilich
I'm,
the
CTO
and
founder
of
pipe
kit
and
also
amateur
Harry
Potter
impersonator
didn't
land
is
quite
as
good
as
I
was
expecting,
but
okay
at
pipekit,
we
are
a
control
plane
for
Argo
workflows,
targeted
at
data
teams.
So,
if
you're
trying
to
manage
several
different
clusters-
like
let's
say
a
Dev
staging
in
a
prog
cluster
or
you
have
different
clusters
for
different
teams
or
you
have
different
clusters
for
different
customers,
we
help
with
that
use
case.
A
Also,
we
do
CI
CD
for
data
pipelines
out
of
the
box
as
part
of
our
SAS
product,
but
within
this
talk,
we're
just
going
to
be
doing
the
open
source
cool
so
without
further
Ado.
What
is
CI
CD
for
data
pipelines
so
could
I
get
a
quick
show
of
hands?
Who
here
is
doing?
Data
pipelines
or
data
processing
cool
that
is
less
hands
than
I
was
expecting
to
see?
A
Can
I
get
another
show
of
hands?
Who
here
is
doing
automated
testing
on
parts
of
those
pipelines
before
pushing
to
prod
a
all
right?
That's
what
I
thought
it's
not
the
most
common,
even
though
a
lot
of
people
are
are
doing,
data
pipelines
most
of
the
time,
we're
kind
of
just
pushing,
maybe
to
like
a
staging
environment
and
then
maybe
do
a
prod
environment
and
really
hoping
for
the
best.
So
really
it's
we're
taking
our
data
Transformations,
just
pushing
them
to
the
higher
environments
and
fingers
crossed.
A
Ideally,
we
should
be
validating
these
transforms
when
we're
making
pull
requests
or
pushing
changes.
Also
rollbacks
are
currently
like
pretty
difficult.
It's
hard
to
go
from
like
hey.
This
is
the
version
that's
currently
on.
Like
my
prod
environment
or
my
staging
environment,
we
introduced
a
bug
with
this
latest
change.
Let's
roll
this
back
to
a
previous
version,
especially
if
you
have
a
big
team
and
you're
trying
to
manage
a
bunch
of
different
people
pushing
to
the
same
clusters
all
at
once.
A
So
a
lot
of
forward
thinkers
in
the
community
are
applying
these
CI
CD
Concepts
that
we're
seeing
with
traditional
software
engineering,
two
data
pipelines
so
they're
factoring
out
their
Transformations
and
other
critical
pieces
into
components.
The
run
tests
on
these
components
during
pull
requests
and
other
events
and
they're
also
versioning
these
components
using
semantic
versioning,
and
this
solves
two
big
problems.
One
is
money
and
the
other
is
time
or
you
can
flip
those
depending
on
which
you
think
is
most
important,
but
the
money
one
is
wasted.
A
Cloud
spend
Cloud
spend
is
pretty
intense,
I
know
a
lot
of
people
have
pretty
high
like
AWS,
gcp,
Azure
budgets
and
data
pipelines
can
really
rack
up
those
budgets
they
take
everywhere
from
several
minutes
to
over
a
month
from
some
customers.
We've
spoken
to
also,
if
you're
running
like
expensive,
gpus,
that's
another
thing
and
then
like
data
scientist
time,
like
let's
say
you
know,
you
push
a
change
and
there's
a
bug.
A
So
let's
say
you
have
a
part
of
a
workflow
and
you
want
to
refactor
it
into
a
piece
that
you
can
reuse
over
and
over
and
over
again,
that's
what
you're
going
to
use
a
workflow
template
for
now.
Why
would
you
want
to
do
this?
Really
you
just
don't
want
to
be
repeating
yourself
all
the
time,
so
one
workflow
template
that
we're
going
to
use
a
little
bit
later
is
one
that
just
clones
down
a
git
repo.
That's
a
common
task
that
you're
going
to
have
to
do
with
a
lot
of
different
workflows.
A
So
it
makes
sense
to
refactor
that
into
a
workflow
template,
also,
if
you're
using
data
or
doing
like
data
Transformations,
that's
another
good
candidate,
so
yeah
data
transforms
setting
up
tearing
down
of
kubernetes
resources
like
dasc
and
Spark
deployments,
if
you're
interested
in
how
to
do
that.
Stick
around
for
my
co-founder
kalin's
talk
coming
up
next
and
also
I,
give
a
talk
last
argocon
on
Das
deployment
and
utilities,
so
testing
these
components.
Now
a
workflow
template
is
really
kind
of
just
a
function
right.
It
has
inputs
and
their
outputs.
A
So
we
can
test
these
inputs
and
outputs,
and
if
we
want
to
do
that,
we
want
to
make
sure
that
the
workflow
templates
or
these
components
are
really
just
pure
functions,
meaning
that
for
a
given
set
of
inputs,
there's
the
same
outputs
right.
So
anything
that's
non-deterministic
or
random.
That's
what's
going
to
like
kick
a
problem
or
if
there
are
side
effects
to
this
function,
but
this
unlocks
testing
for
us,
which
is
great
real,
quick
before
I
get
into
the
demo.
I
want
to
talk
about
semantic,
versioning,
I'm
sure
everyone's
quick
show
of
thumbs.
A
Everyone
familiar
with
what
semantic
versioning
is
yes,
love
to
see
that
so
they're
great
to
have,
but
they're
not
available
in
vanilla,
Argo
workflows.
So
Docker
images
are
semantic
or
can
be
semantically
versioned,
but
workflow
templates
cannot
be
now.
We've
seen
two
ways
of
implementing
that
with
workflow
templates
and
Arco
workflows
the
first,
which
what
is
what
we
do
internally
and
with
our
SAS
platform,
is
appending.
The
version
to
the
name
with
dashes.
A
Are
you
doing
like
template,
Dash,
12-3-9
or
we've
seen
it
done
with
labels
and
annotations
shout
out
to
our
friends
at
Bloomberg
all
right.
So
now
we're
going
to
go
through
a
quick
example
of
doing
that,
CI
step
where
we're
going
to
push
a
change
to
a
workflow
template
in
a
PR
and
then
we're
going
to
use
Argo
events
and
Argo
workflows
to
test
that
out
and
we're
going
to
do
it
fast.
A
Here's
a
quick
architecture
diagram
that
I
don't
have
too
much
time
to
talk
about,
but
you
can
see
that
I'm
taking
like
a
GitHub
in
this
case,
pull
request
event
the
Argo
events.
Deployment
is
set
up
with
an
event
source
to
read
the
webhook
payload
loaded
onto
the
event
bus
and
then
the
sensor
is
going
to
spin
up
an
Argo
workflow
that
is
going
to
spin
up
a
parent
workflow.
A
And
then
the
sensor
is
the
the
start
player
or
the
point
guard
in
this
setup,
where
we're
taking
that
web
hook
and
then
we're
mapping
that
to
an
Argo
workflow
and
we're
also
extracting
some
information
from
the
payload
cool
all
right.
So
the
Argo
workflow
in
that
sensor
is
doing
some
really
cool
stuff.
What
it's
going
to
do
is
it's
going
to
clone
down
that
workflow
template.
A
In
this
case,
it's
going
to
be
a
quick
doubler
like
a
a
workflow
template
that
doubles
inputs,
and
then
it's
going
to
apply
that
workflow
template
to
our
cluster
and
then
once
it
has
applied,
it's
going
to
spin
up
a
additional
set
of
workflows
so
that
it
can
run
the
workflow
template
that
we
have
just
applied
several
times
with
given
inputs
and
outputs,
and
the
reason
that
has
to
happen
in
the
workflow
of
workflows.
Pattern
is
because
at
compile
time,
Argo
workflows
checks
a
workflow
to
validate
that
each
template
that
it
calls
is
presence.
A
So,
if
you're
making
modifications
to
your
templates
within
the
course
of
a
workflow,
you
actually
have
to
spin
up
other
workflows
for
those
modifications
to
take
place
and
I
found
that
out
when
I
was
making
this
demo-
and
that
was
a
fun
thing
to
do
all
right.
So,
let's
rock
through
this
demo
I'm
going
to
have
to
do
it
I
think
it
two
times
speed.
A
So
hopefully
that's
not
too
bad.
So
what
I'm
going
to
do
quickly
is
we're
going
to
make
a
change
to
a
pull
request.
Just
a
quick
readme
update
we're
going
to
push
that
commit,
but
this
file-
or
this
pull
request,
also
contains
a
workflow
template
called
doubler,
which
is
taking
an
input
and
then
multiplying
that
input
times
two.
So
if
we
have
two,
it's
going
to
give
us
four
four
eight
zero
zero
and
those
are
the
test
cases
that
we're
going
to
be
running
on.
A
But
you
see
that
we
have
that
pull
request
the
commit
just
got
updated
and
within
our
Argo
workflows
instance,
we
have
a
workflow
that
was
created
using
Argo
events,
cool
and
UI
is
being
a
UI
and
refreshing
cool.
Now
we
have
the
workflow
that
is
running
and
we
can
see
that
the
first
step
is
checking
out
the
GitHub
PR.
A
So
then
it's
going
to
get
the
template
and
apply
the
template
from
that
template.
It
then
spins
up
the
three
assertions
that
we've
specified
using
with
items
that
I'll
show
in
just
a
minute
cool,
so
those
are
going
to
run
successfully
and
I
will
pause
that,
so
we
should
be
good
on
that
part.
We
successfully
ran
some
assertions.
I
know
that
was
quick
trying
to
be
respectful
of
time.
Lightning
talk,
Bennett
bam.
A
So
now
what
we're
going
to
do
is
push
a
breaking
change
to
that
workflow
template
so
you're
going
to
see
me
pop
open,
vim
and
modify
that
workflow
template
to,
instead
of
multiply
by
two,
it's
going
to
add
two
to
every
input,
so
I'll
go
into
this
doubler
template
and
unfold
it
and
then
we'll
see
where
there's
the
multiply.
I'll
just
switch
that
to
an
ad.
This
is
going
to
be
simulating
a
breaking
change
to
a
data
transformation.
Then
we're
going
to
push
this
guy
again,
one
more
time.
A
A
So
we
can
see
that
the
one
of
the
cases
pass
right.
Two
plus
two
is
four
still,
but
the
other
one's
failed
right
and
then.
Lastly,
if
I
go
to
our
setup
here,
we
can
see
the
test
cases
that
we
did
this
with
with
items
where
we
specified
our
inputs
and
our
expected
outputs.
So
we
didn't
have
a
chance
to
like
cover
all
the
versioning
and
whatnot
and
show
the
example
of
that.
A
But
this
is
how
you
can
specify
a
set
of
inputs,
outputs,
invalify
or
validate
all
of
your
workflow
templates,
using
just
Argo
events
and
Argo
workflows.
Again,
that's
all
the
time
that
I
got.
My
name
is
JP
zivilich.
If
y'all
want
to
chat
a
little
bit
further
about
this
or
about
some
of
the
stuff,
we
do
come
holler.