►
From YouTube: End-to-end Pipeline Tests for GitLab
Description
Presenting the Remote Pipeline Test Framework to the GitLab EMEA Customer Success team
https://gitlab.com/sri-at-gitlab/projects/remote-pipeline-test-framework/framework
Remote Pipeline Test Framework is an end-to-end test framework for pipeline template authors. Trigger remote pipelines and make assertions.
A
All
right
all
right,
I'm
going
to
share
my
screen
now
and
I'm
going
to
share
with
you
a
project
called
remote
pipeline
test
framework.
I
hope
you
can
see
the
readme
file
this
project.
If
I
was
to
describe
it
in
four
words:
it's
it's
a
tool
to
trigger
pipelines
and
make
assertions,
so
it
lets.
You
have
end-to-end
testing
for
pipelines
that
you
author.
Why
would
you
want
this?
A
A
So
we
have
this
tool
called
remote
pipeline
test
framework
you're,
seeing
a
lot
of
snake
art
over
here
and
two
reasons.
Pipelines
are
kind
of
like
snakes
and
that
this
tool
is
written
in
python.
So
I
thought
it's
only
appropriate
to
have
this
so
being
written
in
python.
You
can
install
it
via
pip.
So
if
you
do
pip
install
rptf,
you
get
this
locally
on
your
machine
and
you
can
execute
it.
But
before
you
execute
you're
supposed
to
write
your
test
cases
and
those
are
typically
done
in
a
file
called
rptf.yaml.
A
Here's
a
quick
example.
Each
block
here
is
a
test
case.
A
test
case
has
two
parts.
The
first
is
the
trigger
part,
so
you
specify
which
projects
pipeline
you
want
to
trigger
you
pass
branch
or
tags,
or
things
like
that.
If
you,
if
you
have
a
special
branch
to
work
with
and
then
you
make
a
series
of
assertions,
so
that's
the
rough
idea
of
what
what's
been
built.
A
The
only
thing
you
need
for
this
to
work
is
a
gitlab
access,
token,
with
the
right
permissions
and,
of
course,
a
little
bit
of
a
detailed
example
of
what
you
can
do
when
it
comes
to
authoring
tests,
so
trigger
part
is
easy,
so
you
can
have
a
trigger
for
a
project
id
by
default.
The
branch
would
be
master,
but
you
can
specify
the
specific
branch
that
you
want
to
work
with
as
well.
This
is
actually
a
ref,
so
it
also
can
be
a
tag
name
or
a
comment.
A
If
you
look
at
the
second
section,
that
is
assertions
you
can
assert
for
pipeline
status,
the
number
of
jobs
you
expect
the
pipeline
to
have
individual
status
of
each
of
the
jobs.
So
that's
the
dict
over
here
or
the
number
of
artifacts
or
environment
statuses,
of
the
environments
that
are
provisioned.
A
D
A
A
All
right,
okay,
so
I
have
the
latest
version,
which
means
I
have
my
tool
available
locally.
What
I
also
have
is
a
yaml
file
right.
So
if
I
look
at
my
rptf
yaml
file
here,
you
can
see
that
I've
got
few
jobs
defined
first
job,
it's
supposed
to
trigger
a
pipeline
with
three
jobs
and
should
succeed
second
job,
three
jobs
again,
a
second
test.
Three
jobs
again
pipeline
should
fail
with
the
jest
test.
Failing
and
third
is
that
two
jobs
and
it
should
be
a
success
so
executing
this
is
very
simple.
A
A
Yeah,
that's
that's
a
good
question.
I
think.
Ideally,
you
would
expect
your
pipeline
to
fail
in
certain
situations
right-
and
this
is
this
could
be
like
just
a
test
project
out
there
or
a
test
branch
out
there,
which
is
configured
in
a
way
that
the
pipeline
should
fail.
But
then
you
want
to
assert
that
it's
failing
for
the
right
reasons.
I
mean
it
should
fail
for
the
reason
that
you
expect
it
to
fail.
A
So
this
lets
you,
I
would
say
the
way
I
am
using
it
is
that
I
have
a
main
pipeline
template
project
that
I'm
working
on
and
then
I've
got
other
projects
at
the
side
which
have
got
many
branches.
Some
of
them
should
succeed.
Some
of
them
should
fail,
and
my
main
project
contains
my
test
case.
So
the
main
project
contains
the
rptf
yaml,
which
points
to
these
other
projects
as
test
cases,
so
it
triggers
pipelines
over
there
and
expects
those
to
fail.
A
A
D
A
A
A
But
it's
not
like
a
real
thing.
This
is
a
real
thing.
It
triggers
a
pipeline,
so
you
want
to
make
sure
that
your
test
assumes
that
the
pipeline
is
going
to
be
triggered
and
executed,
so
be
careful
of
what
you
tested
where
you
test.
Sometimes
you
want
to
test
something
without
triggering
the
pipeline.
Then
you
would
not
use
this.
Then
you
would
probably
rely
on
the
unit
testing
the
shell
script
or
whatever
script
you
have
in
your
pipeline.
I.
B
Hope
that
was
clear,
yeah
yeah,
because
why
I'm
asking
I'm
just
thinking
about
a
use
cases
I
could
use
that
framework
and
if
you
could
tell
a
little
bit
more
about
the
use
cases
when
where
this
becomes
applicable.
That
would
be
absolutely
wonderful
right,
because
I
understand
what
it
does,
but
I'm
not
clearly
getting
the
use
case.
A
Yeah
yeah,
that's
that's
a
fair
question.
So
just
imagine
that
you
are,
you
are
part
of
a
devops
team
in
a
large
organization
and
your
team,
or
you
are
responsible
for
creating
pipelines
for
many
other
teams
out
there,
and
you
want
to
test
your
pipelines
that
they
work
as
expected.
So
you
want
to
automate
the
testing
part
you're,
a
pipeline
template
author
you're
responsible
for
writing
pipeline
templates
for
a
lot
of
other
teams,
and
you
want
to
make
sure
they
behave
the
right
way.
A
That's
one
use
case.
Another
possible
use
case
is
basically
pipelines
are
automation
right
and
you
want
to
have.
Let's
say
you
probably
have
recurring
pipelines
every
x
days
x
weeks,
whatever
you
want
to
make
sure
they're
doing
what
they're
doing
we
have
a
simple
approach
of
a
pipeline
failure,
but
then
can
you
go
deeper
in
your
assertions?
Can
you
assert
which
job
of
the
pipeline
has
failed,
and
why
can
you
assert
whether
it's
created
the
artifacts?
It's
supposed
to
create
or
not
things
like
that?
So
when
you
want
to
have
deeper
control?
A
My
personal
story
with
this
was
that
I
was.
I
was
writing
the
five-minute
production
thing
and
I
did
not
have
a
good
way
to
test
it.
I
really
did
not
have
a
nice
way
of
asserting
how
the
pipeline
should
behave
across
different
branches,
different
environments,
different
settings.
So
for
me
it
was
like
I'm
a
I'm.
I
was
a
pipeline
template
author.
I
wanted
to
test
thoroughly
what
my
pipeline
does
under
different
contexts.
B
Okay
and
just
last
question
here,
as
far
as
we
are
talking
about
the
framework,
do
you
have
any
particular
plans
to
extend
in
the
frameworks,
for
example
in
terms
of
parameterizing
the
execution,
so
it
becomes
actually
a
test
case
driven.
So
let
me
just
explain
and
give
you
a
little
bit
more
context
here.
B
So
let's
say
you
now
you
you
define
the
outcome
pretty
much
hard
coded
right
in
in
the
pipeline
definition,
so
expected
results,
I'm
I'm
just
thinking
about
introducing
some
sort
of
a
variable
so
because,
when
you
want
to
assert
how
the
pipeline
is
executed
under
certain
conditions,
you
want
to
be
able
to
parameterize
it
right
somehow
and
based
on
those
parameters.
You
want
to
see
the
outcome
and
assert
that
outcome.
A
A
great
I,
I
agree
so
there's
a
lot
that
can
be
extended
upon
here.
You
know,
I
think
vladimir
salbo
also
had
some
good
ideas
earlier
last
week.
There
are
there's
a
lot
we
can
do.
This
is
basically
you
know,
pipelines
are
automation,
and
this
is
like
an
automation,
testing
framework,
so
the
roadmap
in
theory
could
be
great.
I've
just
released
the
first
version
make
it
available
to
this
kind
of
feedback
and
then
based
on
which
direction
it
should
go.
It
will
thank
you
for
that.
B
Okay,
well
it
it
it's
great.
I
think
it
has
a
great
potential
right.
So
the
only
thing
that
I'm
thinking
of
what
I
would
want
to
see
is
actually
the
mocking
part,
because
in
this
case
that
would
be
absolutely
fantastic
tool
if
you
could
just
mock
out
certain
things,
because
now
it
creates
everything
right.
It
runs
the
pipeline.
So.
A
What
I
would
ask
you
to
do
is
like,
if
you
can,
if
you
can
create
an
issue,
I
will
send
you
a
link,
or
I
will
show
you
a
link
as
part
of
this
demo
to
the
project
and
if
you
can
just
create
an
issue
describing
what
you
mean
by
this,
like
what
you
mean
by
mockery.
It'll,
be
great
it'll.
Give
me
a
good
idea.
A
Yeah,
thank
you
for
those
questions.
While
I
was
answering
them
the
pipeline
actually
executed.
So
let's
just
quickly
have
a
look
at
what
happened.
It's
we
had
these
three
test
cases
right
so
for
each
of
the
test
cases
it
triggered
three
pipelines
so
sorry
for
each
it
triggered
one
pipeline,
so
a
total
of
three
pipelines
were
triggered
and
then
for
a
long
time
it
was
just
waiting
to
see
the
pipeline
finish
execution.
A
So
this
stage
it's
just
waiting
to
see
the
pipeline
finish
execution
and
then,
once
the
pipelines
finished
executing
it
made
all
the
assertions.
So
it
made
three
types
of
assertions
it
made
assertions
on
pipeline
status,
job
count
and
job
statuses
itself.
That's
a
quick
demo
of
the
execution
of
the
cli.
I
will
quickly
bring
your
attention
back
to.
Let's
see
back
to
the
read
me
where
I
can
walk
you
through
the
rest
of
the
readme.
We
saw
the
test
output
in
detail.
I'm
just
going
to
skip
through
this
part.
Fyi.
A
This
tool
creates
a
junit
compatible
format
if
it
has
got
failing
tests,
which
means
that
I
can
specify
the
junit
report
in
my
gitlab
ciaml,
which
means
it
integrates
nicely
with
merge
requests.
So
here's
an
example
of
a
merge
request
which
has
got
some
failing
tests
failing
rptf
tests
and
I
can
expand,
and
I
can
it
is
telling
me
that
hey
listen.
It's
got
three
failed
tests
nicely
and
deeply
integrated
into
my
mr
workflow.
A
I
can
view
the
full
report
which
takes
me
to
my
pipeline
page.
I
select
my
test
tool
and
then
I
can
select
particular
tests
that
I
want
to
see
the
details
for
that's
that.
What
else
do
I
have
yeah?
Finally,
is
the
contributing
section.
So
a
lot
of
you
have
a
lot
of
good
ideas.
The
url
of
the
project
is
is
shared
in
the
document.
You
have
that,
please
add
issues,
tickets,
contributions,
suggestions,
documentation.
These
kind
of
things
are
more
than
welcome.
That's
all.