►
From YouTube: A Picture is Worth a Thousand Words: Bring Data Visualization in Backstage - Olivier Liechti
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
A Picture is Worth a Thousand Words: Bring Data Visualization in Backstage - Olivier Liechti, Avalia Systems
Software analytics is analytics on "software data", with the objective to give development teams insights about their codebase, their work practices, their collaboration patterns. "Software data" is all the data that is already available in the tools they use on a daily basis: code repositories, issue trackers, CI/CD pipelines, observability tools, etc.
Visualization of "software data" is relevant to an effective and enjoyable developer experience. It is useful to get insights, track progress, maintain awareness about technical and non-technical issues. It also makes the user experience visually pleasing.
The goal of this session is to share how we have brought software analytics and data visualization to Backstage. We first look at the overall architecture and process of extracting, storing and presenting information. We then take a closer look at the "last mile" and present the open source toolset that we use to create visualizations. In the presentation, we demonstrate a Backstage plugin that we have created to bring Vega.js visualizations in developer portals. We introduce the notion of "visualization grammar" and illustrate it with concrete examples.
A
So
one
objective
of
the
session
is
to
share
a
recipe
that
you
can
use
and
apply
to
bring
visualization
in
your
developer
portals,
but
before
that
I'm
going
to
provide
some
context
and
explain
why
we
are
interested
in
this
topic.
So
the
broader
context
is
software
analytics,
and
it's
something
that
we've
talked
about
during
the
day.
Even
if
we
have
not
used
the
term
software
analytics.
So
what
is
it?
A
It's
a
research
fields
that
is
active
for
more
than
20
years,
and
it's
basically
analytics
on
software
data
to
provide
information
to
software
development
teams
and
software
development
leaders
and
in
the
previous
sessions
we've
talked
about
all
these
data
that
is
available
and
that
can
be
presented
to
the
to
the
developers.
There
is
data
in
the
code
base.
There
are
metrics,
there
is
data
in
the
activity
that
developers
perform
and
in
green.
There
is
data
about
how
the
people
and
the
developers
feel
and
the
friction
and
the
satisfaction
that
they
that
they
feel.
A
A
So
this
is
a.
This
is
an
article
that
was
published
in
the
New
York
Times
several
years
ago,
and
it's
a
it's
an
article
about
the
the
taxes
that
company
pay
in
the
U.S
and
behind
the
visualization.
We
have
a
data
set
and
what
do
we
have
in
the
data
set?
We
have
companies
who
have
earnings
who
have
a
tax
rate
and
Obey
an
absolute
amount
of
of
tax,
and
what
we
see
here
is
a
non-conventional
data
visualization,
where
the
outer
the
visualization
expert
has
mapped
data
properties
to
graphical
attributes.
A
A
A
A
How
can
you
make
sense
of
this
data
and
how
can
data
visualization
help
with
you?
So
on
the
right?
You
see
some
of
the
benefits,
some
of
the
the
the
advantages
of
using
visualization.
You
get
faster
insights.
You
also
get
deeper
insights.
There
are
things
that
you
see
in
the
data
visualization
that
would
not
stand
out
in
the
standard
table
or
in
text,
and
it's
also
something
that's
very
enjoyable.
A
So
in
the
rest
of
the
session,
I'm
going
to
go
through
three
steps
and
the
idea
is
to
build
an
end-to-end
demo
to
get
some
data
from
a
development
team
to
build
the
data
visualization
using
a
tool
that
I'm
going
to
present,
which
is
Vega
and
finally,
to
bring
this
data
visualization
into
backstage.
With
a
plugin.
A
A
A
A
A
You
have
a
direct
interaction
between
your
plugin
and
the
data
source,
so
there
are
benefits
of
these
of
these
two
first
patterns.
First,
it's
very
simple:
you
don't
need
to
deploy
additional
infrastructure
and
because
you
are
talking
directly
to
the
data
sources,
you're
querying
and
you're
exploring
live
data.
A
The
first
one
you
have
a
decoupled
pipeline
which
is
going
to
query
the
data
sources,
apply
some
processing
and
store
it,
and
here
I
say
it
stores
it
in
a
static
metric
store.
What
do
I
mean
by
that
I
mean
I'm
just
going
to
store
to
generate
files,
maybe
CSV
files,
Json
files
and
these
files
are
then
going
to
be
provided
to
the
renders
dedicated
pattern
on
the
right.
You
have
a
dynamic
metric
store
where
you
can
submit
interactive
queries.
A
A
This
is
the
kind
of
visualizations
that
you
cannot
do
by
interacting
directly
with
the
jira
API
API
gives
you
the
ability
to
query
and
to
obtain
the
list
of
transitions
between
the
state
of
the
the
tickets
of
the
issues,
but
you
definitely
need
to
do
some
processing
if
you
want
to
compute
the
the
chains.
So
it's
a
good
example
for
a
process
where
you
need
to
have
this
decoupled
pipeline.
A
The
pattern
that
I'm
going
to
use
for
the
demo
to
end
up
with
the
visualization
is
much
simpler.
It's
the
keys
Analytics,
where
I'm
going
to
use
the
data
source
being
lit
so
I'm,
not
even
going
to
run
queries
against
an
API
I'm,
going
to
run
the
git
log
command
to
obtain
the
raw
data
from
my
source
code,
history
for
the
pipeline,
I'm
going
to
use
GitHub
actions
and
then
I'm
going
to
store
the
results
directly
in
GitHub
and,
of
course,
there
are
many
reasons
why
this
is
not
a
very
good
idea.
A
The
main
reason
why
I
made
this
Choice
here
was
to
have
an
end-to-end
demo
and
I
will
come
back
to
the
to
the
things
to
to
watch
out.
For
so
the
first
step
is
the
the
the
the
workflow
and
in
the
presentation
the
slides
are
available
on
the
side.
You
have
the
links
to
the
GitHub
repo,
so
everything
that
I'm
showing
today,
you
can
also
access
the
the
code
and
and
re-run
it
yourself.
A
So
the
first
step
is
implemented
as
a
GitHub
action.
It
starts
by
cloning.
The
repo
you
just
need
to
be
aware
that
by
default,
when
you
clone
a
repo,
it
doesn't
clone
the
entire
history.
So
you
need
to
to
have
a
special
parameter
for
that
I
run
the
the
command
and
again
I
store
the
metrics
files
directly
in
git.
A
Why
is
it
not
a
good
idea?
Well,
it
depends
the
kind
of
of
a
file
that
you
generate,
but
if
you
generate
files
that
are
bigger
than
a
few
kilobytes,
then
it's
certainly
not
a
very
good
idea
to
to
use
git
an
easy
Evolution
on
that
pattern
is
to
use
a
bucket
a
cloud
storage
bucket.
So
it's
very
similar
to
what
is
done
with
the
tech
docs
recommended
architecture
where
you
build
the
documentation
in
the
CI
Pipeline,
and
you
put
the
outcome
in
S3
or
in
other
cloud
storage.
A
A
A
A
This
is
a
very
simple
example
in
this
case
I.
Don't
do
any
processing
so
be
aware
that
the
data
I
need
is
never
really
clean.
You
always
have
pseudonyms
you're
going
to
otherwise
always
have
duplicates
in
the
in
the
outer
names.
So
if
you
want
to
do
a
proper
software
analytics
pipelines,
these
are
things
that
you
need
to
to
improve
and
to
and
to
integrate.
A
The
action
the
GitHub
action
has
been
deployed
for
the
demo.
What
I've
done
was
to
Fork
the
backstage
repo,
add
the
action
in
a
custom,
Pipeline
and
so
I'm,
now
able
to
run
this
action
and
to
generate
to
compute
the
metrics
for
the
backstage
code,
and
this
is
what
we
see
on
the
screen
where
we
have
the
list
of
files
that
are
generated
by
the
pipeline.
So
if
we
have
a
look,
we
see
that
the
raw
data
where
we
have
one
line
per
commit
is
2.7
megabyte
and
the
aggregations.
A
A
So
that's
the
preparation
bit
now.
The
interesting
part
is:
how
can
we
create
these
these
these
visualization?
There
are,
of
course,
many
approaches
to
to
to
do
data
visualization,
so
I'm,
going
to
talk
now
about
vega.gs,
which
is
an
open
source
Library
built
on
top
of
D3
and
Vega,
is
described
as
a
visualization
grammar,
a
declarative
language
for
creating
saving
and
sharing
Interactive
visualizations.
A
A
So
what
I'm
going
to
do
now
is
to
show
you
concretely,
our
Vega
spec
looks
like
and
how
you
can
actually
create
the
the
big
aspect.
The
important
part
is,
at
this
stage,
I'm
really
looking
at
the
data
sets
using
Vega
to
create
a
visualization
with
the
idea
that
later
on,
this
will
be
integrated
into
backstage.
A
A
A
A
In
this
case,
it's
the
CSV
file
that
is
generated
in
my
pipeline
via
the
GitHub
action.
What's
interesting
is
that
this
URL
could
point
to
a
URL
exposed
by
backstage.
So
if
we
have
the
tech
insights
or
the
the
metrics
endpoint,
I
could
directly
talk
to
to
Backstage
or
to
a
data
source,
receive
the
CSV
data
and
have
it
ready
for
exploration
and
rendering.
A
You
see
that
I
can
do
Transformations
like
aggregation,
so,
for
example,
I
want
to
group
by
outer
and
count
the
number
of
commits
and
obtain
the
last
the
date
of
the
last
comment,
if
I
specify
a
transformation
like
this,
you
see
that
I'm
transforming
one
data
set
into
another
data
set
and
the
Vega
editor
gives
you
here.
The
list
of
the
data
sets
so
authors
now,
as
this
shape,
where
I
have
a
list
of
authors.
A
I
have
prepared
the
data,
and
you
see
that
you
can
do
some
of
the
data
preparation
in
the
back
end
compute
some
of
the
aggregations
in
the
back
end.
You
can
lose
some
of
them
on
the
front
end.
Of
course,
if
you
do
them
on
the
front
end,
you
have
the
ability
to
do
live
filtering
and
to
give
the
ability
to
the
user
to
specify
the
parameters.
A
A
The
second
visualization
here,
so
it's
exactly
the
same
data
transformation
pipeline,
but
you
see
that
now
we
have
a
graph
that
is
rendered,
and
this
is
the
second
part
of
the
Vega
specification
where
we
have
the
following
elements.
First,
we
Define
scales
in
this
case.
I,
have
three
scales:
I
have
the
scale
that
I
used
for
my
field,
attributes.
A
A
A
You
see
that
I
changed
the
the
attribute,
so
what
you
see
on
the
screen
are
some
of
the
attributes
supported
by
the
Vegas
specific
specification,
but
you
have
access
to
a
lot
of
parameters
that
you
can
use
to
fine-tune
the
visualization.
The
documentation
of
the
project
is
very,
very
good
and
you
have
access
to
to
to
a
lot
of
flexibility.
A
A
A
A
A
So
to
conclude,
because
the
the
question
was,
if
I
have
this
visualization,
what
can
I
do
with
them
in
backstage,
so
we've
created
a
plugin
that
you
can
use
it's
open
source,
you
add
it
to
your
backstage
installation
and
then
in
your
page
or
in
your
tab.
You
just
add
a
fragment
where
you
reference
your
Vega
specification,
you
specify
the
size
of
the
widget
and
it
will
appear
in
your
in
your
portal.
A
A
So
with
that
I'm
coming
to
an
end,
you
have
the
references
to
the
the
plugin
in
the
in
the
slides.
You
have
all
the
examples.
Now
we
are
very
eager
to
to
work
with
people
who
are
interested
in
visualization
people
who
want
to
to
try
the
the
the
plugin
and
ordinary
people
who
are
interested
by
visualization
in
the
context
of
Backstage
or
tens
hundreds
of
of
use
cases,
and
we
would
be
very
interested
to
to
to
discuss
and
to
to
explore
these
ideas.