►
From YouTube: How We Scaled Catalog Ingestion to Hundreds of Thousands of Entities - Damon Kaswell, HP Inc.
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
How We Scaled Catalog Ingestion to Hundreds of Thousands of Entities - Damon Kaswell, HP Inc.
When you need to ingest over 100,000 entities from a data source, you can't just point a Catalog Processor at it and hope for the best. Through a combination of careful planning, properly scaled computing resources, and some clever customization that extends the basic Entity Provider and Catalog Processor functionality, HP Inc. is successfully ingesting hundreds of thousands of separate entities, proving that Backstage is truly ready to scale for the large enterprise. In this presentation, I will:
* Describe the scope of the technical challenge. HP Inc. is using Backstage for many different types of entities from many different sources.
* Do a deep technical dive into how our entity ingestion works, and how we have extended the base Entity Provider mechanism with multiple features that allow us to handle extremely large sets of data.
* Talk about considerations such as entity validation and relations.
A
Can
can
everybody
hear
me
is
this:
is
this
working
okay,
great
yeah,
sort
of
a
technical,
Deep
dive,
I?
Guess
I'm,
not
gonna
I'm,
not
gonna,
throw
throw
a
whole
lot
of
code
at
you
or
anything.
I'll
try
to
go
into
some
details
so
yeah.
My
name
is
Damon
Caswell
I'm,
one
of
the
lead
Developers
for
hp's
internal
developer
portal,
senior,
lead,
engineer
and
I.
Think
I'd
like
to
start
by
talking
about
a
number.
A
Our
stack
includes
a
lot
of
Technology,
some
of
which
you've
seen
today
that
you
you're
all
very
likely
familiar
with
and
a
bunch
of
AWS
Services
as
well,
which
you're
probably
familiar
with
you,
know:
elasticsearch
RDS,
lambda's,
Docker,
et
cetera,
et
cetera,
so
there's
there's
a
lot
of
things
that
go
into
a
typical
backstage
deployment
and
we're
no
different
from
you
know
any
anyone
else
on
that
now.
Some
of
you
are
probably
thinking
HP,
the
printer
company.
A
A
A
We've
got
all
these
all
these
different
things
we
got
web
apps
we've
got
mobile
applications.
Everything
everything
that
you
can
imagine
that
a
modern
tech
company
has
just
because
HP
is
old,
it
doesn't
mean
we
don't
have
them
and
more
got
a
fun
one
here.
For
you,
this
is
one
of
the
largest
printers
in
the
world.
A
So,
yes,
we
are
still
a
printer
company.
This
printer
has
a
doorway
I
think
more
than
one.
Actually,
this
has
a
staircase
and
it
has
over
a
million
inkjets
that
have
to
be
able
to
operate
simultaneously
coordinated
by
some
really
specialized
software.
So
yeah.
The
software
is
a
big
part
of
even
something
big
and
Industrial
like
that.
A
A
So
this
is
the
problem
I'm
going
to
I'm
going
to
break
this
up
into
two
main
sections,
I'm
going
to
outline
the
problem
that
we're
facing,
which
is
really
a
two-part
problem
and
outline
the
two-part
solution
that
we
found
so
ingestion
is
slow,
as
our
ingestion
needs
went
up,
this
the
performance
went
down
it.
Just
it
started
getting
longer
and
longer
from
the
time
an
entity
was
ingested
to
the
time
it
was
visible
in
the
catalog.
A
A
We
we
followed
the
best
practices
and
guidelines
that
are
available
by
default
in
backstage
plugins.
We
parsed
locations
and
emitted
them
from
one
catalog
processor
to
and
then
let
another
one
pick
up
each
location
and
process
the
entities
in
that
location.
It's
a
pretty
common
design
pattern.
Probably
a
lot
of
you
have
actually
can
I
see
a
show
of
hands
how
many
of
you
are
doing
that
now
emitting
locations
and
then
emitting
entities
from
those
locations.
A
A
A
A
We
you
know,
entities
did
eventually
show
up
it,
it
did
work,
but
it
took
a
long
time.
So
we
concluded
this
sort
of
design
pattern.
That's
sort
of
out
of
the
box
design
pattern
was
was
effective
if,
if
you
were
dealing
with
a
small
number
of
entities,
but
as
soon
as
you
started
to
try
to
ramp
that
up,
we
needed
something
else,
and
this
was
this
was
even
back
before.
A
Like
I
said
before,
entity
providers
existed
when
entity
providers
came
our
way
the
we
jumped
on
those
because
they
looked
like
they
would
solve
the
problems,
but
they
really
didn't
I
mean
we
were
thrilled
to
have
a
mechanism
that
was
specifically
designed
to
just
ingest
data.
It's
a
great
idea
and
just
leave
your
processors
in
charge
of
all
the
post-processing,
but
our
design
thinking
hadn't
really
adapted
to
it
yet,
and
then,
on
top
of
that,
it
still
brought
some
problems.
A
If
you
want
to
Target
a
very
large
entity
provider
or
entity,
Source
an
asset
source,
then
it
actually
introduced
new
problems
that
we
had,
that
that
revealed
that
sort
of
an
a
default
entity
provider
didn't
really
scale
comfortably
to
a
very
large
data
source.
We
still
had
locations
polluting
the
catalog.
We
still
had
entities
you
know
taking
a
long
time
to
show
up
sometimes
even
longer
than
before,
now
I
want
to
say
entity
providers
did
offer
a
lot
of
great
things
as
well.
A
I
mean
if
you
I
did
not
mean
to
go
that
that
far
here,
yet
they
they
offered
the
mechanism.
The
first
good
mechanism
for
clearing
orphans
and
that
in
and
of
itself
is
huge
because
data
sources,
change
assets,
get
removed,
assets
get
added
and
if
you
don't
want
any
old
assets
lingering
in
your
catalog,
especially
if
that
asset
is
a
user
who
maybe
has
left
your
company,
you
don't
want
them
still
appearing
in
your
catalog
and,
of
course,
backstage
allows
you
to
ingest
users.
So
this
was
a
this.
A
A
You
know
if
you're
the
entity
at
the
back
of
the
line
you're
at
the
back
of
the
line
so
kind
of
to
summarize
the
issues
that
we
saw
with
both
sort,
both
sort
of
default
out
of
the
box
design
patterns
for
ingestion
so
night,
because
neither
really
you
know
fit
our
needs.
We,
you
know
the
catalog
processor
it
did.
A
It's
really
just
better
for
side
effects
for
for
emitting
relations
is
what
we
found
for
that,
and
you
know
it
wasn't.
Everybody
is
used
to
the
idea
that
backstage
is
eventually
in
sync,
not
real
time.
Well,
HP
kind
of
wanted
to
go
real
time
and
catalog
processors
haven't
gotten
us
weren't
able
to
get
us
there
when
used
for
ingestion.
Now
you
know
entity
providers
are
objectively
better
they're
strictly
dedicated
to
ingestion.
There's
they
don't
do
anything
else.
A
There's
they're
much
more
configurable
with
scheduling
and,
of
course,
there's
the
cleaning
up
of
orphans
that
we
saw,
but
still
can't
admit,
can't
admit
relations
from
it.
There's
only
two
options
for
ingestion,
full
and
Delta
and
I
think
I
included
that
here
yeah,
so
the
full
mutation.
That's
that
very
large
data
set.
A
A
Hey
I
said
it's
nice
to
nice
to
know
our
pain
is
felt
by
somebody
else
out
there
yeah,
if
you,
if
you
ingest
a
very
large
amount
of
data,
you'll,
eventually
run
into
node.js's
own
limits
on
how
much
you
can
shove
into
an
array
a
single
into
a
single
array,
because
that's
what
a
full
mutation
does
you
put
everything
into
a
single
array
to
be
to
be
ingested
and
you
can
work
around
that?
You
can
do
things
like
increase
your
max
Heap
size,
but
you're
gonna
run
into
problems
there
too.
A
Eventually,
it's
a
that's
a
Band-Aid
think
about
your
asset
catalogs.
What
are
they
going
to
do?
They're
going
to
grow
and
they're
going
to
grow
and
grow
some
more
and
keep
growing
and
keep
growing
and
they
grow
because
asset
sources
grow.
You
add
more
apis,
you
add
more
data
assets,
you
add
new
organizations,
new
new,
actual
sources
of
data,
so
entirely
new
entity
types,
more
Docker
images.
You
know
more
Harbor,
stuff,
more
charts,
more
pipelines,
more
everything,
everything
gets.
Everything
grows.
A
A
A
Hey
I've
got
an
entity
to
ingest,
hey
I've
got
an
entity
to
Pro
preprocess,
hey
I've
got
an
entity
to
post
process.
Do
it
do
it?
Do
it
do
it
eventually,
you
know,
postgresism
is
not
not
very
happy
with
that.
Yeah,
it's
a
it's
a!
It
can
be
messy,
because
every
single
entity
needs
to
go
through
the
same
processing
loops
when
we,
when
we,
you
know
before
we
solved
this
issue,
and
we
I
want
to
assure
you
we
did
solve
it.
A
It
could
take
hours
or
in
worst
case
scenarios.
The
worst
we
ever
saw
was
in
days
for
an
entity
that
was
just
added
to
the
to
to
the
catalog
to
actually
show
up
to
be
to
finish
its
post-processing
Loop
and
actually
become
searchable,
and
because,
where,
where
was
I,
oh
my
God,
my
place,
that
never
happens.
A
So
here
we
are
so
we
started
asking
ourselves.
Why
is
that?
Why
we're
post,
pre
and
post
processors
taking
so
long,
we
looked,
we
looked
this?
Is
you
know,
stolen
from
backstage's
own
documentation?
Thank
you
backstage.
This
is
a
their
graph
of
the
processing
loop,
I
I
sourced
it.
It's
the
URLs
at
the
bottom.
A
What
specific
we
asked
ourselves,
what
specific
operations
cause
a
delay
in
the
processor
for
your
post,
we
started
measuring
exactly
which
ones
took
the
longest,
which
was
not
easy,
but
it
was
fruitful
because
we
identified
in
every
case
asynchronous
actions
in
the
in
the
pre
and
post
processor.
You
put
you
got
an
entity
that
you
need
to
post
process.
Oh
hey,
one
of
the
things
you
need
to
put
do
in
post-processing
is
grab
this
additional
piece
of
data
from
this
other
source.
A
It's
just
an
HTTP
get
it'll
only
add
you
know,
500
milliseconds
to
that
entity
and
to
all
200
000
other
entities,
you're
ingesting
and
processing
yeah,
that
that
adds
up.
We,
let's
see
the
here.
It
is
here's
my
calculation
when
there's
a
backlog
of
50
000
entities
and
you've
got
four
pods
processing
entities.
A
So
so
the
summarize
initial
ingestion
very
slow
with
catalog
processors,
entity
providers
can't
really
ingest
very
large
data
sources,
and
you
end
up
with
a
catalog
polluted
with
locations
and
processing.
Asynchronous
operations
cause
a
delay,
standard
patterns
for
enabling
or
for
and
for
emitting
entities
after
reading
locations
are
kind
of
inefficient
because
of
the
asynchronous
operations.
A
A
We
created
our
own
type
of
entity
provider
with
help
from
front
side.
Thank
you
to
Ross,
who
some
of
you
probably
know.
Actually,
all
of
you
should
know
from
the
from
the
backstage
Community.
Now
you
heard
him
talk
earlier,
so
yeah
taras
rocks.
Thank
you
with
their
help.
We
created
something
that
Works
a
little
differently,
so
this
type
of
entity
provider
takes
a
large
data
source
and
ingests
it
in
bite,
size,
configurable
chunks,
with
the
ability
to
back
off.
A
If
the
data
source
is
generating
errors,
anybody
run
into
issues
where
you're
most
of
the
way
through
an
ingestion
and
then
the
API
you're,
trying
to
read
from
crashes
or
gives
an
invalid
response
or
something
goes
wrong
and
better
start
over
yeah.
No,
it's
this
kind
of
entity
provider
has
a
back
off
mechanism
where,
if
an
error
happens
loud.
A
It
uses
paging
to
break
a
large
ingestion
up
into
chunks.
It
pauses
between
chunks
so
that
you're
not
you've
got
constant
pressure
instead
of
spikes
of
really
high
pressure
on
your
data
sources
and
it's
real,
which
is
really
useful
for
data
sources.
That
rate
limit
we're.
We
I'll
just
say
this:
we
do
use
GitHub
and
GitHub
rate
limits,
so
oh
I
I'm
seeing
a
lot
of
nods
and
agreements
an
agreement
here.
Oh
yeah,
yeah,
I'm
glad
I'm
glad
we're
not
the
only
one
I'm
glad
about
that.
A
So
and
all
aspects
of
this
are
configurable,
so
you
can
you
can
figure
how
often
it
runs
you
can
figure
how
long
it's
gonna
it's
gonna
do
a
burst
of
ingestion
before
before
pausing
between
bursts
and
how
long
it
rests
between
full
ingestions.
If
you've
got
a
data
source
that
changes
very
infrequently,
but
it's
really
really
large.
You
might
not
want
to
ingest
it
more
than
once
a
month.
A
A
If
there's
something
wrong,
so
we
also
created
a
suite
of
administrative
tools,
they're
just
basic
rest
endpoints,
to
call
when,
when,
when
you
want
to
manipulate
the
perform
the
activities
being
performed
by
these
incremental
providers-
and
you
can
use
a
you-
could
use
a
web
front
end
for
them
or
any
any
rest
style
tool.
This
is
this
is
a
screenshot
of
Postman,
I'm
I'm
sure
you
all
know
know
what
that
is.
A
We
don't,
we
don't
push
it
and
push
the
stuff
directly
into
the
public,
schema
and
and
postgres.
We
keep
it
in.
We
keep
it
separate
and
there
are
three
main
tables.
There's
the
ingestions
table,
which
tracks
the
status
of
a
running
provider.
The
ingestion
marks
table,
which
is
essentially
a
way
of
tracking
the
cursor
for
the
page
of
data
that
we're
on
and
there
is
the
ingestionmark
entities
table.
This
is
this
is
how
we
accomplished.
A
You
know
splitting
the
difference
between
Delta
and
full
entity
providers.
We
still
needed
a
way
to
get
rid
of
Orphans
and
tracking
the
tracking,
the
state
of
the
entity,
so
that
we
know
whether
it
was
still
there
on
that
last
ingestion
or
not
was
how
we
reached
the
point
of
being
able
to
wipe
out
orphans
in
a
way,
that's
very
similar
to
the
way
that
a
full
mutation
does
with
a
regular
entity
provider.
A
There
are
some
requirements
for
this:
the
your
data
source
has
to
paginate
if
you've
got
a
very,
very
large
data
source
that
does
not
paginate
well
I
mean
first
off.
Why
do
you
have
a
very,
very
large
data
source
that
does
not
paginate,
but
if
you
don't,
the
an
incremental
provider
is
not
going
to
to
work
for
that.
It
requires
pagination
because
it
needs
to
be
able
to
track
where
it
was
in
the
in
the
process.
A
There's
a
we're
also,
because
we're
adding
new
tables
and
one
table
tracks
your
entities.
There
are
storage
considerations,
so
if
you're
always
riding
the
edge
on
how
much
storage
you're
using
for
for
your
database
in
postgres
you're
going
to
need
to
up
that
a
little
bit.
If
you,
if
you
implement
an
incremental
entity
provider,
one
that
did
not
make
it
to
my
slide
here
but
I
should
mention
is
that
it
also
does
not
support
stateful
apis.
A
A
A
So
it's
not
it's
not
a
huge
deal,
but
if,
but
if
you
if,
but
it's
something
to
keep
in
mind
depending
on
what
sorts
of
Legacy
systems
you
want
to
ingest
from
so
that
solved,
half
the
problem
that
got
that
got
us
to
the
point
where
we
could
ingest
those
very
large
sources
of
data
and
do
it
in
small
enough
chunks
that
there
was
never
a
point
where
they
were
just
absolutely
flooding.
The
catalog.
A
It
was
still
taking
too
long
to
get
from
the
point
of
an
ingested
entity
to
the
point
of
it
being
visible
in
the
catalog
we
ident
like
I
mentioned
earlier.
We
identified
asynchronous
operations,
those
all
those
HTTP
gets
that
you
put
in
your
catalog
processors.
Those
are
the
culprit
and
it's
a
little
frustrating,
because
those
are
also
the
natural
places
the
place.
It
feels
right
to
do
those
you're
trying
to
emit
a
relation
we'll
grab
the
data
that
you
need
to
emit
a
relation
with
and
admit
it.
A
Fine,
but
it's
not
efficient
at
it.
So
fixing
the
problem
meant
committing
to
something
that
honestly
looked
a
little
ugly
at
first
front
loading
all
asynchronous
operations
into
the
entity
provider.
We
banned
them
from
from
catalog
processors.
If
we
do
not
use
that
mechanism
at
all
anymore,
our
catalog
processors
are
without
asynchronous
operations
entirely,
and
we
moved
all
of
that
into
the
entity
providers.
These
incremental
entity
providers-
it's
not
a
trivial
task.
A
It
meant
that
for
every
entity
kind
that
we
deal
with,
we
needed
to
include
all
of
the
methods
and
functions
that
are
used
to
get
that
asynchronous
data
right
there
in
the
entity
provider.
It
had
to
do
it
at
that
stage.
That's
okay,
because
you're
only
running
that
you
know
once
in
a
once
in
a
while
and
ideally
you're,
never
since
you're
not
ingesting
a
huge
chunk,
all
at
once,
you're
doing
it
in
bite-sized
chunks.
A
It
never
adds
a
huge
amount
of
data
all
at
once
to
or
a
huge
amount
of
time
all
at
once
to
it,
but
it
was.
It
was
work
to
get
it
there,
every
so.
Every
entity
provider
needed
to
be
Rewritten,
every
catalog
processor
needed
to
have
them
removed
and
the
ugliest
thing
of
all
every
entity
schema
that
we
created
needed
to
be
updated
so
that
the
entity
itself
could
have
all
of
the
data
that
would
eventually
be
used
to
emit
its
relations
or
to
emit
its
side
effects
to
to
run
that
emit
process.
A
That
meant
updating
a
load
of
schemas
with
data
that
was
never
intended
to
be
there
permanently.
It's
just
there.
For
the
point
of
ingestion,
so
that
it
doesn't
have
to
make
an
additional
asynchronous
call
once
it
reaches
the
pre
or
post
processor
and
then
honestly,
all
of
our
pre
and
post
processors.
If
that
data
is
still
there
after
we've
emitted
the
relations
and
such
we
remove,
it
doesn't
need
to
stay
there.
It
doesn't
need
to
stay
as
part
of
the
entity.
A
This
graph
is
showing
the
a
running
average
of
the
discrepancy
between
when
an
entity
is
scheduled
for
processing
and
when
it
actually
passes
through
the
processing
loops
and
that's
in
hours.
That's
that's
a
processing
time
in
hours,
0.57
hours.
So
we've
got
in
this
on
this
graph.
We
were
down
to
just
a
little
bit
more
than
half
an
hour
on
hundreds
of
thousands
of
entities,
so
that
was
the
amount
of
time
total
that
it
would
take
for
an
entity.
A
Before
that
the
same
graph,
we
have
historical
data,
I'm
not
going
to
show
it.
We
have
historical
data
that
shows
the
same
graph
from
a
couple
of
months
ago,
and
it
was,
it
was
different.
It
was
four
hours
or
sometimes
more.
The
like
I
said
earlier.
The
longest
we
saw
was
over
day,
here's
our
Dev
environment.
This
actually
has
a
little
a
few
more
refinements
that
have
gone
into
it.
A
0.25
0.259,
that
is
15
minutes
15
minutes
from
the
point
in
The
Entity
entered
the
catalog
or
entered
the
database
as
an
unprocessed
entity
and
was
scheduled
to
be
processed
to
the
point
where
it
was
actually
processed
15
minutes,
and
then
that
graph
looks
like
it's
going
up
a
little
bit
there.
But
but
I
should
tell
you
something
I
using
those
admin
tools.
I
mentioned
earlier,
I
started.
All
of
our
I
manually
started
all
of
our
entity
providers
before
running
this.
This
check.
So
that's
with
all
of
our
entity
providers
running.
A
So,
in
fact,
if
you
look
at
the
queue
over
that
over
time
in
July
to
July
yeah
I
guess
I
can
do
include
some
of
that
data
6.
4.67
hours.
That
was,
that
was
a
painful
time.
That
was
how
long
it
took
back
in
July.
Oh
my
gosh,
and
we
had
to
tell
all
of
our
stakeholders
you're
going
to
wait
an
average
of
that
much
time.
When
you
add
a
new
entity
before
it
actually
shows
up
to
now
average
of
20
minutes
on
on
average,
in
our
in
our
production
environment,
20
minutes,
92
percent
decrease.
A
We
want
processing,
time
to
drop
further,
the
lowest
I've,
seen
and
I
think
we
can
get
it
down
there.
I've
seen
seven
minutes,
I've
seen
I've
seen
I've
seen
that
that
graph,
that
I
showed
earlier
showed
just
seven
minutes.
I'd
like
it
to
stay
there,
I'd
like
to
go
even
lower
if
possible,
I
I,
don't
know
if
we
can
get
there
super
consistently,
but
we're
sure
going
to
try
so
to
summarize
incremental
entities
and
entity
providers.
You
know
more
flexible
scheduling,
more
more
resilient,
more
manageable
able
to
handle
data
sources.
A
A
A
We
have
the
graphs
to
prove
it.
We
have
the
active
HP
wide
implementation
to
prove
it
backstage
is
very,
very
powerful.
So
this
is
sort
of
a
not
really
a
technical
piece
of
information
that
we
learned
more
of
a
more
of
a
social
piece
of
information.
We
learned
we
need
to
start
talking
to
the
backstage.
Community
needs
to
start
talking
to
potential
adopters,
not
Justin.