►
From YouTube: Parsing Engineering Diagrams for Industrial AI Applications using Deep Learning and Graph Search
Description
In this session, learn how Red Hat, the world’s leading provider of enterprise open source solutions, in partnership with C3.ai, is helping customers develop scalable intelligent applications leveraging AI and machine learning. We’ll walk you through a practical example of digitizing engineering diagrams to enable machine learning-based predictive maintenance. Engineering diagrams are a vital data source for industrial facilities, depicting the configuration and properties of equipment, sensors, and process flow. In this session we will:
- Show how to leverage containers and Kubernetes to provide agility, flexibility, and scalability to enterprise AI application development
- A computer vision pipeline using state-of-the-art deep learning and graph search techniques to parse diagrams and generate structured asset hierarchies
- Show how these hierarchies allow operators to represent their assets digitally and enable intelligent applications
A
Hello
there,
my
name
is
john
senegal
and
I'm
a
solution
architect
at
red
hat,
working
with
partners
around
ai
and
edge
technology
shelvik,
and
I
are
going
to
talk
to
you
about
a
particular
ai
use
case,
parsing
engineering
diagrams,
using
deep
learning
and
graph
search.
But
before
we
get
into
the
use
case,
I'd
like
to
talk
to
you
about
the
technology
that
makes
this
use
case
possible
at
scale.
A
So
today,
I'm
going
to
do
an
introduction
on
the
technology
that
enables
this
particular
use
case.
That
showbik
is
going
to
talk
to
you
about
and
as
part
of
that,
I'm
going
to
talk
to
you
about
why
kubernetes
makes
sense
for
ai
workloads
and
then
shovik
is
gonna.
Get
into
this
very
special
use
case
that
we'd
like
to
talk
to
you
about.
A
A
That
data
make
it
available
to
the
data
scientists,
who
then
build
the
models
who
then
you
work
with
developers
to
integrate
those
models,
and
then
those
models
have
to
be
monitored
and
managed
for
drift
and
make
sure
that
they're
constantly
update,
updated
and
doing
the
things
that
you're
expecting
them
to
do
so.
Taking
a
model
taking
an
idea
and
going
from
idea
to
in
production
is
not
a
trivial
task.
A
Enter
kubernetes,
and
this
is
where
kubernetes
makes
sense
for
ai
workloads.
First
of
all,
kubernetes
brings
agility.
What
does
that
really
mean?
It
means
that
when
you
have
these
types
of
pro
projects,
kubernetes
gives
you
the
ability
for
resource
management.
So
as
your
requirements
and
your
demand
for
resources
seek
compute
power
from
memory
all
of
those
increase
when
you're
deploying
ai
workloads
kubernetes
gives
you
the
opportunity
to
request
resources
on
demand
to
scale
up
and
scale
down
as
required.
A
A
A
You
have
flexibility,
you
have
the
ability,
excuse
me
to
to
spin
up
new
aiml
projects
as
needed,
so
the
data
scientist
has
a
bright
idea
about
creating
a
new
model
in
this
kind
of
platform.
He
has
the
ability
to
spin
up
projects
as
needed,
get
the
resources,
get
the
tools
that
are
required
to
to
play
around
with
his
project,
and
most
important
is
scalability.
A
In
order
to
do
that,
you
need
a
whole
platform
of
things
that
can
support
this
life
cycle
of
building
and
deploying
models.
So
at
the
bottom
you
of
course
need
some
infrastructure.
You
need
a
cloud
public
or
private
virtual.
Maybe
you
want
to
run
this
stuff
on
the
edge,
but
you
need
hardware.
Of
course
you
may
need
compute
acceleration,
so
we
support
gpus
as
part
of
openshift
and
then
the
next
layer
is
where
red
hat
openshift
is.
A
You
need
a
kubernetes
platform
based
on
what
I
described
before,
because
that
platform
will
give
you
the
agility,
flexibility,
scalability
and
portability
we
talked
about
and
then,
of
course
you
need
data
pipeline
services.
You
need
ml,
devops
kinds
of
tools
to
build
and
deploy
those
models.
So,
in
order
to
execute
this
life
cycle,
you
need
an
entire
platform
like
this
one
of
the
things
we
did
when
we
looked
into
this
of
why
kubernetes
makes
sense
is
when
you
talk
to
customers.
A
They
understand
those
issues
that
I
I
talked
about
earlier
and
94
of
them
have
decided
that
or
have
already
decided
that
they
will
containerize
the
ai
ml
workloads
within
the
next
year.
Either
they've
done
it
or
they
will
do
it
within
the
next
year
because
they
understand
the
value
that
containerized
workloads
running
on
a
kubernetes
platform
has
for
them,
and
so
in
order
to
support
that
red
hat
has
the
hybrid
multi-cloud
platform
that
I
talked
about.
A
So
when
we
talk
about
technology
that
will
support
the
use
case,
we're
going
to
cover,
of
course,
a
reference
architecture
is
a
great
place
to
start,
but
actually
this
is
where
we're
going
to
stop
we're
not
going
to
get
any
deeper
into
this.
But
I
must
say
you
can
see
red
hat
in
the
picture
as
a
supporting
common
platform
across
all
of
those
environments.
Public
cloud,
vms,
intel,
nvidia
one
platform
to
support
anywhere.
You
want
to
run
your
ai
ml
workload,
and
so
I'm
going
to
turn
it
over
to
shovik
now.
B
B
B
So
this
the
c3ai
suite
we
expose
this
functionality
using
apis
and
these
apis
are
used
by
the
applications
that
we
build
on
top
of
them.
So
that
will
be
our
focus
for
today.
I
want
to
tell
you
about
a
specific
application
that
that
you
know
we
we
offer
at
c3,
and
I
also
want
to
talk
about
a
specific
capability
of
the
application,
which
is
capability
of
parsing
engineering,
diagrams
and
being
able
to
extract
structured
information
from
highly
unstructured
and
noisy
diagrams.
B
B
B
You
have
a
lot
of
systems
at
your
disposal
and
there
is
a
hierarchy
right,
so
you
may
have
a
compressor
chain
system
and
then,
in
that
system
you
have
multiple
subsystems
right,
so
you
may
have
a
power
turbine
in
that
compressor
train.
You
may
have
a
gearbox,
you
may
have
a
gas
generator
and
so
on
and,
as
you
can
imagine,
you
have
telemetry
from
all
of
these
various
subsystems.
B
Different
levels
of
your
hierarchy-
and
you
know
you
want
a
way
to
understand
the
health
of
your
entire
plant
at
both
the
system
level,
as
well
as
the
subsystem
levels.
So
what
the
c3
ai
reliability
application
does.
Is
it
integrates
all
of
these
disparate
data
sources
such
as
you
know,
time
series
telemetry
data
from
from
pi
sensors.
B
B
B
So
so
these
engineering
diagrams,
are,
you
know
crucial
data
source
for
for
any
industrial
facility
they're
drafted
during
you
know
the
design
of
the
facility
they're
used
during
operations
and
maintenance
to
to
identify
where
equipment
and
sensors
are
and
they're
also
used
during
safety
checks.
So
you
know
in
in
certain
jurisdictions,
there's
actually
requirements
to
keep
these
engineering
diagrams
up
to
date,
because
you
know
they
are
the
most
up-to-date
living
kind
of
representation
of
a
facility,
and
this
is
an
example
of
an
engineering
diagram.
B
B
This
is
some
other
equipment.
So
it's
a
large
large
symbols
are
our
equipment.
Then
you
have
these
circles.
These
circles
are
called
locally
mounted
instruments,
so
these
are
actual
physical
sensors.
So,
like
a
pressure
gauge
or
a
temperature
indicator
which
you'll
find
on
the
facility
floor
and
then
you'll
notice
that
these
circular
symbols
are
often
always
connected
to
this
interesting
symbol,
which
is
a
circle
inside
a
square,
and
these
are
called
tags
and
tags
tags
are
the
the
digital
representation
of
this
sensor.
B
B
The
final
thing
I
want
to
note
note
is
often
you'll
see
lines
in
in
these
diagrams.
So
whenever
you
have
solid
lines,
those
are
physical
process
lines,
so
lines
that
carry
you
know,
fluid
and
sometimes
you'll
see
dashed
lines
and,
and
those
dashed
lines
are
electrical
signals
which
makes
sense
right.
So
you
have
some
sensor
and
you
know
you
have
some
electrical
signal
too.
You
know
it's
it's
like
digital
representation.
B
B
B
Okay,
so
so,
let's
get
started
first,
I'd
like
to
give
you
an
overview
of
our
machine
learning
pipeline,
which
we
use
to
digitize
these
diagrams
as
input
this
pipeline
takes
in
the
image
of
the
diagram,
as
well
as
an
optional
set
of
manually
labeled
symbols
such
as
equipment.
So
so
these
are,
you
know,
one-off
symbols,
which
you
know
we
can't
really
train
the
model
for,
but
you
know
the
user
can
scaffold
the
automated
diagram
parsing
process
by
providing
these
these
kind
of
one-off
symbols
as
labels.
B
B
Number
three
connection:
detection
to
detect
the
connections
between
the
various
tags,
lmi
and
equipment
in
the
diagram,
and
the
output
of
this
pipeline
is
what
we
call
an
asset
hierarchy.
So
essentially
it's
a
structured
representation
of
this
highly
unstructured
image
right.
So
it
tells
you
which
symbols
have
been
found,
whether
they've
been
manually
labeled
by
the
user
or
detected
by
the
pipeline.
B
What
kind
of
symbol
that
is,
whether
it's
an
equipment,
a
tag
or
an
lmi
locally
mounted
instrument,
the
text
which
has
been
parsed
for
that
symbol,
as
well
as
other
symbols
which
it's
connected
to.
So
it
gives
you
all.
You
really
need
to
know
about
this
diagram
and
and
its
contents
before
going
into
more
detail
on.
You
know
the
secret
sauce
and
and
the
ai
models
which
make
this
all
work.
B
B
B
B
As
I
mentioned,
we
let
the
user
specify
a
set
of
manually
labeled
symbols
in
this
case.
They
pass
it
in
as
an
xml
file,
and
here
we
have
a
set
of
five
manually
labeled
symbols
which
the
user
has
specified,
as
well
as
the
text
that
the
user
has
labeled,
and
so
you
see
five
of
these
manually,
labeled
symbols,
zero
or
zero
one,
two,
three
four
and
these
are
equipment,
symbols
which
our
pipeline
otherwise
couldn't
detect,
and
then
we
get
started
with
our
pipeline.
B
The
first
step
in
our
pipeline
is
a
symbol,
detection
step
in
which
we
use
a
cnn
or
a
convolutional
neural
network
to
look
at
crops
of
the
image
and
classify
them
as
a
tag,
symbol,
an
lmi
symbol
or
not
a
symbol.
So
that's
how
you
have
this
one
by
three
dimensional
output
at
the
end
and
notice.
What
we
do
here
is
we
initialize
a
machine
learning
pipe
called
a
symbol,
resection
pipe-
and
you
know
in
in
the
c3x
suite
the
way
we
think
of
machine
learning
way.
We
do.
B
Machine
learning
is
through
this
concept
of
pipelines,
so
we
have
a
pipe
for
symbol,
detection.
We
have
some
hyper
parameters,
some
other
metadata,
the
path
to
the
model
which
we
want
to
use
for
the
step,
and
once
we
have
initialized
this
pipe,
we
passed
the
parcel
diagrams.
Remember.
The
parcel
diagram
is
the
object
which,
which
contains
the
the
diagram
itself,
as
well
as
manually,
labeled
symbols.
B
B
Sometimes
you
know
it.
It
does
make
some
some
mistakes
like
overlapping
symbols.
But
if
you
look
for
the
vast
majority
of
cases,
this
is
really
good
detection
and
we're
able
to
isolate
where
these
sensors
are
and
where
these
lmis
are
and
and
that's
very
exciting
to
someone
who
would
have
to
otherwise.
You
know
parse
each
of
these
individually
to
create
our
systems
of
systems,
asset
hierarchy
and
the
great
thing
right
is
now
we
can
take
this
diagram
and
we
can
convert
it
into
a
structured
acid
hierarchy.
B
Right
so
you
see,
these
are
all
these
five
were
the
manually
labeled
symbols
we
started
out
with,
but
now
we
have
a
whole
set
of
detected
symbols
in
this
diagram,
and
that's
you
know
that's
done
automatically
next
step.
Is
the
text
recognition
step
where
again,
we
pass
in
the
same
parcel
diagram
into
now.
This
text
recognition
pipe
and
the
output
we
get
out
of
that
is
now
you
see
this
column
has
been
filled
in
and
that's
the
text
which
has
been
detected
inside
or
nearby
those
symbols.
B
So,
for
example,
let's
take
a
symbol
like
this
one.
Maybe
this
one
here
pt83030
and
we
see
the
id
is
24.
so
24
should
say
something
like
pt,
eight,
three:
zero,
three
zero.
We
can
look
it
up
in
our
table
here
and
we
see
sure
enough
symbol.
25
was
detected
as
an
lmi
and
we
see
the
associated
text
to
that
symbol.
B
This
is
just
another
visualization
of
the
text
which
has
been
detected
in
green
in
blue.
The
symbols
again
detected
symbols
in
red
and
the
manually
labeled
symbols
in
green
and
finally,
we
have
the
connection
detection
step
in
which
we
use
a
graph
search
to
parse
the
diagram.
You
know
to
traverse
the
diagram
and
detect
connections
between
symbols.
B
First
to
train
this
symbol,
detection
model,
this
cnn
network
that
I
mentioned,
we
first
labeled
a
lot
of
diagrams,
so
we
just
sat
there
labeling
the
data
we
labeled
18
diagrams
in
our
training
set,
and
we
specifically,
we
labeled
all
the
tags.
So
we,
you
know,
drew
bounding
boxes
around
those
tags.
We
drew
all
of
the
lmi
symbols
and
we
randomly
sampled
a
ton
of
not
symbol
crops,
and
so
this
essentially
created
a
data
set
that
we
could
train
on.
B
B
Then
we
know
we.
We
took
these
manually
label
symbols
and
then
on
this
diagram
we
took
a
sliding
window
and
we
passed
it
over
the
diagram
and
each
sliding
window
each
crop.
We
passed
it
through
our
network
and
using
that
we
were
able
to
get
the
actual
predictions
for
where
all
the
sensors
were
located,
and
so
this
is
the
result.
I
showed
you
earlier
of
all
of
the
tag
and
lmi
symbols
in
red
being
detected
by
the
pipeline.
B
B
After
that
we
took
these
regions
and
we
processed
them
using
tesseract,
ocr
and
based
on
proximity.
We
map
them
to
tear
detected
symbols.
So
in
this
case
you
see
this
lmi
symbol
here
and
you
see
inside
it.
You
have
two
of
these
detected
text
regions
so
once
you
process
that
using
ocr
you're
able
to
say
that
okay,
this
pd3030
belongs
to
to
this
lmi
symbol
and
that's
how
we
do
this
association.
B
Finally,
we
have
this
connection
detection
step,
which
I
believe
is
the
most
sophisticated
part
of
this
pipeline
and
a
part
that
excites
me.
The
most
is
data
scientist
so
connection.
Detection
again
is
this
problem
of
finding
which
symbols
are
connected
to
each
other
via
the
lines
and
the
lines
again
they
can
be
physical
process
lines
like
you
see
here
like
a
pipe
or
they
can
be
like
a
electrical
signal
right
like
a
sensor
to
a
database.
B
What
we
do
for
this
connection
detection
is,
we
actually
take
the
image
and
we
represent
this
as
a
graph,
so
you
have
a
graph
of
pixels
in
this
case
and
you
have
symbols
which
are,
you
know,
sets
of
pixels.
So
you
have
four
symbols
here
and
we
do
a
graph
search
going
from
symbol
to
symbol
following
the
lines
which
you
see
as
these
black
dots
in
this
graph.
B
You
can
imagine
right.
This
presents
a
difficulty
right
when
you're
trying
to
traverse
these
dashed
lines,
because
you
know
your
graph
search
right,
it'll
hit
a
roadblock.
It
won't
be
able
to
traverse
these
dashed
lines,
so
we
also
have
an
additional
step
where
we
use
a
probabilistic
huff,
transform
to
fill
in
those
dashed
lines
and
now
suddenly,
your
graph
search
can
traverse
through
those
dashed
lines
and
solid
lines
and
find
connections
along
the
way.
B
So
using
this
graph
search
process,
what
we're
able
to
do
is
for
each
source
symbol,
in
this
case
the
red
symbol,
we're
able
to
find
connections
to
other
symbols
in
the
diagrams
through
these
lines.
So,
for
example,
you
see
here
so
and
then
the
connections
that
the
path
traversed
by
the
graph
search
is
shown
in
blue.
So
here
you
see
that
from
this
symbol
you
know
we
follow
a
traversal
down
this
path
down
this
path
and
we're
able
to
hit
this
connected
symbol.
B
B
B
A
B
B
Unless
we,
you
know
manually,
parse
those
those
things,
but
now
with
this
diagram
digitization
pipeline,
we
can
automatically
detect
the
tags
and
we
can
detect
the
associated
text.
And
if
we
match
this
associated
text
against
the
time
series
data,
then
we
can
suddenly
fetch
data
for
for
any
tags
that
we
wish
in
the
diagram.
B
Finally
presenting
some
some
results
of
this
pipeline.
You
know
in
symbol,
detection
we
had
very
good
performance
for
tag
detection
in
terms
of
precision
and
recall,
as
you
can
see,
in
the
test
side
we
achieved,
you
know
essentially
100
precision
and
recall
perfect
classification,
whereas
for
the
lmi
symbols,
they're
they're
a
little
harder
for
a
problem
to
crack,
so
we
had
a
little
poorer,
but
still
you
can
get
a
nine.
Almost
ninety
percent
recall
with
relatively
high
precision.
B
This
is
a
sample
of
of
the
symbol,
detection
results.
Where
you
see
the
you
know
ground
truth,
labels,
which
you
know
we
labeled
in
green
versus
the
labels
by
the
the
machine,
learning
model,
cnn
and
red,
and
you
can
see
they're
they're,
pretty
close
overall
and
finally,
as
a
next
step
as
a
taste
of
what
we're
working
on
right
now,
we're
working
on
expanding
this
in
in
future
directions,
including
using
rcnn,
which
is
a
region
proposal,
neural
network,
and
this
as
you
can
see
on
the
bottom.
B
B
So,
overall,
you
know
at
c3
we
build
ai
applications.
We
build
applications
that
work
on
a
lot
of
disparate
data
sources
from
time
series,
data
to
maintenance,
data
to
engineering,
diagrams
and
oftentimes.
It
turns
out
that
these
diagrams
are,
you
know,
key
data
structures
to
getting
these
applications
to
to
work
and
and
to
work
as
the
way
you
want
them.