►
From YouTube: Upcoming Feature Alert!! No-Code Extensions in DataHub
Description
Gabe Lyons & Shirshanka Das (Acryl Data) review upcoming No-Code functionality during the November 2021 Community Town Hall.
Referenced Links:
https://datahubproject.io/docs/metadata-modeling/metadata-model
B
B
The
the
thing
gabe
and
I
are
going
to
amaze
you
with,
is
how
we
are
able
to
do
no
code,
extensions
to
the
metadata
model
and
have
both
the
server
and
the
ui
react
to
it.
Without
you
having
to
write
a
single
line
of
code,
we
like
to
play
it
very
live,
so
it's
gonna,
be
a
live
demo
and
we'll
see
what
happens.
B
B
There
are
nodes
in
this
graph
which
we
call
entities.
These
could
be
things
like.
Data
sets,
charts
dashboards
users,
john
just
went
through
a
bunch
of
examples
of
lineage
between
entities
and
entities,
contain
properties
facets,
or
we
call
them
aspects.
Aspects
are
groups
of
information
that
go
together.
For
example,
a
chart
has
information
about
it,
and
so
we've
got
an
aspect
called
chart
info
and
it's
got
a
bunch
of
things
in
it.
Like
name
description,
last,
updated
blah
blah.
Similarly,
we've
got
another
aspect.
Called
ownership
turns
out
to
be
a
very
popular
aspect.
B
It
is
attached
to
a
lot
of
entities
for
good
reason,
so
this
is
how
entities
and
aspects
kind
of
relate
to
each
other
and
form
a
sort
of
document
oriented
model,
and
then
the
graph
comes
together
when
aspects
produce
relationships
and
connect
entities
to
each
other.
So,
for
example,
a
chart
info
aspect
might
include
things
like
who
are
my
inputs
or
where
my
outputs
or
dashboard
info
might
include
those
things
and
that's
how
relationships
get
created
and
these
entities
get
connected
up
into
a
graph.
So
that's
very
brief.
B
We
also
talked
about
how
you
emit
metadata
model
change,
emit
metadata
that
corresponds
to
this
metadata
model.
The
original
approach
it
still
supported
was
to
emit
these
really
big
snapshot.
Mces-
and
these
are
monolithic
large
schema
events
that
have
the
whole
metadata
model,
essentially
captured
in
one
single
event
very
hard
to
work
with
very
hard
to
extend
somewhat
fragile,
because
every
time
you
perturb
the
metadata
model,
this
one
giant
schema
file
kind
of
changes.
B
The
new
way
that
we
are
recommending
everyone
move
to
we've
moved
to
it
in
the
open
source
code
for
sure
is
using
a
yaml-based
registry
to
describe
how
entities
are
connected
up
to
aspects
so
kind
of
wiring
up
this
metadata
model
together
and
then
using
mcps,
which
are
basically
skinny
metadata
change
proposal.
Events
to
emit
metadata
about
the
change
that
just
happened.
So,
for
example,
if
you
want
to
emit,
chart
information
for
a
chart,
so
the
chart
info
aspect
attached
to
a
chart
entity,
you
would
emit
a
skinny
mcp.
B
That
just
has
the
chart
info
aspect
attached
to
the
chart
entity
arm
we'll
get
into
it
in
just
a
bit.
But
I
just
want
to
give
a
quick
brief
intro
to
how
the
metadata
model
works
and
how
you
emit
metadata
around
it
all
right.
So,
as
you
know,
the
data
hub
metadata
model
is
pretty
big.
It's
pretty
complex,
I
encourage
you
to
go,
explore
it,
there's
so
many
entities
in
there
and
so
many
relationships
and
we're
continuously
adding
more.
B
However,
it's
never
enough.
People
always
want
to
model
one
more
thing
or
two
more
things,
or
they
have
a
slightly
different
opinion
for
how
something
is
should
be
modeled
and
today
the
way
that
companies
individuals,
everyone
extends
the
model
is
by
forking
the
repo,
adding
new
models
to
the
existing
model
tree
adding
annotations.
B
So
we
rolled
out
no
code
annotations
earlier
this
year.
They
have
been
a
huge
success.
It
allows
us
to
make
changes
to
the
model
very
quickly
and
not
write
tons
of
code,
but
you're
still
forced
to
add
those
new
models
to
the
existing
model
tree,
which
is
in
the
main
code
base,
you're
forced
to
recompile
the
metadata
service
and
redeploy
it.
B
And
so
what
that
means
is
a
lot
of
people
stop
doing
it
and
very
soon
we
find
people
are
on
like
a
year
old
release
and
they're,
finding
small
bugs
and
guess
what
they're
forever
forked
and
we
want
to
help.
You
not
be
abandoned,
and
that's
that's
really.
The
goal
for
kind
of
the
the
thing
that
we're
going
to
be
demoing
today,
which
is
no
code,
extensions
to
the
model
supported
both
in
the
server
and
the
ui.
B
It's
a
sneak,
peek,
it's
not
in
the
release
yet,
but
we
hope
to
roll
it
out
in
the
coming
month.
So
what's
the
idea?
The
idea
is
individual
companies,
or
teams
or
individuals
can
write
small
model
extensions
to
the
core
model
and
they
can
just
deploy
it
to
the
server.
The
server
is
going
to
continuously
be
merging
these
model
extensions
with
the
core
model
that
it
comes
deployed
with,
and
the
ui
and
the
api
clients
can
consume
the
combined
model.
B
All
right.
If
this
wasn't
ambitious
enough,
I
don't
know
what
is
so
we're
just
gonna
we're
gonna
go
right
into
it
and
talk
about
how
one
of
these
model
repos
might
look
like.
So
I
created
a
small
repo
called
metadata
custom
models.
It'll
actually
show
up
in
the
main
oss
repo,
as
just
a
shell
repository,
so
you
can
kind
of
look
at
it
and
kind
of
clone
it
and
use
it
as
your
base.
B
Don't
please
don't
ever
commit
it
back,
you
can
keep
it
in
your
company
and
you'll
be
happy
for
it
and
here's
an
example
of
what
we
could
be
doing.
So
maybe
I
belong
to
a
company
called
my
company,
and
I've
got
this
idea
for
adding
data
quality
rules
to
data
hub
and
the
data
hub
project
is
moving
very
fast,
but
it's
not
moving
fast
enough
for
me
and
I
want
to
add
data
quality
rules
before
these
people
can
get
to
it.
B
B
So
let's
go
define
a
data
quality
rule.
Don't
pay
too
much
attention
to
what
this
is.
You
know
some
field
level
stuff
check,
definitions,
urls.
You
know
I
designed
it.
It
looks
good
to
me
and
then
I
connect
it
up
as
an
aspect
using
the
aspect.
Annotation,
some
easter
eggs
here
some
render
properties
have
shown
up
and
gabe
will
get
into
why
those
are
magical.
But
the
thing
for
you
to
look
at
is:
I
gave
it
a
name:
it's
called
data
quality
rules
and
it's
really
an
array
of
data
quality
rules
that
I
just
defined.
B
So
I
just
created
an
object
called
the
data
quality
rule
tied
it
up
as
an
array
of
objects
and
then
made
it
an
aspect.
I
still
have
one
important
thing
to
do:
connect
this
aspect
to
an
entity.
So
here's
my
entity
registry,
which
is
checked
in
under
the
registry
directory
in
that
same
model.
In
that
same
repo-
and
I
called
it-
I
gave
it
an
identifier,
my
company
dq
model,
and
here,
as
you
can
see,
I'm
providing
a
patch
to
the
global
entity
registry.
B
B
So
let's
everyone
with
me
so
far,
I
just
covered
extending
the
model,
adding
the
model
extension
to
an
entity
registry
and
then
what
an
entry
that
conforms
to
the
model
looks
like
the
next
thing.
We're
going
to
do
is
go
to
the
metadata
service.
I
have
it
running
helpfully
over
here
on
docker,
so
this
is
running
on
my
local
host
in
docker
and
it's
actually
listening
and
watching
a
directory
called
data
hub
model.
B
Plugins
data
plug-in
models
to
look
for
new
models
to
show
up
there
and
just
to
prove
that
we
don't
have
any
models.
Yet
I'm
just
going
to
hit
the
config
endpoint
and,
as
you
can
see,
there
are
no
models
loaded
up
yet
so
it's
a
lonely
metadata
service.
All
it's
got
is
the
core
oss
models
and
it
doesn't
have
any
new
models.
B
B
So
I've
got
this
data
hub,
plugins
models,
repo
models
directory
which
is
empty,
and
my
custom
model
after
building
looks
like
this.
It's
got
my
company
dq,
which
is
the
identifier.
It's
got
a
version
identifier
and
then
in
there
it's
got
a
lib
directory
which
contains
the
compiled
data
templates,
and
it's
also
got
that
patch
entity
registry
inside
it.
So
what
I'm
going
to
do
is
just
copy
it
bravely
into
the
models.
B
B
B
Now,
let's
try
now,
let's
try
actually
putting
some
data
in
that
conforms
to
that
model.
To
help
you
all.
I
actually
have
a
data
hub
put
command,
which
is
an
extension
to
the
cli
that
we
just
added
where
I
can
actually
add
this
aspect
called
data
quality
rules
with
an
entity
type
and
this
data
set
and
the
dq
rules.json
that
I
just
talked
about
I'm
going
to
hit
it
and
hope
it
works
all
right.
B
B
A
A
Yep,
okay,
so
here
we
introduce
these
data
quality
rules,
but
we
also
introduced
a
little
render
spec,
and
this
is
going
to
tell
the
ui
a
little
bit
of
information
about
how
to
default.
Render
these
aspects.
So
what
we
wanted
to
do
you
know
getting
the
data
into
datahub
is
one
thing,
but
making
sure
that
your
users
can
access.
You
view
it
and
access.
It
is
equally
important,
so
we're
introducing
a
couple
different
ways
that
you
can
render
your
aspects
depending
on
the
format
of
the
data.
A
So
there's
a
tabular
data
format,
and
this
is
for
an
aspect
that
is
a
list
of
something
and
then
there's
also
a
property
bag
style
render-
and
this
would
be
more
for
aspects
that
are
just
a
lot
of
properties
rather
than
a
list
of
elements.
We're
also
going
to
be
driven
by
community
feedback
to
understand
what
are
the
common
types
of
aspects
that
we
need
to
render
and
we'll
probably
build
out
a
few
more
default
renderers
for
different
scenarios
to
make
sure
that
the
majority
of
metadata
cases
can
be
covered
by
these
default
renderers.
A
So
for
our
data
quality
rules,
we
picked
tabular
and
then
we
also
just
gave
it
a
display
name
so
that
we'll
know
what
to
show
in
the
tab
and
then
let's
go
over
to
the
ui,
and
we
can
see
here.
If
I
go
to
this
I've
we've
ingested
a
few
data
quality
rules
on
the
logging
events,
data
set
and
then
we'll
see
a
dq
rules.
Tab
appears
and
it's
this
tabular
data.
We
also
have
a
couple
fun
little
easter
eggs.
So
if
you
we
see
a
url
we'll
render
it
as
a
link.
A
Booleans
will
translate
to
human
readable
and
then
we'll
probably
introduce
a
few
more
fun
renderer
types
as
well,
then,
just
to
demo.
What
another
render
might
look
like.
I
included
also
a
aspect
of
properties,
and
this
just
here
you
can
see
the
property
names
and
then
the
values
on
the
right
hand,
side
and
just
to
reiterate
this
was
done
not
through
editing
any
react
code
or
any
java
code.
All
shashank
I
needed
to
do
was
extend
the
metadata
model
with
those
few
pdl
elements
send
that
package
over
to
his
server.
A
As
soon
as
it
succeeds,
he
can
ingest
that
json
payload
and
then
it's
going
to
show
up
in
the
ui
just
like
this,
and
it's
going
to
look
just
like
any
other
aspect
that
we've
ingested
it'll
look
to
users.
You
know
just
like
it's
part
of
the
metadata
model,
and
if
these
two,
these
two
render
formats,
don't
work
for
you.
We
also
will
have
a
default
fallback
which
just
renders
the
payload
of
the
aspect
as
plain
json.
A
So
just
to
bring
it
all
home
here
are
the
few
short
steps
that
you
need
to
do
to
extend
that
metadata
model
so
clone
the
repo
as
shoshanka
said,
you'll,
add
your
new
models
to
this
custom
model,
module
and
add
the
models
and
change
that
yaml
file
ever
so
slightly
build
and
create
a
jar
and
then
deploy
it
to
your
metadata
service.
You
know,
using
whatever
method
works
for
you
and
we'll
provide
some
examples.
A
A
So
you'll
actually
for
free,
get
a
strongly
typed
graphql
api
with
these
custom
metadata
models
and
then
phase
three
is
gonna,
be,
will
have
a
rest
api
in
addition
to
graphql
types
as
well.
So
that's
what
we've
got
again.
Shushanka
said
this
is
going
to
drop
sometime
next
month
and
we
really
look
forward
to
hearing
your
feedback
and
then
continuing
to
iterate
on
this
with
the
community.
So
thanks
back
to
you
maggie
awesome,
I
hope
that
got
everybody's
heart
rate
going
this
morning.