►
From YouTube: Microsoft Graph Data Connect overview
Description
In this 16-minute developer-focused demo, Nik Charlebois delivers an end-to-end tutorial on extracting information at scale from Microsoft Graph. Starting with an architecture overview – learn about creating an Azure Data Factory, Azure storage account, and establishing the data pipeline connection between the Microsoft 365 dataset and storage account. Once the pipeline is established, define source and sink, trigger the sync, and use Power BI to display data insights. This PnP Community demo is taken from the Microsoft Graph community call – June 2021, recorded on June 1, 2021.
Demo Presenter: Nik Charlebois (Microsoft) | @NikCharlebois
Supporting documents:
• Web - Microsoft Graph Data Connect - https://azure.microsoft.com/en-us/services/graph-data-connect/
• Documentation - Overview of Microsoft Graph Data Connect | https://docs.microsoft.com/en-us/graph/data-connect-concept-overview
Stay connected:
• Twitter - https://twitter.com/microsoft365dev
• YouTube - https://aka.ms/M365DevYouTube
• Microsoft 365 Developer Blog - https://aka.ms/M365DevBlog
• Microsoft 365 PnP Blog - https://aka.ms/m365pnp/community/blog
A
A
So
essentially,
when
you
think
of
the
microsoft
graph
story,
there
are
really
three
main
pieces
to
the
the
entire
graph
story.
Right,
there's
the
rest
apis
which
we're
all
used
to
they've
been
around
for
a
couple
of
years.
Now,
there's
the
graph
connectors,
which
essentially
does
data
ingestion,
allows
you
to
enrich
what
you
have
in
the
graph
and
then
there's
this
third
piece
at
the
bottom
of
the
diagram.
Here.
That's
data
connect
that
lets.
You
extract
information
in
bulk
from
the
microsoft
graph.
A
A
So
today,
if
you
were
to
grant
an
app
registration
access
to
read
information
about
emails,
for
example,
if
you're
not
granting
the
mail
that
read
basic
permissions
by
default,
that
application
will
have
access
to
read
everything
about
the
email,
including
the
buddy,
the
body
preview.
So
all
the
information
that
you
might
not
necessarily
need
or
want
to
grant
the
application
access
to
with
data
connect.
What
you
end
up
doing
is
you
essentially
create
a
contract
between
the
tenant
and
men
and
the
application?
That's
going
to
be
performing
the
extraction,
so
you
specify
all
right.
A
I
want
to
get
information
about,
let's
say
events
or
emails
for
my
users,
but
I
don't
want
to
go
and
extract
information
about
the
buddy,
the
actual
content
of
the
email,
so
you
can
filter
it
out
and
when
the
extraction
happens,
that
property
will
never
be
extracted,
so
it
won't
ever
make
it
to
the
the
end
result
file
when
you're
extracting
data
using
microsoft
graph
data
connect,
essentially
what
you're
doing
is
you're
creating
a
raw
json
file.
That's
going
to
be
hosted
in
an
azure
storage
account
right.
A
So
if
you're
specifying
that
you
don't
want
the
buddy
preview
and
the
buddy
of
the
email,
those
properties
will
never
make
it
into
that
raw
exported
file.
And
then
the
third
focus
area
is
really
data,
governance
and
security.
Because
the
data
is
going
to
be
stored
in
azure
storage
accounts.
You
have
access
to
all
the
audit
logs
to
be
able
to
figure
out
who's
been
reading,
that
data
who's
been
interacting
with
the
data
you've
exported.
A
On
top
of
this,
the
tenants
remember,
I
was
mentioning
that
contract
that
you
need
to
establish
the
tenant
and
men
will
also
be
able
to
review
that
contract
and
grant
or
revoke
access
to
your
application,
to
be
able
to
extract
that
information
or
filter
it
down.
Right
so,
let's
say,
for
example,
you
requested
access
for
the
emails
of
the
legal
and
human
resources
team
for
in
your
tenant.
A
The
tenant
admin
will
receive
that
contract
say
right,
so
we're
trying
to
extract
the
emails
from
everybody
in
those
two
groups-
oh,
but
I
don't
want
them
to
have
access
to
the
legal
team,
so
I
can
then
use
what
we
call
the
denial
list
and
filter
out
the
result
and
say:
look
we're
going
to
grant
you
access
to
the
human
resources
group,
but
not
to
the
legal
team,
so
data
governance
and
security
so,
overall
how
it
works.
There
is
a
an
azure
component.
It's
microsoft
graph
data
connect.
A
That
data
pipeline
will
extract
a
data
source
that
is
of
type
microsoft,
365,
and
you
will
specify
what
data
set
you
want
to
go
and
export
on
the
other
end
of
the
pipeline,
you're
going
to
have
what
we
call
a
sync.
So
essentially,
in
this
case,
it's
an
azure
storage
account
in
the
demo.
I'm
going
to
be
doing
in
just
a
few
minutes.
A
A
Today,
we
support
multiple
data
sets.
We
most
of
them
revolve
around
the
outlook
entities
we
have
planned
to
introduce
new
data
sets
such
as
the
sharepoint
online
file
sharing
file
usage.
We
are
going
to
be
releasing
the
team
chat
that
you
see
at
the
bottom
of
this
slide
here
is
going
to
be
released
within
the
next
few
weeks
and
those
are
the
one-on-one
chats
and
the
one
to
many.
We
have
information
about
calendar
view,
events,
messages,
contacts
and
so
on
right.
A
So
all
the
data
sets
you're
seeing
here
are
available
for
you
to
try
today,
we
recently
updated
our
documentation
with
the
bill
conference
being
last
week,
so
we
just
updated
our
documentation
and
you
can
find
sample
data
sets
for
every
data
set
that
we
support
as
well.
So
if
you
just
go
to
docs.microsoft.com
search
for
microsoft,
graph
data
connect
where
you
have
the
list
of
all
the
supported
data
sets.
A
So
we
mentioned
previously
that
you,
you
have
data
privacy.
I
kind
of
got
ahead
of
myself
there.
I
had
a
slide
on
it,
but
essentially,
when
you
create
the
pipeline,
you
define
the
terms
of
the
contract,
so
you're
extracting
information
for
what
users
and
what
properties
are
you
extracting
the
data
for?
A
So
let's
do
a
quick
demo
of
this.
So
essentially
the
first
thing
I
did
is:
I
created
a
storage
account
so
an
azure
data
lake
storage
that
has
a
container
called
datasets
with
the
messages
folder.
This
is
where
I'm
going
to
go
and
export
the
information
from
so
I'm
going
to
be
extracting
information
about
all
the
emails
in
my
tenant
and
I'm
going
to
be
storing
the
file
in
this
storage
account.
A
In
order
for
you
to
be
able
to
activate
microsoft,
draft
data
connect
as
a
tenant
admin.
What
you
do
is
you
navigate
to
your
office
365
or
microsoft,
365
admin
center,
you
go
to
the
settings
or
settings
and
in
there
you're
going
to
be
able
to
see
microsoft.
Graph
data
connect
listed
right
here,
simply
click
on
it,
and
then
all
you
need
to
do
is
turn
on
microsoft.
Graph
data
connect
for
your
entire
organization.
A
You
also
need
to
specify
a
a
measure
id
group
of
people
that
are
going
to
be
responsible
for
proving
the
request
so
remember
that
contract
that
I
mentioned
people
in
this
team
here
are
going
to
be
the
people
that
are
allowed
to
go
and
approve
or
revoke
the
data
extraction.
So
in
my
case
the
group
is
called
mgdc
approvers.
A
A
Git
integration
in
this
case
just
going
to
go
and
create
a
brand
new
one
that
normally
takes
about
10
seconds
to
complete,
and
what
this
is
going
to
give
me
is
essentially
azure
data
factory
for
those
of
you
that
never
played
with
it.
It's
going
to
give
me
a
new
workbench
where
I
can
go
in
and
define
in
a
visual
fashion.
What
my
etls
or
extract,
transform
and
load
operations
are
going
to
be
on
any
data
set.
A
So
this
is
what
the
azure
data
factory
workbench
looks
like
I'm
going
to
go
in
and
edit
my
pipeline,
or
actually
I'm
going
to
go
and
create
a
new
pipeline
here
and
then
I
have
a
set
of
different
activities
that
I
can
perform
on
the
data.
In
my
case,
microsoft
drive
data
connect
is
actually
a
source
for
the
copy
data
right.
So
just
remember
we're
going
to
be
extracting
that
data
and
storing
it
as
a
raw
json
file.
So
I'm
going
to
drag
and
drop
that
copy
data
activity.
I
can
give
it
whatever
name.
B
A
Example
and
then
I'm
going
to
need
to
go
and
specify
a
source.
So
what
data
am
I
extracting
nsync,
which
is
essentially
where
I'm
going
to
be
storing
the
data
at
the
other
end?
So,
let's
start
with
the
source,
I'm
going
to
go
in
and
define
a
new
data
set,
and
in
here
I'm
going
to
be
searching
for
the
office.
365
data
set
click
continue
on
this,
I'm
going
to
go
and
edit
the
properties
of
it.
What
I
need
to
do
is
create
what
we
call
a
linked
service,
so
essentially
authenticating
against
my
tenant.
A
In
my
case,
what
I've
done
is
I
created
an
app
registration
in
my
tenant
that
doesn't
have
any
permissions
whatsoever,
so
I
didn't
grant
it
any
api
permissions.
We're
only
going
to
be
using
that
app
registration
to
authenticate
back
to
our
sync,
so
my
storage
account
actually
has
granted
a
blob
contributor
role
to
that
service
principle.
A
I'm
going
to
click
I'm
going
to
test
the
connection.
First
make
sure
everything
works,
I'm
going
to
go
and
create
that
so
we're
good.
Now
I
have
a
list
of
all
the
data
sets
available.
In
my
case,
I'm
going
to
go
and
pick
the
messages
data
set
all
right,
so
that
is
going
to
give
me
access
to
all
the
emails
in
my
tenant.
A
It's
going
to
go
back
to
my
pipeline.
What
I'm
going
to
do
is
I'm
going
to
import
the
schema.
So
what
it's
going
to
do
is
it's
going
to
analyze
what
that
data
set
contains
as
far
as
data
fields,
and
it's
going
to
load
them
all
in
here
and
you
can
see
if
you
scroll
down
that
I
do
have
access
right
now
to
the
body
and
body
preview.
A
So
my
organization
has
privacy
concern
with
these
two
I'm
going
to
select
them
out
and
just
remove
them
from
the
exported
schema,
I'm
going
to
specify
a
filter
as
well.
So
I'm
going
to
say,
I
only
want
to
extract
information
from
emails
that
were
sent
in,
let's
say
the
past
quarter.
So
from
march
1st
to
june
1st,
and
then
what
I'm
going
to
do
is
by
default
it's
going
to
go
and
extract
information
about
all
users
in
the
office
365
tenant.
I
want
to
change
that
and
I
to
specify
specific
groups.
A
So,
in
my
case,
I'm
going
to
go
in
and
I'm
going
to
do
the
legal
department,
the
legal
team
and
the
finance
yep.
I
can
only
type
properly
and
finance,
so
it's
only
going
to
go
and
extract
information
from
these
two
groups,
so
it
looks
like
my
source
is
good.
Next
thing
I
need
to
do
is
specify
where
I'm
going
to
be
storing
that
exported
file,
so
I'm
going
to
go
and
create
a
new
data
set
for
the
sync.
A
A
Remember
that
I
need
to
authenticate
with
service
principal
I'm
going
to
pick
my
storage
account,
which
is
community
call,
and
then
I'm
just
going
to
go
and
copy
the
information
about
my
service
principle,
so
the
app
id
and
an
app
secret
that
I
that
I
created
for
my
app
registration,
just
gonna
test
that
make
sure
it
works.
We're
all
good,
I
create
the
link
service.
A
Now
I'm
going
to
go
in
and
go
in
my
data
set
container
that
I
showed
you
previously
and
I'm
going
to
select
that
messages
folder
all
right,
so
we're
all
set.
My
sync
is
created.
My
my
source
is
created.
What
I
can
do
now
is
publish
that
pipeline,
I'm
going
to
click,
publish
on
this
and
then
I'm
going
to
go
and
manually
trigger
it.
What
you
can
do
is,
you
can
actually
have
scheduled
triggers
right.
So
let's
say
you
want
to
do
an
export
every
quarter
or
every
week
you
can
have
a
scheduled
trigger.
A
A
A
A
So
that's
a
little
more
user
friendly,
but
you
can
review
make
sure
that
the
buddy
is
not
there
and
then
let's
say
in
my
case
I
I
want
to
approve
for
the
finance
team,
but
legal
team.
This
is
a
no-no,
so
I'm
just
going
to
go
in
and
pick
legal
from
here,
so
I
can
filter
the
request
and
say
approve
and
boom.
So
essentially,
the
pipeline
will
start
its
execution.
The
data
will
be
stored
in
my
azure
storage
account
and
then
I'm
going
to
end
up
with
a
file
that
looks
like
this
here.
A
That
is
raw
json.
That
represents
the
content
of
the
emails.
I've
extracted
right
and
with
this
really
really
easily,
once
you
have
access
to
that
data,
you
can
start
creating
some
very
interesting
insights,
using
something
like
power
bi,
for
example.
So,
for
example,
in
my
case,
what
I've
done-
and
this
is
a
true
example-
is
my
test.
Tenant
I've
actually
mapped
my
my
exported
json
file
to
power,
bi
dashboard
and
in
here,
for
example,
I'm
getting
inside
the
ceiling
right.
How
are
people
using
attachments
in
my
my
environment?
A
You
can
see
that
you
have
a
link,
so
reference
attachment
is
essentially
when
you're
linking
to
a
file
in
your
onedrive
instead
of
sending
a
the
actual
attachment
where
people
are
creating
those
underscore
v2
underscore
v3
and
so
on.
I
want
people
to
start
using
reference
attachment,
but
you
can
see
that
only
0.04
of
my
organization
is
using
it.
So
there's
probably
room
for
improvement
right.
I
could
probably
do
a
training
program
or
something
email,
statuses
right.
A
So
how
how
many
emails
are
unread,
but
based
on
different
teams
right,
so
you
can
try
to
understand
right.
How
are
teams?
What
teams
are
overloaded
right?
Let's
say
your
human
resources
team
on
average
they
have
300
unread
emails
per
person.
Well,
maybe
there's
something
wrong
there
right.
You
should
probably
try
to
figure
out
what's
going
on.
Why
are
they
overloaded,
like
this
being
able
to
understand
trends
when
are
people
using
emails
throughout
the
day,
and
then
you
can
go
and
do
those
very
interesting,
deep
insights
and
figure
out
all
right?
A
How
are
people
collaborating
inside
of
my
organization
right?
What
are
the
collaboration
diagrams?
How
are
people
talking
with
one
another
across
the
different
teams
and
that
will
easily
surface
very
interesting
insights?
So,
for
example,
this
is
my
retail
team
they're
not
talking
to
my
hr
team
at
all
like.
Why
is
that
right?
A
It
seems
like
my
hr
team
is
only
talking
to
find
this
thing
through
these
two
folks
like
why,
where
is
the
connection
there
and
this
all
the
dashboard?
I
showed
you
there
was
built
within
about
five
minutes
just
by
exposing
that
raw
json
data
that
you've
exported,
and
so
this
is
just
to
give
you
an
idea
of,
what's
possible,
now
brian's
going
to
be
talking
about
our
technical
assessment
program
later
on.
We
do
have
a
specific
program
for
data
connect,
which
is
essentially
going
to
be
giving
access
to
data.