Microsoft 365 Community Community Demos, 22 Mar 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Latest on Microsoft Graph connectors SDK

Description

In this 16-minute developer focused demo – get the latest on connecting data sources into Microsoft Graph from Rajdeep Chandra. There are many OOB connectors, and you can create custom Graph connectors to connect to LOB data. This presentation covers latest SDK features and capabilities, architecture, gRPC, templates (with a project structure), Connector SDK Test Utility, and a step-by-step demo setting up a connector using C#. You are invited to join the private preview today. Expected public release 2nd half 2022. This PnP Community demo is taken from the weekly Microsoft 365 Platform Community call recorded on March 22, 2022.

Demo Presenter: Rajdeep Chandra (Microsoft)

Supporting materials:
• Registration – Microsoft Graph connectors SDK - Preview Sign Up | aka.ms/GCSDKPreview
• Documentation - Overview of Microsoft Graph connectors | aka.ms/GraphConnectors
• Documentation - Microsoft Graph connectors overview | aka.ms/GraphConnectorsAPI
• Documentation - Use the Microsoft Search API to index data | https://docs.microsoft.com/graph/api/resources/indexing-api-overview?view=graph-rest-1.0
• Documentation - Introduction to gRPC | https://www.grpc.io/docs/what-is-grpc/introduction/
• Connectors - Welcome to the Independent Publisher Connector Directory! | https://github.com/microsoft/PowerPlatformConnectors/tree/dev/independent-publisher-connectors

Learn more:
• Learn more about the Microsoft 365 Patterns and Practices community at: http://aka.ms/m365pnp
• Visit the Microsoft 365 Patterns and Practices community YouTube channel: https://aka.ms/m365pnp/videos

A

So hi everyone, I'm rajdeep, I'm from the microsoft graph, connectors team and today I'm going to talk about microsoft, graph, connectors sdk.

A

So previously also, we have talked about this topic, but since then there has been quite some changes that we have made after listening to feedback from you, and we have come up with a new architecture and a few little updates that we have on the sdk and we are going to cover those.

A

So just briefly I'll also talk about what are graph connectors, then we'll jump into the features and capabilities a little bit on the architecture and I'll be showing a demo of how you can actually set up a connector using c sharp to start off so graph connectors lets. You connect all your data sources, which are third-party external data sources and bring the data into microsoft, graph to power.

A

Intelligent microsoft, 365 experiences, so you may have data sources like uh file storage, some project data, finance application, anything right and we have out of the box connectors to connect all these data sources into microsoft. Graph, like for file storage, we have a file share connector for sales and crm. We have a salesforce connector, we have internet connector, we have a servicenow connector, and these connectors come out of the box from microsoft. But let's say you have some line of business applications.

A

Some custom database, some project data that you're storing somewhere, and you want to index that content into microsoft graph as well. uh So that is where microsoft graph connectors and the custom graph connectors come in. Essentially so the ones highlighted in blue are the ones for which you might want to write custom graph connectors on your own, and you can power. Different scenarios like search is one of the primary scenarios and there are also uh upcoming scenarios like let's say, e-discovery and viva topics.

A

All these things will be powered by the data that will come in through your data sources into microsoft, graph right so moving on today, uh you can create your line of business connectors or your custom connectors as such for these data sources using microsoft's graph search apis, but using graph search apis. Although you can create your custom connectors, uh there are a few drawbacks, like the cost of development is high using these, because these are not specifically made for building connectors and connectors. Specific functionalities are absent.

A

You need to develop them on your own, so reduce feature availability like like I was talking like e-discovery viva topics and all those things won't come out of the box. If you use just microsoft graph search apis, you need a. You also need to build a separate admin experience just to monitor all these custom connectors that you have written a separate dashboard, you'll be needing where you'll also do some kind of error, handling error management, which is also difficult today, right using these graph search apis and also on another effect, might be compliance issues.

A

So you might want to reflect the same data that you have in your data source into the microsoft graph index, and once you delete some content from your data source, you might want that exact replication into microsoft graph. So this requires timely purging of deleted content to ensure data freshness and all these logics are pretty complex annuity and you need to write and develop these logics on your own.

A

In case you are using the graph search apis, but what we are doing today with the sdk is packaging a lot of these capabilities together into what we are calling the microsoft graph platform, the graph connector platform right so graph, connector platform is a platform that has been built over the last few years on which our out of the box connectors are working. So all the connectors that I talked about, which come out of the box, the native connectors, like servicenow, confluence jira file, share all these.

A

All these connectors are running on this exact same graph connector platform, which we are enabling, through the sdk for customers, also to uh to leverage this to write their own connectors. So this is a really mature platform and also going forward whatever updates we make will be made available to you through this platform. So this platform today has the capability of getting the data from the connector doing crawl management full, as well as incremental crawls, then scheduling the scrolls triggering the scrolls, the integration with the microsoft admin center right.

A

So you have a m365 admin center, where you are managing your out of the box connectors today. But if you write your custom connector using draft search apis, you need a separate admin center. You can't monitor and manage them from the same m365 admin center. But if you come through the microsoft graph, connector sdk and write your custom connector, you will be able to start monitoring managing all your custom connectors as well through the m65 admin center.

A

Apart from that, we also have delete detection uh difference, detection cycle detection, identity, mapping, graph injection everything is essentially taken care of by the connector platform, and the only thing that we expect from you when you're developing your custom connector is just to tell us how to read from the data, how to read the identities and what is the data source schema?

A

So this is all we need from you in terms of development of a custom connector, and we are abstracting all these capabilities that earlier, you might have to write on your own into the graph connector platform.

A

So apart from that, just to summarize the benefits you get a fast and efficient development, and we are so today you can, with this graph connectors sdk, you can write in any 11 languages of your choice, so we have support for up to 11 languages.

A

Your data sources can be cloud or on-prem data sources, so the connector platform that we already talked about is already really robust and you can leverage the microsoft admin center to manage your custom as well as out of the box connectors all in one single place, so moving on. This is how we have designed the architecture so that you can write your connector code in multiple languages, so the microsoft connector platform hosts the orchestration framework, essentially the framework which manages and schedules all your crawls and it talks via remote.

A

Procedural calls to your line of business connector code that you have written and your line of business connector code is responsible for talking to the data source and getting the data, and our microsoft connector platform will make calls to your line of business connector to get the data and pass it on further through the admin center into microsoft graph.

A

So this rpc framework is actually based on grpc, which is open source and it works in a client server based model. So our microsoft graph, connector platform hosts the client and your connector line of business connector code hosts the server and our client makes calls to this server to get the data and we talk over the tcp port, and this is you: can stream unidirectionally and by bi-directionally, and what enables these 11 languages to make this framework language neutral and platform neutral is something called protocol buffers.

A

So these are ways to share contracts between two systems in a grpc model, and this helps us to easily create distributed applications and services right. So this is a brief on how grpc works, so the server can be implemented in any language and it is, and it is independent of whatever language the client is essentially written in right, so your grpc client can be in ruby or android java, and the server can be in c plus plus server. So, although they are in different languages, it is you can talk to.

A

We can talk to each other using this grpc framework, so how this applies to our entire line of business. Connector sdk is essentially this microsoft connector platform, on which I already said that our out-of-the-box connectors are already running, like let's say, on-prem connectors or file share connectors, so we host the grpc client inside this connector platform and it talks over http to a grpc server which is inside, which again hosts your connector implementation, the connector port, and you can have multiple connectors that are running that can talk to the same connector platform.

A

So let's say you install one microsoft connector platform and you have three different data sources for which you want to write. Let's say line of business connectors or even you want to run your out-of-the-box connectors right, so you can have multiple connectors running on just talking to one single connector platform, and these should reside on the same virtual machine, the connector platform and your line of business connector code. So you need to define the port on over which each of these connectors are running so that this grpc client can talk to each of the servers.

A

And accordingly, if you update the configuration file, these connectors will start working right so just to go quickly over how the entire flow is of developing a line of business connector. So you can download the through id. If you are using c sharp, then you can download a template that we are already providing and it has a project structure. You open up, write the code to read data from the data source as an admin.

A

You need to install, let's say this connector platform and this and update the configuration files and you need to run the line of business connected code that you have written. And finally, you go to the microsoft admin center and it's a very familiar flow of publishing and out of the box connector. So it's same for the custom connector as well.

A

All you need to do is you need to update this manifest file and your and fill in the necessary details to publish your line of business connector so how this works in c sharp is that you already have a template, so you can go to visual studio, extensions and download this template, and once you install the template and search, you go to create a new project and you try to search for the template.

A

Something like this will come up and once you install the template and you create a project using this template, it already has a project structure with some sample code inside it. So once you finish coding update the code, it will generate a exe file and this needs to run on the same vm, where you are running your graph connector platform and you define the connector port over which this connector is running, so that this graph connector platform can talk to your particular connector.

A

And once you update some configuration files, you complete the testing and you feel that it is okay for deploying in your production environment. You can directly go to. Let's say you can directly go to the admin center and you can, through the custom connector option. There is a new option that you'll start seeing.

A

You can update the manifest file and go through the familiar steps to publish the connector.

A

So to give you a brief on how this works directly in visual studio, so in visual studio, I already have this connector template installed, which has this project, so this project has a server cs file which, where you essentially create the server- and you mention the port over which this particular connector is running, and you define a connector id. This will already be defined, automatically be generated for you. You can change it as well.

A

This is a unique connector id for your particular connector, and then we have some dummy database that we are trying to create and reading and over here in connection management service. It's a very familiar flow. First, you define how the authentication works so in validate authentication. You tell us how we can talk to and watch what authentication we can use to get access to the data source.

A

Then next, once this is done, you can tell us how, if there is any custom configuration that we require to access this data source and once that validation done, you tell us what is the data source schema? So what is the schema of the data source that you are that will be expecting?

A

And finally, in the connected crawler implementation? This is where you essentially write the code so that we, you can essentially crawl through the data source, get the data and send it to our connector platform. So we also have a check pointing mechanism so that increase in case there are any crashes. This can automatically resume the crawl. Once your connector server or let's say your connector platform- is up and running wherever the issue went wrong. So once the coding is complete, we also have this testing utility.

A

So in this test utility we have the test cases and you can simply press the different options that are there. Let's say you press one, and you want to test whether the connector platform can talk to the connector service and you it will show options like uh whether it is success where it is failing.

A

So how this works is that I have a configuration file where I've essentially said that this is the connector id which is running on this particular port and this test utility essentially tests, whether this connector is actually running on that port and whether our graph connector platform can talk to that. Similarly, you can run the other test cases and once everything passes, you can go into microsoft. Admin center go to add, and you start seeing the new option that I had just shown.

A

So today you will see all these other options, which are essentially the out of the box connectors, and this is the new option called custom connector that will come up and you can paste the manifest file.

A

So I have a very simple manifest file over here, so the format is present over here as well as will send it over in the documentation, and you can click next next and it's a very similar flow of you can give any name- and this is exactly same as how today's out of the box connectors work and you can click next and follow the process to essentially publish your connection and it.

A

It will hardly take two minutes to essentially publish this connection and once you're done, I already have a custom connector that is published and up and running so once you're done, it will be in something like a ready state and you can open this and actually see what are the crawl statistics. When was it last crawl number of items that were indexed everything from the admin center in one single dashboard, so you don't need to go anywhere else to actually check what is going on with your data source.

A

Like you can see over here, I have an index limit of 700 000 and I have index five thousand four hundred and eighty nine items as well as what is the crawl schedule. If you want to change the schedule, you can go into edit and change it. Similarly, all the all the benefits that are there or all the capabilities that are there for the out of the box. Connectors comes to this. These custom connectors as well, so that is all from my side in terms of demo and in terms of timelines.

A

So today we are in private preview and we expect to be in public preview by calendar year 22, and that is the same timeline around which a lot of our other experiences are also coming up. So in case you integrate your third party custom connected data by then, then you can also onboard into these other experiences like viva topics, integration, e-discovery support around uh calendar year, 2022 h2 timeline, so that is all from my site.

A

I also have a link if you want to sign up for this private preview, I'll share it on the chat and if there are any further questions I'll take it on the chat and I'll be here online. Please feel free to post your questions. Thank you.

A

Thank you. Right, really, really cool stuff and awesome stuff.