Microsoft 365 Community Microsoft Search, 17 Feb 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Bringing in your own data to Microsoft Search

Description

In this 22-minute developer-focused demo, Kevin McDonnell uses Azure Logic Apps and Graph connectors (Microsoft and Ecosystem Partner built, or Graph Search API) for pulling external data sources into your search results. Use Graph search API. Step through the entire setup process: Create connector in Graph, Register connector in search, Create an Adaptive Card for viewing results, and Add items into indexes. Demo shows pulling content from Trello, using Graph calls in 3 Logic Apps (Search, Setup and Trigger). This PnP Community demo is taken from the General Microsoft 365 Developer community call recorded on February 17, 2022.

Demo Presenter: Kevin McDonnell | @kevmcdonk

Supporting materials:
• Documentation - externalConnection resource type | https://docs.microsoft.com/graph/api/resources/externalconnectors-externalconnection?view=graph-rest-1.0
• Repo – Create Schema example | https://github.com/kevmcdonk/S4MSC-Twitter/blob/main/schema.json
• Article - Creating a custom Microsoft Search connector with Logic Apps | https://www.mcd79.com/2020/09/30/creating-custom-microsoft-search-connector-with-logicapps
• Templates - Search Layout designer | https://searchlayoutdesigner.azurewebsites.net/chooseLayout
• Samples - MSSearch-Samples | https://github.com/kevmcdonk/MSSearch-Samples
• Samples - Samples for Microsoft Search Connectors – Twitter | https://github.com/kevmcdonk/S4MSC-Twitter
• Demo – Latest on Cloud Hybrid Search (SSA) & Graph connectors for indexing content from on-premises – Harshit Kumar (Microsoft) – 28:25

Learn more:
• Learn more about the Microsoft 365 Patterns and Practices community at: http://aka.ms/m365pnp
• Visit the Microsoft 365 Patterns and Practices community YouTube channel: https://aka.ms/m365pnp-videos

A

So I could try and keep the slides to a minimum.

A

What do we talk about is how you can bring your own data into microsoft. Search you've, probably seen a new search within there. It's in office.com it's in sharepoint, it's now in teams as well, which is fantastic. So it's about having that one capability that that one bit of search across everything- and it's it's not just about the search, it's some of the ai capabilities it brings with that it's trying to bring all those different things into one place, and some of you come from the sharepoint worlds. Maybe thinking yep.

A

This is all about sharepoint, search and probably using fast, but it's actually been built on the bing technology originally. So it's kind of got all your different client applications pulling that information through um and it's got your content all underneath there. It's pulling things from across um all the areas that the graph looks at uh within this you've got your sharepoint. Your team's chat, all in one place that you can bring things out, but the interesting bit I'm talking about today is that bringing your external data sources.

A

So this could be your third-party systems if you're using salesforce, if you're using your own on-premise once you can connect it up to database with that in azure or even on premise, you can connect it to the same things um with there and there's a few fairly simple rest commands in the graph that can help configure that, but then you trigger and in your own uh way of putting that into there. So it's not about just having that that nightly feed of stuff coming through.

A

It can also be about pulling that uh data as soon as it happens from there, which is what makes it really useful to people as well.

A

I've got some of these are animated. There are three ways you can get those graph connectors uh through the standard ways, and this is going non-divi at the moment, but there's some nice wizards for some microsoft, built ones and there's things like that, you can connect to external websites. You can, I think, salesforce and a lot of the atlassian ones are coming with that as well. If one of the standard ones isn't available, there's a whole load of partner ecosystem ones that have been built out.

A

So if you've got open text, if you've got all sorts of things like that, you can go in, but many of you devs or or connected with devs, and you want to know about how you can do yourself, you don't need to go and spend money with with partners. I'm saying this stupidly as someone who isn't a partner and would happily, if you do want to talk, come talk about that, but this isn't a sales pitch today. I promise this is about how you get your own data into there.

A

So you can use a microsoft graph to register your connector in terms of that, the overall process is the first thing you need to do. Is you need to register what your connector is? So you define your your kind of bucket of information? That's that's going to be in there, so one of the standard ones would be to do a website. I'm going to talk about a couple today. One is twitter and another is actually trello which, when I was putting this together, the the other night.

A

I thought, let's see how easy it would be to bring in something else and had I've been looking the trello api, something else I thought yeah. This should work once you've got that, then you can create your connecting graph and you can start registering those index items so pushing that data into graph. You do it sorry registering it that you want to use in search the next one is creating up your card, so all that great stuff that tim's doing about the adaptive cards.

A

You can use a fair bit of the elements of those to actually see how that appears when you get those search results back, whether you're in office.com you're in sharepoint, hopefully soon more of these will come into teams, but not not quite there yet, and you can render how those results come back quite nicely and then finally, it's about how you add those items to the index, so um it's probably the wrong text there. It's not adding the indexes. It's adding items into that index uh as well.

A

In terms of sending upwards connector getting into the the techie fun and games here, um you've got to set up an app registration in this eurad because you you've got to get your authentication up for the microsoft graph on there you're going to make sure you've granted permissions with that app registration, so that external items have freed right.

A

You then give your external connection a new, unique id uh name and description, so just those three elements there and then really the the interesting bits how you create that schema, so that that schema is the definition of your search index. So, for example, in twitter, you're gonna have who the tweeter is uh you're gonna have what time that tweet went out. You've got the tweet contents. You can even pull in there the number of retweets you could put in the number of uh likes that come in there.

A

So you've got to build out your your kind of object model to define, what's going to be captured within the index as well. In terms of how that schema looks, you can see. I've got an example here of maybe a ticket, so you could be pulling from your uh itil service management system, so whether you're using service. Now, if you've got an on-premise custom built one, you can pull the data from that. So you define your different properties there, such as the ticket title.

A

You define whether you want to be able to search on that text, whether you can retrieve the information. You can add some additional labels um and you can sort of say the types you could have: numbers dates, etc, etc, and and what you want to be returned um when, when you get those quite why you put things in there and not make it retrievable?

A

I haven't entirely worked out, but if that is the kind of thing you want to do, fantastic, knock yourself out once you uh I mentioned, you want to put things into the index so creating those items that come through again you've got another json schema from there, I'm going to come back to the acl in a minute, but it's about putting those properties, so those properties you've defined in the schema, such as the title priority assignee.

A

You need to add what you're going to fill out for those and then down the bottom, you're thinking more about the the content itself. So, if you're, if you imagine, search, usually you're looking at sharepoint pages or you're, looking at um like a document, the content is going to be that document itself when it comes to putting those things. What is the big block that you actually want to get put within there um that it's going to crawl an index from the search will pick up the details within that index as well.

A

The acl up, the top is the access control list. So the great things about your external stuff is you're, not necessarily just looking at um everyone can see this. You can put your own permissions on that, so you think, search within sharepoint is all heavily permissioned um you're going to be putting that thing through. You can do exactly the same with your content.

A

So if we come back to that, the idea of having your uh service management tickets in there, it may be that you have certain tickets, that people can't see- or maybe you can only see your own tickets. You can use the x access control list to define by groups. You can explicitly grant users, you can see we're getting the azure id ids from there or you can set up groups and you can deny people, so you could stop people from seeing things as well. So you've got those two different ways to set up the details.

A

There now a lot of the stuff. I talked about you're, probably thinking. Oh there's, gonna, be a whole load of code and different things like that. I can have to set up a massive infrastructure to either run these uh nightly indexes or I'm gonna have something.

A

That's gonna monitor that, but with this sure, you've got really easy ways of doing that and the way I've got set up and I've got a few samples of these which I'll share in a minute is using logic apps, because the nice thing there is for things like twitter for things like trello and so many other connectors you've already got those connectors so sorting out your authentication sorting out connecting with those apis has actually been done for you.

A

Obviously, if you've got your own custom one, then you may want to do with something else, but again with logic apps. If you've got oauth, it's done a lot of the hard work and you can connect back with that. So I think it's a fantastic way for working with these microsoft search services, assuming you're, not indexing, every microsecond and the cost of those is going to be relatively low and you've got the connectors there. So, assuming you haven't, got a massive volume, then you've got a really easy, effective way of running these.

A

So what I've created with these logic, apps is a setup logic app and that will register the the thing into the the connection.

A

It'll set up the defined schema um and that only really needs to be run once only once per environment you put in there, then I've got a search, one that you can run a regular basis, so this is kind of this could be set up on a timer, but if you're set up as a web one so I can trigger when I want, I kind of use it as a test bed more than anything else, and this goes out and retrieves like 19, the last 99 cards or the last 99 tweets for a particular thing or the last number of cards from trello.

A

You can do it when you want, but really the magic comes with the trigger where you can get trigger as soon as something's created. Assuming it's got that action in there within logic. Apps. You can get that item into the index straight away, but I said I wasn't going to do too much around powerpoint.

A

So let's go into some kind of code low code today, as you can see, I've got a resource group. Here. I've got my I'm going to show the trello one. First, I've got the trello one here, and I've got my three different uh logic: apps set up final one. There is just the api connection that gets set up when you put in uh place your trello connection.

A

If I go and have a look and uh edit that you can see, I've just got when an http request is received. So it's just got the details there and here I've got the four bits that are going to set up that connection. So I'm first going to check the microsoft graph, I'm not going to scroll down too much because it has got my keys in there, but you can see I'm just doing a fairly simple get that's going to retrieve in external connections.

A

I'm using the beta in this case, although actually it's it's out of beta now probably could update that one. But here here is the name of the connection. I've got so it's it's just going to try and get it, and if that fails, then it goes great. I need to create one so do the same, pretty much the same one, except now it's going to do a post and in that post you can see.

A

The body is pretty simple: it's just the id of the ones with it, the name that you're going to give it that people will see and that description, so very quick and easy to set it up. I then do a similar thing for the schema I just check and see. I now know that connections exist because either it existed already or I've just created it. So now I'll go and see if the schema exists.

A

If it doesn't, then I go and create it, and in terms of that schema you can see I've created up one for trello here, so I've, given it a card name. I've said that searchable. I don't want to refine on that, but I can query on it.

A

I've got on there the card id that I've captured probably not really needed, because you'll notice that um you'll notice later when I post and add an item that I use that card id as the unique identifier for each one, because I know each card will be unique as an entry um got things like the date last activity. So I mentioned it's not just strings. You can put dates in there.

A

You can put the card description and I've picked up a couple of things here um with trello you can put you're, probably more used to planner. You can have a checklist of items so where you have those kind of additional checklists, you can do the same here. So all I've done is how many exist and how many of those are completed so that that's neatly defined by schema. I'm now good to go.

A

If I look at search, let's uh edit, that as well even simpler, this is nice thing. If you've used power, automate logic, apps is uh pretty close with that. You've got these blocks. So in this case I'm just doing http record. I can't speak http request or I can. Click on run trigger and do it manually.

A

What it's going to do is look at a specific board that I've got set up in trello and I've just looked. I think I did the last five days. Yeah just looked at the last five days of content within that board and it's going to return return each of those cards and then for each of those cards. I'm just going to make an http call, I'm going to do a put because it may exist there.

A

I want to overwrite any that it already exists on there and it's going to do a post for each of those based on that card. Id that's come from. Trello in this case got my access control list, I'm being a bit lazy. I don't think any of this is important, so I'm just pulling back and making this available for everyone, but this is where I'd start to pull in maybe specific users that want to get in there if, if like with trello- and you haven't, got as your ad hooked up with that.

A

This is where you might want to put an additional step where you um you could. Probably get the emails of the people who have permission to it, go and do a look up using the graph to get list of those people build up your json here and have a more complex access control list. So you could do those same things. um This is just a lazy demo, I'm afraid so. I've just got the everyone within there. I've taken the description from the card and there you can see the properties. I've got that schema.

A

I've set up with the items complete and from there I've just taken the check items again, I'm not having to do any mucking around with json and different elements there, because it's all been created for me by the logic app. So I can just make use of those logic, app items. Of course, if you've got your custom apis, nothing stop you doing that. Http query against your custom api uh even connecting to a database and then pulling out those different items as you want to do in there.

A

But this will then write that external item into the search card itself. um Let's just trigger that off should run, hopefully quickly, always the nervous bits of uh doing a live demo. Let's see where that goes through.

A

In fact, let me just show you a previous one that did work here, my boy, you can do it, let's get it over the line: tick, tick, tick, tick, tick, tick, tick, okay, I'll, come back to that in one sec, just to show you what that looks like, in fact before I do that, I'm gonna jump and show you some of the configuration you need to do um in the admin center.

A

So within your admin center itself for the microsoft 365 gosh everything's running really quickly, just when I want it to um you've, got search and intelligence under there. So you can configure your search, and this is where you you can kind of configure those connectors themselves. Okay, I'm on internet! You can do it I'll, come back to this briefly I'll leave that running and loading. So you can see here. I've got that request. I've got my boards back. I've got my output from there.

A

I can see my trello boards, I can see which cards have come back. It's all sort of things I could do like looking at vote, the number of votes, the number of members that have voted for it. I could pull in there. I've just kept it simple, and here you can see my top 10 that have come back. I've got that access control list and I can see in fact there's a nice example that there was two scroll wheels getting a bit excited.

A

There was two on that checklist, but only one's been checked, so I can sort of show that, um on the results itself, if I come back here now, I've got the data sources, so you can see. Oh look, I've been doing some testing. With that um you can see. I've got my dynamics, connector. Here's I've actually got my website, so I've used that standard connector that you can get from there. But here's my trello connector that I've got set up. You'll, see the dynamics, one that haven't fully set up.

A

It actually tells me that I need to create a vertical and result type before that's actually usable. If I bring up the connect one and scroll down, you can see that it's index 12 items it's not showing on the bottom, but usually it says when you've last retrieved, something uh as well, but uh here I can see that I've got 12 items there, so fantastic I've got stuff in the index. I know I've got things in my quota, so it's putting those details there in terms of the customization you can.

A

You can have different result types. So uh if you wanted within your trello board, you know you might bring back the list of boards as well as a list of the cards, so you can create multiple result, types off the same one that people will search through and then really the the power is in these verticals that define how you want to actually display it. um I'm getting this the wrong way around.

A

What I mean is the results types uh display how you want to display it, so you could have when you got those results back, maybe some of them that have a checklist. You want to display those in a different way. You can- and there is your adaptive card, that you can set up, that, give you a details on how the results can go. What's lovely about that.

A

Is that there's a specific layout designer for these that is designed for microsoft search, so there's a slight difference in some of the things you can and can't do without that. But you can see that a lot of the designs are built out that you can very quickly and easily get one of those.

A

um Let's, let's go with a nice, pretty image one, and you can see it's to find the different blocks within there and all you need to do is define your properties that have come back from your schema, so that would be like card name within that. That would be the card link would define there once I complete that and click on create loud. When I finished out all those properties, um then it gives you the option to copy that json that you then just paste back into this json designer or if you're hardcore.

A

Look, I'm sure many of you are. You can just go straight on and copy your results into there and add it uh directly on there. So what does that look like? You can see here that now you can bring your uh your results, come up straight away and here's the website one. So these are actually linking to my blog. Once I add a vertical. I can see the tweets there. I don't think I have anything for that.

A

But if I look for collab talk, you can see what happens if you don't configure that adaptive card very well and don't connect it up. So in this case I haven't connected up some of those things. I have a feeling the formats changed recently, because the the ones I copied and pasted and the same for trello haven't quite come out as nicely as I'd hoped.

A

But here you can see I've added that trello to search I've got that modified date. I didn't actually add any text to any of these, which was a little bit silly, but here you can see I've got the checklise blah of five.

A

What I know I've done there and I showed this because I don't like to show everything too polished. um What I put there was forget to put uh how many had actually been checked. So I typed that wrongly, but if I click on that, it's going to take me over to trello and take me straight into that board. Item uh assume that I've logged, in which I forgot to do beforehand, but you can see you can link directly into those I'm not going to put my password in just in case.

A

Anyone is watching, um but you can bring those search items and you can add kind of direct links to the things you want uh as well. So you can make these look a bit pretty. You could have the trello logo within your search results and really bring it through from there. I saw todd you said: can you put hot links to tweets? uh Yes, yes, so the the links there you could see, I can put that's a blog post. I've got the ones to my blog post, the same with the trello items.

A

It could be the same with the tweets uh as well from that one.

A

um So pulling through, I just got a quick look through some of the questions before I go back to a couple things on the slide, um so yeah as, as you said, with the acl, if it's not using aad, then you need to build that as well.

A

um Can you make hot links in the tweets within the text of the tweet itself? um That is a good question. Actually um I don't know I'd have to shock I'll, try and get back to you tom on that one. I think they. I think you can, because I'm pretty sure previously, when I had um the kind of one I had was pulling stuff from collab talk there we go. There's an example: it's come back nicely now, so there you can see it's it's regis.

A

It's uh found it as a link and pick that up on there, what it doesn't do is have the um the hashtags, but you could do something within that. So we'll automatically recognize links. You could do something when you create that item to kind of convert hashtags into links as well and provide that link, which is the the bit. I haven't quite gone through uh itself.

A

Gonna uh flip back to the slides one. Second, we've got the demo. Just quick one on licensing only got a minutes left. If you've got e5 fantastic for each license, you've got you can index up to 500 license so a thousand licenses you've got 500 000 items in the index.

A

If you don't have an e5, then you will need to purchase, as it says there. 750 quid per instance or uk prices there. There is a link within there. If you just look at microsoft, search licensing, it will tell you kind of your local prices is within there um that you put up so quite a hefty jump. If you don't have e5s one other thing to notice is as the time of writing. There is a maximum of seven million items that you can have and that's for your whole index within your microsoft search environment.

A

So one thing if you've got a massive amount is do be aware of that. I believe you can speak to microsoft. There are some options there, but that is the default uh link to that. If you want to see more, I've got some links around this. I've got a blog post about how I set that initial one up with logic. Apps I've got some microsoft search samples. So if you are interested in going microsoft search, I've got some kind of csvs that you can input to get going on there and that sample.

A

I haven't created the trello one up, I'm going to do that- hopefully the next week or so to put those examples, but I have got that twitter one. I've got some arm templates, so you can just deploy or you can just copy each of the different elements for the logic apps and uh give it a go yourself. There's a dynamics one in there as well. I would love to hear from others if there's ideas of the sort of thing you'd like to do, especially if there's things like so salesforce is another one.

A

I'm planning to do, because I can get a free instance for developers to look at that, but um yeah, that's about it. If you've got any questions, please do um throw me anything on twitter at kevin mcdunk um or throw it in and I'll take a look at that as well, but otherwise, david back to you.

B

Kevin awesome, very, very cool stuff. Thank you for going a little deeper on that as well, since we had the time very, very great.