Microsoft 365 Community Community Demos, 26 Aug 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Photo ID Web Part using Azure Cognitive Services

Description

In this 17-minute Developer - focused demo, Hugo Bernier introduces us to an intelligent assistant that validates photos. Capturing photos in a web part is not new. What’s new is using Azure Cognitive Services - Computer Vision to evaluate and then approve or reject submitted photos. No need to train the Service just define the evaluation criteria, get and secure a services key and end point, access the API, analyze submission and render results. This PnP Community demo is taken from the SharePoint Framework and JavaScript Special Interest Group (SIG) – Bi-weekly sync call recorded on August 13, 2020.

Demo Presenter: Hugo Bernier (Tahoe Ninjas) | @bernierh

Supporting materials:
• Documentation - Computer Vision API (v3.0) | https://westcentralus.dev.cognitive.microsoft.com/docs/services/computer-vision-v3-ga/operations/56f91f2e778daf14a499f21b
• Repo - Smart Profile Photo Editor | https://github.com/pnp/sp-dev-fx-webparts/tree/master/samples/react-smart-profile-photo-editor
• Overview - Computer Vision | https://azure.microsoft.com/en-us/services/cognitive-services/computer-vision/#product-overview
• Guidance - Reusable property pane controls for the SharePoint Framework solutions | https://pnp.github.io/sp-dev-fx-property-controls/
• Documentation - SharePoint Online tenant properties | https://docs.microsoft.com/en-us/sharepoint/dev/spfx/tenant-properties?tabs=sprest
• Repo - PropertyPaneMarkdownContent control | https://github.com/pnp/sp-dev-fx-property-controls/blob/dev/docs/documentation/docs/controls/PropertyPaneMarkdownContent.md

Learn more:
• Learn more about the Microsoft 365 Patterns and Practices community at: http://aka.ms/m365pnp
• Visit the Microsoft 365 Patterns and Practices community YouTube channel: https://aka.ms/m365pnp-videos

A

My name is hugo. uh This is my mugshot and I can be reached at uh bernieh on twitter and I wouldn't enjoy that blog, I'm an mvp and a proud member of the pmp team, and today I will be talking about a web part that I originally built without spfx.

A

It was a web part that I created for a college that wanted to allow students to upload their own photo id. Now. The problem is at that age. Those students are not always making the best decisions. That's why we have to put messages like do not eat laundry detergent.

A

I mean it's their fault for making the laundry detergent look so delicious, but you know uh you know those people were not necessarily making the best decisions when they were uploading their pictures and they had a person full time reviewing photo ids. That.

B

People were submitting.

A

So the requirements were- and I like this quote by the way by douglas adams. That says you know that, basically, if you try to design something, that's completely foolproof. It's because you underestimate the ingenuity of complete fools.

A

So we tried to do something that was foolproof, but it would require students to be able to upload passport style pictures that would be a one-to-one ratio like a square picture, only one person per picture and it should be the student's own picture, not their children or pet and trust me, you would you wouldn't believe the pictures we got no avatars, no drawing no uh racy images. Yes, we did get racy and adult images, uh no hats, but religious headwear should be permitted and no sunglasses.

A

You know and again the whole idea was to be able to automatically approve the pictures so that we wouldn't have to have someone just manually reviewing pictures, so those people could spend their time doing useful stuff.

A

So the web part that I'll be showing today is a web part that will automatically review those, and you can see here the requirements are validated and tells you what's wrong uh and the idea is to allow students to actually create good pictures, upload pictures that are automatically approved like this one here from visa.

A

I may have overwritten the age there by the way it was more like 29.. uh So how do we do that? Well, spfx to the rescue let's get to the demo?

A

Well, the first thing is this: webcard that you'll be seeing here today is already available uh for you in the spfx webpart samples repo. You can use it as is, and you'll see it's completely customizable.

A

Let's start with talking about the computer vision services, so with azure cognitive services, you have the ability to actually upload pictures or submit pictures and have them automatically reviewed. So, for example, if I look for a picture here- and I take let's say this picture here- if the demo gods are with me so you'll see that it's automatically identifying information about this picture and it's identifying uh keywords uh like a person, man, woman, camera, it's just missing tv really, uh so that's that's great and we can actually do that by also calling the apis directly.

A

So if you go to the api, you can actually make post or get request on different things, and it will document everything you need to do so you pick a location and then you specify the features you want to identify. So we want adult. We want categories, we want description faces and things like that and then you can also detect celebrities and one of the reason why it detects celebrities is because we used to have jokers that would submit pictures of celebrities, pretending that it was them.

A

And then, when you submit the picture, you can determine whether it's a url to a picture, a binary stream or multiform data, which is what I'll be using and the response you get looks a little something like this, where you get a json structure with it's, never a hundred percent right, we're using uh machine learning here. So it's never 100, but it's definitely a high confidence, 99 confidence uh that these things are. You know not racy, for example, and things like that.

A

So what you need to do to get this going is you can go to your azure and you can create a new ai cognitive service resource in azure, and it will create a key and an end point for you. I'm not going to share my key with you, but that's the problem right. So if you have an api key and it's supposed to be secret, you probably don't want to allow your users to or ask your users when they configure the web part to enter the key.

A

At the same time, you probably don't want to hard code those values in your web part, and this is where there's a really cool feature. I don't think many people know about is the sharepoint online tenant properties and what it allows you to do. It allows you to call a rest api from within your web part to retrieve properties that you want to store centrally at your tenant level, and there are no rest apis to that. I know of to set the the properties, but you can actually use things like pnp, powershell or office 365.

A

To do this, so you would call with office 365 you called spo storage entity, and then you would give it a unique name, a unique value, for example I'll. I use the very original azure key and azure endpoint uh and then, when I call the rest api from within the web part, I get the value. So this is a great way to allow me to deploy web parts that have secret apis or secret keys or secret values that I don't want people to see for the purpose of the demo.

A

However, I did customize the web part so that when, if you don't specify the the storage entity, it will automatically show the prompt the user to enter this. So, let's see all right. So this is my.

B

A

Part, and as soon as you do this so you can see here because I didn't use tenant properties, I have to configure it so to configure it here. I have to provide again my key, which I will copy and my endpoint and by the way, if anybody is displaying like asking questions, I can't see the the chat window. I just want to point out here.

A

This element here is you'll you'll notice, it's actually rich decks that I've displayed inside of the property pane, and this is usually not trivial to do, but there's a new pnp, reusable property pain control, that's currently in beta that allows you to use things like markdown to actually create text. This is a great way to provide guidance for your users when you want or help or support information.

A

When you want people to do this and then at the bottom here I can say I want portrait photos, I don't want to allow clip art, racy images, adult images and so on and so forth and I'll come back to the four bin tags. If I have time so now, all I should be able to do here is you should be able to just.

B

A

Drag and drop click or things like. So if I pick, for example, this picture here, which is a real passport picture uh and if I I can just kind of crop it, when I click submit, you'll, see the gratuitous animation and then it says yes, this is a picture and I'll click update later, let's try a different picture.

A

So if I use let's say this picture here, this is actually not a passport picture. This is not a portrait. This is kind of a just a mug shot right. So let's see vesa, I'm sorry, but uh it hasn't been kind to you. It says you're 53., uh just for fun. Let's look at the slightly younger picture here again. This should not get approved. But let's see uh oh, it didn't like that.

A

uh Let's try a different picture here.

B

A

If I look at, for example, I take a picture of me before I've had my first coffee in the morning, and I submit that it should automatically detect that this is a gory picture.

A

Now I know everybody's looking for it for me to show a uh an adult picture, I'm not going to show an adult picture. I will, however, show a picture of my future wife um and if I actually take this picture here, which is as as racy as as I could do in this call- and I submit that it should automatically detect that this is a racy picture.

A

Now uh all the call information that we're getting behind the scene, all the information I'm in the web part sample, I'm actually outputting the the return information in the console. So, if you're ever interested to see what kind of analysis uh we're getting. This is all the information here so see here. It's actually saying. Yes, it is racy content and there's a lot of information.

A

B

A

Take a real picture of me instead, just to show uh you know that this thing works, and I just go like this. I submit it'll actually say hey. This is not a uh this is not a portrait, it'll also say: hey. Did anyone ever tell you uh how much you look like daniel craig? It's uh I get that all the time so I'll just change this to be more like a passport photo, and I can just click and update right.

A

So this is the same uh web part now uh I could actually use the camera if I wanted to, and this works on mobile devices as well, hello, everyone, smile uh yeah, you see the camera at that's 10 pounds, so I'm probably overlaying about three cameras.

A

uh So I'm showing here my profile picture- and I guess I should have showed you when I when I click on update my profile picture- should always update here there. You.

B

A

uh So we've got a few more minutes, so let me get to the code to show you the code unless there's a specific picture that anyone wants to see. But let's do this so the first thing we do is we in the web part, we have a storage entity service that I've created which just gets the api calls the api that we talked about and we use both the the storage key as the name.

A

So that's the api key and the the api key and the api endpoint, and then I just do a regular http response or a request to get the information and then whatever value is coming from the json structure, I'm just looking at the dot value and I'm returning that back to my entity service in my analysis, services, yes I'll, actually verify multiple people as well. In my analysis, services.

A

What I do is, I just grab the api key and endpoint whether it was provided by the tenant storage or by the by the web part configuration, and then I do some very embarrassing, uh search and replace to remove all the the prefix to say that this is an image so that I can get just a base64 uh buffer image that I can pass and you'll see here that I just passed the api key and the api endpoint to azure cognitive services, and then I just say, as you saw in the on the um api documentation.

A

These are the sets of information I'm looking for and for some reason I noticed that I'm passing tags twice. uh It just ignores that, but I want here is it adult? Is it you know what kind of images and so on and so forth?

A

Now the web part has this little bit here at the bottom. That shows you whether your web part meets the criteria or not. This is done in a web in a element called analysis. Dialog content is derived from the spfx dialog, so it allows me to overlay everything on top of the page and block everything else on the page.

A

So what I do here is, I just call my web service or my service, which calls the cognitive services and then I do some analysis, and this is kind of embarrassing. How I do this, but I just go through every single connection. You know.

B

A

Let's look at the categories: is this a valid portrait? Is there only one person and so on and so forth, and then I use state to actually store the information so that the screen refreshes automatically and then I'm using some very simple react. Rendering to render information like? Is this valid.

B

A

Created the custom checklist control because I wanted to be able to- I was too lazy to kind of rinse and repeat every time that I wanted to say. Is it yes or no? So I just created an analysis checklist component that I use to display the list and then, finally, when I want to update the photo, uh I have two methods in there, one that uses the pmpgas one that uses the graph.

A

uh I was honestly having some problems with the pmpgs, if only I knew people on the call that could help me with this, but I I just called the the graph using the regular uh graph api method and I'll show you the page for that. I'm not actually doing so well with handling the error right now. I just write that to the console.

A

I should probably put a nice message or not so nice message when there's an error, but you get the point and then finally uh the part that was telling you the rich text in the I have hyperlinks here and bold in the property pane and if you've used property panes, it's not always easy to do that. So we are using the brand new property pane, markdown content uh control.

B

A

Allows you to pass markdown, and so all I can do here. All I need to do is pass a a string, localized string resource that contains my markdown, and so in my localized content. I have my cool down there, and so you see the hyperlinks and the the two stars for the bold content.

A

That's the documentation for the property pane markdown content, which is currently in beta, uh hopefully you'll, be really soon. You can get that to the pmp github that I owe at the url below.

A

So what we covered today- and I've got a few more minutes, so I'll actually demo some additional images, but what we covered today is azure cognitive services using sharepoint online tenant properties, to hide secrets that you don't necessarily want to ask users to configure using graph to save profile, pictures and the brand new property pane, markdown content control.

A

You can find the code at the url and you can reach me at this. uh My blog and my twitter account. Let's go back to the cool web part for a second, and so we asked someone asked: uh can it detect multiple pictures? So this is a poster from a session that david and I are going to be doing at the pnp uh conference.

A

It should detect two faces. Hopefully uh oh it only detected one. That's embarrassing! Oh yes, I remember because I gotta do this.

A

uh No, it didn't work, oh well. uh Okay, let me just prove that this works. Let me just show this really uh picture of awesome. Cool. Looking people here uh and submit you can see. Bert is not too impressed here that he's on the picture. And yes, it detects four pictures and says you shouldn't do that, and then we had a question to ask a picture. uh Parker. I don't have a picture of parker.

B

Awesome demo very much appreciate it. uh Thank you. Thank you. Thank you very great to see that the integration with cognitive services is uh really exciting to see love seeing that. uh So thank you uh to.

A