Microsoft 365 Community Community Demos, 13 Apr 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Harnessing the Power of OpenAI inside Power Virtual Agents

Description

In this 14-minute developer-focused demo, Matt Jimison focuses on solutions delivered by Power Virtual Agents (PVA) - fallback topics, quick replies and use of Adaptive Cards for formatting answers and by Power Automate – for getting and updating data. These solutions call OpenAI Davinci for text answers and DALL-E for images. Define simple and complex queries. Scope to control/limit source data. Refine script queries using ChatGPT’s 3.5 turbo pushing against Azure OpenAI. UI and code walkthroughs.

This PnP Community demo is taken from the Microsoft 365 & Power Platform Development Community call recorded on March 16, 2023.

Demo Presenter
• Matt Jimison (Takeda) | @mjimison

Community Call Conversation
• https: aka.ms/Mar16-Demo1

Supporting materials
• Documentation – Open AI Completions | https://platform.openai.com/docs/api-reference/completions/create
• Documentation - OpenAI Images | https://platform.openai.com/docs/api-reference/images/create
• Learn - Quickstart: Get started using ChatGPT with Azure OpenAI Service | https://learn.microsoft.com/azure/cognitive-services/openai/chatgpt-quickstart
• Article - Now in Public Preview: Power Virtual Agents’ new advanced authoring canvas | https://powervirtualagents.microsoft.com/blog/now-in-public-preview-power-virtual-agents-new-advanced-authoring-canvas/
• Tool – Adaptive Cards Designer | https://adaptivecards.io/designer/

Learn more
• Microsoft 365 Unified Sample gallery - https://aka.ms/community/samples
• Microsoft 365 Platform Community in YouTube - https://aka.ms/community/videos
• Microsoft 365 Platform Community - http://aka.ms/community/home

A

B

Use cases with PVA and openai, so, let's jump into it. But first let me give you just a little bit of background on myself. So I am a geek at heart and so I actually got my degree in computer. Graphics have been working in the IT world. My whole life and actually was really heavily embedded into SharePoint got into that in 2008.

B

Actually was you know, speaking at SharePoint Saturdays doing all that lovely stuff started working with SharePoint with moss, 2007 and then I got into Consulting and ultimately got further away from the code and started doing more with managing the the team and I made a change late last year to actually get back in the game, get Hands-On with keys and so I'm back to building out Solutions and I. My current role is intelligent experiences, team lead at Takeda and a big big part of that is actually doing a lot with AI and power virtual agents.

B

So that is part of the reason why I'm here? So what we're going to look at and we're going to demo actually quite a bit this morning. I haven't changed this yesterday, just because so much is happening right now in the the AI space and with chat GPT, and do things keep coming out almost on a daily basis. So we're going to look at some of the key things in power virtual agents and talk about how that works.

B

We're going to talk a little bit about power, automate we'll explore what those Solutions look like as far as how we're able to get that data from the open, Ai apis and then we'll talk specifically about those apis as well, and I've got references to all that information at the end, but I want to spend most of the time really demoing and showing you guys what's going on so give me one second and I will switch over all right.

B

So I've got a couple Bots and we're going to look at our first scenario here of essentially looking at open Ai and what we're going to be doing here is just essentially being able to do two things: either having a single chat, single question or query or creating images, and so on. This particular bot I can actually chat if I want to or I can just ask a question. So I'm going to say who was the 17th president of the USA.

B

All right so again back response here. So this is pretty cool. I'll show you the conversation, I think, which is a lot more powerful, but we can get a little bit spiffier. If we want to, we could say, give me a list of Michael Jordan's shoes and the following format.

B

Say year number sold here and then let's go ahead and actually say the shoe, and one of the things that we're doing is: is that you'll see that we're bringing back this as an Adaptive card and so we're repeating basically to the end user, what they asked and then we're putting that in the response.

B

Part of the reason for doing that is because the response coming from the openai apis actually looks quite a bit better when it's put into an Adaptive card, as opposed to just outputting it directly that actually changes in the newer API, which again I'll show you. So we've got this list here and we could look and see the Air Jordan Ones most popular and pretty consistent. I've asked this question to the bot quite a bit. Okay.

B

So now, let's switch to images really quickly and let's go ahead and look for up close photo blade, Jitsu sitting on a quilt with soft Lighting. In the background, it's pretty much what I think everyone is searching for these days now in this particular example, we're choosing to get back four different images, and so that's something that again, when we dig into the code we'll be able to see how we can change that.

B

But these are going to come back just stacked up here and so take a look at this all right and so is on a quilt. And then, let's do one more fuzzy orange monster sitting at a desk, the laptop as an oil painting.

B

So what's coming back to you with these images, is the they're actually URLs that are going to expire in one hour and so actually at Decatur, we're doing something unique where we're actually pushing them into our own Azure, blob storage, but that's something to keep in mind, because when I would come back and revisit this chat in the future. Eventually, these images are going to disappear because that time is going to be met.

B

So there's an orange tree monster and I actually have it too set up so that if we want to, we can click on that and then that's going to load the full-sized image. So all right, so that's images. That's simple text! Let's just kill that! Okay! Now the next thing that I want to show.

B

You is I, think a lot more interesting and so I have developed for this demo, a bot, it's all about PNP, it's about to teach people VMP, so I, just reset the conversation so I'm going to tell the bot High all right and so we're talking now the Peter MC programmer and he likes to tell Dad jokes, and so he can't resist that, but he's all about PMP and only PNP, and so let's just go ahead and ask him a question: that's not related to PNP and see if he's able to help us.

B

Okay, so yeah, he does not want to help right because again he Scopes PNP and PNP.

A

B

So that sounds good, so I'm gonna have him help us write a Powershell script, so write a Powershell script with PMP Powershell.

B

That reads: title and email from a CSV and loads record into SharePoint.

B

Now this is using chat, gpt's, 3.5 turbo and we're actually pushing against the Azure open AI. So this is something it it's similar API if you're, using it directly in open, API. We'll look again at this. How this works, we'll dig in more but what's happening. That's really really cool is that the responses with code are a lot better now, so we get to actually see some sort of format to it and it's a lot easier to read.

B

So we've got this script right where we're reading in we're putting it to a site, CSV uploading, title email that looks good, but let's say that I realized I wanted to make a change, because I might want more than title actually so say. Actually, let's make it read: The Columns dynamically from the CSV.

B

Okay, so now it's actually come back to me and it's actually grabbing those columns from the CSV and then it's now looping through that, and it's going to do that and then I want to wrap it up and say, wrap it up by making this a command lit. Please.

B

So again, the real power of this is the fact that it's conversation right, I, don't have to remember. I, don't have to retype everything that we've talked about and the context is really there which just takes it to another level, as opposed to the first spot. That I showed you, which is good for just single queries, but this now we're actually getting there what's really cool here is.

B

Is that not only has it done what I asked, but it's actually also given me sort of an example of how I could call that, so that's pretty cool, and then one last thing here I can say here show me a list of top SharePoint list, formatting contributors, because again this spot's all about PNP, okay. So, let's see some example list.

B

Now one of the things that's interesting about the fact that the data that's actually captured right with chat GPT is: is that because it's a few years old, some of the links that'll come back, aren't necessarily going to still work today. So that is one thing to keep in mind and that they're not always going to be relevant anymore. Okay, but it's giving us some example code.

B

So that looks good so now we'll go ahead and put this away and let's jump in and look at this, so I'm going to actually jump back really quickly to the first bot that I showed you- and this is all about- essentially capturing a fallback topic. We, we are doing a conversation starter here, so that we're saying hey there feel free to start typing we're using something called quick replies.

B

This is the unified canvas preview, so you could do everything that I am showing you in with the with bot composer, but it's a lot easier to do in unified canvas which hopefully will be GA here in a few months, and so I wanted to kind of show out that capability, which is really really cool and really Blends. Well, those two. You know those two interfaces, so we can chat. We can create images. What we're actually doing here is issuing commands.

B

These are triggers they're, going to trigger other topics and what I want to show you really quickly here we're not going to make any changes. If we go over to system- and we go to the fallback, then here we're actually looking at what's happening. So a topic is not going to be triggered because we have almost no topics that are looking or listening for anything particular.

B

So it's an unknown intent we're going to grab what they put in as our prompt and then we're going to actually send that to a flow and again, if we want to go in and look at some of these flows. For this particular one we're looking at completion, so it comes in as the prompt and then we're initializing a completion, we're actually putting something about not being able to be able to get a response.

B

Because if an error happens, then we have an initial value there and then, as far as the try goes, we're grabbing our API key from Azure key Vault and then we're doing this request and again, the actual API information is in the presentation that I'll share afterwards, but we're sending the prompt we're, saying DaVinci 3 and we're giving it Max tokens 24 8., there's a lot more. You can do with that.

B

You could potentially build in the interfaces even with adaptive cards inside the PVA bot, so that you could let the user decide such as the temperature right, how crazy it can get on responses and whatnot. So that's what that looks like and then really quickly. We're going to jump back over here to Let's. Create images. Show you that so this one in particular, because we are looking at a trigger, we want to get them into this actual flow, and so when they say, let's create images, it's going to ask them to describe an image.

B

It's also going to give them. The quick reply to switch to chat just gives us like sort of a quick menu to go back. If we don't actually want to be in the image UI and so we're going to pass the image text and then we're going to output- and this is another really cool part of the unified canvas. We can do an Adaptive card directly here.

B

So if I click on this and expand, it I've actually got a formula in here, and so what you'll see is I'm getting back the actual raw response from open, AI I'm, taking that as it is parsing the Json and then looking at the data property, then that is going to just be an array of URLs and so I'm going to Loop through all of those and I'm, creating an image type I'm going to pass the URL of the records URLs text size stretch.

B

So that's why we saw those nice big images in team and then I'm actually adding that select action onto it, so that the open URL will allow me to click on the link and then open up a new bot. So that's what the images look like and then to go back over here to the again I think what the more exciting one is, the GPT, and so let's jump back over to this bot and look at it. So this is a hundred percent fallback. There is nothing else in this bot.

B

Besides this value- and me posting this as an example, but we at this point, we really don't want any of the topics to trigger here. If we, if we needed something, we could but it's going to handle everything so we're actually setting a variable, really quick, so user typed in something it's the activities text we're really quickly, seeing if our full conversation and the key to that context of being able to say, write me a script change the script Etc.

B

This is we're actually having to pass it back to the bot and so that that's how it's able to remember and retain it's not going to do that on its own. But if it's blank we're setting a value here and that value just really quickly is going to be chat, markup language. This is where I mentioned it's a little bit different with openai openai does essentially an array of values, and this is really more string format, but we're basically saying a system prompt, we're giving it we're giving it a personality. It's where we're talking about.

B

This is what you do. This is what you respond with when somebody asks for something: that's not part of it, Etc et cetera, and then it's basically waiting for the user query here, and so that starts the conversation user enters something we actually take that and the full conversation we pass it into a flow, and that particular flow is here and then again we're going to call the actual API, and so this is again open.

B

Ai Azure open AI for us over here at cicada and so doing temperature zero, passing in just that specific, prompt, actually and concatenated, with the full value, and we just hover over that the full value and then the the particular question that was asked and then we're going to get back. The response from that and at that point, that is going to be the choices array which is going to always be the first item in that array and it's the text and then we send it back now. What happens after we send it back.

B

Is we're going to send out the response, not as an Adaptive card like the other one, but just the response?

B

That's where we're getting those really nice looking code, examples Etc and then we take that variable and we want to set it to what it was already and then we want to append to it and again, if I get in here and I can go into the formula, so I'm concatenating the full conversation I'm actually putting in then what the user just asked, ending that I am starting it I'm, putting the assistance point and then putting their response in and then ending that starting that and then again we're going to wait for another user query so that just goes on and on and on and what's again happening is every time I ask a question fall back sitting and then it's going from there.

B

So that is what that looks like, and that is it in a nutshell and again back to the actual references.

B

If you want to learn more about the open, Ai apis and the Azure one as well, those are there, the unified canvas which again I'm a huge fan of I, always love the Adaptive card designer. And finally, you can find my blog there Matt jemison.com or look me up on Twitter. So thank you very much.

A

Matt really cool stuff uh I appreciate you diving into that a variety of ways in which you can Implement that how it works uh in all the different capacities, uh the way that you can plug it into Azure, open, Ai and all this stuff, so real, really great job. Thank you. So much.