Microsoft 365 Community Microsoft Teams, 28 Feb 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Turn text to images in Microsoft Teams using OpenAI DALL-E 2 & Teams Toolkit for Visual Studio Code

Description

In this 17-minute developer focused demo, Garry Trinder builds a personal app (DALL-E 2 Image Generator) with a workflow bot that returns images based on text entries. The Teams Toolkit orchestrates creation of welcome message, activity and message handlers, dialog menus, Adaptive Cards, conversation states, querying OpenAI service for images, task modules, publishing to organization’s App Catalog and Azure integration - authentication, provisioning Azure storage, etc. Demo functionality, code walkthrough, plenty of tips and tricks. Sample available.

This PnP Community demo is taken from the weekly Microsoft 365 Platform Community call recorded on February 7, 2023.

Demo Presenter
• Garry Trinder (Microsoft) | @garrytrind

Supporting materials
• Website – OpenAI | https://openai.com/
• Tools - Microsoft Teams Toolkit | aka.ms/ttk
• Repo - DALL-E 2 Image Generator Bot for Microsoft Teams | https://github.com/garrytrinder/msteams-dalle-2-image-generator-bot
• Contributor – Garry Trinder | https://github.com/garrytrinder

Learn more
• Microsoft 365 Unified Sample gallery - https://aka.ms/m365/samples
• Microsoft 365 Platform Community in YouTube - https://aka.ms/m365/videos
• Microsoft 365 Platform Community - http://aka.ms/m365/community

A

Foreign, my name is Gary schinder, I'm, A, Cloud, developer Advocate at Microsoft, and in today we're going to talk about how I integrated Dali 2, which is the open, AI uh image generation model. um So this was the thing that everyone was talking about before chat GPT came along. uh You know the ability to just put text in and get some images back and I thought.

A

Well, wouldn't it be great if we actually had that in teams and how would that look um as a bot, so using teams toolkit for visual studio code I went about creating this um and the result is a sample which you can see in front of you. So I'm going to run through this as a demo.

A

First of all and show you how it's working and then we'll jump into the code and I'll show you all the individual bits and pieces, few tips and tricks as well some things that that I added in there as I was building and just to help my own development and yeah, we'll we'll go through that. So first, let's look at the demo, so I have this bot that I have created.

A

uh I've got my test user at Nesta and in the app catalog I have created my app and I've, published it out so that anyone in this tenant can use it so I'm going to install it. So uh Nesta is going to install the app so just add that it's it's a personal app. So it's it's going to appear on on the side there really nice and accessible, and one thing it's got: It's got a nice onboarding process.

A

So as soon as Nestor is installed, the bot it's gone great I know you've just installed this I'm going to give you some help and so I'm going to show you some information as to what you can use this bot for and also the commands that you can use as well to um to use it and also configuration steps. So one of the things that I wanted to do in this box was actually to have personal preferences.

A

What happens if you had a bot that you wanted to change depending on you know what your preferences were or to actually provide an API key, and so with the open, AI apis, you have to get an API key for for use. You get like free trial.

A

um You can click this button and get a free, API key. So I'll just open that up. It's gonna just go to open it, open AI, just a nice URL launcher you can log in and get get your own key come back and provide your own key. So we've got an Adaptive card here with oh where's. My key okay, just gonna have to quickly go get a key.

A

If that's, rather than not do that, I'll just show the video as a fallback I recorded a video. This is always good. Just in situations like this, so we'll see that the bot is added.

A

And we've got that onboarding message and we can do the configuration paste in your API key change results change the image sages that you want and then ask the bot for some text and it goes away and it generates a number of images based on that description.

A

And then you click on the actual image and it opens up a dialog and you've got the image, nice and easy, but really configurable.

A

Let's look at the actual code here so here we have my project in Visual Studio code, so this was created using uh team's toolkit, so I've got that already installed and I went through the create new app process, create a new app and I chose workflow bot. As my uh my my starting point, I chose that mainly because I have commands in my bot and I also have responses as well. So I want to. You know, get information from from the user.

A

I want to display back uh what they've they've they've sent through to the bottom also store some information in in the background, and why you told teams talk it in the first place. Well, it does a ton of work just to get started. So one of the great things about teams talk. It is as soon as I hit F5. It went away and created my Azure ID app registrations, my bot favorite registrations. It provided me with templates as well for um deploying to Azure later on which I actually did as well. So teams talk.

A

It really gets you up and running very, very quickly with a bot, and we can see if I go to the initialize file here. Here is um what teams talk it gives me. It uses the teams FX Library, which, which is really there, to simplify the authentication and setup of the bot, but also provide an easy way to create commands like the surprise, command and the settings command in here, but also card actions uh like the uh the settings save Handler. So when I click that button I had a Handler there to go.

A

Okay, yeah something's happened and I need to I need to to save that information, so that my my commands I used further down the line that actually generated the the images and called the open AI service, and that could get the API key from a previous previous part of the conversation that that the user had with the bot. So here is uh kind of like the team's FX side of the bot.

A

If you like, and now one of the things that I want to start with, is how did I implement the uh the welcome uh message? Well, one of the things that um the team's effects uh Library, it doesn't do everything. Sometimes we want to fall back to uh bot framework and use some of that functionality and the way that we can do the uh the welcome message is by creating a team's activity Handler. Now this activity, Handler, is specific to bots that are run in teams.

A

So essentially it gives you um it gives you methods that you can override, say: okay, uh the bot's just been installed uh run. This uh run run this bit of code or later on, which I'll show you about the. uh How do we handle the um the pop-up, the the dialogue window, uh when, when we click on the Adaptive card?

A

So when we have that welcome message, this uh this code is fired here, which essentially just sends uh the uh the message of hi there and it renders an Adaptive card called uh called Welcome card.

A

And if we see in this welcome card, um it's pretty standard adaptive card, it's got the information in there for the uh for the commands, but right down at the bottom. It has an action which has a verb when we, uh when we actually execute uh that card, we click on the button.

A

uh It sends a verb to uh to the to the bot and that's called Welcome config, and what that what that does is then I've got a Handler here which essentially then um runs because it's it's it matches that that verb and renders the settings card as well. There's a couple of things here where it's doing it's trying to grab things from State I'll cover that in a minute, um but essentially this is then how we can kind of cascade um one action to another through using these.

A

These trigger verbs, so the settings card here is uh is is rendered. We can show that again as well, where we've got. Please enter your API key and we have the the style there, the password style, so that the key was was masked and the different options for uh the uh number of images that we want and also the size of them and again, we've got some verbs here for setting save and settings uh cancel. So again. These are these are action handlers so in here we can re.

A

We can obtain the information that was submitted through that adaptive card through actions data, so we can take the API key and and the size um as well, and we can set all these values uh into into State. We can also return um the kind of like the summary information. So if you remember, when we hit the save button, the card got the settings card got replaced and the actual information that we submitted in um was was was put put in place of that card.

A

Nice way of kind of reinforcing that um confirmation that it's been submitted.

A

So the activity Handler in if we go into the index.ts um here we've got a web server where this is again generated by teams toolkit where we've got an uh a message: Handler that handles messages sent between our our bot running teams and the the bot framework, and what we can do here is we can run alongside uh the teams, FX bot, that's created for us, we can run our activity Handler as well. So every time something happens, we've also got our activity Handler running and these all work together.

A

Second part is a state um that I want to talk about in here, so by default, when you're using team teams talk it with with a bot everything's starting memory. So if you save a file, your memory will get reset, which is not great when you're dealing with state in a uh in a bot. So one of the things which I did do is install the Azure blobs package and actually set this up.

A

So it was using a local instance of of azure storage and we can configure a new storage instance here we can set up State. This is a pretty standard in in bot framework and we can create State accesses where we can save things like the API key results and sizes settings things like that, so that we can actually push things into uh Interstate. As as we're going along thing, which I know a lot of people have have said: is teams talk? It does a lot of things. We don't know what it's doing.

A

I'd like to add my own services in here. One of the things that I did also do was to actually start a a service and so because I'm, using a local um uh running instance of azure storage to hold my conversation, State I wanted to have um the Azurite service um running when I did an F5 and actually launched my uh my bot so I installed that package, which uh basically creates you know a a local Service uh running on my machine.

A

So it's like I'm I'm, working with a real Azure storage and that's provisioned in the cloud and I can actually extend the the build chain here and and actually reuse a lot of the the work that teams talk. It has has done for me, and so that is a really helpful tip if you were. If you know, if you want to extend that that build chain, then you can do by coming into uh into this task: uh dot dot Json.

A

Now, let's take a look at the actual commands. The one command that we actually ran was the surprise command. So when I put in Surprise, we've got a Handler here, that's looking for that word and what what it's doing is it's actually going away and it's finding the the API key that we've put in state? It's actually checking see you know whether I have on it.

A

If I'd skip that that stage, it would have actually gone yeah actually present you, your API key and go back and use the settings command to uh to to add that value in and and what it's doing is sending some messages back.

A

It's getting a random idea from a random list that I've got here, that's from the open, AI website, and then it's calling this generate image function here and in here is where the open AI stuff takes over in the the open AI service has sdks, and so you can install the open AI package, this one's for node.js, where you can create a client and that you can then use to uh to send a request after the open AI service, passing your API key The Prompt, how many images you want and so on and get a response back of of images that then you can render in um in in your results.

A

So there's a lot going on here. There's a lot of checking to make sure I've got API keys and things I've actually got um the uh the right um preferences as well, such as, let's say a number of images and the size, and we send the request across to open AI. We get a response and we simply render that in an Adaptive card. So let's look at that adaptive card.

A

So in here it's it's actually a loop, so it uses the the data expression to Loop through all of the results and create an image with an action button now every time uh so when, when the the image is actually clicked, it executes this. This task fetch uh activity and it passes the URL to the actual image and also the prompt, so the text that we actually passed into to generate the the image.

A

Now, if I go back to my activity, Handler, we have a another function in here: called handle teams, task module, Fetch and this is triggered and it returns a response to open a task, module that that dialog and it returns just a URL. And so it's actually a URL to a HTML page that I have in my solution and it passes the URL and and the prompt values through is just query string parameters and in here I can show you the HTML page.

A

It's really really basic, it's literally just getting the URL and it's updating the the attributes on an image tag to display that image in the uh in in the task module. And then, when it's closed is the amount we don't need to capture that that actual um event uh coming coming back.

A

One thing to just kind of note, as well as I'm uh rapidly running out of time, um is I. Use teams talk it to get this uh project actually running in Azure as well, and I mentioned at the beginning that it provides uh the uh the templates- and this was really really useful, because you know from from my point of view the defaults. There were enough- the web server that it was using um all of the bot service, all of the identity. I didn't need to touch, but I did add by storage.

A

So I wanted that when this actually was running in in Azure that I had Azure storage there, just like I was using locally in in my development, so I added a bicep file, and this just describes the the storage account that would get provisioned I updated the this provision.bicep file, which is kind of like the the main entry point that says um you know when you. When teams talk, it looks at this file. It's going to look all the resources that it needs to to.

A

Actually, provision for for this project to work um and update the Azure parameters as well in here so I could actually parameterize all the different resources um that uh that was being generated as well, and it was made really easy because you've got different options down here. You've got provision in the cloud, so that will go away. It will run the bicep files.

A

It will basically provision all of the resources and then we've got deploy to Cloud that after we've done the provisioning, this will actually package up and deploy the code and deploy that to the web server and also the uh um the any environment variables that I'll have used.

A

And then finally, we've got published to teams which basically published the app into our organization uh catalog um and then I could basically use any user in my tenant to uh to to use that that bot um and with that I'm, going to finish, and so hopefully you've seen how you know how you can build. Bots like this, uh using teams toolkit and I'm going to put the links in the in the chat and we'll uh it'll be in the in the video description as as well.

A

So you can take a look at the repository and try it out for yourself Hey, sir, back to you excellent, really, really cool stuff. Thank you for the clarification how it's being built and all of the features which we have in the teams toolkit as well, foreign.