Microsoft 365 Community Adaptive Cards, 15 Feb 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Adaptive Cards and the Power Platform to solve captchas

Description

During this 24-minute Developer and Citizen Developer-focused demo, Tomasz Poszytek addresses a roadblock - Bot task execution halted by a captcha challenge. His solution includes Outlook Actionable message capability, an Adaptive Card, 2 Power Automate flows and Robotic Process Automation (RPA) to remove the hurdle. Configure actionable e-mail provider. RPA captures Captcha image and calls a flow that sends Adaptive card prompting user to action. User checks boxes on Adaptive Card that verifies time-out is not exceeded (requiring new capture) and RPA resolves challenge. This demo is taken from the Adaptive Cards community call – January 2022, recorded on January 13, 2022.

Demo Presenter:
• Tomasz Poszytek (MVP) | @TomaszPoszytek

Supporting materials:
• Blog post – Get started with Adaptive Cards | https://techcommunity.microsoft.com/t5/microsoft-365-pnp-blog/get-started-with-adaptive-cards/ba-p/2048786
• Demo - Solving reCaptcha v2 challenge with Power Automate Desktop | https://youtu.be/9pgC02Co2rQ
• Demo - Actionable Messages using Adaptive Cards - ultimate guide | https://youtu.be/x5IXd-g8OXw
• YouTube Channel - Tomasz Poszytek | https://www.youtube.com/c/TomaszPoszytek

Learn more:
• Learn more about the Microsoft 365 Patterns and Practices community at: http://aka.ms/m365pnp
• Visit the Microsoft 365 Patterns and Practices community YouTube channel: https://aka.ms/m365pnp/videos

A

I am thomas poshtek. I know that the last name is quite difficult to pronounce uh I'm working and living in warsaw in poland, europe and I'm mainly focusing in my everyday work on processes, actually, I'm just working with our platform, but mainly with power automate recently, like recently for over a year now, with part of my desktop and adaptive cards. um I think like, since I acknowledge that that technology actually exists. Like three years ago, I think uh became some kind of uh of a technology of of my favorite technology in uh in power outcome.

A

I mean per platform because it's very useful across all the applications that do support adaptive cards, so teams outlook uh well flow as well and well that's what about framework and then pva, and I'm really keeping my fingers crossed that it will be even more popular and become available and more than just this mentioned already. For example, maybe it will appear in power, apps or or somewhere there.

A

Nevertheless, I'm working with other gift cards as well in my process, because this is a really like a easy way to create very sometimes even very complex forms, with a lot of logic.

A

Behind with conditions with some fields which are appearing and being hidden with validation and requirements and then send them directly, for example the teams or to or to outlook, and so I don't really have to take care about the ui layer, it is being done for me because of uh of well how the relative cards are working, but it really speeds up the creation of processes and logic in those processes. Because this way I can very quickly grab response from users and- and so recently I was involved in rpe projects.

A

So for those of you who haven't ever had about rpa, this is robotic press automation. So that's a technology that actually simulates user behavior on the machine. So it helps us to like fill in those gaps between some legacy systems, legacy, alt systems and modern solutions like cloudflows.

A

So where cloudflow is unable to reach out to a desktop application or old database or well anything that doesn't have a digital api, then we can use rpa which is configured to like click, those ui elements, those fields, those text, boxes, buttons, menu items as if there was a regular user. Who is clicking all those ui elements and then populate those fields? Those transactions I mean just, do what would the regular user do and then grab the outcome?

A

Send them back to cloudflow, and this way to be like this integration layer between what we have in a cloud and what we have elsewhere like in on-premises uh servers and uh the solution was about.

A

I mean ultimate automatically acceptance of invoices, and in that case the invoices were actually in uh in a website application still, even though, that the whole system where those invoices were present and were meant to be accepted was online was as a built as a website.

A

It didn't have any kind of rest api or any other api that could be queried so that this acceptance could be done automatically or faster and customer wanted to have that process automated, because uh it was simply taking too much time for employees to go through every invoice and they are having like hundreds of the invoices per month and uh do all those comparisons and acceptance manually.

A

So the goal was just to take a list of invoices to be accepted from sap and then navigate to that website. Compare data from one list with another and accept these invoices that were able to be, or that meant to be accepted, and everything was fine until the day before the solution was meant to be launched productively.

A

The operator of the website introduced recaptcha because they wanted to be more secure and simply secure. The logging procedure with recaptcha so to avoid bots from signing in so recaptcha is that technology. That appears mostly at uh I mean that that appears in in all those scenarios on websites where we as users are meant to provide something to like post a comment sign in um go through some checkout process in in a basket like anywhere. We usually are providing some data and sending this data to the web server.

A

There is captcha to like avoid all these bots that could be doing some things manually or sorry automatically and as well to limit the load that can be then transferred to the web server uh so that only those transactions are really coming from. The real users are uh allowed, so recaptcha is looking like that.

A

uh There are like three types of recaps already uh the most commonly used now is the recaptcha uh v point two, although there is as well recaptcha three point: three, uh which is less visible and doing is doing this kind of um validation, whether the agent is a user or or a bot, based on some kind of automatic discovery.

A

I don't know how this score is calculated, but this recapture v3 is simply trying to discover whether the person, the the agent that is uh operating on the website, is a bot or not, um but basically uh for a regular user. Captcha is not an issue, so you're simply asked to.

A

I mean before this challenge that you saw appears you need to like access the same website or uh recapture a couple of times per day, so that, after I know like a 10th uh 10th approach, 10th selection, that you're not about this challenge is going to be displayed before it is only. You only need to simply select the checkbox and that's it, but then selecting the checkbox for the bot is nothing nothing challenge, although figuring out which tiles it should select.

A

When asked to hey select, all images with stairs is a challenge already, and so I was thinking about maybe doing some kind of image recognition and uh trying to do that with uh some ai builder. But then I didn't know how to actually split this whole image into those small tiles and then analyze one by one.

A

So what I decided to do is to simply make the bot to ask a user, the business user for an input and when speaking about attended- but I mean one additional thing uh in in rpa- is that this technology can be running in like two modes. It can be running in attendance mode, so the one that you're actually in front of a computer and you're, seeing what the body is doing. So in that case, recaptcha, is not going to be an issue, because if it appears, then the bot would simply display like a message.

A

The user hey help me user, resolves recaptcha, clicks, ok and then what continues. But the case I'm talking about was happening in unattended mode, so some were on the remote server on virtual machine in azure, and so business users were not meant even to access that virtual machine. So I was thinking about a solution that would actually help the bot to resolve recaptcha, but without an actual need for the user to log into the virtual machine, and the solution I created is doing that very well, quite quite an easy step.

A

So first is taking a screenshot with that recaptcha displayed, then, is sending an actionable message to user inbox with a question help me resolve that recapture like select, which title should be selected by me and then once the user responds bot, simply selects these images, which are you know, indicated by the user, and this way it can resolve the recaptcha so having that set, let me show you how this solution works. So um first I would run this recaptcha demo.

A

So it close that window, and so what you'll see in just a moment is that the body is going to actually trigger the website that you've seen just a moment ago with a fresh start with a with a form displayed and uh well, it will possibly be again facing that. We capture challenge, so I will not be going into the details about how the whole bot solution is built. If you are interested around that, you can simply navigate to my latest youtube video and there.

A

This solution is described in details like what actual steps the bodies is taking. I will now focus really only this that part around actionable message.

A

Okay, let's see if it is able to now click.

A

Oh so it clicked that checkbox, so the challenge is displayed and now in just a few moments, I should be sent an email with that actionable message. So stay tuned. The another challenge that is uh well happening here is that this. This challenge that you see now is being displayed is only active for like two minutes, so I was trying to do my best to have this solution working as fast as possible.

A

Although there are obviously some delays- and I will tell you where so that's the email with an actual message.

A

uh So here is a screenshot for the user, so they can actually see what is in there and then they're asking us how to suck the layout so that it's like three on three or four on four, so they're, like different check boxes present here and in the end user has to select those which are with motorcycles, which is uh these that and and this one so once it's resolved, there is a confirmation message and then the bodies should now select these styles that I have marked as those containing yeah motorcycles.

A

So this way, but was able to resolve recaptcha, submit the form and actually go through that challenge, which is just impossible to solve by a regular boss, and, as I said, this solution is actually working on virtual machines somewhere in the cloud, so users are not accessing it and the only way of interacting is this kind of adaptive card um all right. So that was the demo. I hope.

B

That's cool that was super.

A

Cool, let me go simply just to the solution like how it was built, because that might be more interesting uh in in terms of uh our call today.

A

So um we have as well as I saw david with us david sorry, I'm just using these old actionable messages. I know that one day it should be replaced with something more modern, with universal actions and so on, but until I am able to use it, I am still relying on on this old approach.

A

So first and the most important thing around actionable messages is that if you want to let power automate to send those kind of dynamic mails to your inbox or to other inboxes in your organization, you need to pro you need to register this actionable email provider.

A

So you need to go to outlook.office.com, connectors, oam and publish, and there you can release a new provider and you have three options on how this provider on the scope of the provider. So the first is just your mailbox and that one is like approved automatically, because this will only allow you to send that email to yourself. Others, even if you will send that mail to others, they won't see the card displayed, so they will suggest a blank mail or only the html contents.

A

Then the second scope is organization, so that you will be able to send this to any person within your tenant. But then you will need an approval from exchange admin and there is a small global which allows you or we will allow you to send that email to anyone within office 365, and that requires approval for microsoft.

A

So I created this organizational use, this organization scope and the next thing that you have to create as well are those target urls. So where data from those actionable messages is going to be sent what I've done.

A

I just used this kind of regex mask so that I simply created um a provider that will allow me to send data from actionable message to any cloud flow or any logic flow. Actually, because uh I'm not a developer and data from my cards is simply being sent to other cloud flows and then it is being handled there. And, lastly, uh you can define as well. It is not visible here, but you can define from which addresses those actionable messages will be sent.

A

So again you should define the providers or the emails, the mailboxes that will be sending those messages, and so once you have it, you need to get this provider id and that's the most important thing around actionable messages when you want to send them uh um to your output during boxes. So then the second thing I built is actually the flow that is being called from the rpa. So rpa is triggering this cloud flow.

A

It is passing to this cloud flow.

A

Some basic information like the instance id so that I am able to like bind specific information, specific requests with specific instance specific rpa instance and the screenshot you saw is being uploaded to onedrive. So first flow is like checking if that file has been uploaded and is available. If not, it is waiting until that screenshot is uploaded. So this is the first place where the the whole solution is actually wasting or losing some time, because this sync between the server and onedrive can take a couple of seconds.

A

I mean second two seconds, uh it's quite fast, usually, but that's the first place where the bot is waiting, then, is creating a recording. Dataverse, where it's storing information about this particular recaptcha challenge, request and then based on, if the bullet was able to find out if the layout of captcha is three on three or four on four or wasn't able to find out is sending a different adaptive card, so the one you saw is actually containing this um option to select.

A

The layout, so first user is requested to select layout because bot wasn't able to discover it and because, based on the layout, the image tiles are different. Therefore, what is asking user to first select this layout right, so first, the layout then, depending on what the layout is set, there are different check boxes to be selected and and and what then.

A

So the next important thing around the design of adaptive card is that is the fact that you need to add this originator parameter, which is just the provider id from that actual messages dashboard. So this is that value without it, uh you won't be able to actually pass through the actionable message and there are as well some other minor differences between the regular adaptive card and actionable message. So, for example, uh the fact that buttons here were cut off is because I didn't choose that option here, so I just pass it again and okay.

A

There are the buttons, uh for example, the fact that in actionable messages, we are able to create a button that is a type of action http that is not present in regular adaptive cards and so in that action so depending on which button is visible and which button user clicks.

A

Those those buttons are sending posts to that. Other action that other cloudflow uh that I have built as well- and I will show you in a moment and as well, they are sending a body and then in body they are sending just a table of all the statuses for all checkboxes which are present. So then, if any checkbox is marked as true, then bot knows that it has to select this particular checkbox.

A

Also, there is information about what is the layout of that of that recaptcha or the instance id of the flow to to have this record in place and as for record id. So the information about the dataverse record that was created by the flow in the first place, so that the second flow I will show in a moment knows what information or what, where to update the information database, which record in dataverse it has to update. Maybe this way right right.

A

So once this whole adaptive card is uh is created, it is then the second thing that is different between adaptive cards, like in teams and those sent in uh in outlook.

A

You need to wrap that content, that adaptive card json in script, type, application additive card plus json so like you need to surround this in the script tags and only then you are able to simply put this script into the body of an email, and, apart from that script, that I mean that that's a actionable message: script code, you're as well able to put other html. So you can really write a regular html email.

A

Another. The thing that, in my case uh is used to, like hide the whole body, is this parameter uh in the beginning, which is telling again the actionable message whether to hide everything else apart from actionable message or maybe to show as well. Whatever else is in that email written, but in my case, even if there was anything written, then it is being hidden and so mail is sent and what else then? The flow is waiting 90 seconds for for the response from the user.

A

So it's like checking every two seconds whether that row it was created um in the beginning is already updated and if it is updated, it means that user click resolve which triggered this second flow to handle the user input for the recaptcha, and this flow is much much more simple because it only well. First, it checks. If uh this whole operation is not timed out.

A

So if user actually resolved that recaptcha after a minute and a half, it means it was too late and then simply bot is sending uh a response telling the user that uh this recapture challenge has expired.

A

And so bot is trying to like five times up to five five times to send again that uh that email uh with a fresh new captcha. So if, after five times user is still unable to resolve that, then the bot will simply uh stop and and wait until someone actually signs into that virtual machine and resolves the captcha manually.

A

But if user is lucky enough to to fit in with those within those minute and a half, then what updates around so that record with information which images were selected and is sending a response back to the user that you saw once I click resolve with information. Thank you for helping the bot to resolve recapture challenge, and so, if the record is resolved, then uh flow is simply continuing and is sending back the response to the rpa with information which images were selected and what is the type of the captcha?

A

If anything went wrong, if user wasn't able to make it within the time or if there were any errors or whatever happened else, then it is uh sending back this code 400, which is again telling the bot that it has to retry the whole process so to refresh the website and get the new captcha and again request user to resolve it from their inbox and also the cool thing about actionable messages is the fact that you don't necessarily need to do that straight from your inbox.

A

You can as well do it on your mainboard on your mobile phone. Sorry. So if, if your outlook on on uh on the phone on your mobile is following the requirements for actionable messages, you are able to simply resolve this challenge straight from your phone, so you don't have to even go to your laptop and check what is what is displayed there. So that's another cool thing about this whole solution and well. This is how it's built.

A

As you already see, there are like several situational places that are actually uh making delays the whole process, so that can impact how fast this is being resolved, but so far it is like running uh since a week. Sorry- and it's quite successful so every day in the morning there is this recapture challenge: mail sent sometimes twice sometimes three times, but in the end users are like waiting.

A

uh If there is anything popping up to your mailboxes and are ready to resolve so so far so good, the bot is uh working without any issues and yeah. That's that's it. I hope you liked it.

B

That was awesome tomas. Thank you. So much for for sharing that demo you're welcome, I'm back to the presentation, and I know tomas. He gave himself a a very humble introduction, but um he he is a very smart mvp and he is very talented. With adaptive cards, um I've gone ahead and in the chat I tossed the link to one of his blog posts that he has on the microsoft tech, community blog site, which is getting started with adaptive cards.

B

I know many users in here are probably very familiar with adaptive cards, but there's always something to learn, and this article really helped me when I was ramping up on the team since I haven't joined um the team for a super long time. His youtube, which is also linked in the chat, also has really good content. I specifically remember watching his video on outlook actionable messages and that also helped with the ramp up when I joined the adaptive cards team. So thank you. Tomas for for joining us in this community call as well.

B

A

You very much for this white introduction.