Microsoft 365 Community Community Demos, 21 Mar 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Solving reCaptcha with Power Automate Desktop

Description

In this 13-minute developer focused demo, Tomasz Poszytek delivers the solution for resolving Google Captcha challenges to remote unattended machines with Power Automate Desktop for RPA (robotic process automation) and Outlook Actionable Messages. In simple terms, a bot captures an image of Captcha screen, sends it to user, user responds, and bot addresses challenge. The solution manages repeated challenges, timeouts, and user contact attempts. Uses an Adaptive Card, 2 Cloud flows and Dataverse. Learn how how it’s done in this demo. This PnP Community demo is taken from the General Microsoft 365 Developer community call recorded on February 17, 2022.

Demo Presenter: Tomasz Poszytek | @TomaszPoszytek

Supporting materials:
• Demo - Solving reCaptcha v2 challenge with Power Automate Desktop | https://www.youtube.com/watch?v=9pgC02Co2rQ
• Demo – Actionable Messages using Adaptive Cards - ultimate guide | https://www.youtube.com/watch?v=x5IXd-g8OXw&t=0s
• Documentation - Send physical clicks on a web element | https://docs.microsoft.com/power-automate/desktop-flows/how-to/send-physical-clicks-web-element
• Documentation - Introduction to Power Automate for desktop | https://docs.microsoft.com/power-automate/desktop-flows/introduction
• Demo - Adaptive Cards and the Power Platform to solve captchas | https://www.youtube.com/watch?v=7xIIjrHoqq0

Learn more:
• Learn more about the Microsoft 365 Patterns and Practices community at: http://aka.ms/m365pnp
• Visit the Microsoft 365 Patterns and Practices community YouTube channel: https://aka.ms/m365pnp-videos

A

Hello, everyone I'm quite nervous because I see that, like over nearly 200 people on the call, so uh I hope as well that the demo is going to launch and then run so anyways.

A

My name is tash poshdek from warsaw, poland, I'm business applications mvp and I'm basically working with processes workflows for four years already, and recently customer asked me to well create another automation this time, using rpa and over time I learned that the website that we were meant to automate was equipped with uh recaptcha with google recapture because of the security reasons and basically uh well, you are as as myself internet user. So possibly you faced recaptcha in various places.

A

Basically, this is a technology that is being used to secure every place that is used to input data from user to the website. That you're browsing like signing in posting comments, uh posting a reply in a forum or other cases where you need to put some data into the form and send to the server.

A

Basically, uh the goal is to avoid bots from sending hundreds of requests at a second so or just to limit bots from from acting as users and writing data on behalf of users, like some fake comments on their blog posts and so on, and nevertheless the goal is that when you open a site, it basically displays like a small checkbox.

A

You have to click it and um if you do not sign in often to the same website from the same machine, then possibly you will never see that challenge thing so challenges uh what you will see in a moment uh this. This screen, where you have to pick or select uh images that are uh corresponding or containing content, that uh that challenge is asking for, like palm trees, cars bikes whatever so for us users, I mean people, it's not really complicated. You just select like three four tiles and that's it for bots.

A

It's well quite impossible, because first, this is in iframe and secondly, uh this is just like one image with an overly on it. So I was thinking about maybe using some cognitive services and image recognition to be able to somehow split these styles, which are containing a specific keyword or specific item, but I failed so far. I wasn't able to make it so I thought about a different uh solution that I'm going to present you right now.

A

The point is that rpa, so the wordpress automation in this case is running in unattended mode, so some are on a virtual machine in azure, where there is no employee, no person who is supervising that run, and so, if that capture challenge is displayed, the bot is simply helpless and cannot proceed. So if that was an attended mode, then simply board could display um a dialogue window asking user hey. Can you just come over here and resolve this captcha and then press ok to continue.

A

However, when this is running in unattended mode, then well, obviously there is no no person on site, so the idea is that the bot is making a print screen, sending a request to a user to their email using actual messages, so um the technology that tim mentioned just moments ago. But this time it is being sent to your outlook to the employees, outlook not to teams or uh or other places, and then uh user is able directly from outlook to select these styles, which contain that specific item.

A

So let me just show you how it works. uh Fingers crossed is going to work because I'm just experiencing some issues with my computer for a longer while anyways, um I just need to press run, and this demo is simply going to open one of those hundreds of websites which does contain uh captcha to be resolved and uh yeah. So now it's running, hopefully in a moment, a website will pop up with the contents, and the body is going to be then able to resolve it right. So there is a website.

A

uh Now bot should select this uh checkbox right, so you can see right now. This challenge has popped up and what is going to happen right now is that the body is calling an endpoint built in cloudflow, which is actually constructing, composing and sending me an email with this actual message. So this is an email. It contains a screenshot that shows me uh that specific challenge and asking me to select these styles and a layout which corresponds to well this one uh image. So here I need to select bicycle.

A

I would select as well these two tiles because they cover the whole image right. So right now I have selected the response has been sent back to the server and the bot is now able to resolve them. However, because I was experimenting a couple of a couple of times, therefore it's not only to I mean the challenge is now getting more complex, so bot has to send another request to myself with another uh screenshot asking again for help, because the challenge is not yet over.

A

So there is a new email with a new request. So again I have to select now the bicycle: um it's somehow containing these four tiles. Sorry, these four tiles and again hit the resolve all right. um So hopefully, this time it will be enough right. So now the bot was able to actually resolve the challenge and post the comment successfully.

A

The goal, or maybe the challenge behind the simple fact it's just challenging for the bot, is the fact that uh user has only like three minutes to resolve this challenge.

A

So again, the more attempts uh are being made to resolve that challenge the more complex it's getting so, for example, it doesn't contain the image or the object in the image that is being requested for or that it asks two or three times to resolve another challenges, or that simply it like replaces one images with others and again asking you to again select the same object in that challenge, and therefore uh this solution I was trying to build, has to really be proactive and be able to resolve or react to every of these scenarios, and so the idea here was that it's simply trying to like up to five times to ask user to help if user is not able to, for example, answer to an email in this three attempts.

A

Then the time simply expires. So five attempts uh sent in around like 20 seconds or it was 30 seconds- no sorry, 20 seconds, yeah or even 10 seconds. Whatever was something that was like fulfilling this time span expected this, this window time window expected uh to react and to resolve the challenge, and so it is not possible, then, uh at the end, bot is simply sending a message to the user.

A

Okay, so you failed uh to resolve the challenge to email so right now you need to sign into this virtual machine and resolve the challenge yourself. uh That way right. So uh one thing um because uh I don't- I don't know if I have enough time to really go into details like how it was built.

B

Yeah you've got. We got some extra time, thomas, if you'd like to to show that.

A

Right, so let me show you, maybe I don't know which piece of the solution is mostly in most interesting for you. I guess that's adaptive cards, so um I won't get into details about the bot, the bots, the box. Behavior is quite simple. It's simply scrapping the website. uh The eye for the recapture challenge is displayed as an iframe, so the bot simply needs to find coordinates of the x and y's width and heights of that recaptcha um image.

A

And then, once user selects specific check boxes in adaptive cart, then the bot is simply calculating what is the x and y for these specific tiles and simply clicking uh there. So that's quite simple and the cloud solution is actually built out of like two two workflows, so the first one uh is the one that is used to request user two uh that is used to request user two uh resolve the captcha and the second one is used to handle that uh response from the adaptive card right.

A

So this is still using the old technology, not the one that the team mentioned. So it's not using a bot underneath it's just using the outlook developer provider id.

A

um So the thing that you can actually do yourself once you navigate to uh outlet connectors, I mean did that that url, that I'm just typing, so it's ultimateoffice.com, slash, connectors, uh slash. Oh, I am publish so here you can. You can register a new provider and with registering this new provider, you need to um specify as well. What is the scope of that submission? So at any time you can use your test submission. So this will be.

A

um This will allow you to send adaptive cards or actionable messages to your inbox or you can use organization, as I did in my case, so I'm able to send these adaptive cards to anyone within the tenant or you can as well request for the global one. But the global one requires microsoft approval, not only your um exchange admin so anyways.

A

uh This is this uh cloudflow, which is being called from rpa. uh It's using http request to call it. um Then it's waiting like five times until the screenshot is uploaded to onedrive and synced. Then it's registering actual information in uh in uh serving database, because every recaptcha challenge has its own record in dataverse so that um those two cloudflows are able to track.

A

What is this request for so this one is writing the request and sending them details about that request through actionable message um to the other flow and then, uh depending uh on what was the layout so three or four on four: it is simply uh building a different active cut uh layout or the content. So one contains three check boxes. I mean three on three, the other one four on four, and uh here is this originator? So that's that's this!

A

uh That's the um provider id that you have to obtain to be able to send actionable messages to outlook. So without this originator uh you won't be able to send adaptive card to outlook and, lastly, uh before it sends the adaptive card it concatenates or wraps that uh content that json in script, type uh application uplift card plus json and sends that email and then actually uh the workflow. This main workflow uh is waiting up to 90 seconds for the response from the user.

A

So if user fails to respond in 90 seconds, then simply the workflow is going to send a response to. Where was it here? It's going to send the response to that rpa that sorry user failed to respond in 90 seconds, so rpa isn't going to make another attempt.

A

Otherwise it's going to send correct response that this information has been obtained. So it simply sensed data like which checkboxes were selected and it knows which boxes were selected because, as said, it is checking that dataverse row it has created in the first step that is being updated by the second flow that handles the response from this adaptive card in outlook and is simply writing um here, like which uh checkboxes have been selected uh for that for that specific case.

A

So that is the whole solution, and so far it's working quite well, so users are able to resolve the captcha without actually need to go and sign into the remote machine.

A

It doesn't happen every day, but at least once a week I have- or I see that there is an email sent with uh with a capture challenge, so that has anyone from business has to help uh bot to proceed. So that's basically it.

B

Awesome, thomas, that's really really cool. I I think everybody loves having to remember those credentials for logging into a remote box or something like that and and go take care of something. So that's really really cool the way you've set that up. uh Thank you very much for sharing and going into depth.

A

I think I shouldn't be sharing that, because google might be really angry that there is a way to crack their captcha.

B

Well, you know you're still doing it right, you're, just you're, just doing a more efficient or effective way, so excellent, very good, very good. All right! Thank you! So much.