Microsoft 365 Community Identity platform, 3 Jun 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Microsoft Graph’s advanced capabilities for querying directory objects – May 2022

Description

This month’s in-depth topic: Microsoft Graph’s advanced capabilities for querying directory objects – Microsoft Graph APIs expose Directory Objects (Users, Groups, Devices, etc.) to interact with Azure AD. Learn how to unlock the full potential of our Directory Object APIs using the Advanced Query Capabilities that allow developers to Count, Filter, Search and Sort Directory Objects on more properties with more OData operators. We cover - what are advanced query capabilities, architecture, walkthrough assorted query examples and SDK support (PowerShell, .NET), call out current limitations and conclude with Q&A. Presenter: Luca Spolidoro – Senior Program Manager, Azure Identity | @Licantrop0. Recorded May 19, 2022.

Learn more
• See the full blog post for this call in the Microsoft 365 platform community blog - https://aka.ms/m365pnp/blog
• Download the recurrent invite for this call | https://aka.ms/IDDevCommunityCalendar
• Microsoft 365 Unified Sample gallery - https://aka.ms/m365/samples
• Microsoft 365 Platform Community in YouTube - https://aka.ms/m365/videos
• Microsoft 365 Platform Community - http://aka.ms/m365/community

A

Hi everyone, I'm lucas, polidoro, I'm a program manager in the microsoft identity platform, and today we are talking about all the capabilities that we added for a querying directory object with microsoft graph. So this is the agenda. For today we introduce what are these advanced query capabilities and then we deep dive a little bit under the hood. I hope you don't mind and we'll talk about some query examples, so you know how to practically use these new capabilities and also we will talk about the sdk support for this.

A

Let's start with it. What are these query? Probabilities.

A

They are additional functionalities that we added to the most common directory objects and their links. We indexed basically additional properties to this directory object. You see in the list and we added new way to filters and use order by and use pound and filter on top of those all the ones you see in the list are the currently ones supported, but that doesn't mean that we will not add more of them in the future.

A

If you give us feedback form, but just to set the discussion, you might come coming off, mute and saying, if you have any experience just to see if you, if you are familiar with this, you you have some app in production. Using that don't be shy.

B

Like myself, boomi uh option yay.

A

Option a perfect, perfect, so more uh one more application in production, cool anybody else um having a application in production. Can you actually raise your hand? Is the reaction you need to see how many of you have apps in production cool? I see see some hands perfect awesome. Thank you very much and we can use the raised hand function for for this and who instead never used this microsoft graph for for directory objects.

A

Okay, let's see some some other hands, that's okay! We are here to learn all right. So then, let's do a quick start on on how to use these capabilities.

A

I guess you you know what graph explorer is graph explorer is a tool that we offer for a query: microsoft graph and it's available on the website.

A

Here there you go so this is graph explorer and you can add your test url with with microsoft graph.

A

Here you can add, headers, etc, and- and this is one of the basic query that you can send- it's counting the number of users in your tenant, I'm using a demo tenant, and if we run this query, we get the response over here. So in this case I have just 40 users in this tenant.

A

Going back to our presentation.

A

The important thing to use these advanced query capabilities is to have a county called true, the query, string and consistency level, equal eventual uh header and I'll. Tell you why in a second, this is a the sample query that I'll just show you.

A

Okay, now now, let's see what actually happened under the hood here is how a call gets from microsoft graph all down to our directories too. So we have microsoft graph that we all know and love and when it receive a request for an identity workload, so a request to users or groups or devices object. It would call internally our service that manage the directory object, called rest directory service that will ultimately execute the read write operation to the aid store that, as you can imagine, being highly available.

A

Highly resistant store has replicas all around the world. Now this is a pretty simple architecture like front-end service layer and and backhand backhand store. But what's the problem with this problem is that the directory store that you see over here is very, very old, was ideated and invented even before azure was was even a thing, so by default, this door doesn't support, even counting. So if you want to know how many user you have in your tenant and your organization with just this query store you, you couldn't get that information.

A

So what we did is, as the fundamental theorem of software engineering say, but all problem computer science can be solved by another level of interaction, and this is exactly what we did. We created another layer to to get data from the directory store, called advanced query service that is able to get this data from another kind of store. That is very performant and a used.

A

uh You know graph query based index and very advanced, sorting and filtering capabilities that the original store couldn't couldn't satisfy and we are pushing the data into the store thanks to another service that does a sync of all the changes that are happening in the original store to this new store.

A

Okay, moreover, as this sync engine is very good at sending change notifications we're using the same sync engine to feed the data into the microsoft graph change notification, to give you the opportunity to subscribe to changes that are happening in the directory, but this is an another tool that that we that we offer so now.

A

Let's talk about eventual consistency, so I show hands how many of you know what eventual consistency is and how it might work in 80., so azure active directory is a highly available and resilient system offering, of course, high performance thanks to distributed servers all around the world, and these servers are maintaining a copy of the data in different partitions, and this is what we call and you to the caf theorem.

A

This implies the needs of eventual consistency. So what ex? What exactly it is? How does it work? Let's say I have my client application and on one end and on the other end, I have all the replicas of these multiple servers of the director.

A

When I read in the eventual consistency model, it means that I may land in a random replica. So I read here: I landed in graphic number one I execute another read and I landed another replica but keep in mind. All of these are kept in sync. So as long as they do just reads, nothing really happens. It's fine, because all the data is copied all over.

A

The problem happens when I'm writing. So, in the version consistency model, when I write into a replica, it may happen that the next stop read lands in a different replica. That was not yet synced because you know with same timing and synchronization I may read in something that is not as up-to-date as I wrote and immediately before and now you'll say, but hey luca. This is but what happened with microsoft graph I can write and read and all it's fine. Well, yes, you're right, because this is not the fault model that aad uses.

A

We use something called session, consistency and session. Consistency is basically a smart way to lock an application plus tenant and geographical location pair into a specific replica so that when you're reading and writing from the same application on the same tenant, you land in the same replica all the time, and that means that it gives you the impression of of having read, write consistency, even if you know an another application.

A

Reading the same data may land in a different replica, but this. This is why, when you use microsoft graph and you read and write the data, you get a sense of consistency, but now you remember that at the beginning I said that, for this advanced query, you need some magic parameter like the dollar count and consistency level. Eventual so why? Why do we need that?

A

Let's get back to our architecture over here. So when the user execute a call to the entity, workload post, the rest directory service, that is the normal service and the advanced query service can answer the question because both of them can and return the list of users right independently.

A

So microsoft graph doesn't know what to do here. Well, the super easy thing is to say: hey if my query contains a dollar count, or my query contains a dollar search that are not supported by the original store. Then okay go go to the advanced query service, but what happened? If I, if I write something when I write something again, the same process as described before the right gets redirected to one of the replica and then the replica are getting in sync all together.

A

The sync engine, then picks up the change from one of the replicas.

A

That may not be the one that the original right was written into and the sync engine does their all in their own calculation and finally push the change to the advanced query store and you notice now that the data in the advanced query store and the data in the original replicas are in completely separate places that they don't have really any correlation between each other, and so all this time of replication, plus all the time for the sync engine to push the change here is something that we want you to account for you as a developer to account for, and so we decided that to agree with advanced capabilities.

A

You also need to add this eventual consistency, header, because I want you to be aware that the data that you will be querying from this store may not be as up-to-date to the original data that is over here, and you know the differences are not like hours or days. The difference of of synchronization are in the order of about five minutes, let's say on average, so that may happen.

A

So that add the user to the list and then, if I use a search of the research dollar account using this, I can get the response that I expect only like two minutes later.

A

So how would you handle this in in your app? Well, there are three scenarios, one. If you do only read that 92 percent apps of apps are doing, then you have does not have any problem, because your app doesn't actually write anything in in the directory. So if you're only reading you, you can even send eventual consistency, header in all the requests and and get a little bit of improvement in latency.

A

But if you're up then right and requires user interaction, you can show some appropriate messaging like hey. This separation might take a while when, when you are the user, so um or, for example, use lock local cache.

A

Well, if your application instead is a like an automation, app, we know that it's not ideal solution like implementing an exponential back off, just to continue querying back and see, for example, the user that you just added it's returned, but we have to have something common that will fix that. Unfortunately, not very soon, it will be next year, first part of next year, but that, as you can imagine, might require a lot of architecture changes and we are working actively working on it to get your developer experience better.

A

So next up we have, for example, okay. Now I'm going back and forth with graph explorer, and so you can, you can see how to query your tenant data and what results you get back so the first one. We already saw the raw counts of the users, and this is the applies that the raw count is just returning a number while for the second example, instead of using a raw count, we use count as a query parameter instead of square segment and also we add a filter in this case.

A

When we do this, we get the list of the users plus an audit account value over here. So you have both in in the response that can be very useful, going back. Another functionality that we added is counting, for example, the transitive member of the group.

A

As you know, a group can contains also nested group and with transitive members, you can count, for example, all the like leaf nodes of of this group, and in this case this group over here has 27 transduced members, and if we remove this transitive name, we only want the direct members to that number would reduce to six now another big functionality that we added is search.

A

I know that many of you were asking for contains, so how? How do I search, for example, users in my in my directory, that has display name and they want only to search, for example, for the second part of my display name and not the first part, not not using start suite. I want only the second part.

A

Well, it turns out that the exact contains operator is super expensive for queering the database and and also it may not give you exactly what you want, and the approach that we took is to separate all the words inside the display name and description properties.

A

Tokenize all these words and let you search for all these tokens, so, for example, for we are separating by space, we are separating by different casing, we're separating by special character and if there are numbers and letters together. So all these are the tokens that are generated in this case and we are going to do a starts with on each of the tokens. So let's say an example: if we run this query so, let's search for old user, we display name, al and and mail start with admin we get. This result.

A

You can see here that al okay, so this was just by the example. Sorry about that. But let's do wi. Instead.

A

I, if I search for this, I I I'm returning this part, because this was separated by a space and I got the starts with. On the second token, there is a lot of documentation regarding search, and if you, if you open our web page with the docs, you can see that there is a lot of explanation on how to use here how to use this capabilities.

A

So next up, we have a little bit of search. Let's see what other example we have search is enabled again on display name and description and when you do an or inside the search, and you select a property that is not display, number or description we fall back to the normal starts with and and that's why I started going going back here over here.

A

Sorry, okay, that's why we say mail starting with admin, because, even though we are inside search mail is not supported as a this tokenization approach, so it fall back with a start with okay. Here this is every service principle with display name sync. You can see that this is found right, even if it, because there is an uppercase difference. This is found because it was tokenized the space and- and this is also found that it was tokenized with a special character that is adult.

C

It's just fun to interject here. I think I think this is great one thing I'll just note is: I was very confused when I started to use the search functions and found that certain object types are limited to the operator, supported and further specific properties of that object type. So if you go back to that page, you were showing sure it's a it's a good page that shows the high level functionality of the search capabilities.

A

C

But the web page, sorry, the one that you broke out.

A

The web page over here.

C

Yeah, so this is good at like showing the high level. Actually, this might be a newer page than the one I'm reviewing yes.

D

There's one that talks.

C

About oh data search capabilities: anyway, there was a table nested in a completely different page. That shows the limitations of various object, types and properties. Oh yeah, yeah,.

A

I mean figure that is this page over here.

C

Yeah, that's the one.

A

Okay, so this is specifically for filter, though not not for filters.

C

Yes, yes, right, right, okay, filtering yeah! Is that related to what you're discussing yes, yes,.

A

These are exactly the advanced career capabilities that I'm talking about, and yes I, unfortunately we have these uh differences for each of the approaches. It.

C

Shows me a little nuts because I didn't know about this up front like I was looking at the main page. So there is a link to this page off the main page I'm talking about, but it's kind of very and so, if you're trying to do this, you might bang your head against the wall, trying to figure out why it's not working and then, when you find this yeah it makes sense. But it's a little bit buried. It's tough, to which I it's good. You know you get it, but it's not as intuitive. Sometimes anyway,.

A

C

A

It's all because of uh the initial architecture right. So if we go back here, remember the indexes are in this in this query: store and that's why we we need to have this advanced advanced player capabilities because they are not in the same one, and if, if you see here, the one with the empty check mark it works by default means that it use rest directory service, the one with the field check mark it. It's you it's using like this service got it that that's uh why it's a that is.

A

There is so much differences, but again we are working on. Is it unifying everything? And so you don't have to to lose your head on understanding what kind of parameters you need to put for for which property and and we are fixing that so this is the good news I can say now.

A

So let's go back over here, okay, okay! So next up we have search transitively. This is a very interesting because transitive search is.

A

Is something that you can limit to a specific type of object, as you know that groups can contain users can contain service principles can contain devices and other groups, and thanks to this name that is called odatacast.

A

You can filter down the list of members by just users and then use you know a search on those on those users and find the one that you that you need all right. So we also added. uh This is just a recent thing. The supports for ends with on the proxy address collection, and this allows you to to do a query like a search every user have in the collection of proxy addresses.

A

That is a list of strings basically or any of the items in this list that ends with this string, and if I run the query, I get all the users with the proxy address ending with this thing.

A

So you can use the advanced query parameters also with the any clause and another thing we also added with advanced query parameters, the not function, so you can negate anything you you need. So if you, if you want to search all the users that are not ending with on microsoft.com, you just add a note here before run the query and you get everybody with empty proxy addresses or proxy addresses that are different from from the search search string.

A

Again, all of these requires these countable through and consistency level. If I forget this one, I remove it and I run the query. I get an error. Oh, this is not supported, but it's not true. It's supported, you just forgot to add the consistency level eventually, and I know that this message is super unhelpful.

A

We are also working towards getting more helpful messages like hey, you might need to try to add consistency level, eventual see if it works. Okay order by so this was another limitation of the original directory store. You cannot get ordering and filtering at the same time.

A

Well now we can order through the same time, always remember to add countable true, even if you don't need it, because that's it it's how microsoft graph knows that we need to use the advanced query service instead of the normal service, and in this case I am searching for my user order by create a datetime and that have an office location starting with the number 20 and then I'm just getting the top top three. This is the result. These are the top last three users that are in building 20.

A

Okay, so batch is another example where struggle a little bit patch is a post operation right. So as a post operation, adding request header here for eventual consistency doesn't make any sense, because I am posting stuff, while the request headers needs to be in the actual post body.

A

Okay, so it's in this yeah, so you can see that the header needs to be in in the actual request of the body and not in the post request. So if you run this query now we have that quest number one. As this result and the request number two, as this other result.

A

Almost done now, let's talk about sdk ports.

A

Powershell is very well supported, because everything that you see in microsoft graph is automatically translated into this powershell sdk, and you can install these using these few commands that you can install microsoft front model and then you import the authentication, for example. If we want to send a request for users, we import the user model and then this one it will show you all the models that are available optional and after we connect to graph.

A

We can send to this commandlet the consistency level, eventual that was shown before and the count that this will get into a variable, and then you can use the other. The advanced query parameter. So, let's see how it works in practice.

A

Okay, so you get mg users blah blah blah and I have oh. I need to connect rough.

A

Perfect and this again and there you go and you'll see every user with name that has room in with even one of the tokens and now, if, as I use the user count as as variable, I do add, user count, oops, sorry, dollar, user coverage, polar user count I'll see that I have six in this variable good.

A

So powershell is in a good place, some of the command. Let's actually do not have the consistency level eventual parameter, but this is just a bug if you, if you encounter some of them, that you cannot add the consistency level, please let us know and we'll fix it right away. Well, the dot net sdk is a little tricky because it doesn't support out of the box.

A

Some of the parameters- and I created this sample application that is published in these github.

A

Over here, so I created this.net application and you can see how to use all the various parameters and if you go to the graph data service, so, for example, if you need to to get the applications and you you want to use search, then you need to use this syntax.

A

Basically, you add into the request parameters the query, option, countable true and then the query option, search as the search is not exposed as one of the normal select and filter. So if I try to do dot search, it doesn't exist, unfortunately, that this application mimics graph explorer or less so when you run this you'll see.

A

That is a very familiar ui with with the various graph explorer commands. So it's the second seems a bit slow building. This very simple up, though okay so sign in with my account over here.

A

Navigation done perfect and.

A

Here you go okay, so I can set my dollar select here I can add the filter. You know I can do this playing a follow-on room as we did before and if you click load now uh not filter, sorry search.

A

Now it would filter so search all the users with with this token, in the display okay, so you can play with it. I think it would be very useful if you're using dotnet as an sdk.

A

And give you a quick good reference? Okay, going back to our presentation, we're almost done. Okay, so conclusion is that all the things that I showed you, unfortunately, are not supported in the current b2c tenants.

A

We are, however, we are working on something called cm tenants that have similar capabilities to the b2c tenants and will support all the capabilities that I showed you before, and many others then, as I was telling you before, search, is only supported to just display name and description, and if you try to do it on any other property, it would default to start suite.

A

We would really like the feedback from you if you need the tokenization approach on any other property that then displayed in the description also ends with filter is limited to these three properties.

A

And again these are just a property name. They are independent of the object, so any object with properties that have this uh name would would work with nz field. Also. Another limitation that we currently have is that if you are using the advanced query capabilities, you cannot expand.

A

We are also fixing that that will come a little sooner in like one or two quarters and we don't support skip. So if you have a pagination of items, so the collection that is over 1999, that is the maximum limit. You have to use skip token to paginate through the various pages, and you cannot do like skip.

A

Let's say the other capabilities where it would support and just recreate it only support. Eventual consistency for now here are the additional resources that I was mentioning you before. We have a help page for for these specific queries and the example of the dotnet application and ask from you if you're using these, please fill this survey.

A

If you feel that you're missing some functionality, I'm personally reading all the answers to this survey and and putting that in our backlog when, when we receive enough feedback, oh that your code of the okay to this concludes our session. I hope you learned something new and you enjoyed the internals of active directory.

A

If you have any other question, I'm here for.

E

Next 10 minutes hi everybody. If you have a question, please feel free to come off, mute and uh just an orderly fashion, we'll try and get through those.

B

Yes, I've got a couple of them: I've just kind of written down, as you went, um because the eventual consistency issue you suggested uh applying an exponential back off is there anything returned in the response that indicates when I write a value and then immediately request it that the response I receive might not be the most up-to-date value.

B

A

B

That I need to do an exponential.

A

Yeah, that's the the work we are currently implementing and.

D

A

Available right now, it would be um something called consistency, token, where, for example, if you receive a notification from like a change and then you you get this cons, you get this consistency token that allows you to understand if the response is in a replica that is as up-to-date as your the notification that you receive, and we are also thinking on on implementing this consistency token for any right, but unfortunately, is not available right now.

A

So yes, it's not ideal, because if, for example, you add and remove a user one after and you do this operation one after the other, and then you query back your your list of users with with the id and then you receive a 404.

A

You don't know if, like this four or four is, is actually true or you know, or is it was due because the delete was not happen yet right, yeah.

B

uh You had said that there's like a five minutes, wait until it's consistent on your side. Is that something we that's somewhat guaranteed or no, it's not.

A

So the replication delay is really susceptible by kind of rights that are happening in the replicas. So, as you can imagine, in one replica, there are many tenants and one of the tenants if they, for example, delete a hundred thousand objects. At the same time, they may impact also other tenants in the same replicas that are trying to replicate the same change all over the other replicas and depending on the type of rights that are happening.

A

The geo where the replica is network, latencies and and so on, the replication delay can can vary a lot and we currently don't have an upper bound.

B

A

B

A

What we can tell you is that on average, like let's say, p99, so not average p99, it will be within five minutes.

B

Okay, uh going back to some of the more recent slides, there was an indication that we could specify the consistency headers ourselves, and I guess I saw that you had said on the the new version. That's coming out. The only option for consistency is eventual on at least the old version. Is there any reason why we wouldn't favor something better than eventual.

A

B

You know, given the choice.

A

So the choice is the default like not specifying the header that is session consistency and if you specify eventual consistency but again with session consistency, you only land into the the normal service, the rest directory service over here. So you, you cannot use the advanced query capabilities if you have this session so by switching.

B

To eventual I, I open up the door to being able to use it.

A

Exactly I gotcha and- and you always need to remember to add count equal through into the query, because adventure consistency is also handled by the rest directory service, so just eventual consistency alone does not guarantee you to to land over here.

A

Okay, so you need both adventure, consistency and count equal to okay.

B

And then, finally, I guess on the powershell sdk: are there any plans to make the the microsoft graph powershell sdk itself available inside of the arm deployment script? Environments so that you know it doesn't have to be installed just because yeah we deploy a lot of arm templates and.

A

B

See you know, value and being able to query the graph as parts of those deployments, but you can't install anything from those environments, so it'd be really cool if that were just available by default in there.

A

I can connect you with our pm for the partial sdk till it upgrades send me an email and, if you just edit this light and then send it in my email, this is my email.

D

Sorry um not directly related to uh this, but do we have any sort of a threshold or idea for things like, um I think, some of the exceptions like bandwidth limit being exceeded or too many requests. I remember from the documentation. This is sorry. If this is in the context of a multi-tenant application, um it seems to imply that it's still per not the owner of the application, uh but the user's tenant is that the way the threshold is applied so the threshold.

A

For directory objects, I, on the throttling documentation is, is an app and tenant layer. So let's go see application plus tenant pair, and these are the limits.

D

ah Okay, so it is uh so the same application if it's a by a different tenant, uh in other words user of the application, it would not affect other users from other. Let's say, businesses.

A

um Well, the application is like application id. So if an application is used by multiple users, it's always the application id that that counts. So what's the trash tool.

D

If uh the same application id is used by tenant a but also used by tenant b, because remember.

A

D

Is a multi-tenant app.

A

Yeah yeah yeah they're they're different things, because okay yeah, thank you for clarifying. Yes, thanks any other questions, no I'm always available by email and again. The survey link is the main uh way that we we use to to get feedback from these items. Okay, thanks so much everyone.

E

Just as a reminder, the next community call is going to be june 16th at 9 a.m, pacific time and you can update your call calendars- aka ms wack id dev community calendar, and this call will be available on the youtube channel later bye. Everyone.

E