Software Engineer – Microsoft 365 SharePoint, Teams, PowerShell, Azure, Search, Governance, Automation etc.
Tag Archives: Microsoft Search
When a user searches, Microsoft Search processes the query and parses search intent from larger phrases, using Artificial Intelligence (AI) to learn common superfluous phrases users add to their queries that don’t impact their search intent. The search results that the user has permission to see are presented on the search results page. Microsoft Search uses intelligent ranking algorithms to order results based on relevance.
There is a Microsoft.Graph PowerShell module provided by Microsoft which simplifies usage of Microsoft Graph API. Below is how to authenticate to MS Graph and how to search within SharePoint and Teams Microsoft 365 content using Microsoft.Graph PowerShell module.
For daemon app authentication we need a certificate configured in Azure App and installed on the user machine. Daemon app authentication code sample (please specify your tenant id, app (client) id and certificate thumbprint:
Microsoft Graph API allows you to work with all the Microsoft 365 content – including search through Exchange e-mail messages, Yammer (Viva Engage) and Teams chat messages and surely OneDrive and SharePoint content (please refer to the original doc). Let me focus on searching in SharePoint Online and OD here but you can use the same technique to search through other Microsoft 365 services. I will use PowerShell but same ideas should work for other platforms/languages – Python, C#, node.js etc.
Assuming we have a registered Azure app configured correctly, including Secrets/Certificates blade and API permissions provided – we should be ready to authenticate and call Graph API unattended – on behalf of application itself.
Let us authenticate as a service/daemon app with client id and client secret:
# Authenticate to M365 as an unattended application
# specify your app id. app secret, tenant id:
$clientID = ""
$clientSc = ""
$TenantId = ""
# Construct URI and body needed for authentication
$uri = "https://login.microsoftonline.com/$tenantId/oauth2/v2.0/token"
$body = @{
client_id = $clientID
client_secret = $clientSc
scope = "https://graph.microsoft.com/.default"
grant_type = "client_credentials"
}
# Get OAuth 2.0 Token
$tokenRequest = Invoke-WebRequest -Method Post -Uri $uri -ContentType "application/x-www-form-urlencoded" -Body $body -UseBasicParsing
$token = ($tokenRequest.Content | ConvertFrom-Json).access_token
$headers = @{Authorization = "Bearer $token" }
Below is how I search Microsoft 365 content programmatically from PowerShell using MS Graph API being authenticates as user.
Notice we use “region” – it is required to search with Graph API under application credentials. Otherwise you will get an error message “SearchRequest Invalid (Region is required when request with application permission.)”:
Parameter “fields” allows you to request only fields you need to be returned. As returning object will be smaller your request will perform faster.
There might be a big number of objects found in m365 upon your request. Graph will not always return to you all the results. AFAIK currently the limit is 500, so if there are more than 500 objects found – only first 500 will be returned. You can specify how many objects you need to be returned per call with “size” parameter.
You can check value of $res.value[0].hitsContainers[0].moreResultsAvailable property and if it’s True – that means there are more results. The value above and parameters “from” and “size” would allow you to organize a loop so you can call search API many times to return all results.
Assuming we have a registered Azure app configured correctly, including Authentication and API permissions provided – we should be ready to authenticate and call Graph API on behalf of a user.
Microsoft Graph API allows you to work with all the Microsoft 365 content – including search through Exchange e-mail messages, Yammer (Viva Engage) and Teams chat messages and surely OneDrive and SharePoint content (please refer to the original doc).
Let me focus on searching in SharePoint Online and OD here but you can use the same technique to search through other Microsoft 365 services. I will use PowerShell but same ideas should work for other platforms/languages – Python, C#, node.js etc.
Let us authenticate first. We’d need a MSAL.PS module for that.
# Ensure we have MSAL.PS module installed
Get-Module MSAL.PS -ListAvailable | ft name, Version, Path
# Install-Module MSAL.PS -Force -Scope CurrentUser -AcceptLicense
Import-Module MSAL.PS
# Authenticate to Microsoft Interactively
$clientid = 'd82858e0-ed99-424f-a00f-cef64125e49c' # your client id
$TenantId = '7ddc7314-9f01-45d5-b012-71665bb1c544' # your tenant id
$token = Get-MsalToken -TenantId $TenantId -ClientId $clientid -Interactive
$headers = @{Authorization = "Bearer $($token.AccessToken)" }
Below is how I search Microsoft 365 content programmatically from PowerShell using MS Graph API being authenticates as user.
# Search
# MS Graph Search API url (beta or v1.0):
$apiUrl = "https://graph.microsoft.com/beta/search/query"
# specify where to search - entity types
$entityTypes = "['driveItem','listItem','list','drive','site']"
$entityTypes = "['driveItem','listItem']"
# query
$query = "test*"
# build a simple request body
$body = @"
{
"requests": [
{
"entityTypes": $entityTypes,
"query": {
"queryString": "$query"
}
}
]
}
"@
# call Graph API:
$res = Invoke-RestMethod -Headers $Headers -Uri $apiUrl -Body $Body -Method Post -ContentType 'application/json'
# explore returned object
$res.value[0].searchTerms
$res.value[0].hitsContainers[0].hits
$res.value[0].hitsContainers[0].hits.Count
$res.value[0].hitsContainers[0].moreResultsAvailable
Why do we need to implement search in our applications?
Use-cases for search on behalf of current user
Along with the usual ones – where you just need your app to search for some data and bring it to user – there is one different scenario I’d like to share:
You need to quickly detect content in SharePoint that is open for everyone
Brute force solution – getting detailed permissions report for all SharePoint sites might not be a feasible option, especially in large environments – it is a very resource-consuming task and might take days and weeks. So consider the following…
Since search is security-trimmed – a user can get only search results he/she already has access to; but what if we create an account and do not grant any SharePoint permissions or group memberships to this account, and then we’d search for everything on behalf of this account? That would mean that all what search returns represent content that is shared with everyone. There are some tricks and gotchas – here is the separate article on the same.
Use-cases for unattended search
What are the use-cases when you need to search in your daemon app or background job? Be aware that when you search on behalf of application credentials – search is NOT security-trimmed and your query would run against ALL SharePoint content… Here are some possible scenarios.
Content detection/Investigation
Let say you want some data is never shared with anyone and never appeared in search for anyone
Or you might want to investigate what is the location some specific data is available at
Imagine you are building sites classification system and you use indexed custom site properties – so you are able to refine search results based on site metadata to get list of specific sites (adaptive scopes used in retention policy are based on the same mechanics)
Automation – let say you have a requirement to configure every tenant site in some ways – for instance – add some hosts to allowed domains to embed video or set some site properties based on who created the site or activate or deactivate some features and so on – how would you do that? You’d probably have a scheduled job that runs let say every hour against only new sites – sites created during that last hour. How would you get these recently created sites? Search with Graph API is the only nice solution today.
In the articles below I’m sharing my techniques on searching in Microsoft 365 SharePoint and Teams from application using Microsoft Graph API. Specifically I’m covering
Microsoft Graph search API
Microsoft.Graph PowerShell module
PnP.PowerShell module
In two flavors:
Search on behalf of currently authenticated user
Unattended Search with daemon (also called service) applications
Below is how do I search Microsoft 365 content programmatically from PowerShell using MS Graph API, PowerShell PnP, Microsoft Graph module, MSAL library being authenticated as user or daemon application. Let me focus on SharePoint content here but you can use the same technique to search through other Microsoft 365 services. Also, I’ll be using PowerShell but same ideas should work for other platforms/languages – Python, C#, node.js etc.
To search on behalf of currently authenticated user we need delegated “Sites.Read.All” API permissions. I recommend you to add both Graph API and SharePoint API permissions as different libraries might use different API’s under the hood. Ensure you add delegated “Sites.Read.All” even if you already have “Sites.FullControl.All” as by some reason “Sites.FullControl.All” does not always work for search.
Here is how an app API permissions to search as current user should look like:
For unattended search – e.g. search on behalf of daemon app – we need application “Sites.Read.All” API permissions. Again, I suggest both Graph API and SharePoint API permissions added. Here is how an app API permissions to search as daemon app should look like:
Ensure you got admin consent for API permissions.
In case you have incorrect permissions in your app – Microsoft Graph will be kind enough to inform you exactly what you need. Example:
“Access to ChatMessage in Graph API requires the following permissions: Chat.Read or Chat.ReadWrite, ChannelMessage.Read.All. However, the application only has the following permissions granted: Sites.Read.All, User.Read”
Assuming we have configured apps – let us get started with
Microsoft Graph API
Microsoft Graph API allows search through all the Microsoft 365 content – including Exchange e-mail messages, Yammer (Viva Engage) and Teams chat messages and surely OneDrive and SharePoint content (please refer to the original doc).
Authenticate as current user to Search with Graph API
I use MSAL.PS PowerShell module to get token, then I build a headers variable
If you are getting error message “SearchRequest Invalid (Region is required when request with application permission.)”:
that’s OK, just modify your body to include region like this (“region”: “NAM” for North America or “GBR” or …). Also, I can modify body with from/size for paging (technique used to iterate through search results if there are many) and return just specific fields to decrease traffic and improve performance:
For daemon app authentication we need a certificate configured in Azure App and installed on the user machine. Daemon app authentication code sample (please specify your tenant id, app (client) id and certificate thumbprint:
I have created a user “Robert Dylan” and never used name “Bob”, but search understands that I’m looking for Robert Dylan when I’m searching for Bob or Bob Dylan or Bobby.
Important thing – result is shown under “All” and “People” verticals.
I know Microsoft claimed m365 search is backed by Turing technology – it understands you, answers your question but not just blindly display keyword occurences (announcement at Ignite Spring, more on Ignite Fall 2021)… So is that it?
But the same trick did not work with Dick for Richard or (of course) Syd for Roger Barrett:
Using Path property in Microsoft 365 Search Query was kind of ambiguous. But now Microsoft implemented update and clarified some details. So below are some tips and tricks on filtering by site Url (path) in query field in Microsoft 365 Search verticals.
Path filter with trailing slash (“/”)
In November 2022 Microsoft rolled out an update for multiple search features, including checks on the path managed property for a trailing slash. Previously path filters were valid with and without trailing slashes.
Consider the following scenario.
Given the path filter with the contain operator (“:”)
SPSiteUrl and Path properties use different matching strategies. When using contains operator (colon sign “:”) – SPSiteUrl will match the full value, while Path will do a “starts with” match.
DepartmentId
DepartmentId is a search managed property used under Hub sites and propagated through all associated sites content.
That means if we want to scope down search to hub site with it’s content – we can use DepartmentId property, e.g.
DepartmentId=4965d9be-929b-411a-9281-5662f5e09d49
instead of iteration through all hub sites and using path: property.