Category Archives: SharePoint

Search in SharePoint using Microsoft Graph API with application credentials

Microsoft Graph API allows you to work with all the Microsoft 365 content – including search through Exchange e-mail messages, Yammer (Viva Engage) and Teams chat messages and surely OneDrive and SharePoint content (please refer to the original doc). Let me focus on searching in SharePoint Online and OD here but you can use the same technique to search through other Microsoft 365 services. I will use PowerShell but same ideas should work for other platforms/languages – Python, C#, node.js etc.

Assuming we have a registered Azure app configured correctly, including Secrets/Certificates blade and API permissions provided – we should be ready to authenticate and call Graph API unattended – on behalf of application itself.

Let us authenticate as a service/daemon app with client id and client secret:

# Authenticate to M365 as an unattended application

# specify your app id. app secret, tenant id:
$clientID = ""
$clientSc = ""
$TenantId = ""

# Construct URI and body needed for authentication
$uri = "https://login.microsoftonline.com/$tenantId/oauth2/v2.0/token"
$body = @{
    client_id     = $clientID
    client_secret = $clientSc
    scope         = "https://graph.microsoft.com/.default"
    grant_type    = "client_credentials" 
}

# Get OAuth 2.0 Token
$tokenRequest = Invoke-WebRequest -Method Post -Uri $uri -ContentType "application/x-www-form-urlencoded" -Body $body -UseBasicParsing
$token = ($tokenRequest.Content | ConvertFrom-Json).access_token
$headers = @{Authorization = "Bearer $token" }

Below is how I search Microsoft 365 content programmatically from PowerShell using MS Graph API being authenticates as user.

# Search
$entityTypes = "['driveItem','listItem','list','drive','site']"
$entityTypes = "['driveItem','listItem']"

$query = "LastModifiedTimeForRetention<2021-01-01"
$apiUrl = "https://graph.microsoft.com/beta/search/query"
$query = "test*"
$body = @"
{ 
  "requests": [
    {
      "entityTypes": $entityTypes,
      "query": {
        "queryString": "$query"
      },
      "from" : 0,
      "size" : 5,
      "fields": ["WebUrl","lastModifiedBy","name" ],
      "region": "NAM"
    }
  ]
}
"@

$res = Invoke-RestMethod -Headers $Headers -Uri $apiUrl -Body $Body -Method Post -ContentType 'application/json'
$res.value[0].searchTerms
$res.value[0].hitsContainers[0].hits
$res.value[0].hitsContainers[0].hits.Count
$res.value[0].hitsContainers[0].moreResultsAvailable

Notice we use “region” – it is required to search with Graph API under application credentials. Otherwise you will get an error message “SearchRequest Invalid (Region is required when request with application permission.)”:

Parameter “fields” allows you to request only fields you need to be returned. As returning object will be smaller your request will perform faster.

There might be a big number of objects found in m365 upon your request. Graph will not always return to you all the results. AFAIK currently the limit is 500, so if there are more than 500 objects found – only first 500 will be returned. You can specify how many objects you need to be returned per call with “size” parameter.

You can check value of $res.value[0].hitsContainers[0].moreResultsAvailable property and if it’s True – that means there are more results. The value above and parameters “from” and “size” would allow you to organize a loop so you can call search API many times to return all results.

Other articles index:
Search m365 SharePoint and Teams content programmatically via MS Graph API

Using Microsoft Graph Search API as current user

Microsoft Graph API allows you to work with all the Microsoft 365 content – including search through Exchange e-mail messages, Yammer (Viva Engage) and Teams chat messages and surely OneDrive and SharePoint content (please refer to the MS’s original doc). After we got a registered Azure app configured correctly, including Authentication and API permissions provided (more on this) – we should be ready to authenticate and call Graph API on behalf of a user.

Let me focus on searching in SharePoint Online and OD here but you can use the same technique to search through other Microsoft 365 services. I will use PowerShell but same ideas should work for other platforms/languages – Python, C#, node.js etc.

Let us authenticate first. We’d need a MSAL.PS module for that.

# Ensure we have MSAL.PS module installed
Get-Module MSAL.PS -ListAvailable | ft name, Version, Path 
# Install-Module MSAL.PS -Force -Scope CurrentUser -AcceptLicense
Import-Module MSAL.PS

# Authenticate to Microsoft Interactively 
$clientid = 'd82858e0-ed99-424f-a00f-cef64125e49c' # your client id
$TenantId = '7ddc7314-9f01-45d5-b012-71665bb1c544' # your tenant id
$token = Get-MsalToken -TenantId $TenantId -ClientId $clientid -Interactive
$headers = @{Authorization = "Bearer $($token.AccessToken)" }

Below is how I search Microsoft 365 content programmatically from PowerShell using MS Graph API being authenticates as user:

# Search
# MS Graph Search API url (beta or v1.0):
$apiUrl = "https://graph.microsoft.com/beta/search/query"

# specify where to search - entity types
$entityTypes = "['driveItem','listItem','list','drive','site']"
$entityTypes = "['driveItem','listItem']"

# query
$query = "test*"

# build a simple request body
$body = @"
{ 
  "requests": [
    {
      "entityTypes": $entityTypes,
      "query": {
        "queryString": "$query"
      }
    }
  ]
}
"@

# call Graph API:
$res = Invoke-RestMethod -Headers $Headers -Uri $apiUrl -Body $Body -Method Post -ContentType 'application/json'

# explore returned object
$res.value[0].searchTerms
$res.value[0].hitsContainers[0].hits
$res.value[0].hitsContainers[0].hits.Count
$res.value[0].hitsContainers[0].moreResultsAvailable

I used “beta” search API to research or make demos, but in production code youd stick with “v1.0”.

You can scope search down using entity types – ‘driveItem’,’listItem’,’list’,’drive’,’site’. “driveitem” here represents document library.

In query you can use KQL.

Always check if more results available with “$res.value[0].hitsContainers[0].moreResultsAvailable”. If there are more results and you need them – consider looping using paging technique.

References

Search M365 content from code: use-cases

Why do we need to implement search in our applications?

Use-cases for search on behalf of current user

Along with the usual ones – where you just need your app to search for some data and bring it to user – there is one different scenario I’d like to share:

You need to quickly detect content in SharePoint that is open for everyone

Brute force solution – getting detailed permissions report for all SharePoint sites might not be a feasible option, especially in large environments – it is a very resource-consuming task and might take days and weeks. So consider the following…

Since search is security-trimmed – a user can get only search results he/she already has access to; but what if we create an account and do not grant any SharePoint permissions or group memberships to this account, and then we’d search for everything on behalf of this account? That would mean that all what search returns represent content that is shared with everyone. There are some tricks and gotchas – here is the separate article on the same.

Use-cases for unattended search

What are the use-cases when you need to search in your daemon app or background job? Be aware that when you search on behalf of application credentials – search is NOT security-trimmed and your query would run against ALL SharePoint content… Here are some possible scenarios.

  • Content detection/Investigation
    • Let say you want some data is never shared with anyone and never appeared in search for anyone
    • Or you might want to investigate what is the location some specific data is available at
  • Imagine you are building sites classification system and
    you use indexed custom site properties – so you are able to refine search results based on site metadata to get list of specific sites (adaptive scopes used in retention policy are based on the same mechanics)
  • Automation – let say you have a requirement to configure every tenant site in some ways – for instance – add some hosts to allowed domains to embed video or set some site properties based on who created the site or activate or deactivate some features and so on – how would you do that? You’d probably have a scheduled job that runs let say every hour against only new sites – sites created during that last hour. How would you get these recently created sites? Search with Graph API is the only nice solution today.

Index of other articles on the subject:

  • Search Microsoft 365 content programmatically: Index
  • Search Microsoft 365 content programmatically: Use-case scenarios
  • Authentication to Microsoft Graph: Azure Registered Apps Certificates and Secrets
  • Authorization to Microsoft Graph: Azure Registered Apps API permissions
  • Calling Microsoft Graph Search API from code as current user
  • Calling Microsoft Graph Search API from daemon/service app
  • Using Microsoft.Graph PowerShell module to Search in Microsoft 365
  • Using PnP.PowerShell module to Search in Microsoft 365

Search m365 SharePoint and Teams content programmatically via MS Graph API

In the articles below I’m sharing my techniques on searching in Microsoft 365 SharePoint and Teams from application using Microsoft Graph API.
Specifically I’m covering

  • Microsoft Graph search API
  • Microsoft.Graph PowerShell module
  • PnP.PowerShell module

In two flavors:

  • Search on behalf of currently authenticated user
  • Unattended Search with daemon (also called service) applications

Index of my articles on the subject:

Video tutorials

Search through Microsoft 365 SharePoint from code

Below is how do I search Microsoft 365 content programmatically from PowerShell using MS Graph API, PowerShell PnP, Microsoft Graph module, MSAL library being authenticated as user or daemon application. Let me focus on SharePoint content here but you can use the same technique to search through other Microsoft 365 services. Also, I’ll be using PowerShell but same ideas should work for other platforms/languages – Python, C#, node.js etc.

First, we need to be authenticated

Here is how to authenticate to Microsoft 365 Graph API.

Second, we need to be authorized

To search on behalf of currently authenticated user we need delegated “Sites.Read.All” API permissions. I recommend you to add both Graph API and SharePoint API permissions as different libraries might use different API’s under the hood. Ensure you add delegated “Sites.Read.All” even if you already have “Sites.FullControl.All” as by some reason “Sites.FullControl.All” does not always work for search.

Here is how an app API permissions to search as current user should look like:

app API permissions to search as current user

Actually, for interactive authentication app ownership is not required, so we can surely use our own registered app, but also it is possible to use any other app registered in Azure and properly configured, e.g. Enterprise “PnP Management Shell” app id: “31359c7f-bd7e-475c-86db-fdb8c937548e”
How do I create and configure Azure App with delegated permissions to SharePoint for PowerShel usage

For unattended search – e.g. search on behalf of daemon app – we need application “Sites.Read.All” API permissions. Again, I suggest both Graph API and SharePoint API permissions added. Here is how an app API permissions to search as daemon app should look like:

Ensure you got admin consent for API permissions.

In case you have incorrect permissions in your app – Microsoft Graph will be kind enough to inform you exactly what you need. Example:

“Access to ChatMessage in Graph API requires the following permissions: Chat.Read or Chat.ReadWrite, ChannelMessage.Read.All. However, the application only has the following permissions granted: Sites.Read.All, User.Read”

Assuming we have configured apps – let us get started with

Microsoft Graph API

Microsoft Graph API allows search through all the Microsoft 365 content – including Exchange e-mail messages, Yammer (Viva Engage) and Teams chat messages and surely OneDrive and SharePoint content (please refer to the original doc).

Authenticate as current user to Search with Graph API

I use MSAL.PS PowerShell module to get token, then I build a headers variable

# Prerequisites
Get-Module MSAL.PS -ListAvailable | ft name, Version, Path 
# Install-Module MSAL.PS -Force -Scope CurrentUser -AcceptLicense
Import-Module MSAL.PS

# Interactive Authentication
$clientid = 'd82858e0-ed99-424f-a00f-cef64125e49c'
$TenantId = '7ddc7314-9f01-45d5-b012-71665bb1c544'
$token = Get-MsalToken -TenantId $TenantId -ClientId $clientid -Interactive
$headers = @{Authorization = "Bearer $($token.AccessToken)" }

Authenticate as service/daemon app

You’d need to update the script providing Tenant id, client (app) id and client (app) secret:

# App Authentication
$clientID = ""
$clientSc = ""
$TenantId = ""

# Construct URI and body needed for authentication
$uri = "https://login.microsoftonline.com/$tenantId/oauth2/v2.0/token"
$body = @{
    client_id     = $clientID
    client_secret = $clientSc
    scope         = "https://graph.microsoft.com/.default"
    grant_type    = "client_credentials" 
}

# Get OAuth 2.0 Token
$tokenRequest = Invoke-WebRequest -Method Post -Uri $uri -ContentType "application/x-www-form-urlencoded" -Body $body -UseBasicParsing
$token = ($tokenRequest.Content | ConvertFrom-Json).access_token
$headers = @{Authorization = "Bearer $token" }
$headers

Search m365 SharePoint and OD content with Microsoft Graph API

In this sample I limited search scope to list items only ($entityTypes = “[‘listItem’]”). Check other entity types here.

Code:

# Search
$entityTypes = "['listItem']"
$apiUrl = "https://graph.microsoft.com/beta/search/query"
$query = "*"
# body for interactive search
$body = @"
{ 
  "requests": [
    {
      "entityTypes": $entityTypes,
      "query": {
        "queryString": "$query"
      }
    }
  ]
}
"@

$res = Invoke-RestMethod -Headers $Headers -Uri $apiUrl -Body $Body -Method Post -ContentType 'application/json'
$res.value[0].hitsContainers[0].hits

If you are getting error message “SearchRequest Invalid (Region is required when request with application permission.)”:

that’s OK, just modify your body to include region like this (“region”: “NAM” for North America or “GBR” or …). Also, I can modify body with from/size for paging (technique used to iterate through search results if there are many) and return just specific fields to decrease traffic and improve performance:

# Search
$entityTypes = "['driveItem','listItem','list','drive','site']"
$entityTypes = "['driveItem','listItem']"

$query = "LastModifiedTimeForRetention<2021-01-01"
$apiUrl = "https://graph.microsoft.com/beta/search/query"
$query = "test*"
$body = @"
{ 
  "requests": [
    {
      "entityTypes": $entityTypes,
      "query": {
        "queryString": "$query"
      },
      "from" : 0,
      "size" : 5,
      "fields": ["WebUrl","lastModifiedBy","name" ],
      "region": "NAM"
    }
  ]
}
"@

$res = Invoke-RestMethod -Headers $Headers -Uri $apiUrl -Body $Body -Method Post -ContentType 'application/json'
$res.value[0].searchTerms
$res.value[0].hitsContainers[0].hits
$res.value[0].hitsContainers[0].hits.Count

We’d not use region for interactive calls or we’ll get “Region is not supported when request with delegated permission.”.

Microsoft.Graph PowerShell module

There is a Microsoft.Graph PowerShell module provided by Microsoft which simplifies authentication and search operations.

Interactive authentication code sample:

# Prerequisites
Get-Module Microsoft.Graph.Authentication -ListAvailable 
Get-Module Microsoft.Graph.Search -ListAvailable 

# Interactive Authentication
$clientid = '31359c7f-bd7e-475c-86db-fdb8c937548e'
$clientid = 'd82858e0-ed99-424f-a00f-cef64125e49c'
$TenantId = '7ddc7314-9f01-45d5-b012-71665bb1c544'
Connect-MgGraph -ClientId $clientid -TenantId $TenantId

For daemon app authentication we need a certificate configured in Azure App and installed on the user machine. Daemon app authentication code sample (please specify your tenant id, app (client) id and certificate thumbprint:

# App Authentication
$clientID = ""
$certThumbprint = ""
$TenantId = ""
Connect-MgGraph -ClientId $clientid -TenantId $TenantId -CertificateThumbprint $certThumbprint

Code sample for SharePoint search with Microsoft.Graph PowerShell module

As currently authenticated user

# Search in the current user context
$params = @{
	requests = @(
		@{
			entityTypes = @(
				"driveItem"
			)
			query = @{
				queryString = "lorem"
			}
			from = 0
			size = 25
			fields = @(
				"title"
				"description"
			)
		}
	)
}

$res = Invoke-MgQuerySearch -Body $params
$res.HitsContainers[0].Hits
   

Again, in case with app authentication – an additional parameter – region – is required:

# Search
$params = @{
	requests = @(
		@{
			entityTypes = @(
				"driveItem"
			)
			query = @{
				queryString = "lorem"
			}
			from = 0
			size = 25
			fields = @(
				"title"
				"description"
			)
                        region = "NAM"
		}
	)
}

$res = Invoke-MgQuerySearch -Body $params
$res.HitsContainers[0].Hits

It’s a good idea to explore returning object.

PnP.PowerShell module

PnP.PowerShell allows you to search through Microsoft 365 SharePoint content with PowerShell style – using command and options.

Let us authenticate interactively:

# Interactive Authentication
$clientid = 'd82858e0-ed99-424f-a00f-cef64125e49c'
$TenantId = '7ddc7314-9f01-45d5-b012-71665bb1c544'
$siteUrl = "https://s5dz3.sharepoint.com"
Connect-PnPOnline -ClientId $clientid -Tenant $TenantId -Url $siteUrl -Interactive

Authentication on behalf of service/daemon app would require certificate installed on the machine and configured in the app and look like:

# Application (daemon) Authentication
$clientID = ""
$certThumbprint = ""
$TenantId = ""
$siteUrl = "https://contoso.sharepoint.com"
Connect-PnPOnline -ClientId $clientid -Tenant $TenantId -Url $siteUrl -Thumbprint $certThumbprint

There are no differences in Microsoft 365 SharePoint Search with PowerShell code samples for interactive and daemon apps (no region parameter).

Examples:

# search
$query = "test*"
$res = Submit-PnPSearchQuery -Query $query 

# examples of query:
$query = "test*"
$query = "* contentclass:STS_ListItem_DocumentLibrary"
$query = "* author:Patti"
$query = "test* site:https://s5dz3.sharepoint.com/teams/sxc"

# examples of submitting request
$res = Submit-PnPSearchQuery -Query $query 
Submit-PnPSearchQuery -Query $query -All 
Submit-PnPSearchQuery -Query $query -MaxResults 5
Submit-PnPSearchQuery -Query $query -SortList @{"LastModifiedTime" = "ascending"} 

# exploring result object:
$res.ResultRows.Count
$res.ResultRows[0]
$res.ResultRows.Title
$res.ResultRows.OriginalPath
$res.ResultRows.LastModifiedTime

Video tutorials

Video tutorials (playlist) on how to authenticate to Microsoft 365 and Search through Microsoft 365 content from code

References

SharePoint Add-Ins and ASC retirement

Microsoft announced retirement for SharePoint Add-Ins and ASC-based app permissions (SharePoint app-only principals). Let me summarize here:

  • SharePoint Add-Ins, ASC-based app permissions and SharePoint app-only principals
  • Timeline of retirement and future plans
  • How to detect if there are legacy service principals used in the environment
  • Migration guidance

Timeline

Known key retirement milestone dates:

  • Mar 2024 – Microsoft will stop accepting new Add-In submission to app store
  • Jul 2024 – Microsoft will stop ability for customers to acquire add-ins from App Store (*)
  • using ACS and Add-ins will be blocked for new tenants since Nov 2024 (**)
  • using ACS and Add-ins will be blocked for all existing tenants since Apr 2, 2026

(*) SPFx based solutions will continue to be available, Installation from a private app catalog stays possible
(**) regardless of their origin (public marketplace, private tenant catalog)

So timeline is generous, and we have plenty of time to

What you need to do to prepare

  1. Detect if there are any legacy service principals used in the environment:
    Use the Microsoft 365 Assessment tool (by PnP) to scan your tenants for SharePoint Add-In usage.
  2. Migrate existing solutions that use Add-Ins or SP App-only spns – Review the guidance for migrating from SharePoint Add-Ins to SharePoint Framework.
  3. Disable SharePoint Add-Ins on your tenant using SharePoint Online Management Shell

How to disable SharePoint Add-Ins on your tenant

Admins can disable SharePoint Add-Ins with the Set-SPOTenant SharePoint Management Shell PowerShell command.

Connect-SPOService -Url https://<tenant>-admin.sharepoint.com

Set-SPOTenant -IsSharePointAddInsDisabled $true

After SharePoint Add-Ins are disabled, users will not be able to add SharePoint Add-Ins to their sites, and admins cannot add new SharePoint Add-Ins to the tenant and site collection app catalogs. SharePoint Add-Ins already added to sites will stay available and can still be used by the site’s users.

SharePoint Add-Ins will not be available from the public marketplace

After July 1, 2024, users browsing the public marketplace (AppSource) will see SharePoint Add-In, but if they select Get it now, a message will explain that SharePoint Add-Ins are retired and cannot be added. If you still require a specific SharePoint Add-In, contact the Add-In creator to understand the possible options.

Resources

Microsoft 365 admin center: Manage ownerless Microsoft 365 groups and teams

There is a new feature published at Microsoft roadmap site:

Microsoft 365 admin center: Manage ownerless Microsoft 365 groups and teams

Teams, Outlook groups, Team Sites etc. powered by Microsoft 365 Groups supports two roles: members and owners. Members can collaborate with others in the group through files, emails, messages etc. Owners manage the group membership and monitor content and conversations. When employees leave an organization or switch projects internally, it results in their existing user accounts getting deleted. If such employees were group owners, keeping track of their groups becomes critical to ensure accountability within the organization. We have introduced a new ownership governance policy to help automate the management of ownerless groups by requesting active members to become owners of the group. Admins can define who is eligible for these notifications and configure what notifications and how often these notifications are sent to active group members. Users, who are members of the ownerless groups can simply accept or decline request via the actionable email message.

  • Feature ID: 180749
  • Added to roadmap: 10/10/2023
  • Last modified: 10/10/2023
  • Product(s): Microsoft 365 Admin Center
  • Cloud instance(s): GCC
  • Platform(s): Web
  • Release phase(s): General Availability


But based on the feature description – all looks exactly as what we already have for years as “Microsoft 365 ownerless groups policy” which you can configure under Microsoft 365 Admin Center -> Settings -> Org settings -> Microsoft 365 groups

More on Microsoft 365 ownerless groups

Massive Microsoft 365 groups update with PowerShell

What if you need to bulk update Microsoft 365 groups membership e.g. to add a group owner or member for tens of thousands m365 groups? Iterating through groups one-by-one is unproductive and could take days. Can we do it faster? Here is what I found.

In my case, it was Microsoft 365 ownerless groups policy implementation for large tenant… Skipping details – I needed to update ownership for 10,000 Microsoft 365 groups and I was looking for a best/fastest possible option maybe some kind of bulk update or with multiple threads. And I figured out that the fastest way is to use PnP.PowerShell that calls Microsoft Graph API but run it against list of groups with PowerShell parallel trick. Here is the sample PowerShell code:

$groups | ForEach-Object -Parallel {
    $owner = "newGroupOwnerUPN@contoso.com"
    Add-PnPMicrosoft365GroupOwner -Identity $_.Id -Users $owner
} -ThrottleLimit 50

That worked for me perfectly and it took ~8 seconds per 1,000 groups.