Using Microsoft Graph API is a preferred and recommended way to connect to SharePoint Online and other Microsoft 365 resources programmatically from Python code. But if by some reason you are required to use classic SharePoint REST API, here is how it is done.
Authentication blade must be configured for authenticate as current user
Certificates and/or secrets must be configured for daemon app (unattended access)
API permissions your Azure app registration must have API permissions configured based on resources you need access to and authentication method chosen here is how to configure API permissions for your app
Office365-REST-Python-Client library installed
Using client Id and client secret to access SharePoint REST API from Python
Errors
Possible errors and diagnostic messages
HTTPError: 401 Client Error: Unauthorized for url… The provided client secret keys for app … are expired. Visit the Azure portal to create new keys for your app or consider using certificate credentials for added security
Microsoft Graph API allows you to work with all the Microsoft 365 content – including search through Exchange e-mail messages, Yammer (Viva Engage) and Teams chat messages and surely OneDrive and SharePoint content (please refer to the original doc). Let me focus on searching in SharePoint Online and OD here but you can use the same technique to search through other Microsoft 365 services. I will use PowerShell but same ideas should work for other platforms/languages – Python, C#, node.js etc.
Assuming we have a registered Azure app configured correctly, including Secrets/Certificates blade and API permissions provided – we should be ready to authenticate and call Graph API unattended – on behalf of application itself.
Let us authenticate as a service/daemon app with client id and client secret:
# Authenticate to M365 as an unattended application
# specify your app id. app secret, tenant id:
$clientID = ""
$clientSc = ""
$TenantId = ""
# Construct URI and body needed for authentication
$uri = "https://login.microsoftonline.com/$tenantId/oauth2/v2.0/token"
$body = @{
client_id = $clientID
client_secret = $clientSc
scope = "https://graph.microsoft.com/.default"
grant_type = "client_credentials"
}
# Get OAuth 2.0 Token
$tokenRequest = Invoke-WebRequest -Method Post -Uri $uri -ContentType "application/x-www-form-urlencoded" -Body $body -UseBasicParsing
$token = ($tokenRequest.Content | ConvertFrom-Json).access_token
$headers = @{Authorization = "Bearer $token" }
Below is how I search Microsoft 365 content programmatically from PowerShell using MS Graph API being authenticates as user.
Notice we use “region” – it is required to search with Graph API under application credentials. Otherwise you will get an error message “SearchRequest Invalid (Region is required when request with application permission.)”:
Parameter “fields” allows you to request only fields you need to be returned. As returning object will be smaller your request will perform faster.
There might be a big number of objects found in m365 upon your request. Graph will not always return to you all the results. AFAIK currently the limit is 500, so if there are more than 500 objects found – only first 500 will be returned. You can specify how many objects you need to be returned per call with “size” parameter.
You can check value of $res.value[0].hitsContainers[0].moreResultsAvailable property and if it’s True – that means there are more results. The value above and parameters “from” and “size” would allow you to organize a loop so you can call search API many times to return all results.
Microsoft Graph API allows you to work with all the Microsoft 365 content – including search through Exchange e-mail messages, Yammer (Viva Engage) and Teams chat messages and surely OneDrive and SharePoint content (please refer to the MS’s original doc). After we got a registered Azure app configured correctly, including Authentication and API permissions provided (more on this) – we should be ready to authenticate and call Graph API on behalf of a user.
Let me focus on searching in SharePoint Online and OD here but you can use the same technique to search through other Microsoft 365 services. I will use PowerShell but same ideas should work for other platforms/languages – Python, C#, node.js etc.
Let us authenticate first. We’d need a MSAL.PS module for that.
# Ensure we have MSAL.PS module installed
Get-Module MSAL.PS -ListAvailable | ft name, Version, Path
# Install-Module MSAL.PS -Force -Scope CurrentUser -AcceptLicense
Import-Module MSAL.PS
# Authenticate to Microsoft Interactively
$clientid = 'd82858e0-ed99-424f-a00f-cef64125e49c' # your client id
$TenantId = '7ddc7314-9f01-45d5-b012-71665bb1c544' # your tenant id
$token = Get-MsalToken -TenantId $TenantId -ClientId $clientid -Interactive
$headers = @{Authorization = "Bearer $($token.AccessToken)" }
Below is how I search Microsoft 365 content programmatically from PowerShell using MS Graph API being authenticates as user:
# Search
# MS Graph Search API url (beta or v1.0):
$apiUrl = "https://graph.microsoft.com/beta/search/query"
# specify where to search - entity types
$entityTypes = "['driveItem','listItem','list','drive','site']"
$entityTypes = "['driveItem','listItem']"
# query
$query = "test*"
# build a simple request body
$body = @"
{
"requests": [
{
"entityTypes": $entityTypes,
"query": {
"queryString": "$query"
}
}
]
}
"@
# call Graph API:
$res = Invoke-RestMethod -Headers $Headers -Uri $apiUrl -Body $Body -Method Post -ContentType 'application/json'
# explore returned object
$res.value[0].searchTerms
$res.value[0].hitsContainers[0].hits
$res.value[0].hitsContainers[0].hits.Count
$res.value[0].hitsContainers[0].moreResultsAvailable
I used “beta” search API to research or make demos, but in production code youd stick with “v1.0”.
You can scope search down using entity types – ‘driveItem’,’listItem’,’list’,’drive’,’site’. “driveitem” here represents document library.
In query you can use KQL.
Always check if more results available with “$res.value[0].hitsContainers[0].moreResultsAvailable”. If there are more results and you need them – consider looping using paging technique.
Being authenticated to Microsoft 365 tenant means Microsoft 365 knows who is trying to get access. To actually be able read/write or manage resource, your app must be Authorized to this resource.
For details – pls refer to MS authorization and Microsoft Graph API permissions. But again, in short in our case that means we need to have an API permission configured for our azure registered app. There are two kinds of API permissions – delegated and application.
Delegated permissions are intended to allow currently authenticated user to have access to the resource. Effective user permissions in this app would be an intersection of user own permissions and app permissions. So if an app have “Sites.FullControl.All” SharePoint delegated API permissions – that does not mean that user will have full control over all sites.
Here is an example of delegated permissions:
Permissions above allow you to search through SharePoint content being authenticated with your personal credentials. In search results you will see only content you already have access to.
Application permissions are what it says – once permissions are configured – application will have access to the resources according to API permissions.
Generally, application permissions allow an app to have access to all resources of the same kind in tenant, e.g. to get one specific groups owners an app must have “GroupMember.Read.All” permission that allows an app to read all tenant groups and their members. There are some exceptions – e.g. for Teams Microsoft developed RSC that allows scoped app access. For SharePoint there is a similar option – “Sites.Selected” API permissions.
API permissions must have an Admin consent. Here is an example of application permissions:
Permissions above allow your app to search all SharePoint content.
Before we can make search call to Microsoft 365 in our code – we need to be authenticated.
There are many kinds of authentication flows Microsoft supports. For more details – please refer to Microsoft Identity Platform documentation but in short – there are two kind of authentications – as current user and as daemon application and in both cases we need an application registration in Azure.
Briefly here is what you need to have for your custom application to authenticate to Microsoft 365:
configure Secrets or Certificates (for daemon apps)
Register Application in Microsoft Azure
1. You’d go to https://portal.azure.com/ 2. Type app registration in search and select App registration
3. Select “New registration”
4. For now, you’d just need to provide your application display name and leave other fields as default – Single tenant and no redirection Url, then click “Register”
5. If you are seeing an error message telling that your are not allowed to register an app – you’d reach your AAD/EntraId/Global admins so they can register an app for you.
6. Under Azure Portal App Registrations you should be able to see your app:
7. Under Overview blade notice you Client Id and tenant Id:
Configure Authentication blade for interactive apps
Azure apps for interactive authentication are configured differently (check MS app types and authentication flows) for different scenarios/platforms.
Here is a good article from MS. Microsoft Graph quick start can register an app for you. Microsoft publish tutorials for .net, Go, Java, JavaScript, PHP, Python, TypeScript…
In a few words, you’d need to add Platform as below
manually if you know how to configure it. Otherwise, Quickstart and/or Integration assistant might help you.
You do not need Certificates or Secrets for authentication as current user.
Configure Secrets or Certificates for daemon apps
Daemon apps (aka service apps, also called background jobs) – all kind unattended access scenarios do not require configuration under Authentication blade – but require Secrets or Certificate. You’d need to be familiar with certificates, as certificates considered as more secure way to authenticate and some authentication flows allow secrets, but some require certificates.
So for daemon apps you need a secret and/or certificate:
Here is how I configure daemon app for PowerShell.
Having an app configured as above – you should be able to authenticate against Microsoft 365 Graph API, but should not have access to resources, as app is not authorized yet.
Why do we need to implement search in our applications?
Use-cases for search on behalf of current user
Along with the usual ones – where you just need your app to search for some data and bring it to user – there is one different scenario I’d like to share:
You need to quickly detect content in SharePoint that is open for everyone
Brute force solution – getting detailed permissions report for all SharePoint sites might not be a feasible option, especially in large environments – it is a very resource-consuming task and might take days and weeks. So consider the following…
Since search is security-trimmed – a user can get only search results he/she already has access to; but what if we create an account and do not grant any SharePoint permissions or group memberships to this account, and then we’d search for everything on behalf of this account? That would mean that all what search returns represent content that is shared with everyone. There are some tricks and gotchas – here is the separate article on the same.
Use-cases for unattended search
What are the use-cases when you need to search in your daemon app or background job? Be aware that when you search on behalf of application credentials – search is NOT security-trimmed and your query would run against ALL SharePoint content… Here are some possible scenarios.
Content detection/Investigation
Let say you want some data is never shared with anyone and never appeared in search for anyone
Or you might want to investigate what is the location some specific data is available at
Imagine you are building sites classification system and you use indexed custom site properties – so you are able to refine search results based on site metadata to get list of specific sites (adaptive scopes used in retention policy are based on the same mechanics)
Automation – let say you have a requirement to configure every tenant site in some ways – for instance – add some hosts to allowed domains to embed video or set some site properties based on who created the site or activate or deactivate some features and so on – how would you do that? You’d probably have a scheduled job that runs let say every hour against only new sites – sites created during that last hour. How would you get these recently created sites? Search with Graph API is the only nice solution today.
In the articles below I’m sharing my techniques on searching in Microsoft 365 SharePoint and Teams from application using Microsoft Graph API. Specifically I’m covering
Microsoft Graph search API
Microsoft.Graph PowerShell module
PnP.PowerShell module
In two flavors:
Search on behalf of currently authenticated user
Unattended Search with daemon (also called service) applications
Below is how do I search Microsoft 365 content programmatically from PowerShell using MS Graph API, PowerShell PnP, Microsoft Graph module, MSAL library being authenticated as user or daemon application. Let me focus on SharePoint content here but you can use the same technique to search through other Microsoft 365 services. Also, I’ll be using PowerShell but same ideas should work for other platforms/languages – Python, C#, node.js etc.
To search on behalf of currently authenticated user we need delegated “Sites.Read.All” API permissions. I recommend you to add both Graph API and SharePoint API permissions as different libraries might use different API’s under the hood. Ensure you add delegated “Sites.Read.All” even if you already have “Sites.FullControl.All” as by some reason “Sites.FullControl.All” does not always work for search.
Here is how an app API permissions to search as current user should look like:
For unattended search – e.g. search on behalf of daemon app – we need application “Sites.Read.All” API permissions. Again, I suggest both Graph API and SharePoint API permissions added. Here is how an app API permissions to search as daemon app should look like:
Ensure you got admin consent for API permissions.
In case you have incorrect permissions in your app – Microsoft Graph will be kind enough to inform you exactly what you need. Example:
“Access to ChatMessage in Graph API requires the following permissions: Chat.Read or Chat.ReadWrite, ChannelMessage.Read.All. However, the application only has the following permissions granted: Sites.Read.All, User.Read”
Assuming we have configured apps – let us get started with
Microsoft Graph API
Microsoft Graph API allows search through all the Microsoft 365 content – including Exchange e-mail messages, Yammer (Viva Engage) and Teams chat messages and surely OneDrive and SharePoint content (please refer to the original doc).
Authenticate as current user to Search with Graph API
I use MSAL.PS PowerShell module to get token, then I build a headers variable
If you are getting error message “SearchRequest Invalid (Region is required when request with application permission.)”:
that’s OK, just modify your body to include region like this (“region”: “NAM” for North America or “GBR” or …). Also, I can modify body with from/size for paging (technique used to iterate through search results if there are many) and return just specific fields to decrease traffic and improve performance:
For daemon app authentication we need a certificate configured in Azure App and installed on the user machine. Daemon app authentication code sample (please specify your tenant id, app (client) id and certificate thumbprint: