Before we can make search call to Microsoft 365 in our code – we need to be authenticated.
There are many kinds of authentication flows Microsoft supports. For more details – please refer to Microsoft Identity Platform documentation but in short – there are two kind of authentications – as current user and as daemon application and in both cases we need an application registration in Azure.
Briefly here is what you need to have for your custom application to authenticate to Microsoft 365:
configure Secrets or Certificates (for daemon apps)
Register Application in Microsoft Azure
1. You’d go to https://portal.azure.com/ 2. Type app registration in search and select App registration
3. Select “New registration”
4. For now, you’d just need to provide your application display name and leave other fields as default – Single tenant and no redirection Url, then click “Register”
5. If you are seeing an error message telling that your are not allowed to register an app – you’d reach your AAD/EntraId/Global admins so they can register an app for you.
6. Under Azure Portal App Registrations you should be able to see your app:
7. Under Overview blade notice you Client Id and tenant Id:
Configure Authentication blade for interactive apps
Azure apps for interactive authentication are configured differently (check MS app types and authentication flows) for different scenarios/platforms.
Here is a good article from MS. Microsoft Graph quick start can register an app for you. Microsoft publish tutorials for .net, Go, Java, JavaScript, PHP, Python, TypeScript…
In a few words, you’d need to add Platform as below
manually if you know how to configure it. Otherwise, Quickstart and/or Integration assistant might help you.
You do not need Certificates or Secrets for authentication as current user.
Configure Secrets or Certificates for daemon apps
Daemon apps (aka service apps, also called background jobs) – all kind unattended access scenarios do not require configuration under Authentication blade – but require Secrets or Certificate. You’d need to be familiar with certificates, as certificates considered as more secure way to authenticate and some authentication flows allow secrets, but some require certificates.
So for daemon apps you need a secret and/or certificate:
Here is how I configure daemon app for PowerShell.
Having an app configured as above – you should be able to authenticate against Microsoft 365 Graph API, but should not have access to resources, as app is not authorized yet.
Why do we need to implement search in our applications?
Use-cases for search on behalf of current user
Along with the usual ones – where you just need your app to search for some data and bring it to user – there is one different scenario I’d like to share:
You need to quickly detect content in SharePoint that is open for everyone
Brute force solution – getting detailed permissions report for all SharePoint sites might not be a feasible option, especially in large environments – it is a very resource-consuming task and might take days and weeks. So consider the following…
Since search is security-trimmed – a user can get only search results he/she already has access to; but what if we create an account and do not grant any SharePoint permissions or group memberships to this account, and then we’d search for everything on behalf of this account? That would mean that all what search returns represent content that is shared with everyone. There are some tricks and gotchas – here is the separate article on the same.
Use-cases for unattended search
What are the use-cases when you need to search in your daemon app or background job? Be aware that when you search on behalf of application credentials – search is NOT security-trimmed and your query would run against ALL SharePoint content… Here are some possible scenarios.
Content detection/Investigation
Let say you want some data is never shared with anyone and never appeared in search for anyone
Or you might want to investigate what is the location some specific data is available at
Imagine you are building sites classification system and you use indexed custom site properties – so you are able to refine search results based on site metadata to get list of specific sites (adaptive scopes used in retention policy are based on the same mechanics)
Automation – let say you have a requirement to configure every tenant site in some ways – for instance – add some hosts to allowed domains to embed video or set some site properties based on who created the site or activate or deactivate some features and so on – how would you do that? You’d probably have a scheduled job that runs let say every hour against only new sites – sites created during that last hour. How would you get these recently created sites? Search with Graph API is the only nice solution today.
In the articles below I’m sharing my techniques on searching in Microsoft 365 SharePoint and Teams from application using Microsoft Graph API. Specifically I’m covering
Microsoft Graph search API
Microsoft.Graph PowerShell module
PnP.PowerShell module
In two flavors:
Search on behalf of currently authenticated user
Unattended Search with daemon (also called service) applications
Below is how do I search Microsoft 365 content programmatically from PowerShell using MS Graph API, PowerShell PnP, Microsoft Graph module, MSAL library being authenticated as user or daemon application. Let me focus on SharePoint content here but you can use the same technique to search through other Microsoft 365 services. Also, I’ll be using PowerShell but same ideas should work for other platforms/languages – Python, C#, node.js etc.
To search on behalf of currently authenticated user we need delegated “Sites.Read.All” API permissions. I recommend you to add both Graph API and SharePoint API permissions as different libraries might use different API’s under the hood. Ensure you add delegated “Sites.Read.All” even if you already have “Sites.FullControl.All” as by some reason “Sites.FullControl.All” does not always work for search.
Here is how an app API permissions to search as current user should look like:
For unattended search – e.g. search on behalf of daemon app – we need application “Sites.Read.All” API permissions. Again, I suggest both Graph API and SharePoint API permissions added. Here is how an app API permissions to search as daemon app should look like:
Ensure you got admin consent for API permissions.
In case you have incorrect permissions in your app – Microsoft Graph will be kind enough to inform you exactly what you need. Example:
“Access to ChatMessage in Graph API requires the following permissions: Chat.Read or Chat.ReadWrite, ChannelMessage.Read.All. However, the application only has the following permissions granted: Sites.Read.All, User.Read”
Assuming we have configured apps – let us get started with
Microsoft Graph API
Microsoft Graph API allows search through all the Microsoft 365 content – including Exchange e-mail messages, Yammer (Viva Engage) and Teams chat messages and surely OneDrive and SharePoint content (please refer to the original doc).
Authenticate as current user to Search with Graph API
I use MSAL.PS PowerShell module to get token, then I build a headers variable
If you are getting error message “SearchRequest Invalid (Region is required when request with application permission.)”:
that’s OK, just modify your body to include region like this (“region”: “NAM” for North America or “GBR” or …). Also, I can modify body with from/size for paging (technique used to iterate through search results if there are many) and return just specific fields to decrease traffic and improve performance:
For daemon app authentication we need a certificate configured in Azure App and installed on the user machine. Daemon app authentication code sample (please specify your tenant id, app (client) id and certificate thumbprint:
Microsoft recently (Oct 2023) announced Microsoft Graph command-line interface (CLI) tool – mgc. Microsoft: “The Microsoft Graph PowerShell command-line interface (CLI) acts as an API wrapper for the Microsoft Graph APIs, exposing the entire API set for use from the command line”. Example:
mgc users list --filter "displayName eq 'John Smith'"
Meantime there is a Microsoft Graph PowerShell SDK (PowerShell module Microsoft.Graph ) since 2020. Example:
Get-MgUser -Filter "displayName eq 'John Smith'"
So, what is the difference? Why Microsoft provides two similar tools? What are the use case scenarios, functionality and scope of each one?
You administer Microsoft 365 SharePoint Online. Part of your daily activities is providing Microsoft Graph and SharePoint Sites.Selected API permissions to other users (developers).
In Aug/Sep 2023 Microsoft pushed an update that prevents site collection admins to create or update an Azure Access Control (ACS) principal (that was the way most of developers used to get Client Id and Client secret to access SharePoint site). So your users are probably getting something like Your SharePoint tenant admin doesn’t allow site collection admins to create or update an Azure Access Control (ACS) principal message attempting to create or update SharePoint App-only principal at AppRegNew.aspx or AppInv.aspx pages. Here are more details on the issue.
Microsoft and MVPs shared some technique how to provide Sites.Selected API permissions, but dealing with scripts manually, elevating individual permissions every time you need to run the script – it all takes time and not very efficient. More and more devs are reaching you on the app. So you want to automate this process.
SharePoint list as a frontend here you can accept intake requests, organize approval workflow and display automation results
Azure Function App as a backend here will be your PowerShell script hosted that runs on scheduled basis and takes care of actual permissions provisioning
Solution details
High-level, getting application permissions to some specific SharePoint site is a two-step process:
get application registration in Azure and properly configure it
get permissions for this application to a specific SharePoint site
For the first step – check this and this articles. I’ll focus on the second step below.
You can provide Sites.Selected permissions for the app to a site with
I will be using second one one. Also PnP.PowerShell will be used to get access to SharePoint intake site and read/update requests from SharePoint list and so on.
Azure App Registration
I registered an admin Application in Azure – “SharePoint Automation App”, added Graph Sites.FullControl.All and SharePoint Sites.FullControl.All permissions, then added Microsoft Graph Directory.Read.All permissions and got tenant admin consent:
I generated a self-signed certificate and added it to the app:
This app will be used to call provide permissions, and to connect to the SharePoint front-end.
Users will register their applications in Azure, add Graph Sites.Selected and SharePoint Sites.Selected permissions, got tenant admin consent, then request permissions to the specific site by creating an intake request – new list item.
Front-End SharePoint Site
I created a SharePoint site for automation. This site will play a front-end role for users. I created a list “Sites.Selected” and updated list columns so I have the following fields:
Target Site Url
Application Id
Permissions (read/write)
Automation Output
In real-world (Prod) – You can (should) also implement approval workflow as you’d provide permissions for the application to the site only with this site owner approval. The PowerShell code behind should also validate site owner’s consent with app access to site. But for the sake of simplicity I’ll skip this in my demo.
Azure Function App
I created an Azure Function App with the following parameters: – Runtime stack: PowerShell Core – Version: 7.2. – OS: Windows – Hosting plan: Consumption
And then PowerShell timer-triggered function in Visual Studio Code.
Function requirements.psd1 (it takes a few hours for Azure to install modules; while modules are installing – you might see “[Warning] The first managed dependency download is in progress, function execution will continue when it’s done. Depending on the content of requirements.psd1, this can take a few minutes. Subsequent function executions will not block and updates will be performed in the background.”):
@{
'Az' = '10.*'
'PnP.PowerShell' = '2.*'
}
Azure Az module to access other Azure resources. PnP.PowerShell module will be used to access SharePoint.
I will keep my admin Azure registered app in a key vault, so need somehow to let the key vault know that this specific app can access this specific credentials. So I enabled system assigned managed Identity for the Function App:
MS: “This resource is registered with Azure Active Directory. The managed identity can be configured to allow access to other resources…”. I’m going to use an object (principal) Id of this function to grant access to keyvault.
Azure key vault
Surely we do not hard-code app secrets. So we need a key vault o store app credentials.
I created a key vault under the same resource group in Azure and named it “SharePointAutomationDemo”. Then I added a roles assignment – “Key Vault Secret User” and “Key vault Reader” to the Function App via it’s managed identity:
I also assigned “Key Vault Administrator” role to the user (developer) who will add certificates/secrets to this key vault and develop Azure function code.
With the new ‘Lists.SelectedOperations.Selected’, ‘ListItems.SelectedOperations.Selected’ and ‘Files.SelectedOperations.Selected’ permissions it is new possible to provide application permissions at a specific list, library or list item levels or at a particular document level, so automation solution would be a little more complicated.
getOffice365GroupsActivityDetail – details about Microsoft 365 groups and activity
getSharePointSiteUsageDetail – details about SharePoint sites and usage
getTeamsTeamActivityDetail – details about Microsoft Teams and activity by teams
Also we know, that Teams sites are group-based, and you can have private and shared channels under Teams – but these sites are not actually group-based and there are group-based SharePoint sites with no Teams behind.
And activities might be different – update document or just visit home page, provide permissions and update channel properties etc.
So the question is what kind of activity at what level is recorded at which report?
As SharePoint or Teams admin you manage Microsoft 365 groups (create, update, delete, manage membership etc.) having your admin role activated. I prefer PowerShell 7 and Microsoft.Graph PowerShell module, and I need an Azure registered app with “Group.ReadWrite.All” Microsoft Graph API delegated permission.
Some findings:
If a user was not a group member or group owner – and the user is added to the group members – this user will get notification “You’ve joined the <Group Name> group” via e-mail that comes from a group e-mail address.
When a user is added to the group owners (or elevated to group owner if user was a group member) – user does not get notification.
When a user was a group owner and now you are adding this user to the group members – user does not get notification.
All the actions are logged into Microsoft 365 audit log under your personal Id.
Script samples:
# This script is just a sample to demonstrate basic technique on getting, updating groups membership and deletion m365 groups with PowerShell and MS Graph
#
# please do not run this script as is, but update it based on your needs
# authentication with personal Id
# app must have as minimum "Group.ReadWrite.All" Microsoft Graph API delegated permission
# user must have SharePoint admin (or Teams admin) roles activated
Connect-MgGraph -ClientId $clientid -TenantId $tenantId
Get-MgContext | Select-Object Scopes -ExpandProperty Scopes
# sample data
$groups = @()
$groups += [PSCustomObject]@{GroupId = '443d22ae-683a-4fe4-8875-7bd78227a026' }
$groups += [PSCustomObject]@{GroupId = 'e5805388-c18c-48c0-b42d-6223cf8f3d82' }
# Get Groups
foreach ($group in $groups) {
Get-MgGroup -GroupId $group.GroupId
}
# add members to the group
$groupId = '443d22ae-683a-4fe4-8875-7bd78227a026'
$userId = 'df74e0d3-d78c-495b-b47a-549437d93cf7' # Adele
New-MgGroupMember -GroupId $groupId -DirectoryObjectId $userId
# add Owner to the group
$groupId = '443d22ae-683a-4fe4-8875-7bd78227a026'
$userId = 'eacd52fb-5ae0-45ec-9d17-5ded9a0b9756' # Megan
New-MgGroupOwner -GroupId $groupId -DirectoryObjectId $userId
# Delete group
$groupId = '443d22ae-683a-4fe4-8875-7bd78227a026'
Remove-MgGroup -GroupId $groupId