Bert Jansen (Microsoft) revealed some details on throttling when you access Microsoft 365 programmatically – via Microsoft Graph or CSOM and guided developers on how to regulate request traffic for optimized throughput using RateLimit headers (Here).
Demystifying SharePoint throttling
Throttling is necessary to ensure that no single user or application consumes too many resources compromising the stability of the entire system, which is used by many clients.
Throttling happens at
User (there are user request limits. Microsoft counts all requests linked to user
Application (Delegated or Application permissions)
Resource units per app per minute
Resource units per app per day
Farm – Spike protection
Very common reason for throttling – when an Application (Delegated or Application permissions) reaches “Resource units per app per minute” threshold.
Usually you catch HTTP errors 429 or 503, wait for some time (respect Retry-after header) and try again.
SharePoint provides various APIs. Different APIs have different costs depending on the complexity of the API, but Microsoft favor Graph API over SharePoint REST/CSOM. The cost of APIs is normalized by SharePoint and expressed by resource units. Application’s limits are also defined using resource units.
Quota depends on tenant size.
Resource unit limits for an application in a tenant (please refer to the Microsoft article)
Predefined costs for Microsoft Graph calls:
Assuming 2 resource units per request is a safe bet.
If I get token with (Graph, MSAL, PnP) and use this token for (Graph API, SharePoint CSOM API, SharePoint REST API) matrix.
An App used in this tests has Sites.FullControl.All MS Graph API and SharePoint API permissions, as well as FullControl ACS based permissions to SharePoint (AppInv.aspx).
How to provide permissions for an Azure registered application with MS Graph SharePoint Sites.Selected API permissions to a specific site via calling Microsoft Graph API from PowerShell.
We need an “admin” application – Azure registered application with with Sites.FullControl.All MS Graph API permissions. This method can use secret, so we need Client Id and Client Secret for this “admin” app.
We also need a Client Id and Application Display Name for an Azure application with Sites.Selected MS Graph and/or SharePoint API permissions provided.
And we need our “target” site Url.
With PowerShell scripts you can:
Get Microsoft Graph Access Token with an “admin” app
Get client (target) site Id
Get current app permissions provided to client site
Add read or write permissions for the client app to the client site
Sites.Selected MS Graph API permissions were introduced by Microsoft in March 2021. It was a good move towards site-level access for non-interactive (daemon) applications, but still developers were limited with only what MS Graph API provides for SharePoint. SharePoint CSOM and REST API still provides much more than MS Graph API.
So developers had to use AppInv.aspx at site level to provide ACS-based permissions to their apps to be able to use SharePoint CSOM and REST APIs. The bad news is ACS-based permissions have some downsides so some SharePoint/m365/security engineers consider them legacy and deprecated. But if we decide to disable SharePoint App-only service principals – all apps with ACS-based permissions provided via AppInv.aspx will stop working.
2021: Microsoft Graph Sites.Selected API
Recently Microsoft introduced Sites.Selected SharePoint API permissions for registered Azure Apps! So from now developers should be fully happy with API permissions provided in Azure (without SharePoint ACS-based permissions).
2022: SharePoint Sites.Selected API
Why is this so important? Because this should allow us to be able to switch from ACS based permissions provided in SharePoint via AppInv.aspx to Azure-provided permissions and as a consequence – disable SharePoint-Apps only principal (‘set-spotenant -DisableCustomAppAuthentication $true’).
Why we are eager to disable Custom App Authentication in SharePoint? Simply say, SharePoint App-only service principals are not trackable (they all appeared as a “app@sharepoint.com” id in all logs) and hard to manage (there is no way to get list of existing/registered SP app-only service principals, sites and their owners) – see more in this article. (Update: there are tools – check Azure ACS retirement: Track down ACS apps).
So, SharePoint Sites.Selected application API permissions provided in Azure is a significant step to make Microsoft 365 SharePoint environment more secure and manageable.
2024: Delegated Sites.Selected Permissions
Since Feb 2024 Microsoft supports also delegated Sites.Selected permissions. Delegated Sites.Selected permissions are assigned the same way as application Sites.selected permissions – through the /permissions endpoint. You still assign only the application id and role. When the call is made the permissions are calculated either as application or delegated, and assuming the request is authorized it will go through.
Microsoft implemented granular permissions ( e.g. to a list, item or file) alongside with Sites.Selected permissions. Original implementations of Sites.Selected allowed access to entire site collection only. With new ‘Lists.SelectedOperations.Selected’, ‘ListItems.SelectedOperations.Selected’ and ‘Files.SelectedOperations.Selected’ permissions it is possible to provide application permissions to list, library or list item or particular document (reference).
Details To Be Provided…
Delegated Sites.Selected API permissions
Since the beginning of 2024 Microsoft supports Delegated Sites.Selected API permissions. This is to support security best practices – the minimally possible access should be provided. I.e. the idea is: even Sites.FullControl.All delegated permissions allows access not to all sites, but to sites current user was provided access to, it would be a good idea to restrict access with only sites this specific app is required access to. That’s good.
Update: Sites.Selected API MS Graph permissions was introduced by Microsoft in 2021. It was a good move towards site-level development, but still developers were limited with only what MS Graph API provides for SharePoint dev. So devs had to use AppInv.aspx at site level to provide ACS permissions to their apps to be able to use SharePoint CSOM and REST APIs. Recently Microsoft introduced Sites.Selected SharePoint API permissions for registered Azure Apps! So now devs should be fully happy without ACS-based permissions.
Scenario
You have an application that needs access to Microsoft 365 SharePoint Online site/list/documents. Application is running without interaction with users – e.g. unattended, as daemon job.
There are two options you can authenticate to Microsoft 365 – with the secret or with the certificate. Authenticating with certificate is considered more secure.
Questions
What happens if SharePoint-Apps only principal is disabled (i.e. ‘set-spotenant -DisableCustomAppAuthentication $true’ )?
Why I’m getting 401 error when authenticating to SPO?
Why I’m getting 403 error when authenticating to SPO with secret?
What permissions to I need to work with SPO?
Findings
Note: we will use PowerShell 7.2 and PnP.PowerShell 1.9 to illustrate it.
Disabled SharePoint-Apps only principal
If SharePoint-Apps only principal is disabled in your tenant (i.e. ‘Get-PnPTenant | select DisableCustomAppAuthentication’ returns $true ), then the only way you work with SPO from code is:
an App registered in Azure
API permissions provided via Azure (MS Graph, SharePoint)
Certificate is used
In all other cases (even your Connect-PnPOnline command complete successfully) – you will be getting error 401 (unauthorized) when trying Get-PnPTenant or Get-PnPTenantSite or Get-PnPSite
Enabled SharePoint-Apps only principal
If SharePoint-Apps only principals are enabled in your tenant (i.e. ‘Get-PnPTenant | select DisableCustomAppAuthentication’ returns $false ), then you have three options to work with SPO from code:
Azure App with a secret (Client Id + Client Secret) and permissions to SharePoint provided via SharePoint (AppInv.aspx) to access SharePoint REST API
Azure App with a certificate (Client Id + Certificate) and permissions provided via Azure to access SharePoint REST API
Azure App with a certificate or secret (Client Id + Secret or Certificate) and permissions provided via Azure to access SharePoint via Microsoft Graph API
There are scenarios when you need to pull only newly created SharePoint sites, e.g. get sites created since yesterday or get last 100 created sites. Usually other articles and existing PowerShell scripts solve this by pulling all sites from tenant and then iterating through sites to get only new sites. That approach is not nice and simply does not work in large environments. How can we get only sites created recently, not all sites? Here is how I use Microsoft Graph API to get only new sites.
Update (6/28/2024): Microsoft announced updates to it’s delta API for SharePoint, so I added option 3 – see below.
Scenario
Let say you administer Teams, OneDrive and SharePoint Online in a Microsoft 365 tenant. You have a pretty big environment – ~10k or more sites and you want to quickly find just new SharePoint sites or teams (e.g. sites created recently – during last hour/day/week/month). This might be required for ad-hoc reports and for automation scenarios – like applying required configurations or assign some property value to all newly created sites.
With GUI it’s done easily: SharePoint Admin Center -> Active Sites -> sort based on “Date Created” – done.
With PowerShell – not so simple. “Get-PnPTenantSite” cmdlet returns a site object but the object does not have “Created” field. It’s a web property (not site property). But to get a web object – you have to connect separately to each site and get root web object to check when the web was created. For small environments it is possible, for large environments it can take days… And still not nice. “Get-PnPTenantSite” with “-Filter” option would help, but “…Currently, you can filter by these properties: Owner, Template, LockState, Url.”
but… 1) it’ll give you group-based sites only 2) it is not easy to automate 3) this might take long for large environments. I know much better solution:
Solution
Microsoft Graph API helps. It returns result in seconds and you can sort or filter results based on created date . Below are two methods: Option 1 is based on Search and filtering and Option 2 is based on Sites Search and sorting. So there are some pros and cons for each method.
Microsoft Graph Search API allows KQL in queries. So we can form a query with something like “created>=1/1/2021” and use entity type = ‘[“site”]’. Search should return only sites created after Jan 01, 2021.
This option is also based on Microsoft Graph API, but sites entry point, which allows search too and sort results by property “createdDateTime”. So we will just search for everything and select how many results we need based on createdDateTime property.
You can use “Get delta” under SharePoint Graph API – check for details here. It says “Get newly created, updated, or deleted sites without having to perform a full read of the entire sites collection”. I’ll do my own testing, but for now check this: Video: Microsoft Graph Delta Capabilities in SharePoint API
UPDATE: this approach only works if the mobile number is stored in SPO UPA. If the mobile number is stored in AAD only, the admin will need to remove the mobile number from AAD for the affected users.
Scenario: you administer SharePoint Online Microsoft 365 tenant business asks you to remove mobile phone numbers from SharePoint User Profiles:
Solution:
As a SharePoint administrator, you can do it: 1. Start Microsoft 365 SharePoint Admin Center 2. Navigate to More Features -> User Profiles -> Manage User Properties
3. Under “Contact Information” -> Mobile phone -> Edit
4. Uncheck “Replicable”, Save, wait a minute or two 5. – Select “Default Privacy Settings”: “Only Me” – Uncheck “User can override” – Uncheck “Allow users to edit values for this property” – Uncheck “Show in the profile properties section of the user’s profile page” – Uncheck “Indexed”
6. Save
Note: Microsoft implemented new MS-Graph API that adds or deletes profile card custom properties using the profile card API in Microsoft Graph