Bert Jansen (Microsoft) revealed some details on throttling when you access Microsoft 365 programmatically – via Microsoft Graph or CSOM and guided developers on how to regulate request traffic for optimized throughput using RateLimit headers (Here).
Demystifying SharePoint throttling
Throttling is necessary to ensure that no single user or application consumes too many resources compromising the stability of the entire system, which is used by many clients.
Throttling happens at
User (there are user request limits. Microsoft counts all requests linked to user
Application (Delegated or Application permissions)
Resource units per app per minute
Resource units per app per day
Farm – Spike protection
Very common reason for throttling – when an Application (Delegated or Application permissions) reaches “Resource units per app per minute” threshold.
Usually you catch HTTP errors 429 or 503, wait for some time (respect Retry-after header) and try again.
SharePoint provides various APIs. Different APIs have different costs depending on the complexity of the API, but Microsoft favor Graph API over SharePoint REST/CSOM. The cost of APIs is normalized by SharePoint and expressed by resource units. Application’s limits are also defined using resource units.
Quota depends on tenant size.
Resource unit limits for an application in a tenant (please refer to the Microsoft article)
Predefined costs for Microsoft Graph calls:
Assuming 2 resource units per request is a safe bet.
Here I’m trying to figure out – how much PowerShell Parallel option is beneficial and how to avoid throttling…
Let us test, how long would it take to create a SharePoint site, if we use regular (sequential) loop or parallelism (I’m creation a sample set of 50 SharePoint Sites in a row):
Regular (Sequential) seconds per site
Parallel, 100 sites in batch seconds per site
Parallel, 500 sites in batch seconds per site
Regular (Sequential)
3.0
Parallel, ThrottleLimit = 2
1.60
0.91
Parallel, ThrottleLimit = 5
0.69
Parallel, ThrottleLimit = 10
0.2 – 0.3
Parallel, ThrottleLimit = 20
0.17
Interesting, but I did not get even one (throttling or any other) error during creation 500 sites.
Get sites details
Now let us test, how long it takes to get sites details with Get-PnPTenantSite (I use a sample set of 500 sites):
Test type
Regular (Sequential), seconds per site
Parallel sample = 100 sites, seconds per site
Parallel sample = 200 sites, seconds per site
Parallel sample = 500 sites, seconds per site
Regular (Sequential)
0.65
Parallel, ThrottleLimit = 2
0.40
0.33
0.31
Parallel, ThrottleLimit = 5
0.17
0.14
0.36 (errors)
Parallel, ThrottleLimit = 10
0.11 (errors)
0.11 (errors)
0.34 (errors)
Parallel, ThrottleLimit = 20
0.12 errors+
0.07 errors+
0.52 (errors)
(errors) means there were small number of errors during test… e.g.
What is the Microsoft Search KQL query field limits for a verticals? Is there limited number of characters or lines?
You know what is Microsoft 365 Search Vertical and what is KQL query in vertical configuration, right?
Under Microsoft 365 admin center Search and intelligence you can configure search verticals. There are some out-of-the-box verticals – like All, Files, Sites, People and you can configure custom one.
As a part of vertical configuration – you can specify KQL query – if you want e.g. limit search with some sites or content types etc.
The question is – how many sites I can specify in this query field? E.g. can I specify 1000 sites? 10k sites?
And the answer is: It does not matter, because the limit is not in number of characters or lines.
In my dev environment I was able to save 50,000 lines (~3M characters). But attempt to save 100K lines (6M symbols) has failed (due to timeout, I believe:
Again, as I said the problem is not here.
The problem is time required for search to apply query. I.e. when you ask search to bring you something – after it gets results from index and before display results to you it applies KQL query configured for the vertical. And this time is the bottleneck.
Here is what I got measuring search response time depending on query size:
Search
response time, seconds
KQL query # of lines
KQL query size, # of symbols
works
1
500
28,000
works
5
1000
59,000
works
9
2000
120,000
works
25
3000
180,000
works/fails
30
3500
208,000
fails
35
3600
214,000
fails
35
50,000
3,000,000
n/a
n/a
100,000 (can’t save KQL query
6,000,000 (can’t save KQL query)
Which means that after ~ 1000 lines (50,000 characters) KQL query size – query becomes too slow, and after ~3000 lines (180k chars) – can fail (due to timeout I’d say).
DepartmentId
If your sites are organized in hierarchy under Hub site – you can use DepartmentId managed property to significantly decrease number of lines in query, as you can cover all sites under the hub with
Microsoft recently implemented “Adaptive” retention policies. At step 2 of “Create retention policy” you’ll be asked “Choose the type of retention policy to create”: “A policy can be adaptive or static. Advantage of an adaptive policy will automatically update where it’s applied based on attributes or properties you’ll define. A static policy is applied to content in a fixed set of locations and must be manually updated if those locations change.”
And if you selected “Adaptive” – on the next step you will need to provide the adaptive scope (so at this moment you should already have created your adaptive scopes):
So, let us create your adaptive scopes. What type of scope do you want to create? SharePoint sites…
And then you’ll have nothing more then set of conditions:
where you can use objects: “Site Url”, “Site Name” and “Refinable String 0″..”Refinable String 99”. Conditions would be “is equal to”, “is not equal to”, “starts with” and “not starts with”. Or you can select “Advanced query builder” and enter KQL query.
Advanced query builder for SharePoint Adaptive Scope
I will be saving my personal gotchas on Microsoft 365 External Access and Guest Access in SharePoint and Teams
We configure external/guest access in AAD, m365 Admin Center, Teams Admin Center, SharePoint Admin Center, specific Group, Team or SharePoint site.
We can configure external guest access directly, or can configure sensitivity labels and policies in Purview (Compliance Admin Center). Configuring sensitivity labels for sites/groups we configure external guest access settings. Configuring sensitivity labels policies we apply labels.
External access via “All Users” group
Be careful with “All users” group created as part of the process. Microsoft: “The dedicated All Users group includes all users in the directory, including guests and external users.” And indeed, “All Users” group by default include external users.
So here is the scenario: we have a site where external sharing is enabled, and someone is sharing a specific file1 or folder1 with some external users. The other site/group member is sharing another file2/folder2 with “All Users” assuming All Users means all this group member. This gives external users access to file2/folder2.
Remediation
Option 0: remove “All Users” group
Option 1: exclude External users or Guest users from “All Users” group:
(user.userPrincipalName -notContains "#EXT#@")
or
(user.userType -ne "Guest")
Option 2: schedule a job that removes “All Users” from all sites UIL. Optionally inform site owners not to use “All Users” but use “Everyone except external users”.
How to exclude a user from “Everyone Except External Users” group?
Let say, you have a public site and you indeed want to provide access to all internal users with the exception of specific relatively small group of people. E.g. 10,000 users in company and only 200 are not fully integrated yet and you do not want to consider as equals in rights to all others and you do not want to provide automatic access. Unfortunately, there is no “deny access” options in SharePoint. All the functionality is about to “allow access” to something. What are you options?
Option 1: Create a security group and include in the group 9,800 users. In this case you’d need to review all sites with access provided to EEEU
Option 2: Change user type in AAD (Entra Id) from Member to Guest. In this case those users will not be a part of EEEU. They will be “Internal Guests”. You’ll still be able to provide direct access to sites and include such users in teams but they will be marked as Guests.
Think of it that there are internal and external users, and there are guests and members. Typically all internal users are members, and all external users are guests. But that does not always reflect real life. And if you change a “User type” property for some internal user from Member to Guest – this users will be an Internal Guest. Check this MS article: Understand and manage the properties of B2B guest users
Some templates can be applied by regular users (site admins) and some templates would require SharePoint tenant admin permissions. But now it’s only via PowerShell. You can get an idea how templates look like at
PnP provisioning engine is something that us used under the hood.
If you are interested in automation of provisioning templates – please let me know in comments below or via site feedback.
===============================
So the information below is obsolete and I will keep it just for the sake of history of SharePoint:
SharePoint Look Book
SharePoint Look Book – a site with a collection of modern SharePoint site templates. You can browse through dozens of good-looking templates… but how do you apply chosen template to your site?
Gotcha #1
There is a button “Add to your tenant>” and it says “You must be a tenant administrator to deploy this template.” Really? No… but Actually, SharePoint Administrator role is required to apply template from lookbook. So yes, tenant-level admin role but just SharePoint service admin role. Site admin role is not enough…
Gotcha #2
Next, when you try to get template by clicking “Add to your tenant>” button, it actually offers you to create a new site. But it also says “…can use existing URL”. Really? No. When you type existing site Url into the “Relative URL to be used for the site” field – You can get “Can’t add this template. The provided site is already in use and the current template cannot be provisioned onto an already existing site. Please provide a different URL” message:
Or, if you managed to enter existing Url, you might get: “Unfortunately your site provisioning at least partially failed!”:
Note: When you follow instructions provided by Microsoft, beware that “Connect-AzureAD” works only in Windows .net framework – i.e. PowerShell 5.1. if you try to run it in PowerShell 7 – you can get “Connect-AzureAD: One or more errors occurred. (Could not load type ‘System.Security.Cryptography.SHA256Cng’ from assembly ‘System.Core, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089’.)” Error. (check Connect-AzureAD Could not load type ‘System.Security.Cryptography.SHA256Cng’ from assembly)
Configuring Sensitivity Labels
Sensitivity labels are configured under Microsoft Purview (Compliance Center), Solutions, Information Protection. You’d need a global admin or “Compliance Administrator” or “Azure Information Protection Administrator” (?) role:
Since we are talking sensitivity labels for SharePoint Sites (not documents), we define label scope as “Groups and Sites”: “Configure privacy, access control, and other settings to protect labeled Teams, Microsoft 365 Groups, and SharePoint sites.”
Then we define which protection settings for groups and sites we should configure on the next steps: – Privacy and external user access settings – Control the level of access that internal and external users will have to labeled teams and Microsoft 365 Groups. – External sharing and conditional access settings – Control external sharing and configure Conditional Access settings to protect labeled SharePoint sites.
If we selected previously “Privacy and external user access settings” – now we need to select group/team privacy (These options apply to all Microsoft 365 Groups and teams, but not standalone sites). When applied, these settings will replace any existing privacy settings for the team or group. If the label is removed, users can change privacy settings again. You can also allow external user access – if group owner will be able to add guests:
Next step – define external sharing and conditional access settings. Specifically, if the content of the SharePoint site can be shared with Anyone (anonymously) or with authenticated users (new or existing) or no external sharing is allowed:
And you can either control the level of access users have from unmanaged devices or select an existing authentication context to enforce restrictions:
Configuring sensitivity labels policies
Sensitivity label policy is basically which label should be available to apply for what users and some other settings like – do users need to provide justification before removing a label or replacing it with one that has a lower-order number or – will users be required to apply labels and optionall the default label
View existing sensitivity labels
“Global reader” role allows view existing sensitivity labels configuration:
Wording would be a little different, but all aspects of the label configuration will be mentioned. E.g. when editing GUI says label scope is “Groups & sites”, read-only label summary defines Scope as “Site, UnifiedGroup”.
Gotchas
Site sensitivity label is applied to site collection only and cannot be applied to subsite (web object).
Applying sensitivity labels programmatically
To apply a label to a m365 group or Teams site with a group behind: MS Graph API support only Delegated permissions.
“Set-PnPSiteSensitivityLabel” works in the current site context. Description says “If the site does not have a Microsoft 365 Group behind it, it will set the label on the SharePoint Online site and will not require Microsoft Graph permissions and will work with both delegate as well as app only logins.” In fact (7/22/2022) app permissions are not working. This cmdlet can assign label to a standalone or a group-based site only with delegated permissions.
“Set-PnPTenantSite” allows you to remove or apply site sensitivity label to both standalone and group-based sites with app permissions. Furthermore, group and team settings respect this. I.e. if you apply label to a group-based site – group will pick this up.
Channel sites should inherit sensitivity label from root site. I’m not sure if it’s a bug but – when you create a team and select sensitivity label as part of team creation process – all the channel sites you create after (Private or Shared) will inherit sensitivity label immediately – when you apply sensitivity label to an existing team – with existing channel sites – in this case Private channel sites inherit team sensitivity label immediately, but with Shared channel sites it’s strange: GUI shows sensitivity label assigned, but site object model does not.
