Bert Jansen (Microsoft) revealed some details on throttling when you access Microsoft 365 programmatically – via Microsoft Graph or CSOM and guided developers on how to regulate request traffic for optimized throughput using RateLimit headers (Here).
Demystifying SharePoint throttling
Throttling is necessary to ensure that no single user or application consumes too many resources compromising the stability of the entire system, which is used by many clients.
Throttling happens at
User (there are user request limits. Microsoft counts all requests linked to user
Application (Delegated or Application permissions)
Resource units per app per minute
Resource units per app per day
Farm – Spike protection
Very common reason for throttling – when an Application (Delegated or Application permissions) reaches “Resource units per app per minute” threshold.
Usually you catch HTTP errors 429 or 503, wait for some time (respect Retry-after header) and try again.
SharePoint provides various APIs. Different APIs have different costs depending on the complexity of the API, but Microsoft favor Graph API over SharePoint REST/CSOM. The cost of APIs is normalized by SharePoint and expressed by resource units. Application’s limits are also defined using resource units.
Quota depends on tenant size.
Resource unit limits for an application in a tenant (please refer to the Microsoft article)
Predefined costs for Microsoft Graph calls:
Assuming 2 resource units per request is a safe bet.
Here I’m trying to figure out – how much PowerShell Parallel option is beneficial and how to avoid throttling…
Let us test, how long would it take to create a SharePoint site, if we use regular (sequential) loop or parallelism (I’m creation a sample set of 50 SharePoint Sites in a row):
Regular (Sequential) seconds per site
Parallel, 100 sites in batch seconds per site
Parallel, 500 sites in batch seconds per site
Regular (Sequential)
3.0
Parallel, ThrottleLimit = 2
1.60
0.91
Parallel, ThrottleLimit = 5
0.69
Parallel, ThrottleLimit = 10
0.2 – 0.3
Parallel, ThrottleLimit = 20
0.17
Interesting, but I did not get even one (throttling or any other) error during creation 500 sites.
Get sites details
Now let us test, how long it takes to get sites details with Get-PnPTenantSite (I use a sample set of 500 sites):
Test type
Regular (Sequential), seconds per site
Parallel sample = 100 sites, seconds per site
Parallel sample = 200 sites, seconds per site
Parallel sample = 500 sites, seconds per site
Regular (Sequential)
0.65
Parallel, ThrottleLimit = 2
0.40
0.33
0.31
Parallel, ThrottleLimit = 5
0.17
0.14
0.36 (errors)
Parallel, ThrottleLimit = 10
0.11 (errors)
0.11 (errors)
0.34 (errors)
Parallel, ThrottleLimit = 20
0.12 errors+
0.07 errors+
0.52 (errors)
(errors) means there were small number of errors during test… e.g.
What is the Microsoft Search KQL query field limits for a verticals? Is there limited number of characters or lines?
You know what is Microsoft 365 Search Vertical and what is KQL query in vertical configuration, right?
Under Microsoft 365 admin center Search and intelligence you can configure search verticals. There are some out-of-the-box verticals – like All, Files, Sites, People and you can configure custom one.
As a part of vertical configuration – you can specify KQL query – if you want e.g. limit search with some sites or content types etc.
The question is – how many sites I can specify in this query field? E.g. can I specify 1000 sites? 10k sites?
And the answer is: It does not matter, because the limit is not in number of characters or lines.
In my dev environment I was able to save 50,000 lines (~3M characters). But attempt to save 100K lines (6M symbols) has failed (due to timeout, I believe:
Again, as I said the problem is not here.
The problem is time required for search to apply query. I.e. when you ask search to bring you something – after it gets results from index and before display results to you it applies KQL query configured for the vertical. And this time is the bottleneck.
Here is what I got measuring search response time depending on query size:
Search
response time, seconds
KQL query # of lines
KQL query size, # of symbols
works
1
500
28,000
works
5
1000
59,000
works
9
2000
120,000
works
25
3000
180,000
works/fails
30
3500
208,000
fails
35
3600
214,000
fails
35
50,000
3,000,000
n/a
n/a
100,000 (can’t save KQL query
6,000,000 (can’t save KQL query)
Which means that after ~ 1000 lines (50,000 characters) KQL query size – query becomes too slow, and after ~3000 lines (180k chars) – can fail (due to timeout I’d say).
DepartmentId
If your sites are organized in hierarchy under Hub site – you can use DepartmentId managed property to significantly decrease number of lines in query, as you can cover all sites under the hub with
Microsoft recently implemented “Adaptive” retention policies. At step 2 of “Create retention policy” you’ll be asked “Choose the type of retention policy to create”: “A policy can be adaptive or static. Advantage of an adaptive policy will automatically update where it’s applied based on attributes or properties you’ll define. A static policy is applied to content in a fixed set of locations and must be manually updated if those locations change.”
And if you selected “Adaptive” – on the next step you will need to provide the adaptive scope (so at this moment you should already have created your adaptive scopes):
So, let us create your adaptive scopes. What type of scope do you want to create? SharePoint sites…
And then you’ll have nothing more then set of conditions:
where you can use objects: “Site Url”, “Site Name” and “Refinable String 0″..”Refinable String 99”. Conditions would be “is equal to”, “is not equal to”, “starts with” and “not starts with”. Or you can select “Advanced query builder” and enter KQL query.
Advanced query builder for SharePoint Adaptive Scope
I will be saving my personal gotchas on Microsoft 365 External Access and Guest Access in SharePoint and Teams
We configure external/guest access in AAD, m365 Admin Center, Teams Admin Center, SharePoint Admin Center, specific Group, Team or SharePoint site.
We can configure external guest access directly, or can configure sensitivity labels and policies in Purview (Compliance Admin Center). Configuring sensitivity labels for sites/groups we configure external guest access settings. Configuring sensitivity labels policies we apply labels.
External access via “All Users” group
Be careful with “All users” group created as part of the process. Microsoft: “The dedicated All Users group includes all users in the directory, including guests and external users.” And indeed, “All Users” group by default include external users.
So here is the scenario: we have a site where external sharing is enabled, and someone is sharing a specific file1 or folder1 with some external users. The other site/group member is sharing another file2/folder2 with “All Users” assuming All Users means all this group member. This gives external users access to file2/folder2.
Remediation
Option 0: remove “All Users” group
Option 1: exclude External users or Guest users from “All Users” group:
(user.userPrincipalName -notContains "#EXT#@")
or
(user.userType -ne "Guest")
Option 2: schedule a job that removes “All Users” from all sites UIL. Optionally inform site owners not to use “All Users” but use “Everyone except external users”.
SharePoint Look Book – a site with a collection of modern SharePoint site templates. You can browse through dozens of good-looking templates… but how do you apply chosen template to your site?
Gotcha #1
There is a button “Add to your tenant>” and it says “You must be a tenant administrator to deploy this template.” Really? No. Actually, SharePoint Administrator role is required to apply template from lookbook.
Gotcha #2
Next, when you try to get template by clicking “Add to your tenant>” button, it actually offers you to create a new site. But it also says “…can use existing URL”. Really? No. When you type existing site Url into the “Relative URL to be used for the site” field – You can get “Can’t add this template. The provided site is already in use and the current template cannot be provisioned onto an already existing site. Please provide a different URL” message:
Or, if you managed to enter existing Url, you might get: “Unfortunately your site provisioning at least partially failed!”:
beware that “Connect-AzureAD” works only in Windows .net framework – i.e. PowerShell 5.1 if you try to run it in PowerShell 7 – you can get “Connect-AzureAD: One or more errors occurred. (Could not load type ‘System.Security.Cryptography.SHA256Cng’ from assembly ‘System.Core, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089’.)” Error. (check Connect-AzureAD Could not load type ‘System.Security.Cryptography.SHA256Cng’ from assembly)
Configuring Sensitivity Labels
Sensitivity labels are configured under Microsoft Purview (Compliance Center), Solutions, Information Protection. You’d need a global admin or “Compliance Administrator” or “Azure Information Protection Administrator” (?) role:
Since we are talking sensitivity labels for SharePoint Sites (not documents), we define label scope as “Groups and Sites”: “Configure privacy, access control, and other settings to protect labeled Teams, Microsoft 365 Groups, and SharePoint sites.”
Then we define which protection settings for groups and sites we should configure on the next steps: – Privacy and external user access settings – Control the level of access that internal and external users will have to labeled teams and Microsoft 365 Groups. – External sharing and conditional access settings – Control external sharing and configure Conditional Access settings to protect labeled SharePoint sites.
If we selected previously “Privacy and external user access settings” – now we need to select group/team privacy (These options apply to all Microsoft 365 Groups and teams, but not standalone sites). When applied, these settings will replace any existing privacy settings for the team or group. If the label is removed, users can change privacy settings again. You can also allow external user access – if group owner will be able to add guests:
Next step – define external sharing and conditional access settings. Specifically, if the content of the SharePoint site can be shared with Anyone (anonymously) or with authenticated users (new or existing) or no external sharing is allowed:
And you can either control the level of access users have from unmanaged devices or select an existing authentication context to enforce restrictions:
Configuring sensitivity labels policies
Sensitivity label policy is basically which label should be available to apply for what users and some other settings like – do users need to provide justification before removing a label or replacing it with one that has a lower-order number or – will users be required to apply labels and optionall the default label
View existing sensitivity labels
“Global reader” role allows view existing sensitivity labels configuration:
Wording would be a little different, but all aspects of the label configuration will be mentioned. E.g. when editing GUI says label scope is “Groups & sites”, read-only label summary defines Scope as “Site, UnifiedGroup”.
Gotchas
Applying sensitivity labels programmatically
To apply a label to a m365 group or Teams site with a group behind: MS Graph API support only Delegated permissions.
“Set-PnPSiteSensitivityLabel” works in the current site context. Description says “If the site does not have a Microsoft 365 Group behind it, it will set the label on the SharePoint Online site and will not require Microsoft Graph permissions and will work with both delegate as well as app only logins.” In fact (7/22/2022) app permissions are not working. This cmdlet can assign label to a standalone or a group-based site only with delegated permissions.
“Set-PnPTenantSite” allows you to remove or apply site sensitivity label to both standalone and group-based sites with app permissions. Furthermore, group and team settings respect this. I.e. if you apply label to a group-based site – group will pick this up.
Breaking news! It turned out that once the policy was activated and generated notifications for the group – the policy consider the group is done forever. Even if you re-activate the policy or change policy scope or other policy parameters – it will not help. Once messages are sent to the group members – no more messages will be generated for the same group ever.
Every resource under Microsoft 365 (Microsoft Teams team, Microsoft 365 group or SharePoint site) must have an owner/owners. Otherwise to whom we communicate on any question – site/group permissions, membership, site/group/team retention policy, content classification etc. Who will be responsible for team/site/group content and configuration and who will provide access to this site for other users.
MS: A team in Microsoft Teams or a Microsoft 365 group and its related services can become ownerless if an owner’s account is deleted or disabled in Microsoft 365. Groups and teams require an owner to add or remove members and change group settings.
Recently Microsoft implemented a new feature: a policy that automatically asks the most active members of an ownerless group or team if they’ll accept ownership. Very important feature. TY Microsoft!
It is important because many other “governance” activities (e.g. permissions attestation, retention policies) rely on site/team ownership. I.e., before we notify site owner that the site is going to be deleted due to inactivity – we want an owner present.
The configuration via wizard is straightforward and intuitive, and Microsoft documented it well, but still we have some questions regarding the policy behavior.
Q: Is it about groups ownership or sites ownership? A: Group ownership and group-based sites ownership (teams, yammer etc.). Non-group based aka Standalone sites (e.g. communication) are not in scope of this feature/policy.
Q: Who can configure this policy? What kind of permissions required to create/update policy? A: Microsoft says “Manage Microsoft 365 groups” permissions required – e.g. admins with Global admin or Groups Admin roles required. “Teams administrator” or “SharePoint Administrator” cannot configure the policy.
Q: After the policy activated – who will receive notification? What exactly “most active members” mean? A: Microsoft only says “most active members” and does not disclose specific algorithm behind.
Q: How about group with no members? What if somebody created a group but did not add any members and then left? A: In this case the policy will not work – as there is nobody who can be notified. This kind of groups must be handled manually, as no owners no members does not mean nobody uses related SharePoint site. What if the group is public and hosts some valuable data?
Q: How do we know the group is ownerless? Only if owner has been deleted from AAD? What if an owner is just blocked or became unlicensed? A: For the policy Microsoft consider blocked or unlicensed users presented in the group owners list as valid users and still group owners; so the policy will not be triggered until the group owners list is empty.
Q: We have implemented Azure AD Settings “EnableGroupCreation” and “GroupCreationAllowedGroupId” (as per Microsoft: Manage who can create Microsoft 365 Groups), so not everyone can create m365 groups. Would this impact ownerless groups policy? In other words – if a user cannot create group – would this keep user from being assigned as a group owners? A: No. Microsoft’s Manage who can create Microsoft 365 Groups trick regulates groups creation only. Later – when a group is created – nothing prevents such user to be added as a group owner.
Q: I support a large Microsoft 365 environment and we already have hundreds and thousands of ownerless groups. I’m concerned how users might react and whether our helpdesk support teams are ready for new type of tickets etc. Implementing the policy in test/stage environment does not make much sense, since there are no really active users etc. So, can I test this policy in production – on real users, but pilot it within a small number of users or ownerless groups before applying to all groups in the environment. A: Yes, you can do a test or pilot implementation in production limiting the impacted users or groups. – if you need to limit users who will be getting notifications – e.g. a “pilot team” – during Step 1 “Notification Options” under “Specify who can receive ownership notifications” you can select “Allow only certain active members” and under “Specify security groups to allow members” you can select a security group – so only members from the specified security group will be sent ownership request. Microsoft 365 groups do not work here.
but be aware – if you choose this option – it is possible that Microsoft 365 groups might have more active members who are not the security group members. E.g. it might make sense to use this option for piloting – against a small number of isolated set of groups/members, but for not for phased implementation. If you have some specific requirements for group ownership – e.g. “only managers could be group owners” or “contractors cannot be group owners” – using security group to limit potential group owners would make sense.
Another option you can use for phased implementation or piloting the policy is to scope it down to a several selected m365 groups – use “Apply policy to” – “Specific groups” option:
NB! After all notifications are sent for a group – you will never ever get any more notifications for the same group. Even if you re-activate the policy or change policy parameters or whatever – it will not help. Once messages sent – it’s done for the group forever.
Q: How many groups I can specify if I select Apply policy to Specific groups option? Is there a maximum? A: Yes, there is a limit. You can specify no more than 50 groups.
Q: I know the policy is applied to Microsoft 365 groups only. But I have many standalone sites with no owners (no site collection administrators). How do I deal with ownerless SharePoint sites? A: Options are: manual intervention, PowerShell, 3-rd party tools – depending on your specific case. E.g. you can elevate some “Site Owners” SharePoint group members to site administrators. For modern sites – how about converting standalone sites to Microsoft 365 group-based sites (TBC – as at the moment it is not clear if it is possible)?
Q: What happens after one of the notified members accepts the ownership request? A: No more notifications will be sent for this group. But previously sent notifications will still be valid.
Q: What happens if several of the notified members accepts the ownership request? A: Only two first served basis. As per Microsoft, only two members can be assigned to group owners via the policy. When a group got two owners – invitation message actionable item for the rest will be converted from “Would you like to be a group owner?” to “MemberName1 and MemberName2 have already agreed to become group owners.” with no “Yes” and “No” actionable buttons.
Q: Can I customize an ownership notification? A: Yes, but – E-mail message body is limited to ~1040 characters – Policy does not provide any WYSIWYG rich text format options (but there are some tricks you can use to format it with headers, bold/italic, links, bullets/lists: more on email template format.)
Q: Can I use shared mailbox as a “send from” e-mail account? A: No. You can use only user or m365 group mailbox.
Q: What if a group become ownerless after policy is activated? A: Policy detects the group is ownerless and start sending notifications within 24 hour. Actually the policy was designed to prevent ownerless groups. So it is recommended to activate the policy once you get the tenant.
Q: We know, that if a user declined ownership once – he will not get any more emails on the same. Is that true for current policy or for any further policies activations? I.e. If the policy updated/re-activated – will it remember user’s decision or it all starts from scratch? A: All the next incarnations of the policy will not trigger e-mail notification for the group if all notifications were sent earlier. I.e. in this case user will not receive any more notifications on the same group.
Q: If nobody accepted ownership – can we reconfigure the policy to sent more notifications – e.g. to wider range of active members or with more strict language in an e-mail template? A: No. After number of emails specified in policy were sent – no more emails will be generated for this group, so the group active members will not get any more emails on the group. Workaround: you can specify policy with e.g. 6 weeks – then after 3 weeks change e-mail template.
Q: What if we specify emails should be sent for 5 weeks, but stop the policy after two weeks? And then we re-activate the policy. A: It is expected the policy will continue sending e-mail notifications until 5 emails sent.
Q: What if we specify 3 weeks in policy, but then re-configure the policy with 5 weeks specified and activate it again? A: tbc – not tested yet
Q: Let say we have an ownerless group with 20 members. Let say we specified security group to limit user who will get invitations. And this security group includes only 6 users from the orphan group out of total 20. Microsoft says the policy will select the most active users. So the question is: will the policy select the most active users from the 20 orphaned group users and if the user in the security group – he/she will get an email? A: No. The policy will select the most active users only from these 6 users that included in the security group, ignoring 14 users, even if they were more active then these 6 selected.
Q: What happens after the policy expires? E.g. after all notifications are sent… A: Policy does not expires. If the policy is activated – it works. If all notifications are sent for the group – so yes, policy is done for this group. But if a new group became ownerless – policy will be triggered for this group again.
Some more findings:
User can forward invitation message, but recipient will not see actionable “Yes” “No” buttons.
If a public group does not have an owner – all requests to joint the team will be declined with “The team does not have an owner” message: (that means no new members, i.e. no new contributors, but read-only visitors access is sill available for everyone, as group is public):
Users can go to My Groups to see groups (Teams, Yammer communities and SharePoint Sites) they are members or owners of.
Proposal to be a group owner lasts forever. So if a user after some time finds an email that asks him “Would you like to be a group owner?” and clicks Yes – he/she will be a group owner, even if the policy is already updated or removed.
As per Microsoft, only first two members can accept the ownership of an ownerless group. No additional members are allowed to accept ownership. If either one or two members accept ownership, other members won’t receive further notifications.
Re-create (or re-activate) the ownerless group policy
You can de-activate the policy and then activate it again. Or you can reconfigure the policy. If you activated the policy again (or re-configured it), but emails are not sent – this might be an expected behavior. Let say you initially specified 3 weeks and 3 notification were sent to the most active ownerless groups members. That means no more emails will be generated for these groups.
De-activate the ownerless group policy
Just uncheck “When there’s no owner…”, and save it to stop the policy:
downside – you’d need to configure the policy from scratch – all previous settings are gone now
Track the ownerless group policy in action via Audit Log
How do I, as an Microsoft 365 administrator, know if the policy works or not, are the emails sent or not and how many (if any) users are accepted “Would you like to be a group owner?” invitation?
Microsoft 365 Audit Search under Microsoft Purview (Compliance center) should help.
OwnerlessGroupNotificationResponse – “Responded to ownerless group notification”
UserId: OwnerlessGroupComplianceAssistant
Record Type (AuditLogRecordType): 126
It seems like event is not added to the Audit log when a policy is created or updated.
Who can create Microsoft 365 Groups
It is possible to limit users – who can create Microsoft 365 Groups (please refer to Microsoft: Manage who can create Microsoft 365 Groups – there is a guide and PowerShell code sample). This might help to keep the environment under control – let say, “only managers can create groups”, or “contractor should not be able to create teams”.
It would be good if the configuration would be consistent in terms “if a user cannot create a group – user cannot be a group owner”. Unfortunately, with current configuration options (Aug 2022), this is not the case. Azure AD Directory Setting “GroupCreationAllowedGroupId” works only for creation. Later, when the group is create – it is possible to add to group as a group owner those who is not able to create group.
Issues
“Ownerless group policy configuration failed” error message. And “Failure in configuring ownerless groups policy” and “Please try again.” – seems like a permissions issue. SharePoint admin, Teams admin or Group admin roles: cannot configure Ownerless Groups Policy. Global admin: yes, can configure Ownerless Microsoft 365 Groups Policy. What is the minimum role required? According to a recent update of the Microsoft’s article – “A Global administrator can create a policy…”
Microsoft recently implemented “Adaptive retention policies” that use “Adaptive scopes”. Adaptive scopes can use “Site Url”, “Site Name” and 100 Refinable Strings from “Refinable String 0″ to ”Refinable String 99”.
How to configure SharePoint the way Refinable Strings are used in the Adaptive retention policies scopes?
The steps are:
Create an indexed site property
Map crawled property to a refinable string managed property
Detailed steps:
Indexed site property
Create an indexed site property or “Adaptive Scope Property” with some values. Ensure you property name (key) is unique, e.g.
Property
SiteRetentionProperty
Value
Y10
with PowerShell Set-PnPAdaptiveScopeProperty or with Set-PnPPropertyBagValue -Indexed:$true. Examples:
Wait until search crawler pics up you site property. Now you have a crawled property.
Search schema mapping
As you know, Refinable Strings are just pre-created by Microsoft refinable managed properties. So you can select one that is not used(*) and map it to crawled property. Assign alias so you could easily identify what is the RefinableString55 about.
(*) Notes
select one that is not used select one that is not used is an important moment, as if you select refinable string that is already taken at the site level – there is a conflict. So before configuring pre-created refinable properties at tenant level – I’d recommend to get report on managed properties taken at sites levels. It would be good idea if you agree with sites owners on properties ranges (e.g. from 00 to 99 – reserved for tenant use, from 100 to 199 – available at sites levels). And/or you can – after getting report on managed properties taken at sites levels – reserve enough unused managed properties by assigning aliases e.g. “this-property-55-is-reserved-do-not-use”.
site custom script If site custom script is enabled (DenyAddAndCustomizePages = false), then site collection admin can change site properties. So if you do not want the property being altered at site level – ensure that noscript site property is enabled (DenyAddAndCustomizePages equals true)
