Category Archives: SharePoint

Ownerless Microsoft 365 groups policy in large environments

Usually Microsoft 365 group can be created by anyone in your org as part of creation a team, Yammer community, Outlook group, SharePoint site etc. If the group owner lefts the company and account got deleted – the group became ownerless.

It would be a nightmare if we’d reach ownerless groups members peron-to-person trying to find out who is a real data owner and who should be a group owner. So we need some kind of automated way.

There is a Microsoft’s ownerless groups policy that detects ownerless groups and sends emails the most active groups members with the question- if they want to become a group owners and in case member accept ownership – policy automatically elevates a person from a group member to group owner. Policy does not cover standalone sites, but majority of orphaned resources in org are usually m365 groups, so that policy should help.

The policy was designed to prevent ownerless groups concept in mind, i.e. to deal with ownerless groups gradually – stretched in time – when they become ownerless. So it is actually recommended to activate the policy once you get the tenant right away. Configuration is done via GUI, it is intuitive and straightforward. Microsoft documented it well, but if you still have questions regarding the policy behavior – here is my Q&A on what is not covered by Microsoft’s FAQ as well as some tips and tricks and gotchas…

The problem is that Microsoft introduced this feature just recently, and if you own the tenant for years, you probably already have some ownerless groups. In small and medium environments with a few dozens of ownerless groups it’s not a big issue, but in a large Microsoft 365 SharePoint Online environment you might end up having hundreds and thousands of ownerless (orphaned) resources you have to deal with.

The challenge is how to implement the policy correctly if there are already many ownerless groups present and then to take care of groups that will become ownerless in the future. Yes, we’d need to address two consecutive issues:

  • Remediate vast amount of existing ownerless groups
  • Prevent groups to become ownerless

Obviously we’d need two different strategies and policies configurations.

There are also 3-rd party tools – like SysKit Point that can help with orphaned resources by enforcing minimum number of owners. There is also “Orphaned resources” policy under SysKit that allows multiple workflow options to resolve the issue – but there is no “fully automated” option -all SysKit options require an interaction from admin/manager.

Microsoft 365 built-in feature – “Ownerless groups policy” allows fully automated process:

  • detects ownerless groups, and for every group found
  • generate e-mail invitations to most active group members
  • assigns users as group owners if they accept invitation

Another problem in large environments is we have strict requirements we want to satisfy:

  • end-users to get only a few emails in a certain period so they can process it
  • end-users get only relevant messages so they will not ignore further notifications
  • high percentage of acceptance and (ideally) no orphaned resources

We want the policy to be tested in production but within a small group first and then we want phased implementation – so we could have a chance to get a feedback on phase 1 and adjust our approach at phase 2 etc.

The policy allows limiting policy scope in two ways:

  • by limiting “who can receive” messages – it’s done by specifying a security group – so only this security group members will be eligible to get invitation and accept or decline it
  • by limiting Microsoft 365 groups that would be in scope for the policy – it’s done by specifying group names

Two options can be specified in the same policy and effective eligible members would be those who satisfy both requirements.

Configuration is done using GUI – i.e. there is no PowerShell commands known on the subject at the moment.

There are a lot of “what if” questions regarding the policy – most of them are resolved in Microsoft’s “Microsoft 365 ownerless group policy FAQ” and my Ownerless m365 groups Q&As, gotchas, findings…

But the most important gotcha for me is that we do not have a chance to re-configure the policy or re-activate it to get more messages for the groups all messages were generated earlier. I.e. if an e-mail messages were generated for a group and the policy stopped working after a specified period of time – it’d done forever. No more e-mails could be generated for the same group.

The other limit is you can specify maximum 50 m365 groups in policy under Apply policy to Specific groups option. And we’d keep in mind exchange’s limit of 10k emals per day.

So, having this said, what would be the proper approach to do phased implementation in terms of configuring policy to scope it down for each step?

First – know your data. Get full report on ownerless groups, analyze it and come up with approach. Let’s assume we have an org with ~100K users and ~5000 ownerless groups. I bet you will find out that you have

  • large m365 groups (50+ members): <1%, i.e. 10-20 groups
  • medium m365 groups (5-50 members): ~25%, i.e. ~1000-2000 groups
  • small m365 groups (1-5 members): ~50%, i.e. ~2500 groups
  • null m365 groups (0 members): ~25%, i.e. ~1000-2000 groups

You’d might have your own classification, but I would propose the following approach to each category.

  • large groups:
    configure policy with “Apply policy to Specific groups” option
    and specify all or several of your large groups (the limit if 50 allowed groups in this field)
  • medium groups:
    configure policy not scoped down (e.g. apply to all groups, all users)
  • small groups:
    elevate all group members to owners
    optionally – elevate specific titles (manager, lead) or salary grade members to owners
  • null groups:
    consider deleting these groups
    optionally – delete only inactive no-members groups or groups with no or small amount of storage/files.

You’d also come up with the ideas on

  • desired min and max number of owners
  • deleting groups/sites phased approach
  • archiving groups/teams/sites

Remember – this is production, so at this moment you should test the policy in non-prod an be fully comfortable with all aspects of configuring the policy and formatting e-mail template etc.

As a remediation part plan I would propose the following:

(WIP)

Wave 0 – piloting

select a few (3-5) ownerless m365 groups came from IT – whose members are your pilot team members, so you could finalize all settings and polish notification message etc.

Implement the policy with settings:

In parallel, while you are waiting weeks for the policy to pause, start developing PowerShell scripts that will 1) delete null (no members) groups and 2) elevate members to owners (get how many members can be elevated if elevate only certain members)

Track user’s response – % of declines and accepts

Get feedback from users – how well the notification message is understandable

Wave 1 – large groups and small groups

Implement the policy with settings:

In parallel, you should already know – how many members can be elevated if elevate only certain members, decide on that and and run PowerShell script that elevates members to owners.

Wave 2 – medium groups and null groups

Implement the policy with settings:

In parallel, run PowerShell script removes groups with no owners and no members (optionally inactive and/or no content).

Wave 3 – all groups left ownerless

Implement the policy with settings:

Wave 4 – permanent policy and deletion script

Implement the policy with settings:

two more moments to consider:
– After all the measures against ownerless groups is done, we will probably still have some groups ownerless
– We will be getting new ownerless groups permanently – during all the waves of policy implementation

Qestions and Answers

Q: Isn’t it a security risk if we elevate members to owners? Would a member get access to more information that he/she did have access to before.
A: 1) Elevating members is the same risk as implementing the ownerless policy, as policy does the same – it elevates member to group owner.
2) When a member is elevated to group owners – a member does not get access to more information, as
a) for standard channels – he/she did have access as a member
b) private channels stays private – new group owner dos not get access to private channels automatically
c) shared channels stays with the same permissions also

TBC

References

SPO Site LastContentModifiedDate vs LastItemModifiedDate vs LastItemUserModifiedDate vs Graph LastModifiedDateTime

How do we know when the SharePoint site was last updated?

We have multiple “when the site was modified last time” properties – e.g. some we can retrieve with SharePoint CSOM:

  • Site LastContentModifiedDate
  • Web LastItemModifiedDate
  • Web LastItemUserModifiedDate

Also we can get

  • MS Graph site object with LastModifiedDateTime property
  • get usage reports via Microsoft Graph (activity reports), and
  • use “Last activity” field via Admin Center GUI

On the other hand – we can view and modify site in multiple ways – visit site home page, open and/or update document/list item, change site/library settings, configure site permissions, assign site sensitivity label, setup site property and so on.

Question: which site “last modified” or “last activity” properties reflect what events/actions?

This might be important if we think of retention policies, or any kind of clean-up processes… Let say, we are getting report on abandoned sites (inactive sites), but we are also assigning sites sensitivity labels, or we are updating site custom properties (e.g. for adaptive scopes), we have an ownerless groups policy working etc.

What if we assign site sensitivity label to an old inactive (5 years old) site – would it affect retention policy since site was updated this way?

Results

So i did some tests and based on detailed results below, it seems like

  • Web LastItemModifiedDate is triggered when user just visited site (but property LastItemUserModifiedDate is not triggered)
  • If a document or list Item updated by user or app – all properties are triggered
  • MS Graph site property LastModifiedDateTime, root web property LastItemModifiedDate and Site LastContentModifiedDate – same values
  • If site custom property is updated – it does not affect any site “last modified” property
  • The same for sensitivity label updated by app – it does not affect any site “last modified” property
  • The same for Microsoft ownerless groups policy – when user accept or decline group membership – no site “last modified” properties are changed (the same is true for Microsoft 365 group last modified date/time property).

Please refer to the table below

Detailed test results

Test results if the event triggers property update:

EventLast Content Modified DateLast Item Modified DateLast Item User Modified DateGraph Last Modified DateTimeGUI Last activity
Page viewed by userYesYesNoYes
Home Page viewed by user
Site Page viewed by user
Document or list item updated by userYesYesYesYes
Document or list item updated by appYesYesYesYes
Site config settings updated by user
Site config settings updated by app
Site custom property updated by appNoNoNoNo
Site Sensitivity label updated by user via SharePointYesNoNoNo
Site/Group Sensitivity label updated by user via Teams
Site/Group Sensitivity label updated by user via AzureNoNoNoNo
Site Sensitivity label updated by appNoNoNoNo
Site collection admin updated by userYesYesNoYes
Site collection admin updated by appYesYesNoYes
SharePoint group membership updated by userYesYesNoYes
Standalone Site connected to a group by userYesYesYesYes
Add Microsoft Teams to Site by UserYesYesYesYes
Update m365 group membership via M365 admin console by adminYesYesNoYes
Update m365 group membership via Azure by admin
Update m365 group membership via Teams by userNoNoNoYes
Update m365 group membership via App
Accept group ownership invitation sent by ownerless groups policyNoNoNoNo
Decline group ownership invitation sent by ownerless groups policyNoNoNoNo

Microsoft 365 Retention Policies SharePoint Adaptive Scopes Advanced Query

Basic query is available as GUI:

where you can use objects: “Site Url”, “Site Name” and “Refinable String 0″..”Refinable String 99”. Conditions would be “is equal to”, “is not equal to”, “starts with” and “not starts with”. Or you can select “Advanced query builder” and enter KQL query.

Advanced query builder

Advanced query builder allows us to use more site properties then “Site Url”, “Site Name” and “Refinable Strings” and more conditions than “is (not) equal to” and “(not) starts with”.

E.g. we can use “Title”, “Created”, “Modified” site properties and “=”,”:”,”<“, “>”, “<=”, “>=” conditions.

Working queries examples:

created>=2022-07-21
modified>1/31/2023
created>12/31/2021 AND modified>=7/31/2022
created<=2020-11-15 OR modified>2023-02-06 (?)
created<=2020-1-15 OR modified>2023-01-31 (?)
created<=11/15/2020 OR modified>1/31/2023
title:test
SiteTitle:test
RefinableString09:Test*
RefinableString09<>Test
RefinableString09=Birding AND RefinableString08<>Included


Not working queries examples:

site:https://contoso.sharepoint.com/sites/test* 
RefinableString11 = Birds # (do not use spaces in advanced query)
Path:https://contoso-my.sharepoint.com
Template:STS
Template:"SITEPAGEPUBLISHING#0"
Template:SITEPAGEPUBLISHING*
? RefinableString09<>Birding AND RefinableString08:Official
modified>31/1/2023 (should be like modified>2023-01-31
)

Query against custom site property (aka property bag value)

You can create custom site property and assign value to the property with
Set-PnPAdaptiveScopeProperty or Set-PnPPropertyBagValue.
Property must be with “Indexed” parameter. Once the property is set up, m365 search crawls site and creates crawled property. Then you map crawled property to some pre-created refinable string managed property. You can assign alias to this managed property.

In my test scenario I used RefinableString09 with alias SiteCustomSubject.

Site property valueQueryresult
BirdingRefinableString09:Birddoes not work
BirdingSiteCustomSubject:Birddoes not work
BirdingRefinableString09:Bird*works
BirdingSiteCustomSubject:Bird*does not work
BirdingRefinableString09:Birdingworks
BirdingSiteCustomSubject:Birdingdoes not work
BirdingRefinableString09:Birding*works
BirdingRefinableString09=Birdingworks
BirdingRefinableString09=Birddoes not work
BirdingRefinableString09=Bird*does not work
BirdingSiteCustomSubject=Birdingdoes not work
RefinableString09<>Birdingworks
RefinableString09=Birding AND RefinableString08<>Includedworks

Query against multi-value property.

Site property valueQueryresult
TestA TestBRefinableString09:TestAworks
TestA TestBRefinableString09 = ‘TestA TestB’does not work
TestA TestB??? RefinableString09=’Test10 Test5′does not work
TestA TestBRefinableString09:TestB ?
TestA,TestBRefinableString09:Test*works
TestA,TestBRefinableString09=Test*does not work
TestA,TestBRefinableString09:Testdoes not work
TestA,TestB
TestA;TestB
TestB TestA
TestA TestB
RefinableString09:TestBworks
TestA, TestB
TestB,TestA
TestA TestB
RefinableString09=TestAdoes not work
TestA,TestB(basic) RefinableString09 starts with testworks

Some more findings

Modify adaptive scope

If you need to modify adaptive scope – you’d better delete it and create a new one. The reason – if you want to validate what sites are included in scope with GUI – via button “Scope details” – you want to see only sites that are in scope, but that’s not the case when you modify the scope, because if you modify the scope – you’d see sites that are not in scope with “Removed” status.

Alternatively you can use filter to filter out removed from scope sites.

what else?

What is the takeaway from this for SharePoint administrators? We would be asked to configure SharePoint the way compliance…

References

SharePoint Sites Lookup

That’s a very common problem in SharePoint world. You are looking for a site owner but there is no tool available for regular user to find who owns the site.

Scenarios.

You get a link to some SharePoint site, but you do not have access to it. You requested access but nobody has responded. You need to find who is the site owner.

(To be continued)

PowerShell scripts for Microsoft 365 SharePoint

After many years working with SharePoint I wrote a lot of PowerShell scripts that help me support, troubleshoot, administer and secure SharePoint. So I’m sharing my scripts with you.

It’s here: https://github.com/VladilenK/Manage-m365-with-PowerShell

Get all SharePoint and Teams sites owners report with PowerShell

This PowerShell script pulls all tenant sites and all sites owners. The script require app authentication with Sites.FullControl.All and Directory.Read.All permissions.
PnP.PowerShell for PowerShell 7 is used.

The script generates two reports

  • Owners report: one user per line, include: Site Url, Title, Owner e-mail, name and type
  • Sites report: one site per line, include: Site Url, Title, list of owners e-mails

Here is the script:


$connAdmin = Connect-PnPOnline -ReturnConnection -Tenant $tenantId  -Url $adminUrl -ClientId $clientid -Thumbprint $certThumbprint
$allTenantSites = Get-PnPTenantSite -Connection $connAdmin | Sort-Object Url
$allTenantSites.count

$sitesReport = @()
$ownersReport = @()
foreach ($tenantSite in $allTenantSites) {
    Write-Host $tenantSite.Url
    $connSite = Connect-PnPOnline -ReturnConnection -Tenant $tenantId  -Url $tenantSite.Url -ClientId $clientid -Thumbprint $certThumbprint
    $site = Get-PnPSite -Connection $connSite -Includes RootWeb, GroupId, Owner
    $siteOwnerEmail = ''
    $siteOwnersReport = @()
    if ($site.GroupId.Guid -eq '00000000-0000-0000-0000-000000000000') {
        $siteAdmins = Get-PnPSiteCollectionAdmin -Connection $connSite | ? { $_.PrincipalType -eq 'User' }
        $ownerType = 'Site Collection Administrator'
        $isGroupSite = $false
    }
    else {
        $siteAdmins = Get-PnPAzureADGroupOwner -Connection $connAdmin -Identity $site.GroupId.Guid
        $ownerType = 'Group Owner'
        $isGroupSite = $true
    }
    foreach ($siteAdmin in $siteAdmins) {
        if (!$siteAdmin.UserPrincipalName) {
            Get-PnPProperty -Connection $connAdmin -ClientObject $siteAdmin -Property UserPrincipalName | Out-Null
        }
        $aadUser = Get-PnPAzureADUser -Connection $connAdmin -Identity $siteAdmin.UserPrincipalName
        if ($aadUser.AccountEnabled) {
            $siteOwnerEmail += $aadUser.Mail + '; '
        }
        $siteOwnersReport += [PSCustomObject]@{
            SiteUrl     = $site.Url
            SiteTitle   = $site.RootWeb.Title
            IsGroupSite = $isGroupSite
            OwnerEmail  = $aadUser.Mail
            OwnerName   = $aadUser.DisplayName
            OwnerType   = $ownerType
            Enabled     = $aadUser.AccountEnabled
        }
    }
    $ownersReport += $siteOwnersReport
    $sitesReport += [PSCustomObject]@{
        SiteUrl     = $site.Url
        SiteTitle   = $site.RootWeb.Title
        IsGroupSite = $isGroupSite
        OwnerEmail  = $siteOwnerEmail
    }
}

$ownersReport.count
$sitesReport.count

Source code: https://github.com/VladilenK/Manage-m365-with-PowerShell

Manage result layouts for SharePoint results in Microsoft Search

Microsoft is improving Search (MC489165):

Manage result layouts for SharePoint results in Microsoft Search

We’re making changes to Microsoft Search. This update will allow Microsoft Search administrators to change result layouts for select SharePoint content using adaptive cards with Result Type feature in Microsoft Search administration.

The default result layouts for SharePoint sites, pages, list items and Portable document format (PDF) results can now be replaced with layouts built using adaptive cards. The changes can be made for Organization level search applicable to Office.com and SharePoint home as well as site level search on SharePoint sites. Changes for Microsoft Search in Bing will be rolled out soon. Note that the feature does not support changing of Office file search results.

This message is associated with Microsoft 365 Roadmap ID 81952

Before the change, when you add a new result type under “Search and intelligence” Customizations – it looked like this:

result type content sources

So there was no built-in “SharePoint” content source as an option – only custom “external” data sources.

But with the new feature implemented list of content sources for the result type will look like this:

SharePoint and OneDrive content source

If you choose “SharePoint and OneDrive” content source – the next option would be to select type of content:

Select type of content and set rules

You also can create different result types for different types of content based on properties-based rules (e.g. one result type for all sites – and a separate result type for a specific site or hub) with optional “Set rules for this type of content”:

Default site result experience would look like

Search results with modified SharePoint result type might look like:

When you modify template via Layout Designer – it is essential to know available object properties.

You can get properties from the “Available properties” below – there is also search through properties feature.

Or you can use SharePoint Search Query Tool to get metadata on search results.

It might take hours and even days for your search to start showing new layouts, but “&cacheClear=true” should help.

DepartmentId 

If your sites are organized in hierarchy under Hub site – you can use DepartmentId managed property to include all hub-associated sites content

DepartmentId is just a hub site Id

… to be continued …

References

Microsoft 365 SharePoint: prevent throttling with RateLimit headers

Bert Jansen (Microsoft) revealed some details on throttling when you access Microsoft 365 programmatically – via Microsoft Graph or CSOM and guided developers on how to regulate request traffic for optimized throughput using RateLimit headers (Here).

Demystifying SharePoint throttling

Throttling is necessary to ensure that no single user or application consumes too many resources compromising the stability of the entire system, which is used by many clients.

Throttling happens at

  • User (there are user request limits. Microsoft counts all requests linked to user
  • Application (Delegated or Application permissions)
    • Resource units per app per minute
    • Resource units per app per day
  • Farm – Spike protection

Very common reason for throttling – when an Application (Delegated or Application permissions) reaches “Resource units per app per minute” threshold.

Usually you catch HTTP errors 429 or 503, wait for some time (respect Retry-after header) and try again.

SharePoint provides various APIs. Different APIs have different costs depending on the complexity of the API, but Microsoft favor Graph API over SharePoint REST/CSOM. The cost of APIs is normalized by SharePoint and expressed by resource units. Application’s limits are also defined using resource units.

Quota depends on tenant size.

Resource unit limits for an application in a tenant (please refer to the Microsoft article)

Predefined costs for Microsoft Graph calls:

Assuming 2 resource units per request is a safe bet.

Links

Update Large Number of SharePoint Sites with PowerShell Parallel

WIP

Here I’m trying to figure out – how much PowerShell Parallel option is beneficial and how to avoid throttling…

Let us test, how long would it take to create a SharePoint site, if we use regular (sequential) loop or parallelism (I’m creation a sample set of 50 SharePoint Sites in a row):

Regular
(Sequential)
seconds per site
Parallel,
100 sites in batch
seconds per site
Parallel,
500 sites in batch
seconds per site
Regular (Sequential)3.0
Parallel,  ThrottleLimit = 21.600.91
Parallel,  ThrottleLimit = 50.69
Parallel,  ThrottleLimit = 100.2 – 0.3
Parallel,  ThrottleLimit = 200.17

Interesting, but I did not get even one (throttling or any other) error during creation 500 sites.

Get sites details

Now let us test, how long it takes to get sites details with Get-PnPTenantSite (I use a sample set of 500 sites):

Test typeRegular
(Sequential),
seconds per site
Parallel
sample = 100 sites,
seconds per site
Parallel
sample = 200 sites,
seconds per site
Parallel
sample = 500 sites,
seconds per site
Regular (Sequential)0.65
Parallel,  ThrottleLimit = 20.400.330.31
Parallel,  ThrottleLimit = 50.170.140.36 (errors)
Parallel,  ThrottleLimit = 100.11 (errors)0.11 (errors)0.34 (errors)
Parallel,  ThrottleLimit = 200.12 errors+0.07 errors+0.52 (errors)

(errors) means there were small number of errors during test… e.g.

Microsoft 365 Search Vertical KQL query field limits

What is the Microsoft Search KQL query field limits for a verticals? Is there limited number of characters or lines?

You know what is Microsoft 365 Search Vertical and what is KQL query in vertical configuration, right?

Microsoft 365 Search Vertical KQL query field limits

Under Microsoft 365 admin center Search and intelligence you can configure search verticals. There are some out-of-the-box verticals – like All, Files, Sites, People and you can configure custom one.

As a part of vertical configuration – you can specify KQL query – if you want e.g. limit search with some sites or content types etc.

The question is – how many sites I can specify in this query field? E.g. can I specify 1000 sites? 10k sites?

And the answer is: It does not matter, because the limit is not in number of characters or lines.

In my dev environment I was able to save 50,000 lines (~3M characters). But attempt to save 100K lines (6M symbols) has failed (due to timeout, I believe:

Again, as I said the problem is not here.

The problem is time required for search to apply query. I.e. when you ask search to bring you something – after it gets results from index and before display results to you it applies KQL query configured for the vertical. And this time is the bottleneck.

Here is what I got measuring search response time depending on query size:

Searchresponse time,
seconds
KQL query
# of lines
KQL query size,
# of symbols
works150028,000
works5100059,000
works92000120,000
works253000180,000
works/fails303500208,000
fails353600214,000
fails3550,0003,000,000
n/an/a100,000
(can’t save KQL query
6,000,000
(can’t save KQL query)

Which means that after ~ 1000 lines (50,000 characters) KQL query size – query becomes too slow, and after ~3000 lines (180k chars) – can fail (due to timeout I’d say).

DepartmentId 

If your sites are organized in hierarchy under Hub site – you can use DepartmentId managed property to significantly decrease number of lines in query,
as you can cover all sites under the hub with

DepartmentId=<HubSiteId>