Category Archives: SPO

Microsoft 365 retention policies: Static vs Adaptive scope

Adaptive scopes are good, but what if both policies are implemented? Which one wins?
The scenario for two policies might be: static retention policy is implemented as default retention policy for all sites, and if site require different retention or deletion – it should fall under one of the adaptive scopes and an adaptive retention policy will be applied.

Implementing Microsoft 365 group expiration policy in large companies

WIP

Scenario

Let say you administer a large Microsoft 365 environment (e.g. ~100k+ users and/or ~50K+ sites) and after some years you have a lot of ownerless groups and sites (around 5k probably), and a lot of inactive groups and sites (maybe 15k). And new ownerless groups are coming at a pace of dozens per week? You a thinking of stopping bleeding and cleaning this up…

Out-of-the-box we have Microsoft 365 groups expiration policy and Microsoft 365 ownerless groups policy. You might also have some 3-rd party tools implemented – e.g. ShareGate, SysKit.

If you do not care – you might just activate both OotB Microsoft policies – via GUI – they are simple to activate. But once you activated policies – they will trigger thousands of emails. Now imagine a person is getting dozens of emails asking him/her to be an owner or to renew the group that maybe does not make much sense… What’ll happen? Right, people will probably ignore these notifications. What will happen next? Groups and sites will be automatically deleted. And then? Right, there will be a noise and many angry users and high-priority tickets and you will have to restore sites/teams and finally you’ll have to look for groups owners manually.

So, what is the right way to clean-up a large Microsoft 365 environment from ownerless and inactive teams, groups sites?

Not a trivial question, hah?

Solution

TBP

Implementing Microsoft 365 groups expiration policy

If you are thinking of activating in an existing environment – you would probably have a spike – all the old groups will be subject to policy. The ide is to avoid situation when a specific person – group owner will get dozens of email. It would be better if a person will receieve, let say one email per week.

Here is my 4 possible approaches to avoid this spike, distribute notifications evenly across the time and ease the pain:

By changing Group Lifetime

You would need to change the policy every, e.g. week, specifying different group lifetime in days period. Consider
– calculate number of days between the oldest group created an today, plus 35 days – it’ll be your first “group lifetime”
– activate the policy with this number of days in “group lifetime” – and within a week you will get notifications on the oldest group/groups
– after a week or two – change the “group lifetime” decreasing it by e.g. 30-60 days and reactivate the policy… and so on

You can easily calculate it all and choose your pace depending on how many groups you have to renew, how much time you need to clean-up. You got the idea.

Downside – in the email notification it will be said “otherwise the group will be deleted on …”, but once you start joggling with dates – this will not be true probably.

By renewing groups as admin

tbp

By sending customized e-mails to users

tbp

By sending users to the groups page

tbp

Microsoft 365 group expiration policy deep dive

Nobody likes garbage, including Microsoft 365 administrators. If any user can create a team or yammer community – they create, but then they leave company and we are getting more and more abandoned groups, teams and SharePoint sites. So we need a way to clean up environment. There is a Microsoft 365 groups expiration policy that can help remove unused groups from the system, but since all Teams and Yammer sites are group-based – it also helps SharePoint admins make things cleaner.

Who can configure the policy and how

The policy lives under Azure Portal, Azure Active Directory, Groups, Expiration:

Microsoft 365 groups expiration policy can be configured by Groups Admin or Global Admin (tenant admin) only. Microsoft 365 Teams or SharePoint admin cannot configure it. Microsoft says that User administrator can do it – so I need to verify it.

Here is the policy config screen:

Microsoft documented it well in the “Microsoft 365 group expiration policy“, but I completed some tests in my lab environment and here is what I found and what is not covered by Microsoft. Let me share it with Questions and Answers format:

Questions and Answers

General questions

Q: How long it takes for policy to start generating notification emails after activation?
A: Immediately, i.e. minutes, maybe up to one hour (in case there groups that are subject for the policy).

Q: Can I customize email that is send to group owners?
A: No, there is no such option at the moment.

Q: What is the email address notifications come from?
A: It’s “msgroupsteam@microsoft.com” with the display name “Microsoft Groups Team”

Q: What does a notification email look like?
A: Please find some examples below, in the end of this article.

Q: Are there any other ways to get notifications? Teams chat? Phone message?
A: I’m not aware of phone messages, but I have seen notification in teams: “TeamName is expiring soon. Renew now”:

but I got only one teams notification, though e-mails notifications I got many.


Q: What happens when a user clicks “Renew group” button in the email notification?
A: User will be sent to a Microsoft’s page and the following “Do you want to renew the group?” window will be shown:

On Yes, it says”<groupName> was successfully renewed. You can close this window now”:

And the group expiration date will be set up as current date.
On “No” it says “Group was not renewed. You can close this window now.”:

And an expiration day will not be changed. No more notifications will be generated. The group will be active until expiration date. Then the group will be deleted.

Q: What if two owners choose opposite?
A: The last action will take effect.

Q: what if one user choose “delete group” but the other one later decided “Renew group”?
A: The one who click “Renew group” will see “<Group Name> successfully renewed. Because the group was deleted, it might take up to 24 hours to be fully restored. You can close this window now.”

Q: What if the group does not have owners?
A: If the group does not have an owner, the expiration emails will go to the email specified in policy configuration. Usually it is a distribution list with admins or other responsible team.

Q: What if I deactivate the policy – will email notifications sent earlier still be actionable?
In other words, would users still be able to renew the group clicking on the “Renew group” button?
A: Yes. Actually “Renew group” button is just a link to the Url:
https://account.activedirectory.windowsazure.com/Group/RenewGroup?tenantId=<tenantId>&id=<groupId>
where a group owner can renew group.

Q: If one of the owners renewed the group – what will happen with notifications sent to other owner? What if other owner click “Renew group” or “delete group”?
A: Notifications sent will stay. Since buttons in the email are just links (not actionable buttons) – user will be redirected to a web-page where he/she will be able to renew or delete the group.

Q: As per MS: “Groups that are actively in use are renewed automatically around 35 days before the group expires. In this case, the owner does not get any renewal notifications. Any of the following actions will automatically renew a group…<list of actions>”. So, what exactly does “Groups that are actively in use” mean?
A: This is not disclosed by Microsoft. They only say “Azure Active Directory (Azure AD), part of Microsoft Entra, uses intelligence to automatically renew groups based on whether they have been in recent use. This renewal decision is based on user activity in groups across Microsoft 365 services like Outlook, SharePoint, Teams, Yammer, and others.” Btw, <list of actions> includes almost all user actions – so basically any action – even just visit site/team is considered as activity.

Q: Can I track the policy in action via audit log?
A: There is no “activity type” for this policy’s specific actions… You also cannot specify user “msgroupsteam@microsoft.com” to get all activities. So no tracks on the policy “before action” – i.e. at the detection and e-mailing stage. If a user clicks “renew” button or “delete group” link – this should be logged as this user action. If it happens that the group is deleted by policy – this should be logged under policy’s account – I’ll follow up on this. TBC.

Q: After the group is deleted, who can restore it?
A: MS says: “A deleted Microsoft 365 group can be restored within 30 days by a group owner or by an Azure AD administrator”.
In fact, SharePoint admin,

Q: What if a user forward this e-mail notification to other user? Can this other user renew or delete the group?
A: When a user receive a notification email forwarded, and he/she click “Renew group” button – his/her experience will be the same if he/she is also a group owner. If a user is not a group owner – he/she will get “You don’t have permission to renew this group because you’re not an owner. To renew , contact a group owner. You can close this window now.”:

Note: if a user with active groups administration permissions receives email and try to renew or delete the group – he/she will also be able to do that.

Q: Can user get information on groups he/her owns, groups expiration data? Can user renew the group before the policy trigger email notification?
A: yes, all that can be done from the page: https://myaccount.microsoft.com/groups/groups-i-own

Scenario with many existing inactive groups

Let say we have a large Microsoft 365 environment with many inactive groups, some of them are inactive for a long time – e.g. 1 or 2 years. We want to implement groups expiration policy, but we want to understand better the policy behavior.

Microsoft says: “The expiration period begins when the group is created, or on the date it was last renewed” and “When you change the expiration policy, the service recalculates the expiration date for each group. It always starts counting from the date when the group was created, and then applies the new expiration policy.”
So in case we implement the policy first time, we know that Renewal Date for all groups is just a Group Creation Date.

Q: What will happen if I activate the policy – will the policy immediately start generating emails to the owners of inactive groups or the policy will wait for expiration period minus 30 days?
A: Immediately. Once activated – policy starts detecting expired groups and sending notifications.

Q: Which groups the policy will be triggered against? All or Inactive only?
A: As per Microsoft, if at around 35 days before expiration it will be determined that group is actually active, the policy can renew the group automatically.
But definition of this activity is not disclosed and might be not the same as group activity status 90 days based on MS Graph data you can see at CA.
(I got notifications for groups that were not active recently but with Active status).

Q: In the case above – what would be the deadline? When the policy will delete the group?
A: If the group expiration period is passed, but the policy was just activated – it does not delete the group immediately. Policy allows ~30-35 days for owners to renew the group.
E.g. My test policy was activated May 3 and I got message for old group immediately, but it said that the group will be deleted on June 7.

Q: What if there are more than 10K emails – will it trigger Exchange throttling?
A: Most likely emails not sent will be sent next day.

Q: Can I specify a distribution list in the policy as an “Email contact for groups with no owners”?
A: Yes

Q: Can I specify an external e-mail address as an “Email contact for groups with no owners”?
A: TBC

Q: Can admin ask user to renew or delete the group by some other custom solution (skipping the policy)?
A: yes. Actually, “Renew group” button is just a link to the following Url:
https://account.activedirectory.windowsazure.com/Group/RenewGroup?tenantId=<tenantId>&id=<groupId>
where <tenantId> is tenant id and <groupId> is group Id. So basically anyone

Microsoft 365 Groups object model

Let me explain the policy behavior in m365 group object model terms.

There is a group property “RenewedDateTime”. When group is created – this property is set up to group created date/time (same as group CreatedDateTime property value).
For the notification purposes the policy calculates “Expected Expiration DateTime” as RenewedDateTime plus “Group LifeTime” (number of days specified in policy, e.g. 180). First notification is triggered about 30 days before “Expected Expiration DateTime”, so the policy simply selects groups with RenewedDateTime property value less then current DateTime minus “Group LifeTime days” minus 30 days and sends notification starting from oldest group:

RenewedDateTime < Today - GroupLifeTime -30

When owner confirms group is still needed – RenewedDateTime is setup to current DateTime.

Q: When a user chose to “Renew group” – will it impact group activity?
A: No. If a user did not visit group – but just clicked “Renew group” – it will not trigger group last activity date. E.g. inactive group will still be inactive.

Q: Is there an API to configure Microsoft 365 groups expiration policy programmatically?
A: Yes, in MS Graph API it is called Group Lifecycle Policy: groupLifecyclePolicy

Q: Can I programmatically renew the group (all groups) as an admin?
A: Yes, consider using PowerShell 7 and PnP.PowerShell module.
PnP Doc says Reset-PnPMicrosoft365GroupExpiration command “Renews the Microsoft 365 Group by extending its expiration with the number of days defined in the group expiration policy set on the Azure Active Directory” – but that does not seem accurate. This command sets up “RenewedDateTime” group property to the current datetime, not related to current policy settings (the policy might even not have been activated).

Q: Is it possible to setup “RenewedDateTime” property to another date/time of my choice (not today)?
A: I could not find a way so far… It says
Property 'renewedDateTime' is read-only and cannot be set.

Q: What permissions are required to renew the group with Reset-PnPMicrosoft365GroupExpiration?
A: TBD

Screenshots

Notification e-mail that comes to group owners “as is” – web outlook view:

Notification e-mail that comes to group owners when content is unblocked (web outlook):

Notification e-mail that comes to group owners when pictures are loaded (desktop Outlook):

Notification e-mail that comes to group owners some key areas:

And I’d add that e-mail says how many members in this group (number of members, not including owners… i.e. if you are the only owner – it’ll be zero members).
Correction: “Renew group” is not an actionable button – it is just a html button with a link.

Screenshot of the notification that comes to email specified in policy for the groups that does not have owners:

References

SPO Site LastContentModifiedDate vs LastItemModifiedDate vs LastItemUserModifiedDate vs Graph LastModifiedDateTime

How do we know when the SharePoint site was last updated?

We have multiple “when the site was modified last time” properties – e.g. some we can retrieve with SharePoint CSOM:

  • Site LastContentModifiedDate
  • Web LastItemModifiedDate
  • Web LastItemUserModifiedDate

Also we can get MS Graph site LastModifiedDateTime property and use “Last activity” field via Admin Center GUI

And we can modify site in a multiple ways – update document or list item on the site, change library or site settings, configure site permissions, assign site sensitivity label, setup site property and so on.

Question: which site “last modified” properties reflect what events/actions?

This might be important if we think of retention policies, or any kind of clean-up processes… Let say, we are getting report on abandoned sites (inactive sites), but we are also assigning sites sensitivity labels, or we are updating site custom properties (e.g. for adaptive scopes), we have an ownerless groups policy working etc.

What if we assign site sensitivity label to an inactive (5 years old site) – would it affect site retention since site was updated?

Results

So i did some tests and based on detailed results below, it seems like

  • Web LastItemModifiedDate is triggered when user visited site (but not LastItemUserModifiedDate)
  • If a document or list Item updated by user or app – all properties are triggered
  • MS Graph site property LastModifiedDateTime, root web property LastItemModifiedDate and Site LastContentModifiedDate – same values
  • If site custom property is updated – it does not affect any site “last modified” property
  • The same for sensitivity label updated by app – it does not affect any site “last modified” property
  • The same for Microsoft ownerless groups policy – when user accept or decline group membership – no site “last modified” properties are changed (the same is true for Microsoft 365 group last modified date/time property).

Please refer to the table below

Detailed test results

Test results if the event triggers property update:

EventLast Content Modified DateLast Item Modified DateLast Item User Modified DateGraph Last Modified DateTimeGUI Last activity
Page viewed by userYesYesNoYes
Home Page viewed by user
Site Page viewed by user
Document or list item updated by userYesYesYesYes
Document or list item updated by appYesYesYesYes
Site config settings updated by user
Site config settings updated by app
Site custom property updated by appNoNoNoNo
Site Sensitivity label updated by user via SharePointYesNoNoNo
Site/Group Sensitivity label updated by user via Teams
Site/Group Sensitivity label updated by user via AzureNoNoNoNo
Site Sensitivity label updated by appNoNoNoNo
Site collection admin updated by userYesYesNoYes
Site collection admin updated by appYesYesNoYes
SharePoint group membership updated by userYesYesNoYes
Standalone Site connected to a group by userYesYesYesYes
Add Microsoft Teams to Site by UserYesYesYesYes
Update m365 group membership via M365 admin console by adminYesYesNoYes
Update m365 group membership via Azure by admin
Update m365 group membership via Teams by userNoNoNoYes
Update m365 group membership via App
Accept group ownership invitation sent by ownerless groups policyNoNoNoNo
Decline group ownership invitation sent by ownerless groups policyNoNoNoNo

Microsoft 365 Retention Policies SharePoint Adaptive Scopes Advanced Query

Basic query is available as GUI:

where you can use objects: “Site Url”, “Site Name” and “Refinable String 0″..”Refinable String 99”. Conditions would be “is equal to”, “is not equal to”, “starts with” and “not starts with”. Or you can select “Advanced query builder” and enter KQL query.

Advanced query builder

Advanced query builder allows us to use more site properties then “Site Url”, “Site Name” and “Refinable Strings” and more conditions than “is (not) equal to” and “(not) starts with”.

E.g. we can use “Title”, “Created”, “Modified” site properties and “=”,”:”,”<“, “>”, “<=”, “>=” conditions.

Working queries examples:

created>=2022-07-21
modified>1/31/2023
created>12/31/2021 AND modified>=7/31/2022
created<=2020-11-15 OR modified>2023-02-06 (?)
created<=2020-1-15 OR modified>2023-01-31 (?)
created<=11/15/2020 OR modified>1/31/2023
title:test
SiteTitle:test
RefinableString09:Test*
RefinableString09<>Test
RefinableString09=Birding AND RefinableString08<>Included


Not working queries examples:

site:https://contoso.sharepoint.com/sites/test* 
RefinableString11 = Birds # (do not use spaces in advanced query)
Path:https://contoso-my.sharepoint.com
Template:STS
Template:"SITEPAGEPUBLISHING#0"
Template:SITEPAGEPUBLISHING*
? RefinableString09<>Birding AND RefinableString08:Official
modified>31/1/2023 (should be like modified>2023-01-31
)

Query against custom site property (aka property bag value)

You can create custom site property and assign value to the property with
Set-PnPAdaptiveScopeProperty or Set-PnPPropertyBagValue.
Property must be with “Indexed” parameter. Once the property is set up, m365 search crawls site and creates crawled property. Then you map crawled property to some pre-created refinable string managed property. You can assign alias to this managed property.

In my test scenario I used RefinableString09 with alias SiteCustomSubject.

Site property valueQueryresult
BirdingRefinableString09:Birddoes not work
BirdingSiteCustomSubject:Birddoes not work
BirdingRefinableString09:Bird*works
BirdingSiteCustomSubject:Bird*does not work
BirdingRefinableString09:Birdingworks
BirdingSiteCustomSubject:Birdingdoes not work
BirdingRefinableString09:Birding*works
BirdingRefinableString09=Birdingworks
BirdingRefinableString09=Birddoes not work
BirdingRefinableString09=Bird*does not work
BirdingSiteCustomSubject=Birdingdoes not work
RefinableString09<>Birdingworks
RefinableString09=Birding AND RefinableString08<>Includedworks

Query against multi-value property.

Site property valueQueryresult
TestA TestBRefinableString09:TestAworks
TestA TestBRefinableString09 = ‘TestA TestB’does not work
TestA TestB??? RefinableString09=’Test10 Test5′does not work
TestA TestBRefinableString09:TestB ?
TestA,TestBRefinableString09:Test*works
TestA,TestBRefinableString09=Test*does not work
TestA,TestBRefinableString09:Testdoes not work
TestA,TestB
TestA;TestB
TestB TestA
TestA TestB
RefinableString09:TestBworks
TestA, TestB
TestB,TestA
TestA TestB
RefinableString09=TestAdoes not work
TestA,TestB(basic) RefinableString09 starts with testworks

Some more findings

Modify adaptive scope

If you need to modify adaptive scope – you’d better delete it and create a new one. The reason – if you want to validate what sites are included in scope with GUI – via button “Scope details” – you want to see only sites that are in scope, but that’s not the case when you modify the scope, because if you modify the scope – you’d see sites that are not in scope with “Removed” status.

Alternatively you can use filter to filter out removed from scope sites.

what else?

What is the takeaway from this for SharePoint administrators? We would be asked to configure SharePoint the way compliance…

References

SharePoint Sites Lookup

That’s a very common problem in SharePoint world. You are looking for a site owner but there is no tool available for regular user to find who owns the site.

Scenarios.

You get a link to some SharePoint site, but you do not have access to it. You requested access but nobody has responded. You need to find who is the site owner.

(To be continued)

Manage result layouts for SharePoint results in Microsoft Search

Microsoft is improving Search (MC489165):

Manage result layouts for SharePoint results in Microsoft Search

We’re making changes to Microsoft Search. This update will allow Microsoft Search administrators to change result layouts for select SharePoint content using adaptive cards with Result Type feature in Microsoft Search administration.

The default result layouts for SharePoint sites, pages, list items and Portable document format (PDF) results can now be replaced with layouts built using adaptive cards. The changes can be made for Organization level search applicable to Office.com and SharePoint home as well as site level search on SharePoint sites. Changes for Microsoft Search in Bing will be rolled out soon. Note that the feature does not support changing of Office file search results.

This message is associated with Microsoft 365 Roadmap ID 81952

Before the change, when you add a new result type under “Search and intelligence” Customizations – it looked like this:

result type content sources

So there was no built-in “SharePoint” content source as an option – only custom “external” data sources.

But with the new feature implemented list of content sources for the result type will look like this:

SharePoint and OneDrive content source

If you choose “SharePoint and OneDrive” content source – the next option would be to select type of content:

Select type of content and set rules

You also can create different result types for different types of content based on properties-based rules (e.g. one result type for all sites – and a separate result type for a specific site or hub) with optional “Set rules for this type of content”:

Default site result experience would look like

Search results with modified SharePoint result type might look like:

When you modify template via Layout Designer – it is essential to know available object properties.

You can get properties from the “Available properties” below – there is also search through properties feature.

Or you can use SharePoint Search Query Tool to get metadata on search results.

It might take hours and even days for your search to start showing new layouts, but “&cacheClear=true” should help.

DepartmentId 

If your sites are organized in hierarchy under Hub site – you can use DepartmentId managed property to include all hub-associated sites content

DepartmentId is just a hub site Id

… to be continued …

References