Tag Archives: Data lifecycle management

Implementing Microsoft 365 group expiration policy in large companies

This post is dedicated to one specific subject: implementing Microsoft 365 groups lifecycle (expiration) policy in large Microsoft 365 environments.

But this post is also a part of a bigger problem – dealing with ownerless resources in Large Microsoft 365 environments. Please refer to the umbrella post.

Scenario

You administer a large Microsoft 365 environment. Let say you have 100k users or more, 50K or more sites. Environment is not new, so after some years you have a lot of ownerless groups and sites (thousands probably), and a lot of inactive groups and sites (probably tens of thousands). You are getting more and more ownerless groups – hundreds each month. You are thinking of stopping bleeding and cleaning this up…

Implementing Microsoft 365 groups expiration policy

If you are thinking of activating in an existing environment – you would probably have a spike – all the old groups will be subject to policy. The ide is to avoid situation when a specific person – group owner will get dozens of email. It would be better if a person will receieve, let say one email per week.

Here is my 4 possible approaches to avoid this spike, distribute notifications evenly across the time and ease the pain:

By changing Group Lifetime

You would need to change the policy every, e.g. week or month, specifying group lifetime in days starting with maximum period. Consider
– calculate number of days between the oldest group created an today, plus 35 days – it’ll be your first “group lifetime”
– activate the policy with this number of days in “group lifetime” – and within a week you will get notifications on the oldest group/groups
– after a week or two – change the “group lifetime” decreasing it by e.g. 30-60 days and reactivate the policy… and so on

You can easily calculate it all and choose your pace depending on how many groups you have to renew, how much time you need to clean-up. You got the idea.

Downside – in the email notification it will be said “otherwise the group will be deleted on …”, but you joggling with lifetime period – so these dates might confuse users

By renewing groups as admin

As an admin, you can use PowerShell “Reset-PnPMicrosoft365GroupExpiration” or graph API “POST /groups/{id}/renew“ to renew any group.

So depending on total number of groups, number of active/inactive groups, number of ownerless groups in your organization – you can come up with a strategy, using one or more of the following techniques:

renew all active and known and important groups (build list of groups to re-activate based on your own criteria) and then trigger the policy
build list of definitely inactive groups and renew all other groups
split groups into chunks and every day or week (depending on numbers) renew groups in a chunk… after that you can activate groups expiration policy, the policy will be triggered against small number of groups every day/week (with this trick you would avoid policy triggering against large number of groups and sending thousands of emails at one)

By sending customized e-mails to users

Another technique to avoid surge in your e-mail system (and most importantly – avoid sudden influx of support requests) – you can send emails to group owners with a link to renew a group. Surely you’d need to build your custom solution for that, but this does not seem like a complicated task. Some PowerShell scripting – and you are good.

“Renew group” link would look like:
https://account.activedirectory.windowsazure.com/Group/RenewGroup?tenantId=<tenantId>&id=<groupId>
where <tenantId> is tenant id and <groupId> is group Id. So with PowerShell you’d just pull groups in question, their owners and send email with the link dynamically built. Consider spreading this activity across the time to avoid spikes.

By sending users to the groups page

Probably the easies option to smooth the expiration policy implementation is to send group owners to a Microsoft “Groups I own” page – where they could renew or delete their groups. The page is:

Groups I Own (https://myaccount.microsoft.com/groups/groups-i-own)

The downside is – the policy should be in active state (otherwise owners will not see when the group is expiring and option to renew it). So consider a maximum lifetime period in policy, then you’d decrease this period).

Groups expiration policy technical details

Check this KBA for a deep dive into the groups lifecycle policy technical details.

Inactive Microsoft 365 groups, teams and SharePoint sites remediation

3 Replies

In Microsoft 365 any users create teams, private/public channels, yammer communities with SharePoint sites behind, as well as standalone SharePoint sites, so in time we – SharePoint engineers – are getting more and more inactive/abandoned groups, teams and SharePoint sites. Dealing with inactive Teams and SharePoint content – as part of Microsoft 365 governance – is a cumbersome, ungrateful and demanding, but necessary and also challenging work, especially in large organizations.

To keep growing content under control – Orgs can use:

Built-in Microsoft 365 group expiration policy
3-rd party tools
Custom solutions, PowerShell scripts, Microsoft Graph API
SharePoint Inactive Site Policies (SAM)

In enterprises it’s most likely all of them.

Built-in Microsoft 365 group expiration policy

There is a “Microsoft 365 groups expiration policy” that comes with every tenant and can help remove unused groups from the system, but since all Teams and Yammer sites are group-based – it also helps SharePoint admins make things cleaner.

This policy does a very simple job: on a regular basis it sends notifications to group owners so a group owner can renew the group, otherwise the group will expire and be deleted. Active groups are renewed automatically.

Although it looks simple, there are some tricks and gotchas, so I highly recommend to check this article, especially if you are going to enable the policy in an existing environment.

3-rd party tools

There are many 3-rd party tools on the market that helps with Microsoft 365 administration, information management and governance, e.g. ShareGate, AvePoint, Quest ControlPoint, SysKit Point etc.

Obviously, these 3-rd party tools exist because they can do what Microsoft ootb cannot do or they can do it better then Microsoft. For example

Archive teams, SharePoint sites (this is a smart idea, because for instance, a site owner feels like content is not relevant anymore, but cannot take a risk to delete the site – so site owner can choose to archive the site – i.e. keep it for a while but not use it.
Delegate tasks to managers or to assigned groups. This is another example of careful attitude to content – e.g. if site owner fails to respond (there might be many reasons) – a tool can reach out to user’s manager or to dedicated resolution group (vs deleting resource blindly) – so somebody can act responsibly and keep the resource or archive or delete it.

Custom solutions – PowerShell scripts, MS Graph API

PowerShell is your best friend when it comes to automation or repeated work or massive updates or ad-hoc reports etc. And PowerShell is your last resort to do something unique, what no other tools can do. PowerShell is very capable and allows to build pretty complex custom solutions.

Microsoft Graph API is a rapidly developing by Microsoft API to manage and work with data in all Microsoft 365 services. You can call MS Graph API from any popular programming language, including PowerShell. Consider Microsoft Graph PowerShell SDK.

PnP – community-based project (not officially supported, but backed by Microsoft). They are doing a really good job providing us with knowledge, guides, tools and SDKs to code against Microsoft 365, including
– Microsoft Graph SDK
– Microsoft Graph Toolkit
– PnP Core SDK
– PnP.PowerShell
– PnPjs

You can use PowerShell to (just a few examples, but there are more scenarios):

find and protect resources that you do not want to be a part of policies, but want to deal with manually, on individual basis, e.g.
– sites or teams owned by top management
– sites or teams with extremely large content
– sites or teams with sensitive or other kind of important content
implement Microsoft 365 group expiration policy graciously, e.g. step-by-step, via small batches (e.g. updating RenewedDateTime group property would allow you to control when this group will expire and Microsoft start sending notification)
deal with non-group-based resources (standalone SharePoint sites), etc.

Solution stack might include VS Code, Microsoft Graph API, Azure Functions, Azure Key Vault, PowerShell, C#, PnP.PowerShell etc.

Microsoft SharePoint Premium (SharePoint Advanced Management)

This is a new, announced in April 2024 and available in 2025 functionality (licensed separately, as Microsoft SharePoint Premium aka SharePoint Advanced Management). Among other benefits, it allows:

Manage site lifecycle policies – an inactive site policy that automatically detects inactive sites and sends notifications to site owners. Sounds familiar? Yes, it resembles the “group expiration policy” but applied to standalone also and there are some more important differences. See my deep dive into SharePoint Inactive Site Policies.

Some more things to consider on the subject

Remediate ownerless SharePoint/Teams resources.

It’s obvious that when you are trying to clean-up inactive resources – you are working with resources owners. Inactive ownerless resource will be simply deleted. That means that before implementing any kind of inactive resources policies – you’d make all efforts to find an owner for every resource to ensure that no important information will be lost in your environment.

I have multiple publications and videos on how to manage ownerless groups in Microsoft 365, including “Deep dive in ownerless Microsoft 365 groups policy“.
Here is the list of orphan Microsoft 365 resources articles.

Retention policies

Though retention policy is something that lives under Purview center (out of SharePoint scope), you should always be aware of it and consider retention configuration, so your settings do not conflict with retention settings.

Microsoft 365 Archive

Again, new functionality. MS says: “Keep your SharePoint content in Microsoft 365 with cost-effective, long-term cold tier storage – without sacrificing manageability, security, and compliance.” In fact, archived sites are no longer accessible by anyone in the organization outside of Microsoft Purview or admin search. Learn more…

Microsoft will be hiding inactive channels.

To keep users channels list relevant, Teams will automatically detect inactive channels user haven’t interacted with in a while, and automatically hide them. Users will have an option to review the list of channels and unhide some or all of them, opt out of automatic hiding from settings, or initiate this process on demand.
Feature ID: 325780

Microsoft 365 Retention Policies SharePoint Adaptive Scopes Advanced Query

1 Reply

Basic query is available as GUI:

where you can use objects: “Site Url”, “Site Name” and “Refinable String 0″..”Refinable String 99”. Conditions would be “is equal to”, “is not equal to”, “starts with” and “not starts with”. Or you can select “Advanced query builder” and enter KQL query.

Advanced query builder

Advanced query builder allows us to use more site properties then “Site Url”, “Site Name” and “Refinable Strings” and more conditions than “is (not) equal to” and “(not) starts with”.

E.g. we can use “Title”, “Created”, “Modified” site properties and “=”,”:”,”<“, “>”, “<=”, “>=” conditions.

Working queries examples:

created>=2022-07-21
modified>1/31/2023
created>12/31/2021 AND modified>=7/31/2022
created<=2020-11-15 OR modified>2023-02-06 (?)
created<=2020-1-15 OR modified>2023-01-31 (?)
created<=11/15/2020 OR modified>1/31/2023
title:test
SiteTitle:test
RefinableString09:Test*
RefinableString09<>Test
RefinableString09=Birding AND RefinableString08<>Included

Not working queries examples:

site:https://contoso.sharepoint.com/sites/test* 
RefinableString11 = Birds # (do not use spaces in advanced query)
Path:https://contoso-my.sharepoint.com
Template:STS
Template:"SITEPAGEPUBLISHING#0"
Template:SITEPAGEPUBLISHING*
? RefinableString09<>Birding AND RefinableString08:Official
modified>31/1/2023 (should be like modified>2023-01-31
)

Query against custom site property (aka property bag value)

You can create custom site property and assign value to the property with
Set-PnPAdaptiveScopeProperty or Set-PnPPropertyBagValue.
Property must be with “Indexed” parameter. Once the property is set up, m365 search crawls site and creates crawled property. Then you map crawled property to some pre-created refinable string managed property. You can assign alias to this managed property.

In my test scenario I used RefinableString09 with alias SiteCustomSubject.

Site property value	Query	result
Birding	RefinableString09:Bird	does not work
Birding	SiteCustomSubject:Bird	does not work
Birding	RefinableString09:Bird*	works
Birding	SiteCustomSubject:Bird*	does not work
Birding	RefinableString09:Birding	works
Birding	SiteCustomSubject:Birding	does not work
Birding	RefinableString09:Birding*	works
Birding	RefinableString09=Birding	works
Birding	RefinableString09=Bird	does not work
Birding	RefinableString09=Bird*	does not work
Birding	SiteCustomSubject=Birding	does not work
	RefinableString09<>Birding	works
	RefinableString09=Birding AND RefinableString08<>Included	works

Query against multi-value property.

Site property value	Query	result
TestA TestB	RefinableString09:TestA	works
TestA TestB	RefinableString09 = ‘TestA TestB’	does not work
TestA TestB	??? RefinableString09=’Test10 Test5′	does not work
TestA TestB	RefinableString09:TestB	?
TestA,TestB	RefinableString09:Test*	works
TestA,TestB	RefinableString09=Test*	does not work
TestA,TestB	RefinableString09:Test	does not work
TestA,TestB TestA;TestB TestB TestA TestA TestB	RefinableString09:TestB	works
TestA, TestB TestB,TestA TestA TestB	RefinableString09=TestA	does not work
TestA,TestB	(basic) RefinableString09 starts with test	works

Some more findings

Modify adaptive scope

If you need to modify adaptive scope – you’d better delete it and create a new one. The reason – if you want to validate what sites are included in scope with GUI – via button “Scope details” – you want to see only sites that are in scope, but that’s not the case when you modify the scope, because if you modify the scope – you’d see sites that are not in scope with “Removed” status.

Alternatively you can use filter to filter out removed from scope sites.

what else?

What is the takeaway from this for SharePoint administrators? We would be asked to configure SharePoint the way compliance…

References

Adaptive scopes Retention Policies Data Lifecycle Purview

Adaptive scopes Retention Policies Data Lifecycle Purview

1 Reply

Microsoft recently implemented “Adaptive” retention policies. At step 2 of “Create retention policy” you’ll be asked “Choose the type of retention policy to create”: “A policy can be adaptive or static. Advantage of an adaptive policy will automatically update where it’s applied based on attributes or properties you’ll define. A static policy is applied to content in a fixed set of locations and must be manually updated if those locations change.”

And if you selected “Adaptive” – on the next step you will need to provide the adaptive scope (so at this moment you should already have created your adaptive scopes):

So, let us create your adaptive scopes.
What type of scope do you want to create? SharePoint sites…

And then you’ll have nothing more then set of conditions:

Advanced query builder for SharePoint Adaptive Scope

Microsoft 365 Retention Policies SharePoint Adaptive Scopes Advanced Query

Configure Site Properties and Refinable Strings for Adaptive Scopes

1 Reply

Microsoft recently implemented “Adaptive Scopes” for retention policies”. Before that we had to use “static” scopes only, i.e. we could apply the policy to all sites or to specific selected sites we had to choose manually. With adaptive scopes we can use rules like “This adaptive scope must include all sites with Site Url starts with A or Site name starts with A” and so on. And then we’d apply the retention policy to all the sites in this adaptive scope. This is nice, but actually site Url and site name does not have much to do with sites categorization for the retention policies. How can we implement sites classification to apply different policies to different sites categories? Luckily, when you configure adaptive scopes, you can use Refinable Strings, and refinable strings is something you can configure to have values from custom site properties. So finally we can assign specific value to custom site property and the site would fall under this or that retention policy based on the value we dynamically assigned to the site.

Note: you can also use site properties like date site created or site last modified date in advanced query builder under adaptive scopes – please check “Microsoft 365 Retention Policies SharePoint Adaptive Scopes Advanced Query“.

How to configure SharePoint custom site properties the way the retention policies adaptive scopes can consume Refinable Strings?

The steps are:

Create an indexed site property with values
Map crawled property to a refinable string managed property

Detailed steps:

Indexed site property

Create an indexed site property or “Adaptive Scope Property” with some values. Ensure you property name (key) is unique, e.g.

Property	Value
SiteRetentionCategory	Y10

You can use PowerShell (with PnP.PowerShell module) commands:
Set-PnPAdaptiveScopeProperty or
Set-PnPPropertyBagValue -Indexed:$true. Examples:

Set-PnPAdaptiveScopeProperty -Key "SiteRetentionProperty" -Value "Y10"

Wait until search crawler picked up you site property. Now you have a crawled property.

Search schema mapping

As you know, Refinable Strings are just pre-created by Microsoft refinable managed properties. So you can select one that is not used(*) and map it to crawled property.
You can assign alias so you could easily identify what is the RefinableString55 about (but aliases do not work in advanced query).

(*) Notes

select one that is not used
select one that is not used is an important, bacause if you select refinable string that is already taken at the some site level – there is a conflict. So before configuring pre-created refinable properties at tenant level – I’d recommend to get report on managed properties taken at sites levels. It would be good idea if you arrange with sites owners on properties ranges (e.g. from 00 to 99 – reserved for tenant use, from 100 to 199 – available at sites level search customizations). And/or you can – after getting report on managed properties taken at sites levels – reserve all unused managed properties by assigning aliases e.g. “this-property-55-is-reserved-by-admin-for-tenant-level-config”.

site custom script
If site custom scripts are enabled (DenyAddAndCustomizePages = false), then site collection admin can change site properties. So if you do not want the property being altered at site level – ensure that noscript site property is enabled (DenyAddAndCustomizePages equals true)

If site custom scripts are disabled (DenyAddAndCustomizePages = true), then an admin must enable them before using “Set-PnPPropertyBagValue” cmdlet (then disable again).
“Set-PnPAdaptiveScopeProperty” cmdlet handles this automatically.

References

Office 365 retention labels and policies for SharePoint

As I am a SharePoint person, and retention policies and labels are not a SharePoint engineer responsibility, I do not go to the m365 Compliance Center frequently. Below are My notes for myself on key moments – how to create and configure Office 365 retention labels and Policies at Compliance Center and use labels in SharePoint Online (SPO).

In SPO at each site collection level you can still work with retention policies the old way – create policies under Site Collection Settings – Content Type Policy – and apply policies at library level under Library Settings/Information Management Policy Settings. There is also Site Retention Policy.

But Microsoft is making efforts to centralize and unify such things – so you can specify retention policies in one place and apply them across all Office 365 content (not only SharePoint). That place was called Office 365 Security and Compliance Center (SCC). Later Microsoft separated Security Center and Compliance Center. So currently Retention Policies are under “Microsoft Purview” (former Microsoft Compliance Center) -> Solutions -> “Data lifecycle management”:

To get access to “Data lifecycle management” solution – you need to have a “” or “” roles. SharePoint or Teams administrator cannot access Purview. Even having “Global reader” or “Security reader” an admin will not be able to see “Data lifecycle management” blade. Here is how Microsoft Purview looks like for a Global reader:

Although SharePoint admins usually do not have access to SCC and do not go to Site content, we still need to know how it all works. And labels are recommended way to specify retention in SharePoint, so here we are.

Labels are applied to documents, documents are kept in libraries, and at each library you can “Apply a label to items in this library”.

Create Labels

Labels are created in SCC under Classification. The main part looks familiar to SharePoint people:

You can

Retain Content forever or for a specified number of days/months/years and then
– delete it or trigger a disposition review or do nothing
Delete content if it’s older than specified number of days/months/years

after it was created/modified/labelled

Apply labels

Now you need to publish created labels – and that is how you create a policy. I.e. policies are where you specify which labels to which content (Exchange, OneDrive, SharePoint, Office 365 groups)

You can also auto-apply labels based on conditions, like

content that contains sensitive info
content that contains specific words or phrases, or properties
content that matches a trainable classifier

but as per Microsoft, “It will take up to 7 days to automatically apply the label to all items that match your conditions.”

Note: “trainable classifier” means an AI ML will be used, and as per Microsoft “Creating machine learning rules requires an Office 365 E5 subscription for your organization”

SharePoint admin center

You can do nothing with labels at SharePoint admin center. Labels are created, published and auto-applied at SCC. At each site collection levels site administrators can apply labels.

SharePoint site

At site collection settings you can still see “Content Type Policy Templates” and “Site Policy”, but that is not the case. Labels are applied at library level under Library Settings/Apply label to items in this list or library.

where you can select a label to apply for all new items in the library. With

You can also apply the label to items that already exist in the library.

You can also apply (change) label for each single item or multiple selected items under Details pop-up page:

or from under Contect Menu/More/Compliance details:

Adaptive retention policies and scopes

And if you selected “Adaptive” – on the next step you will need to provide the adaptive scope (so at this moment you should already have created your adaptive scopes):

So, let us create your adaptive scopes.
What type of scope do you want to create? SharePoint sites…

And then you’ll have nothing more then set of conditions:

What is the takeaway from this for SharePoint administrators? We would be asked to configure SharePoint the way compliance/retention people can use Refinable Strings.

How do users know – what files are going to be deleted

It would be a good idea to let users know, that their files will be deleted, but the next question would be “can I get a list of files that are scheduled to deletion”?

Surely it is possible for admins to go through site content and find files that were modified last time earlier than a specific date, but there is another method – you can recommend your users to search through their site using

See details in “How to figure out – which files are expiring due to age and going to be deleted as result of retention policy.”