Inactive Microsoft 365 groups, teams and SharePoint sites remediation

In Microsoft 365 any users create teams, private/public channels, yammer communities with SharePoint sites behind, as well as standalone SharePoint sites, so in time we – SharePoint engineers – are getting more and more inactive/abandoned groups, teams and SharePoint sites. Dealing with inactive Teams and SharePoint content – as part of Microsoft 365 governance – is a cumbersome, ungrateful and demanding, but necessary and also challenging work, especially in large organizations.

To keep growing content under control – Orgs can use:

In enterprises it’s most likely all of them.

Built-in Microsoft 365 group expiration policy

There is a “Microsoft 365 groups expiration policy” that comes with every tenant and can help remove unused groups from the system, but since all Teams and Yammer sites are group-based – it also helps SharePoint admins make things cleaner.

This policy does a very simple job: on a regular basis it sends notifications to group owners so a group owner can renew the group, otherwise the group will expire and be deleted. Active groups are renewed automatically.

Although it looks simple, there are some tricks and gotchas, so I highly recommend to check this article, especially if you are going to enable the policy in an existing environment.

3-rd party tools

There are many 3-rd party tools on the market that helps with Microsoft 365 administration, information management and governance, e.g. ShareGate, AvePoint, Quest ControlPoint, SysKit Point etc.

Obviously, these 3-rd party tools exist because they can do what Microsoft ootb cannot do or they can do it better then Microsoft. For example

  • Archive teams, SharePoint sites (this is a smart idea, because for instance, a site owner feels like content is not relevant anymore, but cannot take a risk to delete the site – so site owner can choose to archive the site – i.e. keep it for a while but not use it.
  • Delegate tasks to managers or to assigned groups. This is another example of careful attitude to content – e.g. if site owner fails to respond (there might be many reasons) – a tool can reach out to user’s manager or to dedicated resolution group (vs deleting resource blindly) – so somebody can act responsibly and keep the resource or archive or delete it.

Custom solutions – PowerShell scripts, MS Graph API

PowerShell is your best friend when it comes to automation or repeated work or massive updates or ad-hoc reports etc. And PowerShell is your last resort to do something unique, what no other tools can do. PowerShell is very capable and allows to build pretty complex custom solutions.

Microsoft Graph API is a rapidly developing by Microsoft API to manage and work with data in all Microsoft 365 services. You can call MS Graph API from any popular programming language, including PowerShell. Consider Microsoft Graph PowerShell SDK.

PnP – community-based project (not officially supported, but backed by Microsoft). They are doing a really good job providing us with knowledge, guides, tools and SDKs to code against Microsoft 365, including
Microsoft Graph SDK
Microsoft Graph Toolkit
PnP Core SDK
PnP.PowerShell
PnPjs

You can use PowerShell to (just a few examples, but there are more scenarios):

  • find and protect resources that you do not want to be a part of policies, but want to deal with manually, on individual basis, e.g.
    – sites or teams owned by top management
    – sites or teams with extremely large content
    – sites or teams with sensitive or other kind of important content
  • implement Microsoft 365 group expiration policy graciously, e.g. step-by-step, via small batches (e.g. updating RenewedDateTime group property would allow you to control when this group will expire and Microsoft start sending notification)
  • deal with non-group-based resources (standalone SharePoint sites), etc.

Solution stack might include VS Code, Microsoft Graph API, Azure Functions, Azure Key Vault, PowerShell, C#, PnP.PowerShell etc.

Some more things to consider on the subject

Remediate ownerless SharePoint/Teams resources.

It’s obvious that when you are trying to clean-up inactive resources – you are working with resources owners. Inactive ownerless resource will be simply deleted. That means that before implementing any kind of inactive resources policies – you’d make all efforts to find an owner for every resource to ensure that no important information will be lost in your environment.

I have multiple publications and videos on how to manage ownerless groups in Microsoft 365, including “Deep dive in ownerless Microsoft 365 groups policy“.
Here is the list of orphan Microsoft 365 resources articles.

Retention policies

Though retention policy is something that lives under Purview center (out of SharePoint scope), you should always be aware of it and consider retention configuration, so your settings do not conflict with retention settings.

Microsoft SharePoint Premium (SharePoint Advanced Management)

This is new, announced in April 2024 functionality (licensed separately). Among other benefits, it allows:

Manage site lifecycle policies – an inactive site policy that automatically detects inactive sites and sends notifications to site owners. Sounds familiar? Yes, the same as “group expiration policy” but applied to standalone also. Learn more…

Microsoft 365 Archive

Again, new functionality. MS says: “Keep your SharePoint content in Microsoft 365 with cost-effective, long-term cold tier storage – without sacrificing manageability, security, and compliance.” In fact, archived sites are no longer accessible by anyone in the organization outside of Microsoft Purview or admin search. Learn more…

Microsoft will be hiding inactive channels.

To keep users channels list relevant, Teams will automatically detect inactive channels user haven’t interacted with in a while, and automatically hide them. Users will have an option to review the list of channels and unhide some or all of them, opt out of automatic hiding from settings, or initiate this process on demand.
Feature ID: 325780

FastAPI on Azure Functions with Azure API Management

Following Pamela Fox tutorial “FastAPI on Azure Functions with Azure API Management“.

The idea is to deploy FastAPI to Azure Functions the way auto-generated interactive documentation would be public, but actual API would be protected. Pamela solved it with Azure API Management and subscription keys:

“One of my goals was to have the documentation be publicly viewable (with no key) but the FastAPI API calls themselves require a subscription key. That split was the trickiest part of this whole architecture, and it started at the API Management level.”

Pamela published it in 3 parts:
– the idea and solution explained under her blog: FastAPI on Azure Functions with Azure API Management
– code and some initial steps at GitHub: pamelafox/fastapi-azure-function-apim
– video with more deploying details at YouTube: Deploying FastAPI app to Azure Functions + API Management

I will just repeat all the steps in this one-pager.

Environment I use: Linux Ubuntu + VS Code with “Dev Containers” extension and azd

  1. Clone https://github.com/pamelafox/fastapi-azure-function-apim
  2. Start visual studio code and reopen the project with container
  3. ensure it works locally with
    PYTHON_ISOLATE_WORKER_DEPENDENCIES=1 func host start
  4. Deploy it to Azure functions with (you’d answer questions):
    $ azd auth login
    $ azd init
    $ azd up
  5. Go to API management Service, Subscriptions, Add subscription, copy the key (secure it)
  6. From API management service, Overview – open Gateway URL and append it with “/public/docs”
  7. Try GET /generate_name as is – you’ll get “401”
  8. Try the same with subscription key – you’ll get “200”
  9. Save Request Url to call the API from your front-end app

Nest steps:

  • calling other APIs
  • connecting to Databases
  • using secrets

Ownerless Microsoft 365 groups policy in large environments

Usually Microsoft 365 group can be created by anyone in your org as part of creation a team, Yammer community, Outlook group, SharePoint site etc. If the group owner lefts the company and account got deleted – the group became ownerless.

It would be a nightmare if we’d reach ownerless groups members peron-to-person trying to find out who is a real data owner and who should be a group owner. So we need some kind of automated way.

There is a Microsoft’s ownerless groups policy that detects ownerless groups and sends emails the most active groups members with the question- if they want to become a group owners and in case member accept ownership – policy automatically elevates a person from a group member to group owner. Policy does not cover standalone sites, but majority of orphaned resources in org are usually m365 groups, so that policy should help.

The policy was designed to prevent ownerless groups concept in mind, i.e. to deal with ownerless groups gradually – stretched in time – when they become ownerless. So it is actually recommended to activate the policy once you get the tenant right away. Configuration is done via GUI, it is intuitive and straightforward. Microsoft documented it well, but if you still have questions regarding the policy behavior – here is my Q&A on what is not covered by Microsoft’s FAQ as well as some tips and tricks and gotchas…

The problem is that Microsoft introduced this feature just recently, and if you own the tenant for years, you probably already have some ownerless groups. In small and medium environments with a few dozens of ownerless groups it’s not a big issue, but in a large Microsoft 365 SharePoint Online environment you might end up having hundreds and thousands of ownerless (orphaned) resources you have to deal with.

The challenge is how to implement the policy correctly if there are already many ownerless groups present and then to take care of groups that will become ownerless in the future. Yes, we’d need to address two consecutive issues:

  • Remediate vast amount of existing ownerless groups
  • Prevent groups to become ownerless

Obviously we’d need two different strategies and policies configurations.

There are also 3-rd party tools – like SysKit Point that can help with orphaned resources by enforcing minimum number of owners. There is also “Orphaned resources” policy under SysKit that allows multiple workflow options to resolve the issue – but there is no “fully automated” option -all SysKit options require an interaction from admin/manager.

Microsoft 365 built-in feature – “Ownerless groups policy” allows fully automated process:

  • detects ownerless groups, and for every group found
  • generate e-mail invitations to most active group members
  • assigns users as group owners if they accept invitation

Another problem in large environments is we have strict requirements we want to satisfy:

  • end-users to get only a few emails in a certain period so they can process it
  • end-users get only relevant messages so they will not ignore further notifications
  • high percentage of acceptance and (ideally) no orphaned resources

We want the policy to be tested in production but within a small group first and then we want phased implementation – so we could have a chance to get a feedback on phase 1 and adjust our approach at phase 2 etc.

The policy allows limiting policy scope in two ways:

  • by limiting “who can receive” messages – it’s done by specifying a security group – so only this security group members will be eligible to get invitation and accept or decline it
  • by limiting Microsoft 365 groups that would be in scope for the policy – it’s done by specifying group names

Two options can be specified in the same policy and effective eligible members would be those who satisfy both requirements.

Configuration is done using GUI – i.e. there is no PowerShell commands known on the subject at the moment.

There are a lot of “what if” questions regarding the policy – most of them are resolved in Microsoft’s “Microsoft 365 ownerless group policy FAQ” and my Ownerless m365 groups Q&As, gotchas, findings…

But the most important gotcha for me is that we do not have a chance to re-configure the policy or re-activate it to get more messages for the groups all messages were generated earlier. I.e. if an e-mail messages were generated for a group and the policy stopped working after a specified period of time – it’d done forever. No more e-mails could be generated for the same group.

The other limit is you can specify maximum 50 m365 groups in policy under Apply policy to Specific groups option. And we’d keep in mind exchange’s limit of 10k emals per day.

So, having this said, what would be the proper approach to do phased implementation in terms of configuring policy to scope it down for each step?

First – know your data. Get full report on ownerless groups, analyze it and come up with approach. Let’s assume we have an org with ~100K users and ~5000 ownerless groups. I bet you will find out that you have

  • large m365 groups (50+ members): <1%, i.e. 10-20 groups
  • medium m365 groups (5-50 members): ~25%, i.e. ~1000-2000 groups
  • small m365 groups (1-5 members): ~50%, i.e. ~2500 groups
  • null m365 groups (0 members): ~25%, i.e. ~1000-2000 groups

You’d might have your own classification, but I would propose the following approach to each category.

  • large groups:
    configure policy with “Apply policy to Specific groups” option
    and specify all or several of your large groups (the limit if 50 allowed groups in this field)
  • medium groups:
    configure policy not scoped down (e.g. apply to all groups, all users)
  • small groups:
    elevate all group members to owners
    optionally – elevate specific titles (manager, lead) or salary grade members to owners
  • null groups:
    consider deleting these groups
    optionally – delete only inactive no-members groups or groups with no or small amount of storage/files.

You’d also come up with the ideas on

  • desired min and max number of owners
  • deleting groups/sites phased approach
  • archiving groups/teams/sites

Remember – this is production, so at this moment you should test the policy in non-prod an be fully comfortable with all aspects of configuring the policy and formatting e-mail template etc.

As a remediation part plan I would propose the following:

(WIP)

Wave 0 – piloting

select a few (3-5) ownerless m365 groups came from IT – whose members are your pilot team members, so you could finalize all settings and polish notification message etc.

Implement the policy with settings:

In parallel, while you are waiting weeks for the policy to pause, start developing PowerShell scripts that will 1) delete null (no members) groups and 2) elevate members to owners (get how many members can be elevated if elevate only certain members)

Track user’s response – % of declines and accepts

Get feedback from users – how well the notification message is understandable

Wave 1 – large groups and small groups

Implement the policy with settings:

In parallel, you should already know – how many members can be elevated if elevate only certain members, decide on that and and run PowerShell script that elevates members to owners.

Wave 2 – medium groups and null groups

Implement the policy with settings:

In parallel, run PowerShell script removes groups with no owners and no members (optionally inactive and/or no content).

Wave 3 – all groups left ownerless

Implement the policy with settings:

Wave 4 – permanent policy and deletion script

Implement the policy with settings:

two more moments to consider:
– After all the measures against ownerless groups is done, we will probably still have some groups ownerless
– We will be getting new ownerless groups permanently – during all the waves of policy implementation

Qestions and Answers

Q: Isn’t it a security risk if we elevate members to owners? Would a member get access to more information that he/she did have access to before.
A: 1) Elevating members is the same risk as implementing the ownerless policy, as policy does the same – it elevates member to group owner.
2) When a member is elevated to group owners – a member does not get access to more information, as
a) for standard channels – he/she did have access as a member
b) private channels stays private – new group owner dos not get access to private channels automatically
c) shared channels stays with the same permissions also

TBC

References

Site “Birds of Kazakhstan” has moved

One of my favorite sites – “Birds of Kazakhstan” (btw – site I support, to the best of my ability) has migrated again… The new Url is: https://kz.birding.day/

So, please welcome new “Birds of Kazakhstan” site.

Meantime, at the moment, the site hosted 100+ members, who created 2400+ blog posts, uploaded 250K pictures of 500 photographed species out of 524 registered in Kazakhstan.

The original “Birds of Kazakhstan” url: https://birds.kz/ is still in place, but it hosts “Birds of Kazakhstan Best Pictures” now.

LAMP project migration

I have been involved in a birdwatching community web project for a long time. At the moment it has 250k pictures, 2500 blog posts from community members.

The site is a custom development – written by Askar Issabekov with php and MySQL and hosted under Dreamhost dedicated server.

Initially the migration plan was

  • get and configure new hosting
  • copy php scripts and media files
  • backup/restore MySQL databases
  • update hosts files and ensure everything works good
  • set source site in “maintenance” mode
    • put notification on the site header
    • set databases in read-only mode
  • backup/restore MySQL databases again
  • copy media files delta
  • set databases in r/w mode
  • ensure everything works good
  • change DNS from old host to new host

Some more data:

  • MySQL database: 120 MB

It turned out that

  • Source host uses Apache and target host uses Nginx
  • phpMyAdmin cannot import database from SQL backup

So we ended up the following

  • backup and document existing environment
  • get and configure new hosting
    • ensure php version is the same
  • copy php scripts and media files
    • remove all .htaccess and update nginx config files accordingly,
      e.g. Index etc.
    • fix files permissions (chmod)
  • backup/restore MySQL databases
    • if the database is big
      • – consider splitting database – a few tables each backup/restore
      • – consider zipping database while export-import
    • if phpMyAdmin fails – use command-line mysql
    • update database connections (user names, passwords, database names and hosts)
      as temporary measure – it is possible to allow access to old host databases from new host ip
  • update hosts files and ensure everything works good
  • set source site in “maintenance” mode
    • put notification on the site header
    • set databases in read-only mode
  • backup/restore MySQL databases again
  • set databases in r/w mode
  • copy media files delta
  • ensure everything works good
  • change DNS from old host to new host

Question: to avoid hassles with hosts files – is it possible to use a different real target name, then after verifying everything works good – change name

We’d also need to take care of e-mail migration

(WIP)

Microsoft 365 Search: built-in people search by nickname

Did Microsoft silently implement nickname search?

E.g.

I have created a user “Robert Dylan” and never used name “Bob”, but search understands that I’m looking for Robert Dylan when I’m searching for Bob or Bob Dylan or Bobby.

Important thing – result is shown under “All” and “People” verticals.

I know Microsoft claimed m365 search is backed by Turing technology – it understands you, answers your question but not just blindly display keyword occurences (announcement at Ignite Spring, more on Ignite Fall 2021)… So is that it?

But the same trick did not work with Dick for Richard or (of course) Syd for Roger Barrett:

Please check “How to configure Microsoft 365 People Search by Nickname and Full Name

SPO Site LastContentModifiedDate vs LastItemModifiedDate vs LastItemUserModifiedDate vs Graph LastModifiedDateTime

How do we know when the SharePoint site was last updated?

We have multiple “when the site was modified last time” properties – e.g. some we can retrieve with SharePoint CSOM:

  • Site LastContentModifiedDate
  • Web LastItemModifiedDate
  • Web LastItemUserModifiedDate

Also we can get

  • MS Graph site object with LastModifiedDateTime property
  • get usage reports via Microsoft Graph (activity reports), and
  • use “Last activity” field via Admin Center GUI

On the other hand – we can view and modify site in multiple ways – visit site home page, open and/or update document/list item, change site/library settings, configure site permissions, assign site sensitivity label, setup site property and so on.

Question: which site “last modified” or “last activity” properties reflect what events/actions?

This might be important if we think of retention policies, or any kind of clean-up processes… Let say, we are getting report on abandoned sites (inactive sites), but we are also assigning sites sensitivity labels, or we are updating site custom properties (e.g. for adaptive scopes), we have an ownerless groups policy working etc.

What if we assign site sensitivity label to an old inactive (5 years old) site – would it affect retention policy since site was updated this way?

Results

So i did some tests and based on detailed results below, it seems like

  • Web LastItemModifiedDate is triggered when user just visited site (but property LastItemUserModifiedDate is not triggered)
  • If a document or list Item updated by user or app – all properties are triggered
  • MS Graph site property LastModifiedDateTime, root web property LastItemModifiedDate and Site LastContentModifiedDate – same values
  • If site custom property is updated – it does not affect any site “last modified” property
  • The same for sensitivity label updated by app – it does not affect any site “last modified” property
  • The same for Microsoft ownerless groups policy – when user accept or decline group membership – no site “last modified” properties are changed (the same is true for Microsoft 365 group last modified date/time property).

Please refer to the table below

Detailed test results

Test results if the event triggers property update:

EventLast Content Modified DateLast Item Modified DateLast Item User Modified DateGraph Last Modified DateTimeGUI Last activity
Page viewed by userYesYesNoYes
Home Page viewed by user
Site Page viewed by user
Document or list item updated by userYesYesYesYes
Document or list item updated by appYesYesYesYes
Site config settings updated by user
Site config settings updated by app
Site custom property updated by appNoNoNoNo
Site Sensitivity label updated by user via SharePointYesNoNoNo
Site/Group Sensitivity label updated by user via Teams
Site/Group Sensitivity label updated by user via AzureNoNoNoNo
Site Sensitivity label updated by appNoNoNoNo
Site collection admin updated by userYesYesNoYes
Site collection admin updated by appYesYesNoYes
SharePoint group membership updated by userYesYesNoYes
Standalone Site connected to a group by userYesYesYesYes
Add Microsoft Teams to Site by UserYesYesYesYes
Update m365 group membership via M365 admin console by adminYesYesNoYes
Update m365 group membership via Azure by admin
Update m365 group membership via Teams by userNoNoNoYes
Update m365 group membership via App
Accept group ownership invitation sent by ownerless groups policyNoNoNoNo
Decline group ownership invitation sent by ownerless groups policyNoNoNoNo