Sometimes, mostly during PoC or testing policies like retention policy or lifecycle policy you would need some documents created and updated weeks, months or even years ago.
But if you create or upload a document in SharePoint library – it will be just a regular new document. So, how to get old documents in the new environment?
I see two options:
Sync with OneDrive If you sync a library with your local folder (done Microsoft by OneDrive desktop app) and put some old document in your synced folder – the doc will be synchronized back to SharePoint library with Created and Modified properties preserved.
Make the document older with PowerShell With “Set-PnPListItem” PowerShell command you can update not only such properties like Title, but also “Created By”, “Modified By” and even date and time document was created and modified via “Created” and “Modified”. Optionally you can play with document history with “-UpdateType” parameter. UpdateType possible values are:
Update: Sets field values and creates a new version if versioning is enabled for the list
SystemUpdate: Sets field values and does not create a new version. Any events on the list will trigger.
UpdateOverwriteVersion: Sets field values and does not create a new version. No events on the list will trigger
As SharePoint or Teams admin you can manage Microsoft 365 groups (create, update, delete, manage membership etc.) having your admin role activated. I use Azure registered app with “Group.ReadWrite.All” Microsoft Graph API delegated permission and Microsoft.Graph PowerShell module.
When a user was not a group member or group owner – and is added to the group members – user gets notification “You’ve joined the <Group Name> group” via e-mail that comes from a group e-mail address.
When a user is added to the group owners (or elevated to group owner if user was a group member) – user does not get notification.
When a user was a group owner and now is added to the group members – user does not get notification.
All the actions are logged into Microsoft 365 audit log under your personal Id.
Script samples:
# This script is just a sample to demonstrate basic technique on deletion m365 groups with PowerShell and MS Graph
# please do not run this script as is, but update it upon your needs
# authentication with personal Id
# app must have as minimum "Group.ReadWrite.All" Microsoft Graph API delegated permission
# user must have SharePoint admin (or Teams admin) roles activated
Connect-MgGraph -ClientId $clientid -TenantId $tenantId
Get-MgContext | Select-Object Scopes -ExpandProperty Scopes
# sample data
$groups = @()
$groups += [PSCustomObject]@{GroupId = '443d22ae-683a-4fe4-8875-7bd78227a026' }
$groups += [PSCustomObject]@{GroupId = 'e5805388-c18c-48c0-b42d-6223cf8f3d82' }
# Get Groups
foreach ($group in $groups) {
Get-MgGroup -GroupId $group.GroupId
}
# add members to the group
$groupId = '443d22ae-683a-4fe4-8875-7bd78227a026'
$userId = 'df74e0d3-d78c-495b-b47a-549437d93cf7' # Adele
New-MgGroupMember -GroupId $groupId -DirectoryObjectId $userId
# add Owner to the group
$groupId = '443d22ae-683a-4fe4-8875-7bd78227a026'
$userId = 'eacd52fb-5ae0-45ec-9d17-5ded9a0b9756' # Megan
New-MgGroupOwner -GroupId $groupId -DirectoryObjectId $userId
# Delete group
# add Owner to the group
$groupId = '443d22ae-683a-4fe4-8875-7bd78227a026'
Remove-MgGroup -GroupId $groupId
Sites.Selected permissions are needed for the non-interactive application to get access to a specific SharePoint site.
Steps to get access to SharePoint site with Sites.Selected:
1. Register an application in Azure (via Azure portal GUI, PowerShell script or helpdesk/servicedesk request)
2. Update the app so both – MS Graph API Sites.Selected and SharePoint Sites.Selected permissions are configured and consented by tenant admin
3. Obtain and upload client certificate (recommended) or generate client secret
4. Request access for the app to a specific SharePoint site (your SharePoint service admin should be able to do that via PowerShell script or Graph API calls ) Here is the Graph API
5. Validate your app has access to the target SharePoint site with PowerShell tbp
6. Secure your certificate and/or secret
Consider using vault to keep certificate/secret. If you host your application in Azure – consider using managed identity
How do we know when the SharePoint site was last updated?
We have multiple “when the site was modified last time” properties – e.g. some we can retrieve with SharePoint CSOM:
Site LastContentModifiedDate
Web LastItemModifiedDate
Web LastItemUserModifiedDate
Also we can get
MS Graph site object with LastModifiedDateTime property
get usage reports via Microsoft Graph (activity reports), and
use “Last activity” field via Admin Center GUI
On the other hand – we can view and modify site in multiple ways – visit site home page, open and/or update document/list item, change site/library settings, configure site permissions, assign site sensitivity label, setup site property and so on.
Question: which site “last modified” or “last activity” properties reflect what events/actions?
This might be important if we think of retention policies, or any kind of clean-up processes… Let say, we are getting report on abandoned sites (inactive sites), but we are also assigning sites sensitivity labels, or we are updating site custom properties (e.g. for adaptive scopes), we have an ownerless groups policy working etc.
What if we assign site sensitivity label to an old inactive (5 years old) site – would it affect retention policy since site was updated this way?
Results
So i did some tests and based on detailed results below, it seems like
Web LastItemModifiedDate is triggered when user just visited site (but property LastItemUserModifiedDate is not triggered)
If a document or list Item updated by user or app – all properties are triggered
MS Graph site property LastModifiedDateTime, root web property LastItemModifiedDate and Site LastContentModifiedDate – same values
If site custom property is updated – it does not affect any site “last modified” property
The same for sensitivity label updated by app – it does not affect any site “last modified” property
The same for Microsoft ownerless groups policy – when user accept or decline group membership – no site “last modified” properties are changed (the same is true for Microsoft 365 group last modified date/time property).
Please refer to the table below
Detailed test results
Test results if the event triggers property update:
Event
Last Content Modified Date
Last Item Modified Date
Last Item User Modified Date
Graph Last Modified DateTime
GUI Last activity
Page viewed by user
Yes
Yes
No
Yes
Home Page viewed by user
Site Page viewed by user
Document or list item updated by user
Yes
Yes
Yes
Yes
Document or list item updated by app
Yes
Yes
Yes
Yes
Site config settings updated by user
Site config settings updated by app
Site custom property updated by app
No
No
No
No
Site Sensitivity label updated by user via SharePoint
Yes
No
No
No
Site/Group Sensitivity label updated by user via Teams
Site/Group Sensitivity label updated by user via Azure
No
No
No
No
Site Sensitivity label updated by app
No
No
No
No
Site collection admin updated by user
Yes
Yes
No
Yes
Site collection admin updated by app
Yes
Yes
No
Yes
SharePoint group membership updated by user
Yes
Yes
No
Yes
Standalone Site connected to a group by user
Yes
Yes
Yes
Yes
Add Microsoft Teams to Site by User
Yes
Yes
Yes
Yes
Update m365 group membership via M365 admin console by admin
Yes
Yes
No
Yes
Update m365 group membership via Azure by admin
Update m365 group membership via Teams by user
No
No
No
Yes
Update m365 group membership via App
Accept group ownership invitation sent by ownerless groups policy
No
No
No
No
Decline group ownership invitation sent by ownerless groups policy
This PowerShell script pulls all tenant sites and all sites owners. The script require app authentication with Sites.FullControl.All and Directory.Read.All permissions. PnP.PowerShell for PowerShell 7 is used.
It generates two reports
Owners report: one user per line, include: Site Url, Title, Owner e-mail, name and type
Sites report: one site per line, include: Site Url, Title, list of owners e-mails
Bert Jansen (Microsoft) revealed some details on throttling when you access Microsoft 365 programmatically – via Microsoft Graph or CSOM and guided developers on how to regulate request traffic for optimized throughput using RateLimit headers (Here).
Demystifying SharePoint throttling
Throttling is necessary to ensure that no single user or application consumes too many resources compromising the stability of the entire system, which is used by many clients.
Throttling happens at
User (there are user request limits. Microsoft counts all requests linked to user
Application (Delegated or Application permissions)
Resource units per app per minute
Resource units per app per day
Farm – Spike protection
Very common reason for throttling – when an Application (Delegated or Application permissions) reaches “Resource units per app per minute” threshold.
Usually you catch HTTP errors 429 or 503, wait for some time (respect Retry-after header) and try again.
SharePoint provides various APIs. Different APIs have different costs depending on the complexity of the API, but Microsoft favor Graph API over SharePoint REST/CSOM. The cost of APIs is normalized by SharePoint and expressed by resource units. Application’s limits are also defined using resource units.
Quota depends on tenant size.
Resource unit limits for an application in a tenant (please refer to the Microsoft article)
Predefined costs for Microsoft Graph calls:
Assuming 2 resource units per request is a safe bet.
Here I’m trying to figure out – how much PowerShell Parallel option is beneficial and how to avoid throttling…
Let us test, how long would it take to create a SharePoint site, if we use regular (sequential) loop or parallelism (I’m creation a sample set of 50 SharePoint Sites in a row):
Regular (Sequential) seconds per site
Parallel, 100 sites in batch seconds per site
Parallel, 500 sites in batch seconds per site
Regular (Sequential)
3.0
Parallel, ThrottleLimit = 2
1.60
0.91
Parallel, ThrottleLimit = 5
0.69
Parallel, ThrottleLimit = 10
0.2 – 0.3
Parallel, ThrottleLimit = 20
0.17
Interesting, but I did not get even one (throttling or any other) error during creation 500 sites.
Get sites details
Now let us test, how long it takes to get sites details with Get-PnPTenantSite (I use a sample set of 500 sites):
Test type
Regular (Sequential), seconds per site
Parallel sample = 100 sites, seconds per site
Parallel sample = 200 sites, seconds per site
Parallel sample = 500 sites, seconds per site
Regular (Sequential)
0.65
Parallel, ThrottleLimit = 2
0.40
0.33
0.31
Parallel, ThrottleLimit = 5
0.17
0.14
0.36 (errors)
Parallel, ThrottleLimit = 10
0.11 (errors)
0.11 (errors)
0.34 (errors)
Parallel, ThrottleLimit = 20
0.12 errors+
0.07 errors+
0.52 (errors)
(errors) means there were small number of errors during test… e.g.