Yes, sure… But! Since it’s a cloud operation against Microsoft 365 – you will be throttled if you start more than 2 parallel threads! Using just 2 threads does not provide significant performance improvements.
Batching
So, try PnP.PowerShell batches instead. When you use batching, number of requests to the server are much lower. Consider something like:
Scenario: You have a large (>5k items) list in SharePoint Online you need to clean-up, for instance:
You need to delete the entire list
You need to delete all the list items, but keep the list
You need to delete some of list items, but keep the others
Deleting a large SharePoint Online list
There was a problem in SharePoint Online – you could not delete a large list – you had to remove all items first, but removing all items was also a challenge. Microsoft improved SharePoint Online, so now it takes ~1 second to delete any SharePoint list, including 5000+ items list via GUI or PowerShell:
Remove-PnPList -Identity $list
command works very fast – ~1 second to delete entire list with >5000 items.
Delete all items in a large SharePoint Online list
In this scenario we need to keep the list, but make it empty (clean it up).
GUI: You can change the list view settings “Item Limit” to <5000 and try to delete items in chunks, but (at least in my experience) when you try to select, let say, 1000 items and delete them via GUI – it says “775 items were not deleted from large list”:
so this option seems like not a good one.
ShareGate: 3-rd party tools like Sharegate, SysKit give a good results too.
for me both methods gave same good result: ~17 items per second ( ~7 times faster than regular).
Deleting some items from a large SPO list
Consider the following scenario: in a large SharePoint list there are items you need to delete and the rest items must stay (typical case might be to purge old items – e.g. items created last year).
In this case you’d
get all list items (or use query to get some list items)
select items that need to be deleted based on your criteria, e.g. created date or last modified date etc.
use PnP.PowerShell batches to delete only what you need
# to get all list items
$listItems = Get-PnPListItem -List Tasks -PageSize 1000
# or to get some list items
$listItems = Get-PnPListItem -List Tasks -Query <query>
# select items to delete
$itemsToDelete = $listItems | ?{$_.Modified -lt $threshold}
# delete some list items
$batch = New-PnPBatch
$itemsToDelete | Foreach-Object { Remove-PnPListItem -List $list -Identity $_ -Batch $batch }
Invoke-PnPBatch -Batch $batch
PnP.PowerShell batch vs ScriptBlock
How fast are PnP batches? What is better in terms of performance – ScriptBlock or Batching? Here are my measurements:
Time elapsed, seconds
with batches
with scriptBlock
without batches
Add-PnPListItem (100 items)
6-10 seconds
60-120 seconds
Add-PnPListItem (500 items)
20-40 seconds
230-600 seconds
Add-PnPListItem (7000 items)
314-600 seconds
Add-PnPListItem (37000 items)
3200 seconds
Remove-PnPListItem (1000 items)
58-103 seconds
58 seconds
430-1060 seconds
Remove-PnPListItem (7000 items)
395-990 seconds 3000 seconds
397-980 seconds
Remove-PnPListItem (30000 items)
one big batch : 13600 seconds 30 batches 1000 items each: 3500 seconds
both – PnP PowerShell batches and ScriptBlocks are 7-10 times faster than plain PnP PowerShell!
Can we use Microsoft Graph API to complete the same task? TBC…
Note… For the sake of history: It used to be like that for 5k+ lists: “Remove-PnPList” fails with a message “The attempted operation is prohibited because it exceeds the list view threshold enforced by the administrator”. Deleting with GUI failed too.
As I am a SharePoint person, and retention policies and labels are not a SharePoint engineer responsibility, I do not go to the m365 Compliance Center frequently. Below are My notes for myself on key moments – how to create and configure Office 365 retention labels and Policies at Compliance Center and use labels in SharePoint Online (SPO).
In SPO at each site collection level you can still work with retention policies the old way – create policies under Site Collection Settings – Content Type Policy – and apply policies at library level under Library Settings/Information Management Policy Settings. There is also Site Retention Policy.
But Microsoft is making efforts to centralize and unify such things – so you can specify retention policies in one place and apply them across all Office 365 content (not only SharePoint). That place was called Office 365 Security and Compliance Center (SCC). Later Microsoft separated Security Center and Compliance Center. So currently Retention Policies are under “Microsoft Purview” (former Microsoft Compliance Center) -> Solutions -> “Data lifecycle management”:
To get access to “Data lifecycle management” solution – you need to have a “” or “” roles. SharePoint or Teams administrator cannot access Purview. Even having “Global reader” or “Security reader” an admin will not be able to see “Data lifecycle management” blade. Here is how Microsoft Purview looks like for a Global reader:
Although SharePoint admins usually do not have access to SCC and do not go to Site content, we still need to know how it all works. And labels are recommended way to specify retention in SharePoint, so here we are.
Labels are applied to documents, documents are kept in libraries, and at each library you can “Apply a label to items in this library”.
Create Labels
Labels are created in SCC under Classification. The main part looks familiar to SharePoint people:
Label Settings
You can
Retain Content forever or for a specified number of days/months/years and then – delete it or trigger a disposition review or do nothing
Delete content if it’s older than specified number of days/months/years
after it was created/modified/labelled
Apply labels
Now you need to publish created labels – and that is how you create a policy. I.e. policies are where you specify which labels to which content (Exchange, OneDrive, SharePoint, Office 365 groups)
You can also auto-apply labels based on conditions, like
content that contains sensitive info
content that contains specific words or phrases, or properties
content that matches a trainable classifier
but as per Microsoft, “It will take up to 7 days to automatically apply the label to all items that match your conditions.”
Note: “trainable classifier” means an AI ML will be used, and as per Microsoft “Creating machine learning rules requires an Office 365 E5 subscription for your organization”
SharePoint admin center
You can do nothing with labels at SharePoint admin center. Labels are created, published and auto-applied at SCC. At each site collection levels site administrators can apply labels.
SharePoint site
At site collection settings you can still see “Content Type Policy Templates” and “Site Policy”, but that is not the case. Labels are applied at library level under Library Settings/Apply label to items in this list or library.
where you can select a label to apply for all new items in the library. With
You can also apply the label to items that already exist in the library.
You can also apply (change) label for each single item or multiple selected items under Details pop-up page:
or from under Contect Menu/More/Compliance details:
Adaptive retention policies and scopes
Microsoft recently implemented “Adaptive” retention policies. At step 2 of “Create retention policy” you’ll be asked “Choose the type of retention policy to create”: “A policy can be adaptive or static. Advantage of an adaptive policy will automatically update where it’s applied based on attributes or properties you’ll define. A static policy is applied to content in a fixed set of locations and must be manually updated if those locations change.”
And if you selected “Adaptive” – on the next step you will need to provide the adaptive scope (so at this moment you should already have created your adaptive scopes):
So, let us create your adaptive scopes. What type of scope do you want to create? SharePoint sites…
And then you’ll have nothing more then set of conditions:
where you can use objects: “Site Url”, “Site Name” and “Refinable String 0″..”Refinable String 99”. Conditions would be “is equal to”, “is not equal to”, “starts with” and “not starts with”. Or you can select “Advanced query builder” and enter LQL query.
What is the takeaway from this for SharePoint administrators? We would be asked to configure SharePoint the way compliance/retention people can use Refinable Strings.
How do users know – what files are going to be deleted
It would be a good idea to let users know, that their files will be deleted, but the next question would be “can I get a list of files that are scheduled to deletion”?
Surely it is possible for admins to go through site content and find files that were modified last time earlier than a specific date, but there is another method – you can recommend your users to search through their site using
The following solution is for Classic Search only.
What I did is:
User profile services -> Manage User properties: create custom property like “HideFromPeopleSearch”, boolean, do not allow users to edit value, Indexed
Client-side PowerShell script using PnP library:
Connect-PnPOnline -Url https://domain-admin.sharepoint.com
…
$nonPeople = Get-ADUser -filter … # based on what’s in your AD and how you separate people and non-people accounts
foreach($account in $nonPeople) {
Set-PnPUserProfileProperty -Property ‘HideFromPeopleSearch’ -Value ‘True’ -Account $account.UserPrincipalName
}
SearchCenter -> Site Settings -> Search Schema: use any pre-created RefinableString managed property (e.g. RefinableString33), add mapping to crawled property people:HideFromPeopleSearch,
SearchCenter -> Site Settings -> Search Query rules: Local People Results, new Query rule, change ranked results by changing query, {searchTerms} -RefinableString33=True
