Yes, sure… But! Since it’s a cloud operation against Microsoft 365 – you will be throttled if you start more than 2 parallel threads! Using just 2 threads does not provide significant performance improvements.
Batching
So, try PnP.PowerShell batches instead. When you use batching, number of requests to the server are much lower. Consider something like:
PowerShell is our best friend when it comes to ad-hoc and/or scheduled reports in Microsoft 365. PnP team is doing great job providing more and more functionality with PnP PowerShell module for Office 365 SharePoint and Teams.
Small and medium business organizations are mostly good, but for large companies it might be a problem due to just huge amount of data stored in SharePoint. PowerShell reports on all users or all sites might run days… which is probably OK if you run this report once, but totally not acceptable if you need this report e.g. daily/weekly or on-demand.
How can we make heavy PowerShell scripts run faster?
Of course, you start with logic (algorithm) and leveraging full PowerShell functionality (e.g. PowerShell 7 parallelism or PnP batching).
(examples)
What if you did everything, but it still takes too long? You need something like brute force – the closer your code runs to your tenant – the better. What are the option? – Automation account runbook (+workflow) – Azure Function Apps – Azure VM in the region closest to your Tenant
Automation account runbook (+workflow)
Seemed like a good option, but not something Microsoft promotes. Even opposite – automation accounts support only PowerShell 5 (not 7), no plug-ins for VS Code and recently there were messages on some retirement or smth.
Meantime, I tested it – and did not find any significant increasing in speed. In a nutshell, what is behind this service? Same windows machines running somewhere in Azure .
Good news! On Sep, 18 during the SIG community call, PnP Team shared their plans on PnP Sites Core library and PnP Core SDK. “PnP Sites Core v4” library and “PnP Core SDK v1” with .net core support (.net Standard 2.0) – expected in December 2020!
PnP PowerShell v4 for SPO library built for .Net Standard 2.0 / PowerShell 7 will be released in Dec 2020 as well.
Scenario: You have a large (>5k items) list in SharePoint Online you need to clean-up, for instance:
You need to delete the entire list
You need to delete all the list items, but keep the list
You need to delete some of list items, but keep the others
Deleting a large SharePoint Online list
There was a problem in SharePoint Online – you could not delete a large list – you had to remove all items first, but removing all items was also a challenge. Microsoft improved SharePoint Online, so now it takes ~1 second to delete any SharePoint list, including 5000+ items list via GUI or PowerShell:
Remove-PnPList -Identity $list
command works very fast – ~1 second to delete entire list with >5000 items.
Delete all items in a large SharePoint Online list
In this scenario we need to keep the list, but make it empty (clean it up).
GUI: You can change the list view settings “Item Limit” to <5000 and try to delete items in chunks, but (at least in my experience) when you try to select, let say, 1000 items and delete them via GUI – it says “775 items were not deleted from large list”:
so this option seems like not a good one.
ShareGate: 3-rd party tools like Sharegate, SysKit give a good results too.
for me both methods gave same good result: ~17 items per second ( ~7 times faster than regular).
Deleting some items from a large SPO list
Consider the following scenario: in a large SharePoint list there are items you need to delete and the rest items must stay (typical case might be to purge old items – e.g. items created last year).
In this case you’d
get all list items (or use query to get some list items)
select items that need to be deleted based on your criteria, e.g. created date or last modified date etc.
use PnP.PowerShell batches to delete only what you need
# to get all list items
$listItems = Get-PnPListItem -List Tasks -PageSize 1000
# or to get some list items
$listItems = Get-PnPListItem -List Tasks -Query <query>
# select items to delete
$itemsToDelete = $listItems | ?{$_.Modified -lt $threshold}
# delete some list items
$batch = New-PnPBatch
$itemsToDelete | Foreach-Object { Remove-PnPListItem -List $list -Identity $_ -Batch $batch }
Invoke-PnPBatch -Batch $batch
PnP.PowerShell batch vs ScriptBlock
How fast are PnP batches? What is better in terms of performance – ScriptBlock or Batching? Here are my measurements:
Time elapsed, seconds
with batches
with scriptBlock
without batches
Add-PnPListItem (100 items)
6-10 seconds
60-120 seconds
Add-PnPListItem (500 items)
20-40 seconds
230-600 seconds
Add-PnPListItem (7000 items)
314-600 seconds
Add-PnPListItem (37000 items)
3200 seconds
Remove-PnPListItem (1000 items)
58-103 seconds
58 seconds
430-1060 seconds
Remove-PnPListItem (7000 items)
395-990 seconds 3000 seconds
397-980 seconds
Remove-PnPListItem (30000 items)
one big batch : 13600 seconds 30 batches 1000 items each: 3500 seconds
both – PnP PowerShell batches and ScriptBlocks are 7-10 times faster than plain PnP PowerShell!
Can we use Microsoft Graph API to complete the same task? TBC…
Note… For the sake of history: It used to be like that for 5k+ lists: “Remove-PnPList” fails with a message “The attempted operation is prohibited because it exceeds the list view threshold enforced by the administrator”. Deleting with GUI failed too.
Update: Microsoft is deploying an updated version of “Disable Subsites” feature:
This update makes the setting options for new subsite creation easier to understand and prevents users from being able to create subsites using alternate paths when the subsite setting is disabled.
Admins in the SharePoint admin center can choose to either enable or disable subsite creation across sites or enable for classic sites only. Now, when disabling subsite creation, not only will the subsite option be hidden from the command bar including classic but also users will not be able to create new subsites directly through an URL or API.
The option: Hide the Subsite command has been renamed to Disable subsite creation for all sites and will also hide the subsite creation command (including classic) and disable users from being able to create new subsites through a URL or API. The option: Show the Subsite command only for classic sites, has been renamed to Enable subsite creation for classic sites only. The option: Show the Subsite command for all sites, has been renamed to Enable subsite creation for all sites.
Update is applied. What’s Next?
After this update is applied, if you have “Subsite Creation” set to “Disable subsite creation for all sites”, then if any attempt to create a subsite – you’ll get an error message “Sorry, something went wrong. New subsites are not available for your organization. Create a new site instead.”
Site is a new folder
Microsoft recommend “flat structure”, i.e. no subsites. So SPO admins are disabling subsites creation at tenant level. Did you know that you still can create subsite? Let me explain how it is done.
If creation subsites is allowed, you should be able to see it like this:
But actually subsites are not always best practice. Microsoft recommend “flat structure”, i.e. instead of subsite you should have a separate site collection, and if you need a hierarchy and navigation – use Hub sites. So, in Office 365 SharePoint admins usually “disable” SubSites creation:
Now, you see, SubSites are not really disabled, but only the button to create subsites is hidden: “This controls whether the Subsite command appears on the New menu on the Site contents page”.
Anyway, the result is: you are not able to create a SubSite (web) in SharePoint Online:
Actually there are at least 3 options to create a SubSite:
Microsoft: “You can also select to allow or prevent commenting on modern pages. If you allow commenting, it can be turned on or off at the page level“
By default both – Allow users to create new modern pages – Allow commenting on modern pages are turned on (enabled)
Tenant or SharePoint admin can find settings under SharePoint Admin Center -> Settings -> Pages
When you are creating a modern site page – there is an option “Comments” turned On by default:
And page with comments looks like:
Comments on site pages (aka modern pages) can be enabled or disabled at each of the levels: – Tenant level – Site (aka site collection) level – Web (aka subsite ) level – Page level Here is how it is done:
Level
How it’s done
Who can do it
Tenant
GUI ( SharePoint Administration) or PowerShell
Global Administrator or SharePoint Administrator
Site (Site Collection)
PowerShell
Global Administrator or SharePoint Administrator
Web (Subsite)
PowerShell
Site Collection Administrator or Owner (Full Control rights to web)
Page
GUI (Page Editing screen)
Site Member (Edit right to page)
If commenting on modern pages disabled at higher level – lower level settings do not work. E.g. If you disable “Allow commenting on modern pages” at tenant level (it takes minutes) – the functionality will gone from all modern pages of all sites.
When you switch page comments Off – all existing comments will be hidden (but not deleted). If you later turn comments On – comments will reappear, including Likes.
If “Allow commenting on modern pages” disabled at tenant or web level – you will not see “Comments On/Off” switch while editing page. If “Allow commenting on modern pages” disabled at site collection level – you will see “Comments On/Off” switch while editing page, but you will not be able to turn it ON.
PowerShell
When you disable “Allow commenting on modern pages” at tenant level – PowerShell Object (site/web) property “CommentsOnSitePagesDisabled” will not be changed for any site/web. You can still with PowerShell set it to True/False: “Set-PnPWeb -CommentsOnSitePagesDisabled:$false” but it does not take any effect.
If you enable “Allow commenting on modern pages” at tenant level (it takes ~10 minutes) – the functionality will return to all modern pages and all webs and sites properties “CommentsOnSitePagesDisabled” will ???. You can change it with PowerShell: “Set-PnPWeb -CommentsOnSitePagesDisabled:$false”.
# having Site Collection Admin Permissions:
# disable Comments On Site Pages for a subsite:
$webName = "SubSite_02"
Set-PnPWeb -Web $webName -CommentsOnSitePagesDisabled:$true
# enable Comments On Site Pages for a subsite
# (only if comments enabled at tenant level):
Set-PnPWeb -Web $webName -CommentsOnSitePagesDisabled:$false
# having global admin or SharePoint admin permissions:
# site collection:
Set-PnPTenantSite -Url $siteUrl -CommentsOnSitePagesDisabled:$true
# tenant-level Comments:
Set-PnPTenant -CommentsOnSitePagesDisabled:$true # disable
comments
Set-PnPTenant -CommentsOnSitePagesDisabled:$false # enable comments
# does not work:
Set-PnPSite -CommentsOnSitePagesDisabled:$true
How do I know if if the page is modern page or classic page (PowerShell)?
Note: We did not discuss “Wiki Pages” or “Web part Pages”, we discussed only “Modern Pages” (aka Site Pages). I have tested it all personally using Communication sites. MS-Team (group-based) and standalone SharePoint (no-group) sites – TBP.
Remove orphan sites from SharePoint content database
Problem: When you patch SharePoint or perform database-attach migration or just do Test-SPContentDatabase, sometimes you can see errors with category “SiteOrphan”. Although it says “UpgradeBlocking : False”, Ignoring this error may cause severe issues, even data loss. Continue reading →
