Tag Archives: PnP.PowerShell

Testing Sites.Selected SharePoint and MS Graph API

5 Replies

Sites.Selected MS Graph API permissions were introduced by Microsoft in March 2021. One year later, in 2022 they added SharePoint Sites.Selected API permissions.

Azure registered app with SharePoint and MS Graph API Sites.Selected permissions

Why is this so important? Because MS Graph API for SharePoint is still limited and cannot cover all possible needs. I’d estimate: 90% of applications use SharePoint CSOM, so developers have to use AppInv.aspx to provide permissions for their applications to SharePoint API.

But from this moment – having SharePoint API permissions in MS Graph – in theory – we can fully rely on permissions provided in Azure and – in theory – this should allow us disable SharePoint-Apps only principal:

Set-SPOTenant -DisableCustomAppAuthentication $true

My math professor taught me: “before trying to find a solution – ensure the solution exists.” So let us test:

Are we really able to work with a specific SharePoint site using MS Graph and SharePoint API Sites.Selected permissions provided via Microsoft Azure?

What will happen with our new/legacy applications if we disable SharePoint app-only SPNs (DisableCustomAppAuthentication)?

I’m getting controversial test results… maybe PnP.PowerShell 1.10 is not fully support SharePoint Sites.Selected API.

Tech Wizard (Sukhija Vikas) on March 20, 2022 in the article “SharePoint and Graph API APP only permissions for Selected Sites” suggests using pre-release (AllowPrerelease).

So please ignore the following for a while.

Meantime I’ll test providing SharePoint Sites.Selected API permissions via Graph API call.

(wip) Test set #1: Certificate vs Secret

DisableCustomAppAuthentication: $false (SP-app-only spns are enabled).
All applications have “write” access provided to a specific site only.
Connecting with Connect-PnPOnline and then test access with Get-PnPSite

App / Get-PnPSiteSecretCertificate
ACS based (Azure+AppInv)OKThe remote server returned an error: (401) Unauthorized.
MS Graph API Sites.SelectedThe remote server returned an error: (403) Forbidden.The remote server returned an error: (401) Unauthorized.
SharePoint API Sites.SelectedOKOK
MS Graph API + SharePoint API Sites.SelectedAccess is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED))OK
App with no permissionsThe remote server returned an error: (403) ForbiddenThe remote server returned an error: (401) Unauthorized

(wip) Test set #2: Sites.Selected SharePoint vs MS Graph (secret)

  • DisableCustomAppAuthentication = $false
    (SP-app-only spns are enabled).
  • All applications have “write” access provided to a specific site only.
  • Using Client Secret (not a certificate)
  • Using PnP.PowerShell
Action/ViaSharePoint + MS Graph
Sites.Selected
“secret”		SharePoint
Sites.Selected
“secret”		MS Graph
Sites.Selected
“secret”
Connect-PnPOnlineWARNING: Connecting with Client Secret uses legacy authentication and provides limited functionality. We can for instance not execute requests towards the Microsoft Graph, which limits cmdlets related to Microsoft Teams, Microsoft Planner, Microsoft Flow and Microsoft 365 Groups.WARNING: Connecting with Client Secret uses legacy authentication and provides limited functionality. We can for instance not execute requests towards the Microsoft Graph, which limits cmdlets related to Microsoft Teams, Microsoft Planner, Microsoft Flow and Microsoft 365 Groups.WARNING: Connecting with Client Secret uses legacy authentication and provides limited functionality. We can for instance not execute requests towards the Microsoft Graph, which limits cmdlets related to Microsoft Teams, Microsoft Planner, Microsoft Flow and Microsoft 365 Groups.
Get-PnPSiteOKOKThe remote server returned an error: (403) Forbidden.
Get-PnPListOKOK
Get-PnPListItemOKOK
Set-PnPSiteAttempted to perform an unauthorized operation.
Set-PnPListAttempted to perform an unauthorized operation.
Set-PnPListItemOKOK
New-PnPListAccess is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED))
Add-PnPListItemOK

(wip) Test set #3: Read vs Write vs FullControl

DisableCustomAppAuthentication = $false
(SP-app-only spns are enabled).
All applications have Sites.Selected SharePoint and MS Graph API permissions.
Using Client Secret (not a certificate)
Using PnP.PowerShell

ReadWriteFullControl
Connect-PnPOnlineWARNING: Connecting with Client Secret uses legacy authentication and provides limited functionality. We can for instance not execute requests towards the Microsoft Graph, which limits cmdlets related to Microsoft Teams, Microsoft Planner, Microsoft Flow and Microsoft 365 Groups.WARNING: Connecting with Client Secret uses legacy authentication and provides limited functionality. We can for instance not execute requests towards the Microsoft Graph, which limits cmdlets related to Microsoft Teams, Microsoft Planner, Microsoft Flow and Microsoft 365 Groups.WARNING: Connecting with Client Secret uses legacy authentication and provides limited functionality. We can for instance not execute requests towards the Microsoft Graph, which limits cmdlets related to Microsoft Teams, Microsoft Planner, Microsoft Flow and Microsoft 365 Groups.
Get-PnPSiteAccess is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED))
Get-PnPListAccess is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED))
Get-PnPListItem
Set-PnPSite
Set-PnPList
Set-PnPListItem
New-PnPList
Add-PnPListItem

(wip) Test set #5: Certificate vs Secret

C#, SharePoint CSOM, PnP.Framework

Findings

PnP.PowerShell Get-, Grant-, Set- and Revoke-PnPAzureADAppSitePermission cmdlets require Azure App with MS Graph Sites.FullControl.All app permissions (otherwise it says “Access denied”) and authentication via certificate (otherwise it says “This cmdlet does not work with a ACS based connection towards SharePoint.”)

The same actions – managing permissions for the client app to the specific site collections – could be done via Microsoft Graph Sites Permissions API using just secret-based authentication.

If an azure app does not have Sites.Selected API permissions configured – “Grant-PnPAzureADAppSitePermission” works as expected – no error messages – the output is normal – as if Sites.Selected API permissions were configured in the app. The same for Get-, -Set and Revoke-. Permissions provided for the app to the site are not effective though: Connect-PnPOnline works well, but all other commands – starting from Get-PnPSite – returns “The remote server returned an error: (403) Forbidden.”

If an app have no permissions to SharePoint – “Connect-PnPOnline” works ok, but “Get-PnPSite” return an error: “The remote server returned an error: (403) Forbidden.”

Set-PnPAzureADAppSitePermission gives an error message “code”:”generalException”,”message”:”General exception while processing”
if the site is not specified.

AppInv is not working?

Error: Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED))

References

Testing environment

  • Microsoft 365 E5 Dev environment
  • PowerShell 7.2.2
  • PnP.PowerShell 1.10
  • “write” permissions to the specific sites for client apps were assigned via PnP.PowerShell

Connecting to SharePoint Online programmatically: Secret vs Certificate

3 Replies

Update: Sites.Selected API MS Graph permissions was introduced by Microsoft in 2021. It was a good move towards site-level development, but still developers were limited with only what MS Graph API provides for SharePoint dev.
So devs had to use AppInv.aspx at site level to provide ACS permissions to their apps to be able to use SharePoint CSOM and REST APIs.
Recently Microsoft introduced Sites.Selected SharePoint API permissions for registered Azure Apps! So now devs should be fully happy without ACS-based permissions.

Scenario

You have an application that needs access to Microsoft 365 SharePoint Online site/list/documents. Application is running without interaction with users – e.g. unattended, as daemon job.

There are two options you can authenticate to Microsoft 365 – with the secret or with the certificate. Authenticating with certificate is considered more secure.

Questions

  • What happens if SharePoint-Apps only principal is disabled
    (i.e. ‘set-spotenant -DisableCustomAppAuthentication $true’ )?
  • Why I’m getting 401 error when authenticating to SPO?
  • Why I’m getting 403 error when authenticating to SPO with secret?
  • What permissions to I need to work with SPO?

Findings

Note: we will use PowerShell 7.2 and PnP.PowerShell 1.9 to illustrate it.

Disabled SharePoint-Apps only principal

If SharePoint-Apps only principal is disabled in your tenant
(i.e. ‘Get-PnPTenant | select DisableCustomAppAuthentication’ returns $true ), then the only way you work with SPO from code is:

  • an App registered in Azure
  • API permissions provided via Azure (MS Graph, SharePoint)
  • Certificate is used

In all other cases (even your Connect-PnPOnline command complete successfully) – you will be getting error 401 (unauthorized) when trying Get-PnPTenant or Get-PnPTenantSite or Get-PnPSite

Enabled SharePoint-Apps only principal

If SharePoint-Apps only principals are enabled in your tenant
(i.e. ‘Get-PnPTenant | select DisableCustomAppAuthentication’ returns $false ), then you have three options to work with SPO from code:

  • Azure App with a secret (Client Id + Client Secret) and permissions to SharePoint provided via SharePoint (AppInv.aspx) to access SharePoint REST API
  • Azure App with a certificate (Client Id + Certificate) and permissions provided via Azure to access SharePoint REST API
  • Azure App with a certificate or secret (Client Id + Secret or Certificate) and permissions provided via Azure to access SharePoint via Microsoft Graph API

PowerShell Script to Fetch All Alerts from SharePoint Online Site

1 Reply

PowerShell Script to get All Alerts of all Users from a specific SharePoint Online Site Collection, including subsites:

https://github.com/VladilenK/PowerShell/blob/main/reports/Site/Fetch-All-Alerts-from-SPO-Site.ps1

https://raw.githubusercontent.com/VladilenK/PowerShell/main/reports/Site/Fetch-All-Alerts-from-SPO-Site.ps1

based on Salaudeen Rajack:
“SharePoint Online: Get All Alerts from a Site Collection using PowerShell“

Connect to SharePoint Online and MS Graph Interactively with Client App and MSAL token

3 Replies

Scenario

You have a Microsoft 365 subscription with SharePoint Online. You use PowerShell, PnP.PowerShell module and MS Graph API to work with SharePoint under current user’s credential. You need to authenticate to SharePoint Online via Connect-PnPOnline and to Microsoft Graph API interactively on behalf of a current user.

Problem

Unfortunately, both “Connect-PnPOnline -Interactive -Url <siteUrl>” or “Connect-PnPOnline -UseWebLogin -Url <siteUrl>” might fail with something like “Need admin approval”, “App needs permission to access resources in your organization that only an admin can grant. Please ask an admin to grant permission to this app before you can use it.” or “Permissions requested” or similar

Solution

  • register an Azure App. Choose “single tenant”
  • configure authentication blade:
    – add platform – “Mobile and Desktop app”
    – select “https://login.microsoftonline.com/common/oauth2/nativeclient”
    – add custom Redirect URI: “http://localhost”
  • configure API permissions blade:
    – add delegated permissions you need (refer to specific API you’ll use)
    e.g. Microsoft Graph Sites.FullControl.All and SharePoint AllSites.FullControl
  • use the following code samples

PnP.PowerShell

$siteUrl = "https://contoso.sharepoint.com/teams/myTeamsSite"
$appId = "" # Client Id
Connect-PnPOnline -ClientId $appId -Url $siteUrl -Interactive
Get-PnPSite

A pop-up window will appear to authenticate interactively. If you are already authenticated with another credentials (or single-sigh-on) – an interactive window might pop up and disappear – that prevents you enter your other id.
To ensure Connect-PnPOnline prompts you for your credentials – use ” -ForceAuthentication” option.

If you are a SharePoint tenant admin – you can connect to a tenant with:

$orgName = "yourTenantPrefix" 
$adminUrl = "https://$orgName-admin.sharepoint.com" 
$appId = "" # Client Id 
$connection = Connect-PnPOnline -ClientId $appId -Url $adminUrl -Interactive -ReturnConnection # -ForceAuthentication 
$connection

Microsoft Graph API

Use MSAL.PS module to get an msal token then use token in Microsoft graph-based requests:

$tenantId = ""
$clientid = ""
$url = ""
$token = Get-MsalToken -ClientId $clientid -TenantId $tenantId -Interactive

By default token expires in ~ 1 hour. But you can refresh it silently.
This helps you in long-running PowerShell scripts that takes hours to complete.
So you can include something like this in the loop:

if ($token.ExpiresOn.LocalDateTime -lt $(get-date).AddMinutes(10)) {    
  $token = Get-MsalToken -ClientId $clientid -TenantId $tenantId -ForceRefresh -Silent    
  Write-Host "Token will expire on:" $token.ExpiresOn.LocalDateTime
}

Application permissions

Somehow using Connect-PnPOnline with AccessToken option did not work if the token was acquired with MSAL.PS interactively. But it did work when you get msal.ps token unattended (using App credentials). So…

If you can get an Application (non Delegated) permissions to your azure-registerd-app,
you can use msal token to connect to site with PnP

=========================

NB: For delegated permissions, the effective permissions of your app are the intersection of the delegated permissions the app has been granted (via consent) and the privileges of the currently signed-in user. Your app can never have more privileges than the signed-in user.

Update item Author field with PowerShell and Item-level list permissions

Leave a reply

Scenario:

You have imported a list into SharePoint Online. Every list item contains information for specific users and users’ accounts. You want list items be visible to specific users only.
You want to leverage Item-Level Permissions under List Advanced settings: “Read access: Read items that were created by the user”. But the problem is – since this list was imported – it was not users who created items. So the item-level access feature to work properly – you’d need to update the list so for each item the field “Created By” will have a user account you want the item be visible for.

Solution:

PnP.PowerShell helps. Using “Set-PnPListItem”, you can re-write “Author” field in the list item. In the example below I’m using just static user’s UPN:

Set-PnPListItem -List "Test" -Identity 1 -Values @{"Author"="testuser@domain.com"}

but in reality you’d do it dynamically – based on your specific case.

Then you can turn on Item-Level Permissions under List Advanced settings: “Read access: Read items that were created by the user”:

Add users to “Site Visitors” group for read-only access:

References

  • https://pnp.github.io/powershell/cmdlets/Set-PnPListItem.html
  • https://sharepointmaven.com/how-to-enable-item-level-permissions-in-sharepoint/

Connect-PnPOnline with a certificate stored in Azure Key Vault

Leave a reply

Scenario

You run some PnP PowerShell code unattended e.g. daemon/service app, background job – under application permissions – with no user interaction.
Your app needs to connect to SharePoint and/or Microsoft Graph API. Your organization require authentication with a certificate (no secrets). You want certificate stored securely in Azure Key Vault.

Solution (Step-by-step process)

  1. Obtain a certificate (create a self-signed or request trusted)
  2. In Azure where you have Microsoft 365 SharePoint tenant
    1. Create a new Registered App in Azure; save App (client) id, Directory (Tenant) Id
    2. Configure App: add MS Graph and SharePoint API application (not delegated) permissions
    3. Upload the certificate to the app under “Certificates & secrets”
  3. In Azure where you have paid subscription (could be same or different)
    1. Create an Azure Key Vault
    2. Upload certificate to the Key Vault manually (with GUI)
  4. While you develop/debug your custom daemon application at your local machine
    1. Provide permissions to the Key Vault via Access Control and Access Policies to your personal account
    2. Connect to Azure (the one where your Key Vault is) running Connect-AzAccount
      – so your app can get a Certificate to authenticate to SharePoint Online
  5. For your application deployed to Azure (e.g. Azure Function App )
    1. Turn On managed identity (Your Function App -> Identity -> Status:On) and Save; notice an Object (Principal) Id just created
    2. Provide for your managed identity principal Id permissions to the Key Vault via Key Vault Access Policies, so when your daemon app is running in the cloud – it could go to the key Vault and retrieve Certificate

Here is the sample PowerShell code to get certificate from Azure Key Vault and Connect to SharePoint with PnP (Connect-PnPOnline):

# ensure you use PowerShell 7
$PSVersionTable

# connect to your Azure subscription
Connect-AzAccount -Subscription "<subscription id>" -Tenant "<tenant id>"
Get-AzSubscription | fl
Get-AzContext

# Specify Key Vault Name and Certificate Name
$VaultName = "<azure key vault name>"
$certName = "certificate name as it stored in key vault"

# Get certificate stored in KeyVault (Yes, get it as SECRET)
$secret = Get-AzKeyVaultSecret -VaultName $vaultName -Name $certName
$secretValueText = ($secret.SecretValue | ConvertFrom-SecureString -AsPlainText )

# connect to PnP
$tenant = "contoso.onmicrosoft.com" # or tenant Id
$siteUrl = "https://contoso.sharepoint.com"
$clientID = "<App (client) Id>" # Azure Registered App with the same certificate and API permissions configured
Connect-PnPOnline -Url $siteUrl -ClientId $clientID -Tenant $tenant -CertificateBase64Encoded $secretValueText

Get-PnPSite

The same PowerShell code in GitHub: Connect-PnPOnline-with-certificate.ps1

References:

PnP.PowerShell Batches and PowerShell 7 Parallel

1 Reply

Parallelism

Can I use PowerShell 7 “-Parallel” option against SharePoint list items with PnP.PowerShell? Can I run something like:

$items | ForEach-Object -Parallel {
    $listItem = Set-PnPListItem -List "LargeList" -Identity $_ -Values @{"Number" = $(Get-Random -Minimum 100 -Maximum 200 ) }
}

Yes, sure… But! Since it’s a cloud operation against Microsoft 365 – you will be throttled if you start more than 2 parallel threads! Using just 2 threads does not provide significant performance improvements.

Batching

So, try PnP.PowerShell batches instead. When you use batching, number of requests to the server are much lower. Consider something like:

$batch = New-PnPBatch
1..100 | ForEach-Object{ Add-PnPListItem -List "ItemTest" -Values @{"Title"="Test Item Batched $_"} -Batch $batch }
Invoke-PnPBatch -Batch $batch


Measurements

Adding and setting 100 items with “Add-PnPListItem” and “Set-PnPListItem” in a large (more than 5000 items ) SharePoint list measurements:

Add-PnPListItem
Time per item, seconds		Set-PnPListItem
Time per item, seconds
Regular, without batching1.261.55
Using batches (New-PnPBatch)0.100.80
Using “Parallel” option, with ThrottleLimit 20.690.79
Using “Parallel” option, with ThrottleLimit 30.44 (fails level: ~4/100) 0.53 (fails level: ~3/100)

Adding items with PnP.PowerShell batching is much faster than without batching.

More:

How to delete a large SPO list or all/some items in a large SPO list

2 Replies

Scenario: You have a large (>5k items) list in SharePoint Online you need to clean-up, for instance:

  • You need to delete the entire list
  • You need to delete all the list items, but keep the list
  • You need to delete some of list items, but keep the others

Deleting a large SharePoint Online list

There was a problem in SharePoint Online – you could not delete a large list – you had to remove all items first, but removing all items was also a challenge. Microsoft improved SharePoint Online, so now it takes ~1 second to delete any SharePoint list, including 5000+ items list via GUI or PowerShell: 

Remove-PnPList -Identity $list

command works very fast – ~1 second to delete entire list with >5000 items.

Delete all items in a large SharePoint Online list

In this scenario we need to keep the list, but make it empty (clean it up).

GUI: You can change the list view settings “Item Limit” to <5000 and try to delete items in chunks, but (at least in my experience) when you try to select, let say, 1000 items and delete them via GUI – it says “775 items were not deleted from large list”:

so this option seems like not a good one.

ShareGate: 3-rd party tools like Sharegate, SysKit give a good results too.

PowerShell

Try this PowerShell command with ScriptBlock:

Get-PnPListItem -List $list -Fields "ID" -PageSize 100 -ScriptBlock { Param($items) $items | Sort-Object -Property Id -Descending | ForEach-Object{ $_.DeleteObject() } } 

or this PowerShell with batches:

$batch = New-PnPBatch
1..12000 | Foreach-Object { Remove-PnPListItem -List $list -Identity $_ -Batch $batch }
Invoke-PnPBatch -Batch $batch

for me both methods gave same good result: ~17 items per second ( ~7 times faster than regular).

Deleting some items from a large SPO list

Consider the following scenario: in a large SharePoint list there are items you need to delete and the rest items must stay (typical case might be to purge old items – e.g. items created last year).

In this case you’d

  • get all list items (or use query to get some list items)
  • select items that need to be deleted based on your criteria, e.g. created date or last modified date etc.
  • use PnP.PowerShell batches to delete only what you need
# to get all list items
$listItems = Get-PnPListItem -List Tasks -PageSize 1000
# or to get some list items 
$listItems = Get-PnPListItem -List Tasks -Query <query>
# select items to delete
$itemsToDelete = $listItems | ?{$_.Modified -lt $threshold}
# delete some list items
$batch = New-PnPBatch 
$itemsToDelete | Foreach-Object { Remove-PnPListItem -List $list -Identity $_ -Batch $batch } 
Invoke-PnPBatch -Batch $batch

PnP.PowerShell batch vs ScriptBlock

How fast are PnP batches? What is better in terms of performance – ScriptBlock or Batching? Here are my measurements:

Time elapsed, secondswith batcheswith scriptBlockwithout batches
Add-PnPListItem (100 items)6-10 seconds60-120 seconds
Add-PnPListItem (500 items)20-40 seconds230-600 seconds
Add-PnPListItem (7000 items)314-600 seconds
Add-PnPListItem (37000 items)3200 seconds
Remove-PnPListItem (1000 items)58-103 seconds58 seconds430-1060 seconds
Remove-PnPListItem (7000 items)395-990 seconds
3000 seconds		397-980 seconds
Remove-PnPListItem (30000 items)one big batch : 13600 seconds
30 batches 1000 items each: 3500 seconds

both – PnP PowerShell batches and ScriptBlocks are 7-10 times faster than plain PnP PowerShell!

Can we use Microsoft Graph API to complete the same task? TBC…

Note… For the sake of history: It used to be like that for 5k+ lists:
“Remove-PnPList” fails with a message “The attempted operation is prohibited because it exceeds the list view threshold enforced by the administrator”. Deleting with GUI failed too.

References: