Here: https://github.com/VladilenK/Manage-m365-with-PowerShell
PowerShell scripts for Microsoft 365 SharePoint
Leave a reply
Tag Archives: PnP
Token – SharePoint API compatibility matrix
If I get token with (Graph, MSAL, PnP) and use this token for (Graph API, SharePoint CSOM API, SharePoint REST API) matrix.
An App used in this tests has Sites.FullControl.All MS Graph API and SharePoint API permissions, as well as FullControl ACS based permissions to SharePoint (AppInv.aspx).
| Token/API | MS Graph /v1.0/sites | SharePoint CSOM PnP.PowerShell Get-PnPSite Get-PnPTenantSite | SharePoint REST API PnP.PowerShell Invoke-PnPSPRestMethod Invoke-RestMethod |
| MS Graph /oauth2/v2.0/token secret | OK | (401) Unauthorized | AudienceUriValidationFailedException |
| MSAL.PS Get-MsalToken with secret | OK | (401) Unauthorized | AudienceUriValidationFailedException |
| MSAL.PS Get-MsalToken with certificate | OK | (401) Unauthorized | AudienceUriValidationFailedException |
| PnP.PowerShell Get-PnPAccessToken with Certificate | OK | OK OK | OK AudienceUriValidationFailedException |
| PnP.PowerShell Get-PnPGraphAccessToken with Certificate | OK | OK OK | OK AudienceUriValidationFailedException |
| PnP.PowerShell Get-PnPAppAuthAccessToken with Certificate or secret | InvalidAuthenticationToken | OK OK | OK OK |
| PnP.PowerShell Request-PnPAccessToken with Certificate | InvalidAuthenticationToken | OK OK | OK AudienceUriValidationFailedException |
| PnP.PowerShell Request-PnPAccessToken with Secret | InvalidAuthenticationToken | OK OK | OK OK |
Testing Sites.Selected SharePoint and MS Graph API
Sites.Selected MS Graph API permissions were introduced by Microsoft in March 2021. One year later, in 2022 they added SharePoint Sites.Selected API permissions.
Why is this so important? Because MS Graph API for SharePoint is still limited and cannot cover all possible needs. I’d estimate: 90% of applications use SharePoint CSOM, so developers have to use AppInv.aspx to provide permissions for their applications to SharePoint API.
But from this moment – having SharePoint API permissions in MS Graph – in theory – we can fully rely on permissions provided in Azure and – in theory – this should allow us disable SharePoint-Apps only principal:
Set-SPOTenant -DisableCustomAppAuthentication $trueMy math professor taught me: “before trying to find a solution – ensure the solution exists.” So let us test:
Are we really able to work with a specific SharePoint site using MS Graph and SharePoint API Sites.Selected permissions provided via Microsoft Azure?
What will happen with our new/legacy applications if we disable SharePoint app-only SPNs (DisableCustomAppAuthentication)?
I’m getting controversial test results… maybe PnP.PowerShell 1.10 is not fully support SharePoint Sites.Selected API.
Tech Wizard (Sukhija Vikas) on March 20, 2022 in the article “SharePoint and Graph API APP only permissions for Selected Sites” suggests using pre-release (AllowPrerelease).
So please ignore the following for a while.
Meantime I’ll test providing SharePoint Sites.Selected API permissions via Graph API call.
(wip) Test set #1: Certificate vs Secret
DisableCustomAppAuthentication: $false (SP-app-only spns are enabled).
All applications have “write” access provided to a specific site only.
Connecting with Connect-PnPOnline and then test access with Get-PnPSite
| App / Get-PnPSite | Secret | Certificate |
| ACS based (Azure+AppInv) | OK | The remote server returned an error: (401) Unauthorized. |
| MS Graph API Sites.Selected | The remote server returned an error: (403) Forbidden. | The remote server returned an error: (401) Unauthorized. |
| SharePoint API Sites.Selected | OK | OK |
| MS Graph API + SharePoint API Sites.Selected | Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED)) | OK |
| App with no permissions | The remote server returned an error: (403) Forbidden | The remote server returned an error: (401) Unauthorized |
(wip) Test set #2: Sites.Selected SharePoint vs MS Graph (secret)
- DisableCustomAppAuthentication = $false
(SP-app-only spns are enabled). - All applications have “write” access provided to a specific site only.
- Using Client Secret (not a certificate)
- Using PnP.PowerShell
| Action/Via | SharePoint + MS Graph Sites.Selected “secret” | SharePoint Sites.Selected “secret” | MS Graph Sites.Selected “secret” |
| Connect-PnPOnline | WARNING: Connecting with Client Secret uses legacy authentication and provides limited functionality. We can for instance not execute requests towards the Microsoft Graph, which limits cmdlets related to Microsoft Teams, Microsoft Planner, Microsoft Flow and Microsoft 365 Groups. | WARNING: Connecting with Client Secret uses legacy authentication and provides limited functionality. We can for instance not execute requests towards the Microsoft Graph, which limits cmdlets related to Microsoft Teams, Microsoft Planner, Microsoft Flow and Microsoft 365 Groups. | WARNING: Connecting with Client Secret uses legacy authentication and provides limited functionality. We can for instance not execute requests towards the Microsoft Graph, which limits cmdlets related to Microsoft Teams, Microsoft Planner, Microsoft Flow and Microsoft 365 Groups. |
| Get-PnPSite | OK | OK | The remote server returned an error: (403) Forbidden. |
| Get-PnPList | OK | OK | |
| Get-PnPListItem | OK | OK | |
| Set-PnPSite | Attempted to perform an unauthorized operation. | ||
| Set-PnPList | Attempted to perform an unauthorized operation. | ||
| Set-PnPListItem | OK | OK | |
| New-PnPList | Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED)) | ||
| Add-PnPListItem | OK |
(wip) Test set #3: Read vs Write vs FullControl
DisableCustomAppAuthentication = $false
(SP-app-only spns are enabled).
All applications have Sites.Selected SharePoint and MS Graph API permissions.
Using Client Secret (not a certificate)
Using PnP.PowerShell
| Read | Write | FullControl | |
| Connect-PnPOnline | WARNING: Connecting with Client Secret uses legacy authentication and provides limited functionality. We can for instance not execute requests towards the Microsoft Graph, which limits cmdlets related to Microsoft Teams, Microsoft Planner, Microsoft Flow and Microsoft 365 Groups. | WARNING: Connecting with Client Secret uses legacy authentication and provides limited functionality. We can for instance not execute requests towards the Microsoft Graph, which limits cmdlets related to Microsoft Teams, Microsoft Planner, Microsoft Flow and Microsoft 365 Groups. | WARNING: Connecting with Client Secret uses legacy authentication and provides limited functionality. We can for instance not execute requests towards the Microsoft Graph, which limits cmdlets related to Microsoft Teams, Microsoft Planner, Microsoft Flow and Microsoft 365 Groups. |
| Get-PnPSite | Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED)) | ||
| Get-PnPList | Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED)) | ||
| Get-PnPListItem | |||
| Set-PnPSite | |||
| Set-PnPList | |||
| Set-PnPListItem | |||
| New-PnPList | |||
| Add-PnPListItem |
(wip) Test set #5: Certificate vs Secret
C#, SharePoint CSOM, PnP.Framework
Findings
PnP.PowerShell Get-, Grant-, Set- and Revoke-PnPAzureADAppSitePermission cmdlets require Azure App with MS Graph Sites.FullControl.All app permissions (otherwise it says “Access denied”) and authentication via certificate (otherwise it says “This cmdlet does not work with a ACS based connection towards SharePoint.”)
The same actions – managing permissions for the client app to the specific site collections – could be done via Microsoft Graph Sites Permissions API using just secret-based authentication.
If an azure app does not have Sites.Selected API permissions configured – “Grant-PnPAzureADAppSitePermission” works as expected – no error messages – the output is normal – as if Sites.Selected API permissions were configured in the app. The same for Get-, -Set and Revoke-. Permissions provided for the app to the site are not effective though: Connect-PnPOnline works well, but all other commands – starting from Get-PnPSite – returns “The remote server returned an error: (403) Forbidden.”
If an app have no permissions to SharePoint – “Connect-PnPOnline” works ok, but “Get-PnPSite” return an error: “The remote server returned an error: (403) Forbidden.”
Set-PnPAzureADAppSitePermission gives an error message “code”:”generalException”,”message”:”General exception while processing”
if the site is not specified.
AppInv is not working?
Error: Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED))
References
- Track SharePoint App-only Service Principals in Microsoft 365
- Microsoft Graph Sites Permissions API
- “SharePoint and Graph API APP only permissions for Selected Sites”
Testing environment
- Microsoft 365 E5 Dev environment
- PowerShell 7.2.2
- PnP.PowerShell 1.10
- “write” permissions to the specific sites for client apps were assigned via PnP.PowerShell
Providing Permissions to a Site for Sites.Selected App
How to provide permissions for an Azure registered application with MS Graph SharePoint Sites.Selected API permissions to a specific site via calling Microsoft Graph API from PowerShell.
We need an “admin” application – Azure registered application with with Sites.FullControl.All MS Graph API permissions. This method can use secret, so we need Client Id and Client Secret for this “admin” app.
We also need a Client Id and Application Display Name for an Azure application with Sites.Selected MS Graph and/or SharePoint API permissions provided.
And we need our “target” site Url.
With PowerShell scripts you can:
- Get Microsoft Graph Access Token with an “admin” app
- Get client (target) site Id
- Get current app permissions provided to client site
- Add read or write permissions for the client app to the client site
- Revoke one specific permission from site
- Revoke all app permissions provided to site
– please refer to the GitHub Repo Sites.Selected
References
Connecting to SharePoint Online programmatically: Secret vs Certificate
Update: Sites.Selected API MS Graph permissions was introduced by Microsoft in 2021. It was a good move towards site-level development, but still developers were limited with only what MS Graph API provides for SharePoint dev.
So devs had to use AppInv.aspx at site level to provide ACS permissions to their apps to be able to use SharePoint CSOM and REST APIs.
Recently Microsoft introduced Sites.Selected SharePoint API permissions for registered Azure Apps! So now devs should be fully happy without ACS-based permissions.
Scenario
You have an application that needs access to Microsoft 365 SharePoint Online site/list/documents. Application is running without interaction with users – e.g. unattended, as daemon job.
There are two options you can authenticate to Microsoft 365 – with the secret or with the certificate. Authenticating with certificate is considered more secure.
Questions
- What happens if SharePoint-Apps only principal is disabled
(i.e. ‘set-spotenant -DisableCustomAppAuthentication $true’ )? - Why I’m getting 401 error when authenticating to SPO?
- Why I’m getting 403 error when authenticating to SPO with secret?
- What permissions to I need to work with SPO?
Findings
Note: we will use PowerShell 7.2 and PnP.PowerShell 1.9 to illustrate it.
Disabled SharePoint-Apps only principal
If SharePoint-Apps only principal is disabled in your tenant
(i.e. ‘Get-PnPTenant | select DisableCustomAppAuthentication’ returns $true ), then the only way you work with SPO from code is:
- an App registered in Azure
- API permissions provided via Azure (MS Graph, SharePoint)
- Certificate is used
In all other cases (even your Connect-PnPOnline command complete successfully) – you will be getting error 401 (unauthorized) when trying Get-PnPTenant or Get-PnPTenantSite or Get-PnPSite
Enabled SharePoint-Apps only principal
If SharePoint-Apps only principals are enabled in your tenant
(i.e. ‘Get-PnPTenant | select DisableCustomAppAuthentication’ returns $false ), then you have two options to work with SPO from code:
- Azure App with a secret (Client Id + Client Secret) and permissions to SharePoint provided via SharePoint ( AppInv.aspx )
- Azure App with a certificate (Client Id + Certificate) and permissions provided via Azure (Microsoft Graph and/or SharePoint)
Error 401 while accessing SharePoint Online with PnP
(Get-PnPTenant, Get-PnPTenantSite)
Get list of new SPO sites with PowerShell
Scenario
Let say you administer Microsoft 365 SharePoint Online and you want to get a list of new SharePoint sites (e.g. sites created recently – during last day/week/month).
With GUI it’s done easily: SharePoint Admin Center -> Active Sites -> sort based on “Date Created” – done.
With PowerShell – not so simple.
“Get-PnPTenantSite” cmdlet returns a site object but the object does not have “Created” field. It’s a web property. But to get a web object – you have to connect separately to each site and get root web object to check when the web was created. For small environments it is possible, for large environments it can take days… And still not nice.
“Get-PnPTenantSite” with “-Filter” option would help, but “…Currently, you can filter by these properties: Owner, Template, LockState, Url.”
Get-SPOSite – similar experience.
Solution
Microsoft Graph API helps. It returns result in seconds and you can sort or filter results based on created date . Below are two methods: Option 1 is based on Search and filtering and Option 2 is based on Sites Search and sorting. So there are some pros and cons for each method.
Option #1: Microsoft Graph Search API.
Entry point: https://graph.microsoft.com/v1.0/search/query
Microsoft Graph Search API allows KQL in queries. So we can form a query with something like “created>=1/1/2021” and use entity type = ‘[“site”]’. Search should return only sites created after Jan 01, 2021.
Check PowerShell script sample here: Get-NewSites.ps1
https://github.com/VladilenK/PowerShell/blob/main/reports/SharePoint/Get-NewSites.ps1
If you are getting more than 500 results – think of paging.
Option #2: Microsoft Graph Sites API
Entry point: https://graph.microsoft.com/v1.0/sites
This option is also based on Microsoft Graph API, but sites entry point, which allows search too and sort results by property “createdDateTime”. So we will just search for everything and select how many results we need based on createdDateTime property.
Check PowerShell script sample here: Get-NewSites.ps1
https://github.com/VladilenK/PowerShell/blob/main/reports/SharePoint/Get-NewSites.ps1
References
Retrieve SharePoint Online system page html content programmatically (PowerShell)
How can I get HTML content of a SharePoint online page from code, e.g. PowerShell?
Invoke-WebRequest returns “Sign in to your account” page, not a real page, even with -Token option.
Thanks to Denis Molodtsov, the solution is found. It turns out the “Invoke-PnPSPRestMethod” PnP cmdlet works not only against /api endpoints, but also against site pages and system pages.
But (as per my experience) it works only with PnP.PowerShell and with -UseWebLogin authentication option and with -raw parameter.
Connect-PnPOnline -url $siteUrl -UseWebLogin
Invoke-PnPSPRestMethod -url /_layouts/15/viewlsts.aspx -Raw
Other combination of authentication options ( -interactive, -clientId, -Token, -SPOManagementShell, -PnPManagementShell ) – worked well, but only for /_api endpoints, and gave me “401 UNAUTHORIZED” against system/site pages.
Unattended authentication (with clientId, clientSecret and certificate) – same.
Legacy PnP module SharePointPnPPowerShellOnline did not work at all: “EXCEPTION,PnP.PowerShell.Commands.Admin.InvokeSPRestMethod”.
I tested it with
– SharePointPnPPowerShellOnline v 3.29.2101.0 (under Windows PowerShell 5.1) and
– PnP.PowerShell 1.8.0. (both Windows PowerShell 5.1 and .net core PowerShell 7.1.5)
PowerShell Script to Fetch All Alerts from SharePoint Online Site
PowerShell Script to get All Alerts of all Users from a specific SharePoint Online Site Collection, including subsites:
https://github.com/VladilenK/PowerShell/blob/main/reports/Site/Fetch-All-Alerts-from-SPO-Site.ps1
based on Salaudeen Rajack:
“SharePoint Online: Get All Alerts from a Site Collection using PowerShell“
Connect to SharePoint Online and MS Graph Interactively with Client App and MSAL token
Scenario
You have got a Microsoft 365 subscription with SharePoint Online. You use PowerShell, PnP.PowerShell module and MS Graph API to work with SharePoint under current user’s credential. So you need to authenticate to SharePoint Online via Connect-PnPOnline and to Microsoft Graph API interactively on behalf of a user.
Problem
Unfortunately, both “Connect-PnPOnline -Interactive -Url <siteUrl>” or “Connect-PnPOnline -UseWebLogin -Url <siteUrl>” might fail with something like “Need admin approval”, “App needs permission to access resources in your organization that only an admin can grant. Please ask an admin to grant permission to this app before you can use it.” or similar
Solution
- register an Azure App. Choose “single tenant”
- configure authentication blade:
– add platform – “Mobile and Desktop app”
– select “https://login.microsoftonline.com/common/oauth2/nativeclient”
– add custom Redirect URI: “http://localhost” - configure API permissions blade:
– add delegated permissions you need (refer to specific API you’ll use)
e.g. Microsoft Graph Sites.FullControl.All and SharePoint AllSites.FullControl - use the following code samples
PnP.PowerShell
$siteUrl = "https://contoso.sharepoint.com/teams/myTeamsSite"
$appId = "" # Client Id
Connect-PnPOnline -ClientId $appId -Url $siteUrl -Interactive
Get-PnPSiteA pop-up window will appear to authenticate interactively. If you are already authenticated with another credentials (or single-sigh-on) – an interactive window might pop up and disappear – that prevents you enter your other id.
To ensure Connect-PnPOnline prompts you for your credentials – use ” -ForceAuthentication” option.
If you are a SharePoint tenant admin – you can connect to a tenant with:
$orgName = "yourTenantPrefix"
$adminUrl = "https://$orgName-admin.sharepoint.com"
$appId = "" # Client Id
$connection = Connect-PnPOnline -ClientId $appId -Url $adminUrl -Interactive -ReturnConnection # -ForceAuthentication
$connection Microsoft Graph API
Use MSAL.PS module to get an msal token then use token in Microsoft graph-based requests:
$tenantId = ""
$clientid = ""
$url = ""
$token = Get-MsalToken -ClientId $clientid -TenantId $tenantId -InteractiveBy default token expires in ~ 1 hour. But you can refresh it silently.
This helps you in long-running PowerShell scripts that takes hours to complete.
So you can include something like this in the loop:
if ($token.ExpiresOn.LocalDateTime -lt $(get-date).AddMinutes(10)) {
$token = Get-MsalToken -ClientId $clientid -TenantId $tenantId -ForceRefresh -Silent
Write-Host "Token will expire on:" $token.ExpiresOn.LocalDateTime
}Application permissions
Somehow using Connect-PnPOnline with AccessToken option did not work if the token was acquired with MSAL.PS interactively. But it did work when you get msal.ps token unattended (using App credentials). So…
If you can get an Application (non Delegated) permissions to your azure-registerd-app,
you can use msal token to connect to site with PnP
=========================
NB: For delegated permissions, the effective permissions of your app are the intersection of the delegated permissions the app has been granted (via consent) and the privileges of the currently signed-in user. Your app can never have more privileges than the signed-in user.
Connect-PnPOnline with a certificate stored in Azure Key Vault
Scenario
You run some PnP PowerShell code unattended e.g. daemon/service app, background job – under application permissions – with no user interaction.
Your app needs to connect to SharePoint and/or Microsoft Graph API. Your organization require authentication with a certificate (no secrets). You want certificate stored securely in Azure Key Vault.
Solution (Step-by-step process)
- Obtain a certificate (create a self-signed or request trusted)
- In Azure where you have Microsoft 365 SharePoint tenant
- Create a new Registered App in Azure; save App (client) id, Directory (Tenant) Id
- Configure App: add MS Graph and SharePoint API application (not delegated) permissions
- Upload the certificate to the app under “Certificates & secrets”
- In Azure where you have paid subscription (could be same or different)
- Create an Azure Key Vault
- Upload certificate to the Key Vault manually (with GUI)
- While you develop/debug your custom daemon application at your local machine
- Provide permissions to the Key Vault via Access Control and Access Policies to your personal account
- Connect to Azure (the one where your Key Vault is) running Connect-AzAccount
– so your app can get a Certificate to authenticate to SharePoint Online
- For your application deployed to Azure (e.g. Azure Function App )
- Turn On managed identity (Your Function App -> Identity -> Status:On) and Save; notice an Object (Principal) Id just created
- Provide for your managed identity principal Id permissions to the Key Vault via Key Vault Access Policies, so when your daemon app is running in the cloud – it could go to the key Vault and retrieve Certificate
Here is the sample PowerShell code to get certificate from Azure Key Vault and Connect to SharePoint with PnP (Connect-PnPOnline):
# ensure you use PowerShell 7
$PSVersionTable
# connect to your Azure subscription
Connect-AzAccount -Subscription "<subscription id>" -Tenant "<tenant id>"
Get-AzSubscription | fl
Get-AzContext
# Specify Key Vault Name and Certificate Name
$VaultName = "<azure key vault name>"
$certName = "certificate name as it stored in key vault"
# Get certificate stored in KeyVault (Yes, get it as SECRET)
$secret = Get-AzKeyVaultSecret -VaultName $vaultName -Name $certName
$secretValueText = ($secret.SecretValue | ConvertFrom-SecureString -AsPlainText )
# connect to PnP
$tenant = "contoso.onmicrosoft.com" # or tenant Id
$siteUrl = "https://contoso.sharepoint.com"
$clientID = "<App (client) Id>" # Azure Registered App with the same certificate and API permissions configured
Connect-PnPOnline -Url $siteUrl -ClientId $clientID -Tenant $tenant -CertificateBase64Encoded $secretValueText
Get-PnPSite
The same PowerShell code in GitHub: Connect-PnPOnline-with-certificate.ps1
References:
- https://docs.microsoft.com/en-us/powershell/module/az.keyvault/get-azkeyvaultcertificate?view=azps-5.3.0
- https://stackoverflow.com/questions/43837362/keyvault-generated-certificate-with-exportable-private-key
- Do I need certificate or secret to Connect to SharePoint Online from code
- Register and configure Azure App to Get access to an SPO site from code