Category Archives: Microsoft 365

Get list of new SPO sites with PowerShell

Scenario

Let say you administer Microsoft 365 SharePoint Online and you want to get a list of new SharePoint sites (e.g. sites created recently – during last day/week/month).

With GUI it’s done easily: SharePoint Admin Center -> Active Sites -> sort based on “Date Created” – done.

With PowerShell – not so simple.
“Get-PnPTenantSite” cmdlet returns a site object but the object does not have “Created” field. It’s a web property. But to get a web object – you have to connect separately to each site and get root web object to check when the web was created. For small environments it is possible, for large environments it can take days… And still not nice.
“Get-PnPTenantSite” with “-Filter” option would help, but “…Currently, you can filter by these properties: Owner, Template, LockState, Url.”

Get-SPOSite – similar experience.

Solution

Microsoft Graph API helps. It returns result in seconds and you can sort or filter results based on created date . Below are two methods: Option 1 is based on Search and filtering and Option 2 is based on Sites Search and sorting. So there are some pros and cons for each method.

Option #1: Microsoft Graph Search API.

Entry point: https://graph.microsoft.com/v1.0/search/query

Microsoft Graph Search API allows KQL in queries. So we can form a query with something like “created>=1/1/2021” and use entity type = ‘[“site”]’. Search should return only sites created after Jan 01, 2021.

Check PowerShell script sample here: Get-NewSites.ps1
https://github.com/VladilenK/PowerShell/blob/main/reports/SharePoint/Get-NewSites.ps1

If you are getting more than 500 results – think of paging.

Option #2: Microsoft Graph Sites API

Entry point: https://graph.microsoft.com/v1.0/sites

This option is also based on Microsoft Graph API, but sites entry point, which allows search too and sort results by property “createdDateTime”. So we will just search for everything and select how many results we need based on createdDateTime property.

Check PowerShell script sample here: Get-NewSites.ps1
https://github.com/VladilenK/PowerShell/blob/main/reports/SharePoint/Get-NewSites.ps1

References

Retrieve SharePoint Online system page html content programmatically (PowerShell)

How can I get HTML content of a SharePoint online page from code, e.g. PowerShell?

Invoke-WebRequest returns “Sign in to your account” page, not a real page, even with -Token option.

Thanks to Denis Molodtsov, the solution is found. It turns out the “Invoke-PnPSPRestMethod” PnP cmdlet works not only against /api endpoints, but also against site pages and system pages.

But (as per my experience) it works only with PnP.PowerShell and with -UseWebLogin authentication option and with -raw parameter.

Connect-PnPOnline -url $siteUrl -UseWebLogin
Invoke-PnPSPRestMethod -url /_layouts/15/viewlsts.aspx -Raw

Other combination of authentication options ( -interactive, -clientId, -Token, -SPOManagementShell, -PnPManagementShell ) – worked well, but only for /_api endpoints, and gave me “401 UNAUTHORIZED” against system/site pages.
Unattended authentication (with clientId, clientSecret and certificate) – same.

Legacy PnP module SharePointPnPPowerShellOnline did not work at all: “EXCEPTION,PnP.PowerShell.Commands.Admin.InvokeSPRestMethod”.

I tested it with
– SharePointPnPPowerShellOnline v 3.29.2101.0 (under Windows PowerShell 5.1) and
– PnP.PowerShell 1.8.0. (both Windows PowerShell 5.1 and .net core PowerShell 7.1.5)

Search Unified Audit Log Daemon Job

How to run “Search-UnifiedAuditLog” in unattended way, i.e. non-interactive.
What are the minimal permissions required?

The PowerShell code:

$clientId = ""
$cPwd = ConvertTo-SecureString -String "" -AsPlainText -Force

$cPath = ""C:\Users\UserName\Certificates\Cert.pfx""
$organization = "contoso.onmicrosoft.com"

Connect-ExchangeOnline -CertificateFilePath $cPath -CertificatePassword $cPwd -AppID $clientId -Organization $organization

[DateTime]$start = [DateTime]::UtcNow.AddMinutes(-45)
[DateTime]$end = [DateTime]::UtcNow
$resultSize = 1000

$results = $null
$results = Search-UnifiedAuditLog -StartDate $start -EndDate $end -ResultSize $results.Count
$results | Select-Object RecordType, CreationDate, UserIds, Operations -First 3

Disconnect-ExchangeOnline -Confirm:$false

Troubleshooting

The error “The term ‘Search-UnifiedAuditLog’ is not recognized”:

Search-UnifiedAuditLog: C:\scripts\PowerShell.auth\Search-AuditLog-w-App.ps1:16:12
Line |
16 | $results = Search-UnifiedAuditLog -StartDate $start -EndDate $end -Re …
| ~~~~~~
| The term 'Search-UnifiedAuditLog' is not recognized as a name of a cmdlet, function, script file, or executable
| program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again.

means a proper administrative role (e.g. “Exchange administrator”) is not assigned to the app.

References

PowerShell Script to Fetch All Alerts from SharePoint Online Site

PowerShell Script to get All Alerts of all Users from a specific SharePoint Online Site Collection, including subsites:

https://github.com/VladilenK/PowerShell/blob/main/reports/Site/Fetch-All-Alerts-from-SPO-Site.ps1

https://raw.githubusercontent.com/VladilenK/PowerShell/main/reports/Site/Fetch-All-Alerts-from-SPO-Site.ps1

based on Salaudeen Rajack:
SharePoint Online: Get All Alerts from a Site Collection using PowerShell

Connect to SharePoint Online and MS Graph Interactively with Client App and MSAL token

Scenario

You have got a Microsoft 365 subscription with SharePoint Online. You use PowerShell, PnP.PowerShell module and MS Graph API to work with SharePoint under current user’s credential. So you need to authenticate to SharePoint Online via Connect-PnPOnline and to Microsoft Graph API interactively on behalf of a user.

Problem

Unfortunately, both “Connect-PnPOnline -Interactive -Url <siteUrl>” or “Connect-PnPOnline -UseWebLogin -Url <siteUrl>” might fail with something like “Need admin approval”, “App needs permission to access resources in your organization that only an admin can grant. Please ask an admin to grant permission to this app before you can use it.” or similar

Solution

  • register an Azure App. Choose “single tenant”
  • configure authentication blade:
    – add platform – “Mobile and Desktop app”
    select “https://login.microsoftonline.com/common/oauth2/nativeclient”
    add custom Redirect URI: “http://localhost”
  • configure API permissions blade:
    – add delegated permissions you need (refer to specific API you’ll use)
    e.g. Microsoft Graph Sites.FullControl.All and SharePoint AllSites.FullControl
  • use the following code samples

PnP.PowerShell

$siteUrl = "https://contoso.sharepoint.com/teams/myTeamsSite"
$appId = "" # Client Id
Connect-PnPOnline -ClientId $appId -Url $siteUrl -Interactive
Get-PnPSite

A pop-up window will appear to authenticate interactively. If you are already authenticated with another credentials (or single-sigh-on) – an interactive window might pop up and disappear – that prevents you enter your other id.
To ensure Connect-PnPOnline prompts you for your credentials – use ” -ForceAuthentication” option.

If you are a SharePoint tenant admin – you can connect to a tenant with:

$orgName = "yourTenantPrefix" 
$adminUrl = "https://$orgName-admin.sharepoint.com" 
$appId = "" # Client Id 
$connection = Connect-PnPOnline -ClientId $appId -Url $adminUrl -Interactive -ReturnConnection # -ForceAuthentication 
$connection 

Microsoft Graph API

Use MSAL.PS module to get an msal token then use token in Microsoft graph-based requests:

$tenantId = ""
$clientid = ""
$url = ""
$token = Get-MsalToken -ClientId $clientid -TenantId $tenantId -Interactive

By default token expires in ~ 1 hour. But you can refresh it silently.
This helps you in long-running PowerShell scripts that takes hours to complete.
So you can include something like this in the loop:

if ($token.ExpiresOn.LocalDateTime -lt $(get-date).AddMinutes(10)) {    
  $token = Get-MsalToken -ClientId $clientid -TenantId $tenantId -ForceRefresh -Silent    
  Write-Host "Token will expire on:" $token.ExpiresOn.LocalDateTime
}

Application permissions

Somehow using Connect-PnPOnline with AccessToken option did not work if the token was acquired with MSAL.PS interactively. But it did work when you get msal.ps token unattended (using App credentials). So…

If you can get an Application (non Delegated) permissions to your azure-registerd-app,
you can use msal token to connect to site with PnP

=========================

NB: For delegated permissions, the effective permissions of your app are the intersection of the delegated permissions the app has been granted (via consent) and the privileges of the currently signed-in user. Your app can never have more privileges than the signed-in user.

Track SharePoint App-only Service Principals in Microsoft 365

Scenario

Developers in the organization can use both – Azure Apps and SharePoint Apps to work with SharePoint sites in their “daemon” applications.

It is recommended to use Azure apps so, you want to know – what are SharePoint Apps registered and their owners, who registered SharePoint Apps. Eventually you would disable SharePoint Apps-only principal but before that you’d move Devs from SP-App-only to Azure App (see Disable Custom App Authentication).

(SharePoint App-only service principals aka SP-App-Only are SPN or App registered from within SharePoint using AppRegNew.aspx system page).

One of the approaches – track Apps/Owners with Unified Audit Log

Use Unified Audit Logs

The following PowerShell code:

$operations = 'Add service principal.'
$recordType = 'AzureActiveDirectory'
Search-UnifiedAuditLog -StartDate $start -EndDate $end -ResultSize $resultSize -Formatted -Operations $operations -RecordType $recordType

returns events with operation = ‘Add service principal.’ Nice, but…
if an app was registered in Azure – event contains an UPN under UserIds property:

Unfortunately, in case with registering app in SharePoint, an audit log event will be like:

i.e. UserId registerd is “spo_service@support.onmicrosoft.com”, so we do not know who registered a SharePoint-only app

In theory – we could use events recorded immediately before and after “Add service principal” event to track a user and site who has registered a SharePoint-only app… But for me it seems like too complicated for automation.

Instead we can do simple search through audit log for events “AppRegNew.aspx page visited”. This gives us a good approximation of who registered SP-App-only principal. Worst scenario – we reach more people than we really need (including those who started registering sp-app-only but did not complete) but all of them would be definitely our target auditory.

Consider the following code:

$freeText = "appregnew"
$operations = 'PageViewed'
$recordType = 'SharePoint'

$results = Search-UnifiedAuditLog -StartDate $start -EndDate $end -ResultSize $resultSize -FreeText $freeText -Operations $operations -RecordType $recordType 

this would give you all users who loaded “/_layouts/15/appregnew.aspx” page

Update: Sites.Selected API MS Graph permissions was introduced by Microsoft in 2021. It was a good move towards site-level development, but still developers were limited with only what MS Graph API provides for SharePoint dev.
So devs had to use AppInv.aspx at site level to provide ACS permissions to their apps to be able to use SharePoint CSOM and REST APIs.
Recently Microsoft introduced Sites.Selected SharePoint API permissions for registered Azure Apps! So now devs should be fully happy without ACS-based permissions.

References

How to Find Content Shared with Everyone in SharePoint and Teams

There is a known problem in SharePoint – complicated permissions system. Site owners/administrators provide access, site contributors upload documents and nobody knows – who has access to their sites. As a result – sometimes sensitive documents become overshared (over-exposed).

The biggest concern is sites content shared with “Everyone”. How do we find sites shared with “Everyone” in a large Microsoft 365 environment?

NB. When I say “shared with Everyone” – I actually mean 3 possible “everyone” logins:

  • Everyone
  • Everyone except external users
  • All users

Approach #1 (Brute force)

We can get full permissions report at tenant level (or permissions provided only to “Everyone”). There are 3-rd party tools (e.g. ShareGate, SysKit, AvePoint, Metalogix etc.), or you can run PowerShell script…

Sounds easy? Well, if you have less than 1000 sites – probably it will work. But if your environment is 10K+ sites – it will take forever. Permission report might run hours for an average site with site/subsite, list/library and list item details level. So the approach will not work for large enterprise environments.

One might say – we can limit report with root web permissions only to get it faster. But this would not be accurate. And what is not accurate in the IT security – lead to even bigger risks. So, we need report detailed up to every item level deep, as even one file with sensitive info shared with everyone can cause security issue. (3-rd party tools usually by default limit it to libraries level.)

Ok, if this approach is not really working – what’s working?

Clever idea: why do we need to iterate through all the tenant documents/items if all the content is already crawled by search? Search is also respect permissions. Can we just use search to get files shared with Everyone? Let us see.

What if we use some dummy user account with no specific permissions provided and no group membership and try to search content on behalf of that account. The idea is if this user can see data – it means that data is open for everyone.

Check this and this articles. Can we get results programmatically (e.g. with PowerShell)? Can we use Microsoft Graph search API? Sure.
Check this article “How to search against SharePoint Online Content with Microsoft Graph search API with PowerShell”.

But! We have some problems here.

Search Problem #1. Again, for small environments or if there are not much “Open” sites – it would work. But for large enterprise environments the problem is the same as in “brute force”. Search returns too many results – it’ll take weeks to get all of them. (There are team sites “legally” shared with everyone, public Office 365 group based sites, communication sites… ).

Search Problem #2. Even if we get all search results – we do not know – at what level permissions are provided to everyone. So we will need to build list of sites based on the search results – ant then still need to run permissions report against these sites.

Search Problem #3. We are getting results with paging. But recently Microsoft started limiting number of returning results. E.g. your search request result might say like “there are 3659735 total hits” but after result number 1000 it just stops returning anything, even with paging.

Approach # 3 Hybrid

The idea: why do we need to get all search results if even one result from a site would be enough to add the site to the list of sites require permissions review. In other words, we do not need all results from site, we only need one to know the site is open.

So, consider (imho, the best) approach (Solution):

  1. You get list of all sites in tenant.
  2. You run search request against each site in the loop
    (e.g. consider KQL option “Site: https://yourTenant.SharePoint.com/sites/YourSite”.
    If at least something found in the site – add the site to the “Open Sites” list.
    With this approach you will get list of sites shared with “Everyone…” in a couple of minutes.
  3. Run permissions report against this shortlist

Note: consider there are resources like “Styles Library” shared with everyone by default.

Note: You can refine the list you get at step 1 – e.g., excluding sites connected to public teams or known communication sites…

Note: consider implementing sensitivity labels. At least you can start with high-sensitive sites. Site owners/member will know – what kind of site they are working on.

Pros and cons of the Approach # 3 Hybrid

Pro: the only fast and accurate enough to rely on

Con 1 : crawling and indexing takes time, so search-based reports can miss recent changes in data and permissions

Con 2: this approach cannot be automated (since we need an interactive authentication).

How to communicate to site owners

The Next step would be “How to let site owners know that there are resources shared with Everyone… on their sites”.

References

Access SPO Site Programmatically via MS Graph API and SharePoint API

Scenario

You are a software developer. Your company uses Microsoft Office 365 (SharePoint, Teams etc.). The need is to work with a specific site collection programmatically (from code – Python, C#, Java, PowerShell, JavaScript etc.) – e.g. upload/download documents, update list items, search etc.

The code must run without user interaction (unattended, aka daemon app). Sometimes this is also called “SharePoint Automation”.

The solution is based on a new Graph API feature – Sites.Selected and a classic SP-Only app.

Solution

  1. Register an Azure App and configure it as usual.
    Select API Permissions blade and add two permissions:
    – Microsoft Graph -> Applications Permissions -> “sites.selected
    – SharePoint -> Applications Permissions -> “sites.selected
  2. Request “Grant admin consent” from a tenant/global admin
  3. Request SharePoint admin to run PowerShell code (e.g. this one) to assign proper permissions to your azure app for a specific site collection (consider site owner consent)
  4. (optionally) Provide SharePoint API permissions:
    (require Site Collection Owner/Admin account) – use
    https://YourTenant.sharepoint.com/teams/YourSite/_layouts/15/appinv.aspx
    to add SharePoint API permissions to your app. E.g. full control permissions to site collection would be
<AppPermissionRequests AllowAppOnlyPolicy="true">  
   <AppPermissionRequest Scope="http://sharepoint/content/sitecollection" 
    Right="FullControl" />
</AppPermissionRequests>

Consider minimal permissions (e.g. as per Sumit)

Problem Solved

  • you get access to one and only one site collection (“least privilege” principal)
  • you get both – SharePoint API and Microsoft Graph API permissions to SharePoint
  • you can use app secret or certificate to authenticate – depending on what are your security requirements

Note: if your scenario require authenticated user present – the solution would be a little different: Connect-PnPOnline Interactive with Client App Id

Update:

Sites.Selected API MS Graph permissions was introduced by Microsoft in 2021. It was a huge step forward, but still devs were limited with MS Graph API against SharePoint.
So devs had to use AppInv at site level to provide ACS permissions to their apps to use SharePoint CSOM and REST APIs.
Recently Microsoft introduced Sites.Selected SharePoint API permissions for registered Azure Apps! So now devs should be fully happy without ACS-based permissions AppInv.aspx. (See more here on disabling SP Apps Only SPNs)

Thanks to Leon Armston and Scott Murdock

References:

Microsoft Search: country-targeted bookmarks

If you have country-specific content – Microsoft Search allows bookmarks to be configured to pop-up only for users from a specific country.

But it does not work (or works only for US). What is problem? How do we troubleshoot it? How can we simulate user to be located in a specific country?

How does Microsoft define “this user is from that country”. What are the criteria to correlate User <-> Country? Physical IP address? License assigned country? Locale? Browser settings? Azure AD properties?

Any ideas/experience will be appreciated.
Please let me know.

Resources: