Tag Archives: SPO

How to Find Content Shared with Everyone in SharePoint and Teams

There is a known problem in SharePoint – complicated permissions system. Site owners/administrators provide access, site contributors upload documents and nobody knows – who has access to their sites. As a result – sometimes sensitive documents become overshared (over-exposed).

The biggest concern is sites content shared with “Everyone”. How do we find sites shared with “Everyone” in a large Microsoft 365 environment?

NB. When I say “shared with Everyone” – I actually mean 3 possible “everyone” logins:

  • Everyone
  • Everyone except external users
  • All users

Approach #1 (Brute force)

We can get full permissions report at tenant level (or permissions provided only to “Everyone”). There are 3-rd party tools (e.g. ShareGate, SysKit, AvePoint, Metalogix etc.), or you can run PowerShell script…

Sounds easy? Well, if you have less than 1000 sites – probably it will work. But if your environment is 10K+ sites – it will take forever. Permission report might run hours for an average site with site/subsite, list/library and list item details level. So the approach will not work for large enterprise environments.

One might say – we can limit report with root web permissions only to get it faster. But this would not be accurate. And what is not accurate in the IT security – lead to even bigger risks. So, we need report detailed up to every item level deep, as even one file with sensitive info shared with everyone can cause security issue. (3-rd party tools usually by default limit it to libraries level.)

Ok, if this approach is not really working – what’s working?

Clever idea: why do we need to iterate through all the tenant documents/items if all the content is already crawled by search? Search is also respect permissions. Can we just use search to get files shared with Everyone? Let us see.

What if we use some dummy user account with no specific permissions provided and no group membership and try to search content on behalf of that account. The idea is if this user can see data – it means that data is open for everyone.

Check this and this articles. Can we get results programmatically (e.g. with PowerShell)? Can we use Microsoft Graph search API? Sure.
Check this article “How to search against SharePoint Online Content with Microsoft Graph search API with PowerShell”.

But! We have some problems here.

Search Problem #1. Again, for small environments or if there are not much “Open” sites – it would work. But for large enterprise environments the problem is the same as in “brute force”. Search returns too many results – it’ll take weeks to get all of them. (There are team sites “legally” shared with everyone, public Office 365 group based sites, communication sites… ).

Search Problem #2. Even if we get all search results – we do not know – at what level permissions are provided to everyone. So we will need to build list of sites based on the search results – ant then still need to run permissions report against these sites.

Search Problem #3. We are getting results with paging. But recently Microsoft started limiting number of returning results. E.g. your search request result might say like “there are 3659735 total hits” but after result number 1000 it just stops returning anything, even with paging.

Approach # 3 Hybrid

The idea: why do we need to get all search results if even one result from a site would be enough to add the site to the list of sites require permissions review. In other words, we do not need all results from site, we only need one to know the site is open.

So, consider (imho, the best) approach (Solution):

  1. You get list of all sites in tenant.
  2. You run search request against each site in the loop
    (e.g. consider KQL option “Site: https://yourTenant.SharePoint.com/sites/YourSite”.
    If at least something found in the site – add the site to the “Open Sites” list.
    With this approach you will get list of sites shared with “Everyone…” in a couple of minutes.
  3. Run permissions report against this shortlist

Note: consider there are resources like “Styles Library” shared with everyone by default.

Note: You can refine the list you get at step 1 – e.g., excluding sites connected to public teams or known communication sites…

Note: consider implementing sensitivity labels. At least you can start with high-sensitive sites. Site owners/member will know – what kind of site they are working on.

Pros and cons of the Approach # 3 Hybrid

Pro: the only fast and accurate enough to rely on

Con 1 : crawling and indexing takes time, so search-based reports can miss recent changes in data and permissions

Con 2: this approach cannot be automated (since we need an interactive authentication).

How to communicate to site owners

The Next step would be “How to let site owners know that there are resources shared with Everyone… on their sites”.

References

Access SPO Site Programmatically via MS Graph API and SharePoint API

Scenario

You are a software developer. Your company uses Microsoft Office 365 (SharePoint, Teams etc.). The need is to work with a specific site collection programmatically (from code – Python, C#, Java, PowerShell, JavaScript etc.) – e.g. upload/download documents, update list items, search etc.

The code must run without user interaction (unattended, aka daemon app). Sometimes this is also called “SharePoint Automation”.

The solution is based on a new Graph API feature – Sites.Selected and a classic SP-Only app.

Solution

  1. Register an Azure App and configure it as usual.
    Select API Permissions blade and add two permissions:
    – Microsoft Graph -> Applications Permissions -> “sites.selected
    – SharePoint -> Applications Permissions -> “sites.selected
  2. Request “Grant admin consent” from a tenant/global admin
  3. Request SharePoint admin to run PowerShell code (e.g. this one) to assign proper permissions to your azure app for a specific site collection (consider site owner consent)
  4. (optionally) Provide SharePoint API permissions:
    (require Site Collection Owner/Admin account) – use
    https://YourTenant.sharepoint.com/teams/YourSite/_layouts/15/appinv.aspx
    to add SharePoint API permissions to your app. E.g. full control permissions to site collection would be
<AppPermissionRequests AllowAppOnlyPolicy="true">  
   <AppPermissionRequest Scope="http://sharepoint/content/sitecollection" 
    Right="FullControl" />
</AppPermissionRequests>

Consider minimal permissions (e.g. as per Sumit)

Problem Solved

  • you get access to one and only one site collection (“least privilege” principal)
  • you get both – SharePoint API and Microsoft Graph API permissions to SharePoint
  • you can use app secret or certificate to authenticate – depending on what are your security requirements

Note: if your scenario require authenticated user present – the solution would be a little different: Connect-PnPOnline Interactive with Client App Id

Update:

Sites.Selected API MS Graph permissions was introduced by Microsoft in 2021. It was a huge step forward, but still devs were limited with MS Graph API against SharePoint.
So devs had to use AppInv at site level to provide ACS permissions to their apps to use SharePoint CSOM and REST APIs.
Recently Microsoft introduced Sites.Selected SharePoint API permissions for registered Azure Apps! So now devs should be fully happy without ACS-based permissions AppInv.aspx. (See more here on disabling SP Apps Only SPNs)

Thanks to Leon Armston and Scott Murdock

References:

PnP.PowerShell Release 1.3.0

Great news:

Added -Interactive login option to Connect-PnPOnline which is similar to -UseWebLogin but without the limitations of the latter. The -UseWebLogin is using cookie based authentication towards SharePoint and cannot access Graph tokens. Using -Interactive we use Azure AD Authentication and as a result we are able to acquire Graph tokens.

more changes: https://github.com/pnp/powershell/releases/tag/1.3.0

PnP.PowerShell Batches and PowerShell 7 Parallel

Parallelism

Can I use PowerShell 7 “-Parallel” option against SharePoint list items with PnP.PowerShell? Can I run something like:

$items | ForEach-Object -Parallel {
    $listItem = Set-PnPListItem -List "LargeList" -Identity $_ -Values @{"Number" = $(Get-Random -Minimum 100 -Maximum 200 ) }
} 

Yes, sure… But! Since it’s a cloud operation against Microsoft 365 – you will be throttled if you start more than 2 parallel threads! Using just 2 threads does not provide significant performance improvements.

Batching

So, try PnP.PowerShell batches instead. When you use batching, number of requests to the server are much lower. Consider something like:

$batch = New-PnPBatch
1..100 | ForEach-Object{ Add-PnPListItem -List "ItemTest" -Values @{"Title"="Test Item Batched $_"} -Batch $batch }
Invoke-PnPBatch -Batch $batch


Measurements

Adding and setting 100 items with “Add-PnPListItem” and “Set-PnPListItem” in a large (more than 5000 items ) SharePoint list measurements:

Add-PnPListItem
Time per item, seconds
Set-PnPListItem
Time per item, seconds
Regular, without batching1.261.55
Using batches (New-PnPBatch)0.100.80
Using “Parallel” option, with ThrottleLimit 20.690.79
Using “Parallel” option, with ThrottleLimit 30.44 (fails level: ~4/100) 0.53 (fails level: ~3/100)

Adding items with PnP.PowerShell batching is much faster than without batching.

More:

SPO: Allow users to create modern pages

Microsoft: “Using modern pages in Microsoft SharePoint is a great way to share ideas using images, Office files, video, and more. Users can Add a page to a site quickly and easily, and modern pages look great on any device.
If you’re a global or SharePoint admin in Microsoft 365, you can allow or prevent users from creating modern pages. You can do this at the organization level by changing settings in the SharePoint admin center. If you allow the creation of site pages as the organization level, site owners can turn it on or off at the site level.

By default both
– Allow users to create new modern pages
– Allow commenting on modern pages
are turned on (enabled)

Tenant or SharePoint admin can find settings under
SharePoint Admin Center -> Settings -> Pages

How it looks like:

Site Pages are created under “Pages” Library.

Let us test it, with:
– (tenant-level) Allow users to create new modern pages: ON
– (tenant-level) Allow commenting on modern pages: ON
– web feature “Site Pages” – “Allows users to add new site pages to a site”: Activated

User
Permissions
can create Pagecan edit pagecan Enable/Disable
page comments
can comment on Page
Full Control (Owner)YesYesYesYes
Edit (Member)YesYesYesYes
Read (Visitor)NoNoNoYes

There is a web feature “Site Pages” – “Allows users to add new site pages to a site”.
The feature is activated by default:

What if we disable this feature?
“New -> Page” has disappeared from “New” menu under “Site Contents” for Owners and Members…
From “Home” and “Pages” you still can see “New -> Page” options.
You can still create a new page from but if you try to create a page from Pages – “Sorry, something went wrong” “Cannot create a Site Page. Please have your administrator enable the required feature on this site.” :

Office 365 behavior, with:
– (tenant-level) Allow users to create new modern pages: ON
– (tenant-level) Allow commenting on modern pages: ON
– web feature “Site Pages” – “Allows users to add new site pages to a site”: Deactivated

User
Permissions
can create Pagecan edit pagecan Enable/Disable
page comments
can comment on Page
Full Control (Owner)Yes,
but only from “Home”
not from “Site Contents” or “Pages”
YesYesYes
Edit (Member)Yes,
but only from “Home”
not from “Site Contents” or “Pages”
YesYesYes
Read (Visitor)NoNoNoYes


If we disable feature “Site Pages” – “Allows users to add new site pages to a site” on the root web – it does not affect subsites (subwebs).

Can we Activate/Deactivate the feature “Site Pages” using PowerShell?

PowerShell

(TBP)

References
– Microsoft “Allow users to create and comment modern pages

See also:
Allow commenting on modern pages

How to delete a large SPO list and/or all items in a large SPO list

Scenario 1: You have a large (>5k items) list in SharePoint Online.
You need to delete this list.

Scenario 2: You have a large (>5k items) list in SharePoint Online.
You need to delete all the list items, but keep the list.

Deleting a large SharePoint Online list

GUI: Microsoft improved SharePoint, so now it takes ~1 second to delete any SharePoint list, including 5000+ items list via GUI.

PowerShell: “Remove-PnPList -Identity $list” command works very fast – ~1 second to delete entire list with >5000 items.

Delete all items in a large SharePoint Online list

In this scenario we need to keep the list, but make it empty (clean it out).

GUI: You can change the list view settings “Item Limit” to <5000, but (at least in my experience) when you try to select, let say, 1000 items and delete them via GUI – it says “775 items were not deleted from large list”:

so this option seems like not a good one.

ShareGate: 3-rd party tools like Sharegate, SysKit give a good results too.

PowerShell

Try this PowerShell command with ScriptBlock:

Get-PnPListItem -List $list -Fields "ID" -PageSize 100 -ScriptBlock { Param($items) $items | Sort-Object -Property Id -Descending | ForEach-Object{ $_.DeleteObject() } } 

or this PowerShell with batches:

$batch = New-PnPBatch
1..12000 | Foreach-Object { Remove-PnPListItem -List $list -Identity $_ -Batch $batch }
Invoke-PnPBatch -Batch $batch

for me both methods gave same good result: ~17 items per second ( ~7 times faster than regular).

PnP.PowerShell batch vs ScriptBlock

How fast are PnP batches? What is better in terms of performance – ScriptBlock or Batching? Here are my measurements:

Time elapsed, secondswith batcheswith scriptBlockwithout batches
Add-PnPListItem (100 items)4-10 seconds42-120 seconds
Add-PnPListItem (500 items)20-40 seconds230-600 seconds
Add-PnPListItem (7000 items)314-560 seconds
Remove-PnPListItem (1000 items)58-103 seconds58 seconds430-1060 seconds
Remove-PnPListItem (7000 items)395-990 seconds397-980 seconds

both – PnP PowerShell batches and ScriptBlocks are 7-10 times faster than plain PnP PowerShell!

Note… For the sake of history: It used to be like that for 5k+ lists:
“Remove-PnPList” fails with a message “The attempted operation is prohibited because it exceeds the list view threshold enforced by the administrator”. Deleting with GUI fails too.

References:

Office 365 retention labels and policies for SharePoint

As I am a SharePoint person, and retention policies and labels are not a SharePoint engineer responsibility, I do not go to the m365 Compliance Center frequently. Below are My notes for myself on key moments – how to create and configure Office 365 retention labels and Policies at Compliance Center and use labels in SharePoint Online (SPO).

In SPO at each site collection level you can still work with retention policies the old way – create policies under Site Collection Settings – Content Type Policy – and apply policies at library level under Library Settings/Information Management Policy Settings. There is also Site Retention Policy.

But Microsoft is making efforts to centralize and unify such things – so you can specify retention policies in one place and apply them across all Office 365 content (not only SharePoint). That place was called Office 365 Security and Compliance Center (SCC). Later Microsoft separated Security Center and Compliance Center. So currently Retention Policies are under “Microsoft Purview” (former Microsoft Compliance Center) -> Solutions -> “Data lifecycle management”:

To get access to “Data lifecycle management” solution – you need to have a “” or “” roles. SharePoint or Teams administrator cannot access Purview. Even having “Global reader” or “Security reader” an admin will not be able to see “Data lifecycle management” blade. Here is how Microsoft Purview looks like for a Global reader:

Although SharePoint admins usually do not have access to SCC and do not go to Site content, we still need to know how it all works. And labels are recommended way to specify retention in SharePoint, so here we are.

Labels are applied to documents, documents are kept in libraries, and at each library you can “Apply a label to items in this library”.

Create Labels

Labels are created in SCC under Classification. The main part looks familiar to SharePoint people:

Label Settings

You can

  • Retain Content forever or for a specified number of days/months/years and then
    – delete it or trigger a disposition review or do nothing
  • Delete content if it’s older than specified number of days/months/years

after it was created/modified/labelled

Apply labels

Now you need to publish created labels – and that is how you create a policy. I.e. policies are where you specify which labels to which content (Exchange, OneDrive, SharePoint, Office 365 groups)

You can also auto-apply labels based on conditions, like

  • content that contains sensitive info
  • content that contains specific words or phrases, or properties
  • content that matches a trainable classifier

but as per Microsoft, “It will take up to 7 days to automatically apply the label to all items that match your conditions.”

Note: “trainable classifier” means an AI ML will be used, and as per Microsoft “Creating machine learning rules requires an Office 365 E5 subscription for your organization”

SharePoint admin center

You can do nothing with labels at SharePoint admin center. Labels are created, published and auto-applied at SCC. At each site collection levels site administrators can apply labels.

SharePoint site

At site collection settings you can still see “Content Type Policy Templates” and “Site Policy”, but that is not the case. Labels are applied at library level under Library Settings/Apply label to items in this list or library.

where you can select a label to apply for all new items in the library. With

You can also apply the label to items that already exist in the library.

You can also apply (change) label for each single item or multiple selected items under Details pop-up page:

or from under Contect Menu/More/Compliance details:

Adaptive retention policies and scopes

Microsoft recently implemented “Adaptive” retention policies. At step 2 of “Create retention policy” you’ll be asked “Choose the type of retention policy to create”: “A policy can be adaptive or static. Advantage of an adaptive policy will automatically update where it’s applied based on attributes or properties you’ll define. A static policy is applied to content in a fixed set of locations and must be manually updated if those locations change.”

And if you selected “Adaptive” – on the next step you will need to provide the adaptive scope (so at this moment you should already have created your adaptive scopes):

So, let us create your adaptive scopes.
What type of scope do you want to create? SharePoint sites…

And then you’ll have nothing more then set of conditions:

where you can use objects: “Site Url”, “Site Name” and “Refinable String 0″..”Refinable String 99”. Conditions would be “is equal to”, “is not equal to”, “starts with” and “not starts with”. Or you can select “Advanced query builder” and enter LQL query.

What is the takeaway from this for SharePoint administrators? We would be asked to configure SharePoint the way compliance/retention people can use Refinable Strings.



References