Category Archives: SPO

How to Find Content Shared with Everyone in SharePoint and Teams

There is a known problem in SharePoint – complicated permissions system. Site owners/administrators provide access, site contributors upload documents and nobody knows – who has access to their sites. As a result – sometimes sensitive documents become overshared (over-exposed).

The biggest concern is sites content shared with “Everyone”. How do we find sites shared with “Everyone” in a large Microsoft 365 environment?

NB. When I say “shared with Everyone” – I actually mean 3 possible “everyone” logins:

  • Everyone
  • Everyone except external users
  • All users

Approach #1 (Brute force)

We can get full permissions report at tenant level (or permissions provided only to “Everyone”). There are 3-rd party tools (e.g. ShareGate, SysKit, AvePoint, Metalogix etc.), or you can run PowerShell script…

Sounds easy? Well, if you have less than 1000 sites – probably it will work. But if your environment is 10K+ sites – it will take forever. Permission report might run hours for an average site with site/subsite, list/library and list item details level. So the approach will not work for large enterprise environments.

One might say – we can limit report with root web permissions only to get it faster. But this would not be accurate. And what is not accurate in the IT security – lead to even bigger risks. So, we need report detailed up to every item level deep, as even one file with sensitive info shared with everyone can cause security issue. (3-rd party tools usually by default limit it to libraries level.)

Ok, if this approach is not really working – what’s working?

Clever idea: why do we need to iterate through all the tenant documents/items if all the content is already crawled by search? Search is also respect permissions. Can we just use search to get files shared with Everyone? Let us see.

What if we use some dummy user account with no specific permissions provided and no group membership and try to search content on behalf of that account. The idea is if this user can see data – it means that data is open for everyone.

Check this and this articles. Can we get results programmatically (e.g. with PowerShell)? Can we use Microsoft Graph search API? Sure.
Check this article “How to search against SharePoint Online Content with Microsoft Graph search API with PowerShell”.

But! We have some problems here.

Search Problem #1. Again, for small environments or if there are not much “Open” sites – it would work. But for large enterprise environments the problem is the same as in “brute force”. Search returns too many results – it’ll take weeks to get all of them. (There are team sites “legally” shared with everyone, public Office 365 group based sites, communication sites… ).

Search Problem #2. Even if we get all search results – we do not know – at what level permissions are provided to everyone. So we will need to build list of sites based on the search results – ant then still need to run permissions report against these sites.

Search Problem #3. We are getting results with paging. But recently Microsoft started limiting number of returning results. E.g. your search request result might say like “there are 3659735 total hits” but after result number 1000 it just stops returning anything, even with paging.

Approach # 3 Hybrid

The idea: why do we need to get all search results if even one result from a site would be enough to add the site to the list of sites require permissions review. In other words, we do not need all results from site, we only need one to know the site is open.

So, consider (imho, the best) approach (Solution):

  1. You get list of all sites in tenant.
  2. You run search request against each site in the loop
    (e.g. consider KQL option “Site: https://yourTenant.SharePoint.com/sites/YourSite”.
    If at least something found in the site – add the site to the “Open Sites” list.
    With this approach you will get list of sites shared with “Everyone…” in a couple of minutes.
  3. Run permissions report against this shortlist

Note: consider there are resources like “Styles Library” shared with everyone by default.

Note: You can refine the list you get at step 1 – e.g., excluding sites connected to public teams or known communication sites…

Note: consider implementing sensitivity labels. At least you can start with high-sensitive sites. Site owners/member will know – what kind of site they are working on.

Pros and cons of the Approach # 3 Hybrid

Pro: the only fast and accurate enough to rely on

Con 1 : crawling and indexing takes time, so search-based reports can miss recent changes in data and permissions

Con 2: this approach cannot be automated (since we need an interactive authentication).

How to communicate to site owners

The Next step would be “How to let site owners know that there are resources shared with Everyone… on their sites”.

References

Access SPO Site Programmatically via MS Graph API and SharePoint API

Scenario

You are a software developer. Your company uses Microsoft Office 365 (SharePoint, Teams etc.). The need is to work with a specific site collection programmatically (from code – Python, C#, Java, PowerShell, JavaScript etc.) – e.g. upload/download documents, update list items, search etc.

The code must run without user interaction (unattended, aka daemon app). Sometimes this is also called “SharePoint Automation”.

The solution is based on a new Graph API feature – Sites.Selected and a classic SP-Only app.

Solution

  1. Register an Azure App and configure it as usual.
    Select API Permissions blade and add two permissions:
    – Microsoft Graph -> Applications Permissions -> “sites.selected
    – SharePoint -> Applications Permissions -> “sites.selected
  2. Request “Grant admin consent” from a tenant/global admin
  3. Request SharePoint admin to run PowerShell code (e.g. this one) to assign proper permissions to your azure app for a specific site collection (consider site owner consent)
  4. (optionally) Provide SharePoint API permissions:
    (require Site Collection Owner/Admin account) – use
    https://YourTenant.sharepoint.com/teams/YourSite/_layouts/15/appinv.aspx
    to add SharePoint API permissions to your app. E.g. full control permissions to site collection would be
<AppPermissionRequests AllowAppOnlyPolicy="true">  
   <AppPermissionRequest Scope="http://sharepoint/content/sitecollection" 
    Right="FullControl" />
</AppPermissionRequests>

Consider minimal permissions (e.g. as per Sumit)

Problem Solved

  • you get access to one and only one site collection (“least privilege” principal)
  • you get both – SharePoint API and Microsoft Graph API permissions to SharePoint
  • you can use app secret or certificate to authenticate – depending on what are your security requirements

Note: if your scenario require authenticated user present – the solution would be a little different: Connect-PnPOnline Interactive with Client App Id

Update:

Sites.Selected API MS Graph permissions was introduced by Microsoft in 2021. It was a huge step forward, but still devs were limited with MS Graph API against SharePoint.
So devs had to use AppInv at site level to provide ACS permissions to their apps to use SharePoint CSOM and REST APIs.
Recently Microsoft introduced Sites.Selected SharePoint API permissions for registered Azure Apps! So now devs should be fully happy without ACS-based permissions AppInv.aspx. (See more here on disabling SP Apps Only SPNs)

Thanks to Leon Armston and Scott Murdock

References:

Authenticate to Microsoft Graph from PowerShell Interactively

Scenario

You are a developer or power user in a company with Microsoft 365 tenant.
You need to connect to Microsoft Graph and then call Microsoft Graph API to consume some MS Graph resources on behalf of authenticated user programmatically with PowerShell – e.g. add/remove documents or list items, search for sites or documents content etc. – whatever available with Graph API.

You do not have tenant admin permissions or any tenant-level admin permissions (SharePoint, Teams, Exchange etc. ). But you can register an Azure App and request tenant admin consent.

Solution

  • register an Azure App
  • under authentication blade – add platform – “Mobile and Desktop app”
    add “http://localhost” (and select …/nativeclient Url ?)
  • under API permissions blade – add delegated permissions you need
    (refer to specific API you’ll use)
  • install MSAL.PS PowerShell module
  • use the following code to get graph access token and call graph API
$AppId = ""
$TenantId = ""
$connectionDetails = @{
    'TenantId'    = $AppId
    'ClientId'    = $TenantId
    'Interactive' = $true
}

$token = Get-MsalToken @connectionDetails
# or 
$token = Get-MsalToken -TenantId $TenantId -ClientId $appId -Interactive 

$Headers = @{
    'Authorization' = "bearer $($token.AccessToken)"
}

Invoke-RestMethod -Uri 'https://graph.microsoft.com/v1.0/me' -Headers $Headers

You can find the code sample here: https://github.com/VladilenK/

Did not work:

Az PowerShell module did not work for me:

Connect-AzAccount -Tenant ""
$azAccessToken = Get-AzAccessToken -Resource "https://graph.microsoft.com" 

$Headers = @{
  'Authorization' = "$($azAccessToken.Type) $($azAccessToken.Token)"
}

Invoke-RestMethod -Uri 'https://graph.microsoft.com/v1.0/me' -Headers $Headers

As I understand we need somehow let Azure know API permissions we want (e.g. via app registerd)…

PnP did not work for me too:

$url = "https://orgname.sharepoint.com"
Connect-PnPOnline -ClientId "" -Url $url -Interactive 
$pnpToken = Get-PnPGraphAccessToken 
$Headers = @{
    'Authorization' = "bearer $($pnpToken)"
}
Invoke-RestMethod -Uri 'https://graph.microsoft.com/v1.0/me' -Headers $Headers

# did not work as well:
$pnpToken = Get-PnPAppAuthAccessToken
$pnpToken = Get-PnPAccessToken 

the error message was (maybe I missed something – please let me know):

“code”: “InvalidAuthenticationToken”, “message”: “Access token validation failure. Invalid audience.”

References

Microsoft Search country-targeted bookmarks: new “Use Azure AD locations” flag

If you have country-specific content – Microsoft Search allows bookmarks to be configured to pop-up only for users from a specific country.

And “Use Azure AD locations” flag is a new option that make it actually works…
For a long time country settings were the same but without “Use Azure AD locations” flag. So what does “Use Azure AD locations” flag do?

Use Azure AD locations

“Use Azure AD locations” flag is a straight-forward configuration settings. It says: “This bookmark will only appear for users with Azure AD locations that match selected countries or regions. If cleared, the user’s IP address will be used to determine location. This checkbox can be altered from both Country or region setting and Targeted variations setting.”

I have tested this new “Use Azure AD locations” flag – it works. Once you configure user’s country in AAD and country-targeted bookmarks – all works good. Bookmarks appear for the user.

What if we do not “Use Azure AD locations” flag

to be provided

What was before “Use Azure AD locations” flag

What was before Microsoft implemented this “Use Azure AD locations” option? How did Microsoft understand “this user is from that country”. What was the criteria to correlate User <-> Country? License assigned country? Windows locale? Browser settings? Azure AD properties?

How does Microsoft define “this user is from that country”. What are the criteria to correlate User <-> Country? Physical IP address? License assigned country? Locale? Browser settings? Azure AD properties?

It turned out, the way it was designed previously:
– configure Microsoft 365 integration with Bing
– in Bing -> Settings : select Country/Region
– search from Bing
that was the only way to make it work! So yes, do not leave “Use Azure AD locations” option unchecked. Microsoft confirmed it was poor design.

Always “Use Azure AD locations” flag.

Resources:

SharePoint sites shared with Everyone and Microsoft Delve issue

There is a known problem with Microsoft Delve. It’s not a technology problem though.

We know SharePoint site permissions are not easy to manage. E.g. you can break permissions inheritance at any level – subsite, library, list, folder, list item or specific document. Anybody with full permissions can do that. The worst thing is there is was (*1) no native ability for site owner to get full site permissions report. We must have used third-party tools or PowerShell to have all permissions in one document.

So no wonder SharePoint sites were heavily over-exposed. Especially when a site owner tired with complexity of SharePoint permissions system decided to share resource with “Everyone”. And the other person, not knowing site is shared with everyone, might save some sensitive data. That is the real issue.

Now, what is Delve? It’s a service that
– get signals from allover Office 365 – who did what etc.
– based on that, using AI and Office Graph, generates suggestions – “what others do”.
Of course, Delve is security-trimmed, i.e. it will neve suggest you a document you do not have access to. But some sites might be overshared. Delve works as it should work – it suggests you documents it believes related to you (based on Microsoft Graph insights) and you already have access to.

Now bad thing happens – people start seeing documents they never new they have access to. Where are these documents from? Of course from sites shared with Everyone. Who to blame for the security breach? Delve? Microsoft Graph? Microsoft 365 SharePoint Online?

Strictly says, it is not Delve’s problem. It’s more human problem than technological.
Delve just does it’s job, and does perfectly. Delve simply displays the information already shared widely.

How do we solve the issue?

  1. Disable Delve?
  2. Disable search (stop sites crawling and remove results)?
  3. Restrict users who can provide signals via item insights privacy?
    see Microsoft KBA on how to disable MS Graph for a specific User

Those methods are half-measure. Methods above are just hiding the problem – not solving it. Agree it helps stop the deterioration, bud does not fix the root cause.

How do we solve the real problem and what is the root cause?

  1. Of course, we need remove incorrectly provided permissions. How?
  2. Only site owner (data owner) knows which content should be shared with whom with which access rights. So we need to ask sites owners to review their permissions. How?
  3. First, we need a list of over-exposed sites. How?
  4. There are two methods (more details – check this article)
    • Brute force – use PowerShell or 3-rd party tool to get permission report on all sites in tenant, select permissions provided for Everyone…
    • Smart move – use Microsoft search. As search is security-trimmed, we can search for available content on behalf of a user with no permissions provided.
  5. Then we find owners for each wide-open site. How?
    • for group-based sites we get member of the “owners” group
    • for non-group based sites we get site collection administrators
  6. We would also sort sites by “is it supposed to be public?”. I.e. if the site was born as public – e.g. Public Team or Public Yammer community – or Communication site – maybe it’s less concern.
  7. It would be a good idea to bring DLP and/or automatic content sensitivity labelling, so we could start remediation from sites labelled as storing most sensitive data.
  8. Finally, we need to let site owner know that his site is Open to everybody and ask to fix it. How?


References

Bill Baer’s on search and “prevent sensitive files from being exposed in search”

PnP.PowerShell Release 1.3.0

Great news:

Added -Interactive login option to Connect-PnPOnline which is similar to -UseWebLogin but without the limitations of the latter. The -UseWebLogin is using cookie based authentication towards SharePoint and cannot access Graph tokens. Using -Interactive we use Azure AD Authentication and as a result we are able to acquire Graph tokens.

more changes: https://github.com/pnp/powershell/releases/tag/1.3.0

Connect-PnPOnline with a certificate stored in Azure Key Vault

Scenario

You run some PnP PowerShell code unattended e.g. daemon/service app, background job – under application permissions – with no user interaction.
Your app needs to connect to SharePoint and/or Microsoft Graph API. Your organization require authentication with a certificate (no secrets). You want certificate stored securely in Azure Key Vault.

Solution (Step-by-step process)

  1. Obtain a certificate (create a self-signed or request trusted)
  2. In Azure where you have Microsoft 365 SharePoint tenant
    1. Create a new Registered App in Azure; save App (client) id, Directory (Tenant) Id
    2. Configure App: add MS Graph and SharePoint API application (not delegated) permissions
    3. Upload the certificate to the app under “Certificates & secrets”
  3. In Azure where you have paid subscription (could be same or different)
    1. Create an Azure Key Vault
    2. Upload certificate to the Key Vault manually (with GUI)
  4. While you develop/debug your custom daemon application at your local machine
    1. Provide permissions to the Key Vault via Access Control and Access Policies to your personal account
    2. Connect to Azure (the one where your Key Vault is) running Connect-AzAccount
      – so your app can get a Certificate to authenticate to SharePoint Online
  5. For your application deployed to Azure (e.g. Azure Function App )
    1. Turn On managed identity (Your Function App -> Identity -> Status:On) and Save; notice an Object (Principal) Id just created
    2. Provide for your managed identity principal Id permissions to the Key Vault via Key Vault Access Policies, so when your daemon app is running in the cloud – it could go to the key Vault and retrieve Certificate

Here is the sample PowerShell code to get certificate from Azure Key Vault and Connect to SharePoint with PnP (Connect-PnPOnline):

# ensure you use PowerShell 7
$PSVersionTable

# connect to your Azure subscription
Connect-AzAccount -Subscription "<subscription id>" -Tenant "<tenant id>"
Get-AzSubscription | fl
Get-AzContext

# Specify Key Vault Name and Certificate Name
$VaultName = "<azure key vault name>"
$certName = "certificate name as it stored in key vault"

# Get certificate stored in KeyVault (Yes, get it as SECRET)
$secret = Get-AzKeyVaultSecret -VaultName $vaultName -Name $certName
$secretValueText = ($secret.SecretValue | ConvertFrom-SecureString -AsPlainText )

# connect to PnP
$tenant = "contoso.onmicrosoft.com" # or tenant Id
$siteUrl = "https://contoso.sharepoint.com"
$clientID = "<App (client) Id>" # Azure Registered App with the same certificate and API permissions configured
Connect-PnPOnline -Url $siteUrl -ClientId $clientID -Tenant $tenant -CertificateBase64Encoded $secretValueText

Get-PnPSite

The same PowerShell code in GitHub: Connect-PnPOnline-with-certificate.ps1

References:

Yammer API with PowerShell: Private groups and messages

As an Office 365 administrator, I would like to get some reports on Yammer with PowerShell. How it’s done?

Patrick Lamber wrote a good article here: “Access the Yammer REST API through PowerShell“. The only I would add (important!) is:

By default, even with a Verified Admin token, you do not have access to private messages and private groups content.
To get private stuff, you need select “Private Content Mode” under Yammer Admin Center -> Content and Security -> Content Mode:

Check Microsoft: “Monitor private content in Yammer” and
Yammer: “Verified Admin Private Content Mode

If you do not have “Private Content Mode” set up, you might see some weird “Invoke-WebRequest” errors like:

PnP.PowerShell Batches and PowerShell 7 Parallel

Parallelism

Can I use PowerShell 7 “-Parallel” option against SharePoint list items with PnP.PowerShell? Can I run something like:

$items | ForEach-Object -Parallel {
    $listItem = Set-PnPListItem -List "LargeList" -Identity $_ -Values @{"Number" = $(Get-Random -Minimum 100 -Maximum 200 ) }
} 

Yes, sure… But! Since it’s a cloud operation against Microsoft 365 – you will be throttled if you start more than 2 parallel threads! Using just 2 threads does not provide significant performance improvements.

Batching

So, try PnP.PowerShell batches instead. When you use batching, number of requests to the server are much lower. Consider something like:

$batch = New-PnPBatch
1..100 | ForEach-Object{ Add-PnPListItem -List "ItemTest" -Values @{"Title"="Test Item Batched $_"} -Batch $batch }
Invoke-PnPBatch -Batch $batch


Measurements

Adding and setting 100 items with “Add-PnPListItem” and “Set-PnPListItem” in a large (more than 5000 items ) SharePoint list measurements:

Add-PnPListItem
Time per item, seconds
Set-PnPListItem
Time per item, seconds
Regular, without batching1.261.55
Using batches (New-PnPBatch)0.100.80
Using “Parallel” option, with ThrottleLimit 20.690.79
Using “Parallel” option, with ThrottleLimit 30.44 (fails level: ~4/100) 0.53 (fails level: ~3/100)

Adding items with PnP.PowerShell batching is much faster than without batching.

More:

Long-running PowerShell Office 365 reports

(WIP)

PowerShell is our best friend when it comes to ad-hoc and/or scheduled reports in Microsoft 365. PnP team is doing great job providing more and more functionality with PnP PowerShell module for Office 365 SharePoint and Teams.

Small and medium business organizations are mostly good, but for large companies it might be a problem due to just huge amount of data stored in SharePoint. PowerShell reports on all users or all sites might run days… which is probably OK if you run this report once, but totally not acceptable if you need this report e.g. daily/weekly or on-demand.

How can we make heavy PowerShell scripts run faster?

Of course, you start with logic (algorithm) and leveraging full PowerShell functionality (e.g. PowerShell 7 parallelism or PnP batching).

(examples)

What if you did everything, but it still takes too long? You need something like brute force – the closer your code runs to your tenant – the better.
What are the option?
– Automation account runbook (+workflow)
– Azure Function Apps
– Azure VM in the region closest to your Tenant

Automation account runbook (+workflow)

Seemed like a good option, but not something Microsoft promotes. Even opposite – automation accounts support only PowerShell 5 (not 7), no plug-ins for VS Code and recently there were messages on some retirement or smth.

Meantime, I tested it – and did not find any significant increasing in speed. In a nutshell, what is behind this service? Same windows machines running somewhere in Azure .

TBC

References
PnP PowerShell