There are some scenarios you need to quickly get only newly created SharePoint sites, but traditional methods do not work as there is no “filter” options and it’s too long to iterate through all sites. Here is how to us Microsoft Graph API to get it.
Scenario
Let say you administer Teams, OneDrive and SharePoint Online in a Microsoft 365 tenant. You have a pretty big environment – ~10k or more sites and you want to quickly find just new SharePoint sites or teams (e.g. sites created recently – during last hour/day/week/month). This might be required for ad-hoc reports and for automation scenarios – like applying required configurations or assign some property value to all newly created sites.
With GUI it’s done easily: SharePoint Admin Center -> Active Sites -> sort based on “Date Created” – done.
With PowerShell – not so simple. “Get-PnPTenantSite” cmdlet returns a site object but the object does not have “Created” field. It’s a web property. But to get a web object – you have to connect separately to each site and get root web object to check when the web was created. For small environments it is possible, for large environments it can take days… And still not nice. “Get-PnPTenantSite” with “-Filter” option would help, but “…Currently, you can filter by these properties: Owner, Template, LockState, Url.”
but… 1) it’ll give you group-based sites only 2) it is not easy to automate 3) this might take long for large environments. I know much better solution:
Solution
Microsoft Graph API helps. It returns result in seconds and you can sort or filter results based on created date . Below are two methods: Option 1 is based on Search and filtering and Option 2 is based on Sites Search and sorting. So there are some pros and cons for each method.
Microsoft Graph Search API allows KQL in queries. So we can form a query with something like “created>=1/1/2021” and use entity type = ‘[“site”]’. Search should return only sites created after Jan 01, 2021.
This option is also based on Microsoft Graph API, but sites entry point, which allows search too and sort results by property “createdDateTime”. So we will just search for everything and select how many results we need based on createdDateTime property.
Your organization use Microsoft 365. You are implementing or configuring an Intranet Portal (Home Site). Search plays an important role here – you want search be relevant to the context – i.e.
Official Results – if a user searches something on a company’s intranet portal – user expect “official” results, not a something from somebody’s OneDrive or Yammer chat
Promoted Results – so information management team can adjust search with search answers – Bookmarks, Acronyms and Q&As
Problem
Microsoft Bookmarks are working only at tenant level search – i.e. if you want bookmarks work on site level search – you need to set up site search scope as tenant.
So if you configure the Intranet Portal site (Home site) search scope to “site” or “hub” to limit results with site/hub content – you will loose “answers” functionality.
Solution
The solution is very simple:
Keep site search scope as tenant-wide to use answers (boormarks), and
Configure search verticals and query to limit results to “official” sites only
Update Query field with KQL – e.g., with something like
(path:http//contoso.sharepoint.com/sites/IntranetPortal/ OR path:http//contoso.sharepoint.com/sites/CompanyNews/ OR path:http//contoso.sharepoint.com/sites/Onboarding)/)
to get results only from “Intranet Portal” and “Company News” sites.
Keep in mind that this will affect all other SharePoint search entry points – SharePoint landing page, Office.com etc. – so although you can configure All (and Files) verticals, but it’s not recommended. It will confuse users – they expect to search for everything under “All” vertical. Instead, consider custom vertical – e.g. “Official” scope.
After configuring – It might take 1-24 hours for the change to take effect, depending on tenant size.
Service vs Site search
If you configure that at the tenant level – i.e., Microsoft 365 Administration -> Settings -> Search and intelligence -> Customizations -> Verticals then search results will be trimmed everywhere – SharePoint Landing Page, Office landing page (Office.com), Office App, Bing search. Teams search will not be affected as from Teams you only search for teams content. Same for Onedrive and Yammer.
If you want the “official” search results only under Intranet Portal and leave other search entry points unaffected – then you need to configure the same at Home (Intranet Portal) site level: Site Settings -> Microsoft Search -> Configure search settings -> Verticals and configure site search scope to site or hub scope. But in this case you will loose answers functionality.
Global search settings – like acronyms, bookmarks and verticals – works only if site search scope is tenant. If site search scope is site or hub – then site-level search verticals will apply (and no answers functionality will be possible).
Home site is a root site
There might be two problems with that:
if a home site configured as a root site – you KQL will look like(path:http//contoso.sharepoint.com/ OR path:http//contoso.sharepoint.com/sites/CompanyNews)and that query will not work as any site Url will match the root site Url.
if you need to mention many sites in KQL – like 50 sites with an Official Information – you might hit the “number of allowed character” limit
The solution is DepartmentId property:
DepartmentId
Use DepartmentId={<Hub Site ID>} in the KQL qury and your search results will be limited to your hub content while answers will still be working too. You can even combine DepartmentId with other conditions to add more sites (that are not in hub) to search scope:
(DepartmentId={4965d9be-929b-411a-9281-5662f5e09d49} OR path:http//contoso.sharepoint.com/sites/Onboarding OR path:http//contoso.sharepoint.com/sites/CompanyNews)
It worth to mention, that DepartmentId is the property that propagated from the root of the hub site to all associated sites and their content – list items, documents and pages.
Site Property Bag
Another possible option would be – site property bag… The ultimate goal is to provide users with “Official” results only. But official sites might not be all part of one Home site hub. We can include in All search vertical query 10, 50, 100 sites, but what if we have 10k official sites in enterprise – e.g. operated by different departments – and all of them might want to be present in search results. So, how about – if the site is considered official – we create an indexed site property, e.g. “SiteIsOfficial” with a value “Yes”. Then we map the crawled property to a managed property – e.g. RefinableString89 – and use this managed search property in query – e.g. (SiteIsOfficial=Yes).
This is actually clever idea, but this does not work… This query would only return sites, not sites content. E.g. what was indexed as site object – will be included (including home page). But site items – i.a. documents, lists, other pages – all site content – will not be included…
So let’s get back to DepartmentId…
Rename All Vertical
Again – the ultimate goal is to give users option to have “Official” results. But they still might want to be able to search through all content. What if we rename the default “All” vertical to “Home Site” and configure query for official results only. Then we can create a custom vertical called “Everything” or “All” with no query limitations to give users all reasults
Update: not a good idea either… If the home site search scope is tenant – so verticals are configured and be visible at tenant level – i.e. everywhere…
Separate Official Vertical
My personal preference is to keep All vertical as real All, and create a custom Vertical “Official” for official results only where we would use query trick.
UPDATE: this approach only works if the mobile number is stored in SPO UPA. If the mobile number is stored in AAD only, the admin will need to remove the mobile number from AAD for the affected users.
Scenario: you administer SharePoint Online Microsoft 365 tenant business asks you to remove mobile phone numbers from SharePoint User Profiles:
Solution:
As a SharePoint administrator, you can do it: 1. Start Microsoft 365 SharePoint Admin Center 2. Navigate to More Features -> User Profiles -> Manage User Properties
3. Under “Contact Information” -> Mobile phone -> Edit
4. Uncheck “Replicable”, Save, wait a minute or two 5. – Select “Default Privacy Settings”: “Only Me” – Uncheck “User can override” – Uncheck “Allow users to edit values for this property” – Uncheck “Show in the profile properties section of the user’s profile page” – Uncheck “Indexed”
6. Save
Note: Microsoft implemented new MS-Graph API that adds or deletes profile card custom properties using the profile card API in Microsoft Graph
How can I get HTML content of a SharePoint online page from code, e.g. PowerShell?
Invoke-WebRequest returns “Sign in to your account” page, not a real page, even with -Token option.
Thanks to Denis Molodtsov, the solution is found. It turns out the “Invoke-PnPSPRestMethod” PnP cmdlet works not only against /api endpoints, but also against site pages and system pages.
But (as per my experience) it works only with PnP.PowerShell and with -UseWebLogin authentication option and with -raw parameter.
Other combination of authentication options ( -interactive, -clientId, -Token, -SPOManagementShell, -PnPManagementShell ) – worked well, but only for /_api endpoints, and gave me “401 UNAUTHORIZED” against system/site pages. Unattended authentication (with clientId, clientSecret and certificate) – same.
Legacy PnP module SharePointPnPPowerShellOnline did not work at all: “EXCEPTION,PnP.PowerShell.Commands.Admin.InvokeSPRestMethod”.
I tested it with – SharePointPnPPowerShellOnline v 3.29.2101.0 (under Windows PowerShell 5.1) and – PnP.PowerShell 1.8.0. (both Windows PowerShell 5.1 and .net core PowerShell 7.1.5)
You have a Microsoft 365 subscription with SharePoint Online. You use PowerShell, PnP.PowerShell module and MS Graph API to work with SharePoint under current user’s credential. You need to authenticate to SharePoint Online via Connect-PnPOnline and to Microsoft Graph API interactively on behalf of a current user.
Problem
Unfortunately, both “Connect-PnPOnline -Interactive -Url <siteUrl>” or “Connect-PnPOnline -UseWebLogin -Url <siteUrl>” might fail with something like “Need admin approval”, “App needs permission to access resources in your organization that only an admin can grant. Please ask an admin to grant permission to this app before you can use it.” or “Permissions requested” or similar
configure API permissions blade: – add delegatedpermissions you need (refer to specific API you’ll use) e.g. Microsoft Graph Sites.FullControl.All and SharePoint AllSites.FullControl
A pop-up window will appear to authenticate interactively. If you are already authenticated with another credentials (or single-sigh-on) – an interactive window might pop up and disappear – that prevents you enter your other id. To ensure Connect-PnPOnline prompts you for your credentials – use ” -ForceAuthentication” option.
If you are a SharePoint tenant admin – you can connect to a tenant with:
By default token expires in ~ 1 hour. But you can refresh it silently. This helps you in long-running PowerShell scripts that takes hours to complete. So you can include something like this in the loop:
Somehow using Connect-PnPOnline with AccessToken option did not work if the token was acquired with MSAL.PS interactively. But it did work when you get msal.ps token unattended (using App credentials). So…
If you can get an Application (non Delegated) permissions to your azure-registerd-app, you can use msal token to connect to site with PnP
=========================
NB: For delegated permissions, the effective permissions of your app are the intersection of the delegated permissions the app has been granted (via consent) and the privileges of the currently signed-in user. Your app can never have more privileges than the signed-in user.
Update (May 2023): You can use Get-PnPAzureACSPrincipal to returns the lists of all Azure ACS principals installed in your Tenant including subsites.
Scenario
Developers in the organization can use both – Azure Apps and SharePoint Apps to work with SharePoint sites in their “daemon” applications.
It is recommended to use Azure apps so, you want to know – what are SharePoint Apps registered and their owners, who registered SharePoint Apps. Eventually you would disable SharePoint Apps-only principal but before that you’d move Devs from SP-App-only to Azure App (see Disable Custom App Authentication).
(SharePoint App-only service principals aka SP-App-Only are SPN or App registered from within SharePoint using AppRegNew.aspx system page).
One of the approaches is to track Apps/Owners with Unified Audit Log
returns events with operation = ‘Add service principal.’ Nice, but… if an app was registered in Azure – event contains an UPN under UserIds property:
Unfortunately, in case with registering app in SharePoint, an audit log event will be like:
i.e. UserId registerd is “spo_service@support.onmicrosoft.com”, so we do not know who registered a SharePoint-only app
In theory – we could use events recorded immediately before and after “Add service principal” event to track a user and site who has registered a SharePoint-only app… But for me it seems like too complicated for automation.
Instead we can do simple search through audit log for events “AppRegNew.aspx page visited”. This gives us a good approximation of who registered SP-App-only principal. Worst scenario – we reach more people than we really need (including those who started registering sp-app-only but did not complete) but all of them would be definitely our target auditory.
this would give you all users who loaded “/_layouts/15/appregnew.aspx” page
Update: Sites.Selected API MS Graph permissions was introduced by Microsoft in 2021. It was a good move towards site-level development, but still developers were limited with only what MS Graph API provides for SharePoint dev. So devs had to use AppInv.aspx at site level to provide ACS permissions to their apps to be able to use SharePoint CSOM and REST APIs. Recently Microsoft introduced Sites.Selected SharePoint API permissions for registered Azure Apps! So now devs should be fully happy without ACS-based permissions.
There is a known problem in SharePoint (and Teams*) – complicated permissions system. Site owners/administrators provide access, site contributors upload documents and nobody knows – who has access to their sites. As a result – sometimes sensitive documents become overshared (over-exposed).
The biggest concern is when sites content is shared with “Everyone”. How do we find sites shared with “Everyone” in a large Microsoft 365 environment so for this sites we can request permissions review?
(*) Microsoft with the introduction of Teams had to simplify permissions in SharePoint – since there should only be 3 types of access levels – owner, member and visitor. It was… in some ways, but in other ways it made things worse.
Solution #1 (3-rd party tools)
You are lucky if you can use 3-rd party tools (e.g. ShareGate, SysKit Point, AvePoint, Metalogix etc.), with the ability to get full permissions report. Though there might be a problem to get full permissions report for all tenant sites if your m365 environment is not small. Some tools allow you to get tenant-wide permissions report for specific Ids – this option should work better for large environments.
Still there might be another problem. Consider the following. When I say “shared with Everyone” – I actually mean at least 3 possible “everyone” system logins:
Everyone
Everyone except external users
All users
– those are system id’s, but what if there are other ids – e.g. migrated from on-prem or cloud-born custom security or Microsoft 365 groups in tenant that also includes everyone (e.g. dynamic security group that includes all org accounts)? How do you think this group will be identified as “Everyone” group? So – you’d also know which groups include “all” or “almost all” users and get report for these groups also.
Obviously this option #1 is not free, as it requires licenses to be obtained. And still it worth to consider option 3.
Solution #2 (PowerShell “Brute force”)
You can get full permissions report per site or for entire tenant with PowerShell, which if free… The only you need is to write a script yourself or find existing one. Sounds easy?
Well, first problem is it takes a decent amount of time and competences to write such script. If if you find one – it would require some skills to adopt it and run. Frankly say, I have not seen so far scripts that are ready OotB to do that job. And it is not a good idea to run scripts you are not fully confident with against production environment.
Another possible problem – size of environment. The script I designed and use to get comprehensive permissions report might run hours against a good site – if I need full details on site/subsites, lists/libraries, folders and list items levels. So if you have less than 1000 sites – probably this approach can fly. But if your environment is 10K+ sites – it will take forever. So the approach might not work for large enterprise environments.
One might say – we can limit report with root web permissions only to get it faster. But this would not be accurate. And what is not accurate in the IT security – lead to even bigger risks. So, we need check permissions up to every item level deep, as even one file with sensitive info shared inappropriately can cause security issue. (Btw, 3-rd party tools usually by default limit reports to libraries level, so check reporting options)
The other issue with this approach… Let say you got full permissions report… It would look like “resource -> group -> permissions”… How do you know for each group – what is the group in terms of membership?
Ok, if this solution is not easy to get working – what are other options?
Solution #3 (Search-based)
This solution is based on simple but clever idea: why do we need to iterate through all the tenant documents/items if all the content is already crawled by search? Search is also respect permissions. Can we just use search to get files shared with Everyone? Let us see.
What if we use some dummy user account with no specific permissions provided and no group memberships and try to search content on behalf of that account. The idea is if this user can see some data – then these data is open to everyone.
With this option we would use search query “*” and all 5 possible SharePoint entities – driveItem’,’listItem’,’list’,’drive’,’site’ to find everything that is shared with everyone. We’d pull results with paging (we’d use “from” option in a loop to pull all results). After we get all results – we’d select only unique site collections. But! We might have some problems here.
Problem #1. Again, for small environments or if there are not much “Open” sites – it would work. But for large enterprise environments the problem is the same as in “brute force”. Search would returns too many results – and it might take weeks (exact time is unpredictable) to get all of them. (Surely there are sites “legally” shared with everyone, public Office 365 group based sites, communication sites… So your search will be flooded with content from sites you already know are shared with all).
Problem #2. We are getting results with paging. But recently Microsoft started limiting number of returning results. E.g. your search request result might say like “there are 3659735 total hits” but after result number 1000 it just stops returning anything, even with paging.
Solution#3 Option #2 – loop sites
The idea is: why do we need to get all search results if even one result from a site would be enough to put the site to the list of “open” sites. In other words, we do not need all results from the site, we only need to know if there are any results from the site, at least one – so we know if the site is open for everyone or not.
So, consider the following approach:
You get list of all sites in tenant.
You run search request against each site in the loop (e.g. consider KQL option “Site: https://yourTenant.SharePoint.com/sites/YourSite”. If at least something found in the site – add the site to the “Open Sites” list. With this approach you will get list of sites shared with “Everyone…” in a predictable time.
Solution#3 Option #3 – exclude known “open” sites
There are sites “legally” shared with everyone – e.g. corporate portal, department communication sites, public teams, public Viva Engage communities etc. If it is know that these sites are public – you can exclude them from all sites list – so in the “Solution#3 Option #2 – loop sites” – you’d loop only through sites that are not supposed to be public. I know – percentage of “legally public” sites in tenant to all sites is a relatively small number, so should not significantly decrease elapsed time… but still.
Pros and cons of the Solution # 3
Pro: the only fast enough (at least predictable time to complete) and accurate enough to rely on solution.
Con 1 : crawling and indexing takes time, so search-based reports can miss recent changes in data and permissions
Con 2: this approach cannot be automated (since we need an interactive authentication). I.e. we need to run it manually every time.
Con 3: After we get all sites shared with everyone – we do not know – at what level permissions are broken and provided to everyone. It might be entire site or one file. If you want to know what exactly is shared with everyone – run permissions report against this shortlist.
Notes
Note 1: consider there are resources like “Styles Library” shared with everyone by default, especially on migrated sites
Note 2: There are might be security groups intended to hold all or part of the enterprise (e.g. “All employee” or “all contractors”). If the enterprise comprises from several businesses or regions – it might be “All Business 1” or “All EMEA”… you got the idea. You can modify search-based solution if you add your dummy account you are running search on behalf of to some of theses groups to find out if there are resources shared maybe not with everyone but with all “North America based” users or with “all employees”, which might make sense also.
Note 3: separate service, but consider implementing/using sensitivity labels. At least you can start with high-sensitive sites. With sensitivity labels – site owners/member would know – what kind of site they are working on.
What’s next
Ok, we know list of SharePoint resources shared with everyone, but what would be the next step? Should we communicate to site owners – if so how to let site owners know that there are resources shared with Everyone… on their sites. To be continued…
