I have created a user “Robert Dylan” and never used name “Bob”, but search understands that I’m looking for Robert Dylan when I’m searching for Bob or Bob Dylan or Bobby.
Important thing – result is shown under “All” and “People” verticals.
I know Microsoft claimed m365 search is backed by Turing technology – it understands you, answers your question but not just blindly display keyword occurences (announcement at Ignite Spring, more on Ignite Fall 2021)… So is that it?
But the same trick did not work with Dick for Richard or (of course) Syd for Roger Barrett:
Some tips and tricks on filtering by site Url (path) in query field in Microsoft 365 Search verticals
Path filter with trailing slash (“/”)
In November 2022 Microsoft rolled out an update for multiple search features, including checks on the path managed property for a trailing slash. Previously path filters were valid with and without trailing slashes.
Consider the following scenario.
Given the path filter with the contain operator (“:”)
SPSiteUrl and Path properties use different matching strategies. When using contains operator (colon sign “:”) – SPSiteUrl will match the full value, while Path will do a “starts with” match.
DepartmentId
DepartmentId is a search managed property used under Hub sites and propagated through all associated sites content.
That means if we want to scope down search to hub site with it’s content – we can use DepartmentId property, e.g.
DepartmentId=4965d9be-929b-411a-9281-5662f5e09d49
instead of iteration through all hub sites and using path: property.
Nickname (or alias or moniker or hypocorism etc.) is some another name – usually shorter than original name and widely used in maybe all countries in the world. E.g. Bob is a nickname for Robert in the US and GB, Checo is the other name for Sergio in Mexico. There might be personal names also – e.g. Bapu (father) is an word that is usually associated with Mohandas Karamchand Gandhi, also known as the Mahatma Gandhi. More examples: David “Noodles” Aaronson, Roger Keith “Syd” Barrett etc.
In Microsoft 365 we want to search for a person’s name we know – and in many cases it’s a nickname – e.g. Beth (Bethany) or Alex (Alexander). Can m365 search do that?
It takes a few steps to implement search by nickname:
create a custom property under SharePoint Online User Profiles service
fill this property with values
configure Search Schema – map crawled property to managed property
Step-by-Step
Create a custom property under SharePoint Online User Profiles service
Ensure you have a SharePoint Administrator role activated
Navigate to SharePoint Admin Center – more features – User Profiles – Manage user properties – New Property
Configure custom property according to your needs, – ensure “Policy Settings” “Default Privacy Setting:” Everyone is selected – ensure “Search Settings” “Indexed” is selected
Fill SharePoint Online User Profile Properties with values
That would be a custom solution – e.g. manual work from SPO Admin Center GUI or PowerShell script ( e.g. with some dictionary). This is required for search to pick it up, crawl the property and create crawled property – so you could proceed with search schema mapping.
Configure Search Schema – map crawled property to managed property
For the nickname to be generally available in full-text-search – i.e. user simply enter value in a search bar and gets results – here are the steps:
Ensure you have a SharePoint Administrator role activated
Navigate to SharePoint Admin Center – More features – Search – Manage Search Schema
Select Crawled Properties and ensure search picked up your custom property and crawled it – check your crawled property name under Category: People.
Under Managed Properties – create a new managed property
select “Searchable”
under Advanced searchable settings – select Full-text index: PeopleIdx
If you configure Verticals Query at the tenant level – i.e., Microsoft 365 Administration -> Settings -> Search and intelligence -> Customizations -> Verticals then search results will be trimmed everywhere – SharePoint Landing Page, Office landing page (Office.com), Office App, Bing search (but not other sites).
Teams search will not be affected as from Teams you only search for teams content. Same for Onedrive and Yammer. Sites with site or hub search scope will not be affected too.
If you configure verticals at site level: Site Settings -> Microsoft Search -> Configure search settings -> Verticals and want this be in effect – ensure site search scope is set to site or hub scope. But in this case you will loose answers functionality.
Global search settings – like acronyms, bookmarks and verticals – works only at tenant level search or at site leve if the site search scope is set to tenant. If site search scope is site or hub – then site-level search verticals will apply (and no answers functionality will be possible).
Your organization use Microsoft 365. You are implementing or configuring an Intranet Portal (Home Site). Search plays an important role here – you want search be relevant to the context – i.e.
Official Results – if a user searches something on a company’s intranet portal – user expect “official” results, not a something from somebody’s OneDrive or Yammer chat
Promoted Results – so information management team can adjust search with search answers – Bookmarks, Acronyms and Q&As
Problem
Microsoft Bookmarks are working only at tenant level search – i.e. if you want bookmarks work on site level search – you need to set up site search scope as tenant.
So if you configure the Intranet Portal site (Home site) search scope to “site” or “hub” to limit results with site/hub content – you will loose “answers” functionality.
Solution
The solution is very simple:
Keep site search scope as tenant-wide to use answers (boormarks), and
Configure search verticals and query to limit results to “official” sites only
Update Query field with KQL – e.g., with something like
(path:http//contoso.sharepoint.com/sites/IntranetPortal/ OR path:http//contoso.sharepoint.com/sites/CompanyNews/ OR path:http//contoso.sharepoint.com/sites/Onboarding)/)
to get results only from “Intranet Portal” and “Company News” sites.
Keep in mind that this will affect all other SharePoint search entry points – SharePoint landing page, Office.com etc. – so although you can configure All (and Files) verticals, but it’s not recommended. It will confuse users – they expect to search for everything under “All” vertical. Instead, consider custom vertical – e.g. “Official” scope.
After configuring – It might take 1-24 hours for the change to take effect, depending on tenant size.
Service vs Site search
If you configure that at the tenant level – i.e., Microsoft 365 Administration -> Settings -> Search and intelligence -> Customizations -> Verticals then search results will be trimmed everywhere – SharePoint Landing Page, Office landing page (Office.com), Office App, Bing search. Teams search will not be affected as from Teams you only search for teams content. Same for Onedrive and Yammer.
If you want the “official” search results only under Intranet Portal and leave other search entry points unaffected – then you need to configure the same at Home (Intranet Portal) site level: Site Settings -> Microsoft Search -> Configure search settings -> Verticals and configure site search scope to site or hub scope. But in this case you will loose answers functionality.
Global search settings – like acronyms, bookmarks and verticals – works only if site search scope is tenant. If site search scope is site or hub – then site-level search verticals will apply (and no answers functionality will be possible).
Home site is a root site
There might be two problems with that:
if a home site configured as a root site – you KQL will look like(path:http//contoso.sharepoint.com/ OR path:http//contoso.sharepoint.com/sites/CompanyNews)and that query will not work as any site Url will match the root site Url.
if you need to mention many sites in KQL – like 50 sites with an Official Information – you might hit the “number of allowed character” limit
The solution is DepartmentId property:
DepartmentId
Use DepartmentId={<Hub Site ID>} in the KQL qury and your search results will be limited to your hub content while answers will still be working too. You can even combine DepartmentId with other conditions to add more sites (that are not in hub) to search scope:
(DepartmentId={4965d9be-929b-411a-9281-5662f5e09d49} OR path:http//contoso.sharepoint.com/sites/Onboarding OR path:http//contoso.sharepoint.com/sites/CompanyNews)
It worth to mention, that DepartmentId is the property that propagated from the root of the hub site to all associated sites and their content – list items, documents and pages.
Site Property Bag
Another possible option would be – site property bag… The ultimate goal is to provide users with “Official” results only. But official sites might not be all part of one Home site hub. We can include in All search vertical query 10, 50, 100 sites, but what if we have 10k official sites in enterprise – e.g. operated by different departments – and all of them might want to be present in search results. So, how about – if the site is considered official – we create an indexed site property, e.g. “SiteIsOfficial” with a value “Yes”. Then we map the crawled property to a managed property – e.g. RefinableString89 – and use this managed search property in query – e.g. (SiteIsOfficial=Yes).
This is actually clever idea, but this does not work… This query would only return sites, not sites content. E.g. what was indexed as site object – will be included (including home page). But site items – i.a. documents, lists, other pages – all site content – will not be included…
So let’s get back to DepartmentId…
Rename All Vertical
Again – the ultimate goal is to give users option to have “Official” results. But they still might want to be able to search through all content. What if we rename the default “All” vertical to “Home Site” and configure query for official results only. Then we can create a custom vertical called “Everything” or “All” with no query limitations to give users all reasults
Update: not a good idea either… If the home site search scope is tenant – so verticals are configured and be visible at tenant level – i.e. everywhere…
Separate Official Vertical
My personal preference is to keep All vertical as real All, and create a custom Vertical “Official” for official results only where we would use query trick.
Office graph = codename for collective set of services and insights we generate on top of the infrastructure that fast office graph group developed = social Intel concepts (SharePoint home, Delve, OneDrive Discoverview) are derivatives of Office graph
Microsoft Search API provides one unified search endpoint that you can use to query data in the Microsoft cloud – messages and events in Outlook mailboxes, and files on OneDrive and SharePoint – that Microsoft Search already indexes.
Turing technology – understands you, answers your question e.g. hover over doc -> doc summary (based on “deep speed” AI model) announcement at Ignite Spring, more on Ignite Fall 2021
Modern Search: MS nailed the fundamentals, now start bringing it everywhere – to Teams first, then SharePoint (said Nov 2020).
Modern Search Customizations – we’ll take the best from Classic SharePoint Search, a lot will retire – investing in more flexibility
Bill Baer: “People use search in a different ways 1) you have organisations who have a well-established intranet built around set of governance controls, a very clean architecture and they want to build a search into that intranet scenario; that’s why a lot of SharePoint capabilities are going to come along with Microsoft search for that particular endpoint 2) then you have other people who live their day in teams“
Updates
Shared search engine results page (developed once – transitioned everywhere) Ctrl-F to search through teams (chats?) (contextual search) Natural language search (starting from Outlook) Image search (before eoy), +
Metadata content type search Syntex customers will be able to use content type to search in the advanced search flyout in document libraries that contain content types beyond the default types. Syntex only.
Conversations: teams chats, outlook groups conversations, yammer conversation can be found under Conversations vertical in Bing search. Later – in Office.com and SharePoint landing page. E-mail messages are added to Conversations vertical in Bing search
Bookmarks (new promoted results), acronyms, Q&A – all under “Answers”
BookmarksTargeting – for the specific audience based on device/OS, Country/Region, security groups…
SharePoint Search Admin Center -> will be migrated from SharePoint admin center to to Microsoft Search Admin Center transitioning (Search and Intelligence Admin Center) – long-running project custom dictionaries, spelling suggestions – will retire, (move to a graph-driven speller)
+ Viva Topics – based search capabilities
Create Topic Answers with Microsoft Viva Topics to bring together people, content, and information (including synonyms and acronyms)
Knowledge answers provide a direct answer to questions authoritative information in an organization across SharePoint and OneDrive content
Files/Calendars/Links answers
Graph Connectors Graph Connectors are generally available (ADLS – Azure Data Lake Storage Gen2, Azure DevOps, Azure SQL and Microsoft SQL Server, Enterprise websites, MediaWiki, File share, Oracle SQL, Salesforce, Jira, Confluence, ServiceNow + 100+ from partners; New connectors coming to Microsoft Search: Jira Graph connector, Confluence Graph connector).
Graph Connector allows to connect external source of information to Microsoft 365 and makes that data available across all m365 apps and services so you can find what you need wherever you’re working, whether in one of your favorite productivity apps or one of the many Microsoft 365 services such as SharePoint or Office.com
Graph Connectors roadmap:
Actionable experiences Search results on select Graph connectors will soon support actions that will allow users to interact with the result and perform changes to the Connector content within the Search application.
Results clusters The results shown in a result cluster are grouped together based on the search vertical configuration.
Profile Query variables Define any attribute from the user’s Profile, as a query variable and it would be resolved during query evaluation (This feature is currently in preview)
Profile enrichment with Graph connectors …you will soon be able to enrich Microsoft 365 profile properties like Job title, Phone numbers, Skills etc. with data from HRMS systems using graph connectors. …then surface this rich profile information on people experiences like profile cards.
Search Federation federation capabilities will allow enterprises build and integrate their custom LOB search experiences, customized search providers, into the overall Microsoft Search. With federated search, you can make information from systems where the data cannot leave the systems boundaries available to search across in Microsoft 365 productivity apps and services, without indexing its data with Microsoft Search.
Standalone Search – AAD identity – Graph connector – Ingest your data – use Search = in Windows 10, Office.com ( e.g. for those who have their data in other productivity suite, have no intent to use m365, but want to search)
There is a known problem in SharePoint – complicated permissions system. Site owners/administrators provide access, site contributors upload documents and nobody knows – who has access to their sites. As a result – sometimes sensitive documents become overshared (over-exposed).
The biggest concern is sites content shared with “Everyone”. How do we find sites shared with “Everyone” in a large Microsoft 365 environment?
NB. When I say “shared with Everyone” – I actually mean 3 possible “everyone” logins:
Everyone
Everyone except external users
All users
Approach #1 (Brute force)
We can get full permissions report at tenant level (or permissions provided only to “Everyone”). There are 3-rd party tools (e.g. ShareGate, SysKit, AvePoint, Metalogix etc.), or you can run PowerShell script…
Sounds easy? Well, if you have less than 1000 sites – probably it will work. But if your environment is 10K+ sites – it will take forever. Permission report might run hours for an average site with site/subsite, list/library and list item details level. So the approach will not work for large enterprise environments.
One might say – we can limit report with root web permissions only to get it faster. But this would not be accurate. And what is not accurate in the IT security – lead to even bigger risks. So, we need report detailed up to every item level deep, as even one file with sensitive info shared with everyone can cause security issue. (3-rd party tools usually by default limit it to libraries level.)
Ok, if this approach is not really working – what’s working?
Approach #2 (Search)
Clever idea: why do we need to iterate through all the tenant documents/items if all the content is already crawled by search? Search is also respect permissions. Can we just use search to get files shared with Everyone? Let us see.
What if we use some dummy user account with no specific permissions provided and no group membership and try to search content on behalf of that account. The idea is if this user can see data – it means that data is open for everyone.
Check this and this articles. Can we get results programmatically (e.g. with PowerShell)? Can we use Microsoft Graph search API? Sure. Check this article “How to search against SharePoint Online Content with Microsoft Graph search API with PowerShell”.
But! We have some problems here.
Search Problem #1. Again, for small environments or if there are not much “Open” sites – it would work. But for large enterprise environments the problem is the same as in “brute force”. Search returns too many results – it’ll take weeks to get all of them. (There are team sites “legally” shared with everyone, public Office 365 group based sites, communication sites… ).
Search Problem #2. Even if we get all search results – we do not know – at what level permissions are provided to everyone. So we will need to build list of sites based on the search results – ant then still need to run permissions report against these sites.
Search Problem #3. We are getting results with paging. But recently Microsoft started limiting number of returning results. E.g. your search request result might say like “there are 3659735 total hits” but after result number 1000 it just stops returning anything, even with paging.
Approach # 3 Hybrid
The idea: why do we need to get all search results if even one result from a site would be enough to add the site to the list of sites require permissions review. In other words, we do not need all results from site, we only need one to know the site is open.
So, consider (imho, the best) approach (Solution):
You get list of all sites in tenant.
You run search request against each site in the loop (e.g. consider KQL option “Site: https://yourTenant.SharePoint.com/sites/YourSite”. If at least something found in the site – add the site to the “Open Sites” list. With this approach you will get list of sites shared with “Everyone…” in a couple of minutes.
Run permissions report against this shortlist
Note: consider there are resources like “Styles Library” shared with everyone by default.
Note: You can refine the list you get at step 1 – e.g., excluding sites connected to public teams or known communication sites…
Note: consider implementing sensitivity labels. At least you can start with high-sensitive sites. Site owners/member will know – what kind of site they are working on.
Pros and cons of the Approach # 3 Hybrid
Pro: the only fast and accurate enough to rely on
Con 1 : crawling and indexing takes time, so search-based reports can miss recent changes in data and permissions
Con 2: this approach cannot be automated (since we need an interactive authentication).
How to communicate to site owners
The Next step would be “How to let site owners know that there are resources shared with Everyone… on their sites”.
your organization maintain custom active directory attributes – e.g. Campus Name, Employee Id, Hiring Category etc.
your organization use Microsoft 365 for collaboration
users want to search for custom properties like “Employee id” or refine search by custom properties like Campus Name
What is Microsoft saying
“The following Azure AD user attributes are synced to the UPA.” : (Azure AD attributes) – UserPrincipalName, DisplayName, GivenName, sn, telephoneNumber, proxyAddresses, PhysicalDeliveryOfficeName, Title, Department, WWWHomePage, PreferredLanguage, msExchHideFromAddressList, Manager Respective User profile property display names: Account Name, User Name, User Principal Name, Name, FirstName, LastName, Work phone, Work Email, SIP Address, Office, Title, Job Title, Department, Public site redirect, Language Preferences, SPS-HideFromAddressLists, Manager (here)
Q: “Why isn’t it possible to map additional properties for UPA synchronization to sync from Azure AD to the User Profile Application?” A: “UPA synchronization is limited to a preconfigured set of properties to guarantee consistent performance across the service.” (here)
“You can make the following attributes from Azure Active Directory (Azure AD) visible on users’ profile cards. These attributes are not case-sensitive: UserPrincipalName, Fax, StreetAddress, PostalCode, StateOrProvince, Alias” (here)
“You can add any of the 15 Azure AD custom extension attributes to users’ profile cards… Custom properties are not searchable and can’t be used to search for people across Microsoft apps and services.” (here)
Solution
It takes a few steps to solve the problem:
create a custom property under SharePoint Online User Profiles service
synchronize AD/AAD attribute with SPO User Profile custom property
configure Search Schema – map crawled property to managed property
Detailed:
Create a custom property under SharePoint Online User Profiles service
Ensure you have a SharePoint Administrator role activated
Navigate to SharePoint Admin Center – more features – User Profiles – Manage user properties – New Property
Configure custom property according to your needs, – ensure “Policy Settings” “Default Privacy Setting:” Everyone is selected – ensure “Search Settings” “Indexed” is selected
Hint: you can fill this property for some user profiles – for search to pick it up and crawl the property – so you could configure search schema mapping before synchronizing property from Active Directory
Synchronize Active Directory attribute to SharePoint Online User Profile
That would be a custom solution – e.g. scheduled PowerShell script. You can host this script in Azure Functions if you sync Azure AD attributes to SPO or use on-prem machine if you need access to local AD.
PowerShell Script example (TBP)
Configure Search Schema – map crawled property to managed property
Ensure you have a SharePoint Administrator role activated
Navigate to SharePoint Admin Center – More features – Search – Manage Search Schema
Select Crawled Properties and ensure search picked up your custom property and crawled it – you should see your property name under Category: People.
Full-text search and/or query-based search
If you want your custom property is generally available in full-text-search – i.e. user simply enter value in a search bar and gets results – typical scenario might be an employee id – here are the steps (under Search Schema)
create a new managed property
for Full-text search
select Searchable
under Advanced searchable settings – select Full-text index: PeopleIdx
for Query-based search – select Queryable and Retrievable
map this managed property to crawled property
Free-text search: you just enter what you search for into search bar and click search.
Query-based search allows you to use KQL – e.g. you are searching for keyword “PowerShell” with full-text search, but want only people with PowerShell skills located in a specific building or campus – search query might look like “PowerShell campus:Stanford”
Refine search with custom property
If you want to be able to refine your search with custom property – in this case the steps are (under Search Schema):
under Managed Properties – select a refinable string property that is not taken (not mapped) – e.g. RefinableSting53
setup alias – so you could refer to this RefinableSting53 by meaningful name
If you have country-specific content – Microsoft Search allows bookmarks to be configured to pop-up only for users from a specific country.
And “Use Azure AD locations” flag is a new option that make it actually works… For a long time country settings were the same but without “Use Azure AD locations” flag. So what does “Use Azure AD locations” flag do?
Use Azure AD locations
“Use Azure AD locations” flag is a straight-forward configuration settings. It says: “This bookmark will only appear for users with Azure AD locations that match selected countries or regions. If cleared, the user’s IP address will be used to determine location. This checkbox can be altered from both Country or region setting and Targeted variations setting.”
I have tested this new “Use Azure AD locations” flag – it works. Once you configure user’s country in AAD and country-targeted bookmarks – all works good. Bookmarks appear for the user.
What if we do not “Use Azure AD locations” flag
to be provided
What was before “Use Azure AD locations” flag
What was before Microsoft implemented this “Use Azure AD locations” option? How did Microsoft understand “this user is from that country”. What was the criteria to correlate User <-> Country? License assigned country? Windows locale? Browser settings? Azure AD properties?
How does Microsoft define “this user is from that country”. What are the criteria to correlate User <-> Country? Physical IP address? License assigned country? Locale? Browser settings? Azure AD properties?
It turned out, the way it was designed previously: – configure Microsoft 365 integration with Bing – in Bing -> Settings : select Country/Region – search from Bing that was the only way to make it work! So yes, do not leave “Use Azure AD locations” option unchecked. Microsoft confirmed it was poor design.
