Tag Archives: Search

Manage result layouts for SharePoint results in Microsoft Search

Microsoft is improving Search (MC489165):

Manage result layouts for SharePoint results in Microsoft Search

We’re making changes to Microsoft Search. This update will allow Microsoft Search administrators to change result layouts for select SharePoint content using adaptive cards with Result Type feature in Microsoft Search administration.

The default result layouts for SharePoint sites, pages, list items and Portable document format (PDF) results can now be replaced with layouts built using adaptive cards. The changes can be made for Organization level search applicable to Office.com and SharePoint home as well as site level search on SharePoint sites. Changes for Microsoft Search in Bing will be rolled out soon. Note that the feature does not support changing of Office file search results.

This message is associated with Microsoft 365 Roadmap ID 81952

Before the change, when you add a new result type under “Search and intelligence” Customizations – it looked like this:

result type content sources

So there was no built-in “SharePoint” content source as an option – only custom “external” data sources.

But with the new feature implemented list of content sources for the result type will look like this:

SharePoint and OneDrive content source

If you choose “SharePoint and OneDrive” content source – the next option would be to select type of content:

Select type of content and set rules

You also can create different result types for different types of content based on properties-based rules (e.g. one result type for all sites – and a separate result type for a specific site or hub) with optional “Set rules for this type of content”:

Default site result experience would look like

Search results with modified SharePoint result type might look like:

When you modify template via Layout Designer – it is essential to know available object properties.

You can get properties from the “Available properties” below – there is also search through properties feature.

Or you can use SharePoint Search Query Tool to get metadata on search results.

It might take hours and even days for your search to start showing new layouts, but “&cacheClear=true” should help.

… to be continued …

References

Microsoft 365 Search Vertical KQL query field limits

What is the Microsoft Search KQL query field limits for a verticals? Is there limited number of characters or lines?

You know what is Microsoft 365 Search Vertical and what is KQL query in vertical configuration, right?

Microsoft 365 Search Vertical KQL query field limits

Under Microsoft 365 admin center Search and intelligence you can configure search verticals. There are some out-of-the-box verticals – like All, Files, Sites, People and you can configure custom one.

As a part of vertical configuration – you can specify KQL query – if you want e.g. limit search with some sites or content types etc.

The question is – how many sites I can specify in this query field? E.g. can I specify 1000 sites? 10k sites?

And the answer is: It does not matter, because the limit is not in number of characters or lines.

In my dev environment I was able to save 50,000 lines (~3M characters). But attempt to save 100K lines (6M symbols) has failed (due to timeout, I believe:

Again, as I said the problem is not here.

The problem is time required for search to apply query. I.e. when you ask search to bring you something – after it gets results from index and before display results to you it applies KQL query configured for the vertical. And this time is the bottleneck.

Here is what I got measuring search response time depending on query size:

Searchresponse time,
seconds
KQL query
# of lines
KQL query size,
# of symbols
works150028,000
works5100059,000
works92000120,000
works253000180,000
works/fails303500208,000
fails353600214,000
fails3550,0003,000,000
n/an/a100,000
(can’t save KQL query
6,000,000
(can’t save KQL query)

Which means that after ~ 1000 lines (50,000 characters) KQL query size – query becomes too slow, and after ~3000 lines (180k chars) – can fail (due to timeout I’d say).


Hide User Accounts from Microsoft 365 People Search

What if you want specific users do not appear in Microsoft 365 SharePoint, Teams or Delve search results?

Solution

Set “ShowInAddressList” Azure AD User object property to false

Details

In many cases we do not need some accounts appear in Microsoft 365 Search. Examples of are:

a) secondary/admin accounts
e.g. a person have several roles and several accounts under the same name, e.g.
regular user: John Smith John.Smith@contoso.com
administrative account: John Smith John.Smith.2@contoso.com

b) role or service accounts like SharePoint_Support@contoso.com

c) non-mail-enabled objects

d) disabled accounts

Getting multiple results for the same one person might confuse users and even lead to miscommunication and broken processes.

There is a good article by Tania Menice (Microsoft): Exclude Users From Delve and SharePoint Online People Search with the latest updates stating that currently it is not possible, but Microsoft is working on it.

Basically, the article says:

  • Set the profiles AD property msExchHideFromAddressLists to True or Yes,
  • Sync/wait, so finally SharePoint UPA service SPS-HideFromAddressLists property will be set (msExchHideFromAddressLists AD property is mapped to the UPA SPS-HideFromAddressLists)
  • Under SharePoint classic search – update query:
    {searchboxquery} to {searchboxquery} -“SPS-HideFromAddressLists”:1

It works perfect for classic search. The problem is it does not work as expected in modern Microsoft Search.

“People” vertical is not customizable so far. So we cannot change query to do the same trick. But… it seems like Microsoft is working on it so finally it should be done by ootb means.

Here is the current situation on how different services or search entry points respect SPS-HideFromAddressLists property:

Microsoft 365 Service or Search Entry pointrespect SPS-HideFromAddressLists
(msExchHideFromAddressLists)
web Outlook “New message” user pickerYes
web Outlook “Contacts”Yes
Office.com “All” verticalYes
Office.com “People” verticalNo
SharePoint landing page “All” verticalYes
SharePoint landing page “People” verticalNo
Bing Work All VerticalYes
Bing Work People VerticalYes

So only “People” vertical in Microsoft search does not respect SPS-HideFromAddressLists (msExchHideFromAddressLists).

What about cloud-based accounts (not synchronized from local AD)?

There is a configuration setting “Show in global address list” that does the same job. It’s under Microsoft 365 admin center -> Active Users -> User – Edit -> Mail -> Show in global address list:

And another configuration settings “Hide from global address list (GAL)” under Exchange Admin Center:

(WIP)

User Account12345
EnabledYesNoYesYesYes
Licensed (E5)NoYesYesYesYes
m365 Admin Center: Show in Global Address Listn/aNoNoYesNo
Exchange Admin Center: Hide from global address list (GAL)n/aOnOnOffOn
Get-AzureADUser -ObjectId … | Select -ExpandProperty ShowInAddressListnullnullnullnullFalse
SPO UPA ‘SPS-HideFromAddressLists‘ valueFalseFalseFalseFalseTrue
Outlook Address List “All Users” Shown
Office.com Search: Vertical “All”Shown
Office.com Search: Vertical “People”ShownShownShownShown
Bing Work Search: All/People verticalsShownShown
Teams Search: “All” VerticalShownShown
Teams Search: “People” verticalShownShownShownShown
Microsoft 365 Profile card – OrganizationShownShown
Teams Profile card – OrganizationShownShownShownShown
Teams People PickerShownShownShownShown
SharePoint People PickerShown
Outlook People Picker:Shown
* – some users can see changes after hours, for some it takes days

Doesn’t it seem confusing we have properties:

  • “Show in Global Address List” under m365 Admin Center
  • “Hide from global address list (GAL)” under Exchange Admin Center
  • “ShowInAddressList” Azure AD User object property
  • “SPS-HideFromAddressLists” SharePoint User Profile property

Are these properties related to each other?

(WIP) Let’s test it:

Action-Consequences
(immediate reaction – minutes if not other mentioned)
“Show in Global Address List”
under m365 Admin Center
“Hide from global address list (GAL)”
under Exchange Admin Center
“ShowInAddressList”
Azure AD User object property
“SPS-HideFromAddressLists”
SharePoint User Profile property
New user created, license assignedYesOffnullFalse
Uncheck “Show in my organization address list” under Microsoft 365 admin centerNoOnafter one minute: null
after 24 hours: null
after one minute: False
after 24 hours: False
Set “ShowInAddressList”
Azure AD User object property to “True”
YesOffTrueFalse
Set “ShowInAddressList”
Azure AD User object property to “False”
NoOnFalseTrue

Some findings:

  • “Show in Global Address List” under m365 Admin Center and “Hide from global address list (GAL)” under Exchange Admin Center – same, i.e. if you change one – another will be updated momentarily
    But both does not affect “ShowInAddressList” Azure AD User object property and “SPS-HideFromAddressLists” SharePoint User Profile property
    and vise versa (???)
  • “SPS-HideFromAddressLists” SharePoint User Profile property is not changeable.
    If you try to change the property value you’ll get an error message:
    Set-PnPUserProfileProperty : Property Not Editable: This property can not be modified.
  • “ShowInAddressList” Azure AD User object property is editable and synchronized to “SPS-HideFromAddressLists” SharePoint User Profile property (takes minutes)
    but then search crawler must pick this change up (takes hours) to hide/show the user
  • here Microsoft says: regarding showInAddressList – Do not use in Microsoft Graph. Manage this property through the Microsoft 365 admin center instead. Represents whether the user should be included in the Outlook global address list. See Known issue.
  • Known issue (Microsoft): showInAddressList property is out of sync with Microsoft Exchange. When querying users through Microsoft Graph, the showInAddressList property may not indicate the same status shown in Microsoft Exchange. We recommend you manage this functionality directly with Microsoft Exchange through the Microsoft 365 admin center and not to use this property in Microsoft Graph.
  • Results are inconsistent 🙁

Note: Az module works fine too. I.e. Get-AzADUser instead of Get-AzureADUser and Set-AzADUser instead of Set-AzureADUser.

References

Get list of new SPO sites with PowerShell

Scenario

Let say you administer Microsoft 365 SharePoint Online and you want to get a list of new SharePoint sites (e.g. sites created recently – during last day/week/month).

With GUI it’s done easily: SharePoint Admin Center -> Active Sites -> sort based on “Date Created” – done.

With PowerShell – not so simple.
“Get-PnPTenantSite” cmdlet returns a site object but the object does not have “Created” field. It’s a web property. But to get a web object – you have to connect separately to each site and get root web object to check when the web was created. For small environments it is possible, for large environments it can take days… And still not nice.
“Get-PnPTenantSite” with “-Filter” option would help, but “…Currently, you can filter by these properties: Owner, Template, LockState, Url.”

Get-SPOSite – similar experience.

Solution

Microsoft Graph API helps. It returns result in seconds and you can sort or filter results based on created date . Below are two methods: Option 1 is based on Search and filtering and Option 2 is based on Sites Search and sorting. So there are some pros and cons for each method.

Option #1: Microsoft Graph Search API.

Entry point: https://graph.microsoft.com/v1.0/search/query

Microsoft Graph Search API allows KQL in queries. So we can form a query with something like “created>=1/1/2021” and use entity type = ‘[“site”]’. Search should return only sites created after Jan 01, 2021.

Check PowerShell script sample here: Get-NewSites.ps1
https://github.com/VladilenK/PowerShell/blob/main/reports/SharePoint/Get-NewSites.ps1

If you are getting more than 500 results – think of paging.

Option #2: Microsoft Graph Sites API

Entry point: https://graph.microsoft.com/v1.0/sites

This option is also based on Microsoft Graph API, but sites entry point, which allows search too and sort results by property “createdDateTime”. So we will just search for everything and select how many results we need based on createdDateTime property.

Check PowerShell script sample here: Get-NewSites.ps1
https://github.com/VladilenK/PowerShell/blob/main/reports/SharePoint/Get-NewSites.ps1

References

DepartmentId for Enterprise Intranet Portal – Home Site and Search Scope and Relevance

Scenario

Your organization use Microsoft 365. You are implementing or configuring an Intranet Portal (Home Site). Search plays an important role here – you want search be relevant to the context – i.e.

  • Official Results – if a user searches something on a company’s intranet portal – user expect “official” results, not a something from somebody’s OneDrive or Yammer chat
  • Promoted Results – so information management team can adjust search with search answers – Bookmarks, Acronyms and Q&As

Problem

Microsoft Bookmarks are working only at tenant level search – i.e. if you want bookmarks work on site level search – you need to set up site search scope as tenant.

So if you configure the Intranet Portal site (Home site) search scope to “site” or “hub” to limit results with site/hub content – you will loose “answers” functionality.

Solution

The solution is very simple:

  • Keep site search scope as tenant-wide to use answers (boormarks), and
  • Configure search verticals and query to limit results to “official” sites only

Update Query field with KQL – e.g., with something like

(path:http//contoso.sharepoint.com/sites/IntranetPortal OR path:http//contoso.sharepoint.com/sites/CompanyNews OR path:http//contoso.sharepoint.com/sites/Onboarding)

to get results only from “Intranet Portal” and “Company News” sites.

Keep in mind that this will affect all other SharePoint search entry points – SharePoint landing page, Office.com etc. – so although you can configure All (and Files) verticals, but it’s not recommended. It will confuse users – they expect to search for everything under “All” vertical. Instead, consider custom vertical – e.g. “Official” scope.

After configuring – It might take 1-24 hours for the change to take effect, depending on tenant size.

Service vs Site search

If you configure that at the tenant level – i.e., Microsoft 365 Administration -> Settings -> Search and intelligence ->  Customizations -> Verticals
then search results will be trimmed everywhere  – SharePoint Landing Page, Office landing page (Office.com), Office App, Bing search.
Teams search will not be affected as from Teams you only search for teams content. Same for Onedrive and Yammer.

If you want the “official” search results only under Intranet Portal and leave other search entry points unaffected – then
you need to configure the same at Home (Intranet Portal) site level: Site Settings -> Microsoft Search -> Configure search settings -> Verticals
and configure site search scope to site or hub scope. But in this case you will loose answers functionality.

Global search settings – like acronyms, bookmarks and verticals – works only if site search scope is tenant.
If site search scope is site or hub – then site-level search verticals will apply (and no answers functionality will be possible).

Home site is a root site

There might be two problems with that:

  1. if a home site configured as a root site – you KQL will look like(path:http//contoso.sharepoint.com/ OR path:http//contoso.sharepoint.com/sites/CompanyNews)and that query will not work as any site Url will match the root site Url.
  2. if you need to mention many sites in KQL – like 50 sites with an Official Information – you might hit the “number of allowed character” limit

The solution is DepartmentId property:

DepartmentId

Use DepartmentId={<Hub Site ID>} in the KQL qury and your search results will be limited to your hub content while answers will still be working too. You can even combine DepartmentId with other conditions to add more sites (that are not in hub) to search scope:

(DepartmentId={4965d9be-929b-411a-9281-5662f5e09d49} OR path:http//contoso.sharepoint.com/sites/Onboarding OR path:http//contoso.sharepoint.com/sites/CompanyNews)

Site Property Bag

Another possible option – site property bag.
Remember – the ultimate goal is to give users “Official” results. But official sites might not be all part of one Home site hub. There might be many official sites – e.g. operated by different departments – and all of them might want to be present in search results.
So, how about – if the site is considered official – we create an indexed site property, e.g. “SiteIsOfficial” with a value “Yes”. Then we map the crawled property to a managed property – e.g. RefinableString89 – and use this managed search property in query – e.g. (SiteIsOfficial=Yes).

Update: this is actually clever idea, but it does not work… This query would only return sites, not content.

Official Vertical

Again – the ultimate goal is to give users option to have “Official” results. But they still might want to be able to search through all content.
What if we rename the default “All” vertical to “Home Site” and configure query for official results only.
Then we can create a custom vertical called “Everything” or “All” with no query limitations to give users all reasults

Update: not a good idea either… If the home site search scope is tenant – so verticals are configured and be visible at tenant leve – i.e. everywhere…

Resources:

Microsoft 365 Search: roadmap and announcements

updated: Feb 13, 2022

(Old/Classic) SharePoint Search: content-centric (SharePoint Search Center)
(New/Modern) Microsoft Search: people-centric (Teams, Office, OneDrive, Delve etc.)

Office graph = codename for collective set of services and insights we generate on top of the infrastructure that fast office graph group developed 
= social Intel concepts (SharePoint home, Delve, OneDrive Discoverview) are derivatives of Office graph 

Microsoft Graph = API ( including universal search API)
The Graph Search API went General Availability (GA):
– Microsoft Search API in Microsoft Graph
Use the Microsoft Search API to query data
Microsoft Search API Code samples, Tutorials at github

Microsoft Search API provides one unified search endpoint that you can use to query data in the Microsoft cloud – messages and events in Outlook mailboxes, and files on OneDrive and SharePoint – that Microsoft Search already indexes.

Turing technology – understands you, answers your question e.g. hover over doc -> doc summary (based on “deep speed” AI model)
announcement at Ignite Spring, more on Ignite Fall 2021

Modern Search: MS nailed the fundamentals, now start bringing it everywhere  – to Teams first, then SharePoint (said Nov 2020).

Modern Search Customizations  – we’ll take the best from Classic SharePoint Search,
a lot will retire – investing in more flexibility  

PnP modern Search
– custom result pages, webparts, branding theme; filters, refiners, scoping control  ) 
pnp modern search – webparts (video)
https://microsoft-search.github.io/pnp-modern-search

Core idea behind Microsoft search is coherence 

Bill Baer:
People use search in a different ways 
1) you have organisations who have a well-established intranet built around set of governance controls, a very clean architecture and they want to build a search into that intranet scenario; that’s why a lot of SharePoint capabilities are going to come along with Microsoft search for that particular endpoint
2) then you have other people who live their day in teams

Updates

Shared search engine results page (developed once – transitioned everywhere)
Ctrl-F to search through teams (chats?) (contextual search)
Natural language search (starting from Outlook)
Image search (before eoy), + 

Metadata content type search Syntex customers will be able to use content type to search in the advanced search flyout in document libraries that contain content types beyond the default types. Syntex only.

Conversations: teams chats, outlook groups conversations, yammer conversation can be found under Conversations vertical in Bing search. Later – in Office.com and SharePoint landing page.
E-mail messages are added to Conversations vertical in Bing search

Bookmarks (new promoted results), acronyms, Q&A – all under “Answers”

Bookmarks Targeting – for the specific audience based on device/OS, Country/Region, security groups…

SharePoint Search Admin Center -> will be migrated from SharePoint admin center to to Microsoft Search Admin Center transitioning (Search and Intelligence Admin Center) – long-running project custom dictionaries, spelling suggestions – will retire, (move to a graph-driven speller) 

+ Viva Topics – based search capabilities

  • Create Topic Answers with Microsoft Viva Topics to bring together people, content, and information (including synonyms and acronyms)
  • Knowledge answers provide a direct answer to questions authoritative information in an organization across SharePoint and OneDrive content
  • Files/Calendars/Links answers

Graph Connectors
Graph Connectors are generally available (ADLS – Azure Data Lake Storage Gen2, Azure DevOps, Azure SQL and Microsoft SQL Server, Enterprise websites, MediaWiki, File share, Oracle SQL, Salesforce, Jira, Confluence, ServiceNow + 100+ from partners; New connectors coming to Microsoft Search: Jira Graph connector, Confluence Graph connector).

Graph Connector allows to connect external source of information to Microsoft 365 and makes that data available across all m365 apps and services so you can find what you need wherever you’re working, whether in one of your favorite productivity apps or one of the many Microsoft 365 services such as SharePoint or Office.com

Graph Connectors roadmap:

Actionable experiences
Search results on select Graph connectors will soon support actions that will allow users to interact with the result and perform changes to the Connector content within the Search application.

Results clusters
The results shown in a result cluster are grouped together based on the search vertical configuration.

Profile Query variables
Define any attribute from the user’s Profile, as a query variable and it would be resolved during query evaluation (This feature is currently in preview)

Profile enrichment with Graph connectors
…you will soon be able to enrich Microsoft 365 profile properties like Job title, Phone numbers, Skills etc. with data from HRMS systems using graph connectors. …then surface this rich profile information on people experiences like profile cards.

Search Federation
federation capabilities will allow enterprises build and integrate their custom LOB search experiences, customized search providers, into the overall Microsoft Search. With federated search, you can make information from systems where the data cannot leave the systems boundaries available to search across in Microsoft 365 productivity apps and services, without indexing its data with Microsoft Search.

Azure Cognitive Search Federation

PowerBI search vertical

Custom verticals and custom refiners

Custom result templates – search layout designer – wysiwyg editor
Manage search result layouts
Microsoft Search Layout Designer

Standalone Search  – AAD identity – Graph connector – Ingest your data – use Search = in Windows 10, Office.com  ( e.g. for those who have their data in other productivity suite, have no intent to use m365, but want to search)

More info:

References

Bill Baer “Making the most of Microsoft Search” @ MS Ingnite fall 2021

Current state of SharePoint Search and Microsoft Search scopes

https://techcommunity.microsoft.com/t5/microsoft-search-blog/microsoft-search-at-ignite-2020/ba-p/1651098

https://techcommunity.microsoft.com/t5/microsoft-search-blog/what-s-new-for-microsoft-search-ignite-2020-edition/ba-p/1675291

Bill Baer: What’s new and what’s next for Microsoft Search (May 25, 2021)

Bill Baer on Search:

Microsoft 365 Search Roadmap

How to Find Content Shared with Everyone in SharePoint and Teams

There is a known problem in SharePoint – complicated permissions system. Site owners/administrators provide access, site contributors upload documents and nobody knows – who has access to their sites. As a result – sometimes sensitive documents become overshared (over-exposed).

The biggest concern is sites content shared with “Everyone”. How do we find sites shared with “Everyone” in a large Microsoft 365 environment?

NB. When I say “shared with Everyone” – I actually mean 3 possible “everyone” logins:

  • Everyone
  • Everyone except external users
  • All users

Approach #1 (Brute force)

We can get full permissions report at tenant level (or permissions provided only to “Everyone”). There are 3-rd party tools (e.g. ShareGate, SysKit, AvePoint, Metalogix etc.), or you can run PowerShell script…

Sounds easy? Well, if you have less than 1000 sites – probably it will work. But if your environment is 10K+ sites – it will take forever. Permission report might run hours for an average site with site/subsite, list/library and list item details level. So the approach will not work for large enterprise environments.

One might say – we can limit report with root web permissions only to get it faster. But this would not be accurate. And what is not accurate in the IT security – lead to even bigger risks. So, we need report detailed up to every item level deep, as even one file with sensitive info shared with everyone can cause security issue. (3-rd party tools usually by default limit it to libraries level.)

Ok, if this approach is not really working – what’s working?

Clever idea: why do we need to iterate through all the tenant documents/items if all the content is already crawled by search? Search is also respect permissions. Can we just use search to get files shared with Everyone? Let us see.

What if we use some dummy user account with no specific permissions provided and no group membership and try to search content on behalf of that account. The idea is if this user can see data – it means that data is open for everyone.

Check this and this articles. Can we get results programmatically (e.g. with PowerShell)? Can we use Microsoft Graph search API? Sure.
Check this article “How to search against SharePoint Online Content with Microsoft Graph search API with PowerShell”.

But! We have some problems here.

Search Problem #1. Again, for small environments or if there are not much “Open” sites – it would work. But for large enterprise environments the problem is the same as in “brute force”. Search returns too many results – it’ll take weeks to get all of them. (There are team sites “legally” shared with everyone, public Office 365 group based sites, communication sites… ).

Search Problem #2. Even if we get all search results – we do not know – at what level permissions are provided to everyone. So we will need to build list of sites based on the search results – ant then still need to run permissions report against these sites.

Search Problem #3. We are getting results with paging. But recently Microsoft started limiting number of returning results. E.g. your search request result might say like “there are 3659735 total hits” but after result number 1000 it just stops returning anything, even with paging.

Approach # 3 Hybrid

The idea: why do we need to get all search results if even one result from a site would be enough to add the site to the list of sites require permissions review. In other words, we do not need all results from site, we only need one to know the site is open.

So, consider (imho, the best) approach (Solution):

  1. You get list of all sites in tenant.
  2. You run search request against each site in the loop
    (e.g. consider KQL option “Site: https://yourTenant.SharePoint.com/sites/YourSite”.
    If at least something found in the site – add the site to the “Open Sites” list.
    With this approach you will get list of sites shared with “Everyone…” in a couple of minutes.
  3. Run permissions report against this shortlist

Note: consider there are resources like “Styles Library” shared with everyone by default.

Note: You can refine the list you get at step 1 – e.g., excluding sites connected to public teams or known communication sites…

Note: consider implementing sensitivity labels. At least you can start with high-sensitive sites. Site owners/member will know – what kind of site they are working on.

Pros and cons of the Approach # 3 Hybrid

Pro: the only fast and accurate enough to rely on

Con 1 : crawling and indexing takes time, so search-based reports can miss recent changes in data and permissions

Con 2: this approach cannot be automated (since we need an interactive authentication).

How to communicate to site owners

The Next step would be “How to let site owners know that there are resources shared with Everyone… on their sites”.

References