Search Unified Audit Log Daemon Job

How to run “Search-UnifiedAuditLog” in unattended way, i.e. non-interactive.
What are the minimal permissions required?

The PowerShell code:

$clientId = ""
$cPwd = ConvertTo-SecureString -String "" -AsPlainText -Force

$cPath = ""C:\Users\UserName\Certificates\Cert.pfx""
$organization = "contoso.onmicrosoft.com"

Connect-ExchangeOnline -CertificateFilePath $cPath -CertificatePassword $cPwd -AppID $clientId -Organization $organization

[DateTime]$start = [DateTime]::UtcNow.AddMinutes(-45)
[DateTime]$end = [DateTime]::UtcNow
$resultSize = 1000

$results = $null
$results = Search-UnifiedAuditLog -StartDate $start -EndDate $end -ResultSize $results.Count
$results | Select-Object RecordType, CreationDate, UserIds, Operations -First 3

Disconnect-ExchangeOnline -Confirm:$false

Troubleshooting

The error “The term ‘Search-UnifiedAuditLog’ is not recognized”:

Search-UnifiedAuditLog: C:\scripts\PowerShell.auth\Search-AuditLog-w-App.ps1:16:12
Line |
16 | $results = Search-UnifiedAuditLog -StartDate $start -EndDate $end -Re …
| ~~~~~~
| The term 'Search-UnifiedAuditLog' is not recognized as a name of a cmdlet, function, script file, or executable
| program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again.

means a proper administrative role (e.g. “Exchange administrator”) is not assigned.

References

PowerShell Script to Fetch All Alerts from SharePoint Online Site

PowerShell Script to get All Alerts of all Users from a specific SharePoint Online Site Collection, including subsites:

https://github.com/VladilenK/PowerShell/blob/main/reports/Site/Fetch-All-Alerts-from-SPO-Site.ps1

https://raw.githubusercontent.com/VladilenK/PowerShell/main/reports/Site/Fetch-All-Alerts-from-SPO-Site.ps1

based on Salaudeen Rajack:
SharePoint Online: Get All Alerts from a Site Collection using PowerShell

Microsoft 365 Search: roadmap and announcements

updated: Feb 13, 2022

(Old/Classic) SharePoint Search: content-centric (SharePoint Search Center)
(New/Modern) Microsoft Search: people-centric (Teams, Office, OneDrive, Delve etc.)

Office graph = codename for collective set of services and insights we generate on top of the infrastructure that fast office graph group developed 
= social Intel concepts (SharePoint home, Delve, OneDrive Discoverview) are derivatives of Office graph 

Microsoft Graph = API ( including universal search API)
The Graph Search API went General Availability (GA):
– Microsoft Search API in Microsoft Graph
Use the Microsoft Search API to query data
Microsoft Search API Code samples, Tutorials at github

Microsoft Search API provides one unified search endpoint that you can use to query data in the Microsoft cloud – messages and events in Outlook mailboxes, and files on OneDrive and SharePoint – that Microsoft Search already indexes.

Turing technology – understands you, answers your question e.g. hover over doc -> doc summary (based on “deep speed” AI model)
announcement at Ignite Spring, more on Ignite Fall 2021

Modern Search: MS nailed the fundamentals, now start bringing it everywhere  – to Teams first, then SharePoint (said Nov 2020).

Modern Search Customizations  – we’ll take the best from Classic SharePoint Search,
a lot will retire – investing in more flexibility  

PnP modern Search
– custom result pages, webparts, branding theme; filters, refiners, scoping control  ) 
pnp modern search – webparts (video)
https://microsoft-search.github.io/pnp-modern-search

Core idea behind Microsoft search is coherence 

Bill Baer:
People use search in a different ways 
1) you have organisations who have a well-established intranet built around set of governance controls, a very clean architecture and they want to build a search into that intranet scenario; that’s why a lot of SharePoint capabilities are going to come along with Microsoft search for that particular endpoint
2) then you have other people who live their day in teams

Updates

Shared search engine results page (developed once – transitioned everywhere)
Ctrl-F to search through teams (chats?) (contextual search)
Natural language search (starting from Outlook)
Image search (before eoy), + 

Metadata content type search Syntex customers will be able to use content type to search in the advanced search flyout in document libraries that contain content types beyond the default types. Syntex only.

Conversations: teams chats, outlook groups conversations, yammer conversation can be found under Conversations vertical in Bing search. Later – in Office.com and SharePoint landing page.
E-mail messages are added to Conversations vertical in Bing search

Bookmarks (new promoted results), acronyms, Q&A – all under “Answers”

Bookmarks Targeting – for the specific audience based on device/OS, Country/Region, security groups…

SharePoint Search Admin Center -> will be migrated from SharePoint admin center to to Microsoft Search Admin Center transitioning (Search and Intelligence Admin Center) – long-running project custom dictionaries, spelling suggestions – will retire, (move to a graph-driven speller) 

+ Viva Topics – based search capabilities

  • Create Topic Answers with Microsoft Viva Topics to bring together people, content, and information (including synonyms and acronyms)
  • Knowledge answers provide a direct answer to questions authoritative information in an organization across SharePoint and OneDrive content
  • Files/Calendars/Links answers

Graph Connectors
Graph Connectors are generally available (ADLS – Azure Data Lake Storage Gen2, Azure DevOps, Azure SQL and Microsoft SQL Server, Enterprise websites, MediaWiki, File share, Oracle SQL, Salesforce, Jira, Confluence, ServiceNow + 100+ from partners; New connectors coming to Microsoft Search: Jira Graph connector, Confluence Graph connector).

Graph Connector allows to connect external source of information to Microsoft 365 and makes that data available across all m365 apps and services so you can find what you need wherever you’re working, whether in one of your favorite productivity apps or one of the many Microsoft 365 services such as SharePoint or Office.com

Graph Connectors roadmap:

Actionable experiences
Search results on select Graph connectors will soon support actions that will allow users to interact with the result and perform changes to the Connector content within the Search application.

Results clusters
The results shown in a result cluster are grouped together based on the search vertical configuration.

Profile Query variables
Define any attribute from the user’s Profile, as a query variable and it would be resolved during query evaluation (This feature is currently in preview)

Profile enrichment with Graph connectors
…you will soon be able to enrich Microsoft 365 profile properties like Job title, Phone numbers, Skills etc. with data from HRMS systems using graph connectors. …then surface this rich profile information on people experiences like profile cards.

Search Federation
federation capabilities will allow enterprises build and integrate their custom LOB search experiences, customized search providers, into the overall Microsoft Search. With federated search, you can make information from systems where the data cannot leave the systems boundaries available to search across in Microsoft 365 productivity apps and services, without indexing its data with Microsoft Search.

Azure Cognitive Search Federation

PowerBI search vertical

Custom verticals and custom refiners

Custom result templates – search layout designer – wysiwyg editor
Manage search result layouts
Microsoft Search Layout Designer

Standalone Search  – AAD identity – Graph connector – Ingest your data – use Search = in Windows 10, Office.com  ( e.g. for those who have their data in other productivity suite, have no intent to use m365, but want to search)

More info:

References

Bill Baer “Making the most of Microsoft Search” @ MS Ingnite fall 2021

Current state of SharePoint Search and Microsoft Search scopes

https://techcommunity.microsoft.com/t5/microsoft-search-blog/microsoft-search-at-ignite-2020/ba-p/1651098

https://techcommunity.microsoft.com/t5/microsoft-search-blog/what-s-new-for-microsoft-search-ignite-2020-edition/ba-p/1675291

Bill Baer: What’s new and what’s next for Microsoft Search (May 25, 2021)

Bill Baer on Search:

Microsoft 365 Search Roadmap

Hair Care

One I decided to grow hair… It was just for the sake of experience – I always wanted to see “what if…”. Do I have split ends? What is my hair type? How to brush? etc… Long story short – I grew hair…

I thought all that marketing tricks like “this shampoo is only for that type of hair” or “you should use conditioner after shampoo” do not make sense. Honestly, I thought all these hair care products are just an instrument to swindle money. I did not know that I need to dry my hair…

Lucki me, I have a friend who is a professional hair stylist. She’s in her business for 20+ years, and she with her husband recently moved to Minnesota and opened a beauty studio in Ridgedale center in Minnetonka. So I got some useful advices.

But what’s the best is that they have a hair care online store where you can find hair care products from top professional brands – Alterna, Aluram, BaByLiss, Global Keratin,
Hempz, JBH, Keratherapy, Keratin Complex, Matrix Biolage, Milbon, Moroccanoil, Redken, STMNT, Verb etc. I recommend.

References

Connect to SharePoint Online and MS Graph Interactively with Client App and MSAL token

Scenario

You have a Microsoft 365 subscription with SharePoint Online. You use PowerShell, PnP.PowerShell module and MS Graph API to work with SharePoint under current user’s credential. You need to authenticate to SharePoint Online via Connect-PnPOnline and to Microsoft Graph API interactively on behalf of a current user.

Problem

Unfortunately, both “Connect-PnPOnline -Interactive -Url <siteUrl>” or “Connect-PnPOnline -UseWebLogin -Url <siteUrl>” might fail with something like “Need admin approval”, “App needs permission to access resources in your organization that only an admin can grant. Please ask an admin to grant permission to this app before you can use it.” or “Permissions requested” or similar

Solution

  • register an Azure App. Choose “single tenant”
  • configure authentication blade:
    – add platform – “Mobile and Desktop app”
    select “https://login.microsoftonline.com/common/oauth2/nativeclient”
    add custom Redirect URI: “http://localhost”
  • configure API permissions blade:
    – add delegated permissions you need (refer to specific API you’ll use)
    e.g. Microsoft Graph Sites.FullControl.All and SharePoint AllSites.FullControl
  • use the following code samples

PnP.PowerShell

$siteUrl = "https://contoso.sharepoint.com/teams/myTeamsSite"
$appId = "" # Client Id
Connect-PnPOnline -ClientId $appId -Url $siteUrl -Interactive
Get-PnPSite

A pop-up window will appear to authenticate interactively. If you are already authenticated with another credentials (or single-sigh-on) – an interactive window might pop up and disappear – that prevents you enter your other id.
To ensure Connect-PnPOnline prompts you for your credentials – use ” -ForceAuthentication” option.

If you are a SharePoint tenant admin – you can connect to a tenant with:

$orgName = "yourTenantPrefix" 
$adminUrl = "https://$orgName-admin.sharepoint.com" 
$appId = "" # Client Id 
$connection = Connect-PnPOnline -ClientId $appId -Url $adminUrl -Interactive -ReturnConnection # -ForceAuthentication 
$connection 

Microsoft Graph API

Use MSAL.PS module to get an msal token then use token in Microsoft graph-based requests:

$tenantId = ""
$clientid = ""
$url = ""
$token = Get-MsalToken -ClientId $clientid -TenantId $tenantId -Interactive

By default token expires in ~ 1 hour. But you can refresh it silently.
This helps you in long-running PowerShell scripts that takes hours to complete.
So you can include something like this in the loop:

if ($token.ExpiresOn.LocalDateTime -lt $(get-date).AddMinutes(10)) {    
  $token = Get-MsalToken -ClientId $clientid -TenantId $tenantId -ForceRefresh -Silent    
  Write-Host "Token will expire on:" $token.ExpiresOn.LocalDateTime
}

Application permissions

Somehow using Connect-PnPOnline with AccessToken option did not work if the token was acquired with MSAL.PS interactively. But it did work when you get msal.ps token unattended (using App credentials). So…

If you can get an Application (non Delegated) permissions to your azure-registerd-app,
you can use msal token to connect to site with PnP

=========================

NB: For delegated permissions, the effective permissions of your app are the intersection of the delegated permissions the app has been granted (via consent) and the privileges of the currently signed-in user. Your app can never have more privileges than the signed-in user.

Track SharePoint App-only Service Principals in Microsoft 365

Update (May 2023):
You can use Get-PnPAzureACSPrincipal to returns the lists of all Azure ACS principals installed in your Tenant including subsites.

Scenario

Developers in the organization can use both – Azure Apps and SharePoint Apps to work with SharePoint sites in their “daemon” applications.

It is recommended to use Azure apps so, you want to know – what are SharePoint Apps registered and their owners, who registered SharePoint Apps. Eventually you would disable SharePoint Apps-only principal but before that you’d move Devs from SP-App-only to Azure App (see Disable Custom App Authentication).

(SharePoint App-only service principals aka SP-App-Only are SPN or App registered from within SharePoint using AppRegNew.aspx system page).

One of the approaches is to track Apps/Owners with Unified Audit Log

Use Unified Audit Logs

The following PowerShell code:

$operations = 'Add service principal.'
$recordType = 'AzureActiveDirectory'
Search-UnifiedAuditLog -StartDate $start -EndDate $end -ResultSize $resultSize -Formatted -Operations $operations -RecordType $recordType

returns events with operation = ‘Add service principal.’ Nice, but…
if an app was registered in Azure – event contains an UPN under UserIds property:

Unfortunately, in case with registering app in SharePoint, an audit log event will be like:

i.e. UserId registerd is “spo_service@support.onmicrosoft.com”, so we do not know who registered a SharePoint-only app

In theory – we could use events recorded immediately before and after “Add service principal” event to track a user and site who has registered a SharePoint-only app… But for me it seems like too complicated for automation.

Instead we can do simple search through audit log for events “AppRegNew.aspx page visited”. This gives us a good approximation of who registered SP-App-only principal. Worst scenario – we reach more people than we really need (including those who started registering sp-app-only but did not complete) but all of them would be definitely our target auditory.

Consider the following code:

$freeText = "appregnew"
$operations = 'PageViewed'
$recordType = 'SharePoint'

$results = Search-UnifiedAuditLog -StartDate $start -EndDate $end -ResultSize $resultSize -FreeText $freeText -Operations $operations -RecordType $recordType 

this would give you all users who loaded “/_layouts/15/appregnew.aspx” page

Update: Sites.Selected API MS Graph permissions was introduced by Microsoft in 2021. It was a good move towards site-level development, but still developers were limited with only what MS Graph API provides for SharePoint dev.
So devs had to use AppInv.aspx at site level to provide ACS permissions to their apps to be able to use SharePoint CSOM and REST APIs.
Recently Microsoft introduced Sites.Selected SharePoint API permissions for registered Azure Apps! So now devs should be fully happy without ACS-based permissions.

Update: I’m not sure when Microsoft implemented that, but I recently found another Record Type – “SharePointAppPermissionOperation”. There is an Activity type – AppPermissionGrant…
To be explored…

References

Update item Author field with PowerShell and Item-level list permissions

Scenario:

You have imported a list into SharePoint Online. Every list item contains information for specific users and users’ accounts. You want list items be visible to specific users only.
You want to leverage Item-Level Permissions under List Advanced settings: “Read access: Read items that were created by the user”. But the problem is – since this list was imported – it was not users who created items. So the item-level access feature to work properly – you’d need to update the list so for each item the field “Created By” will have a user account you want the item be visible for.

Solution:

PnP.PowerShell helps. Using “Set-PnPListItem”, you can re-write “Author” field in the list item. In the example below I’m using just static user’s UPN:

Set-PnPListItem -List "Test" -Identity 1 -Values @{"Author"="testuser@domain.com"}

but in reality you’d do it dynamically – based on your specific case.

Then you can turn on Item-Level Permissions under List Advanced settings: “Read access: Read items that were created by the user”:

Add users to “Site Visitors” group for read-only access:

References

  • https://pnp.github.io/powershell/cmdlets/Set-PnPListItem.html
  • https://sharepointmaven.com/how-to-enable-item-level-permissions-in-sharepoint/