Get list of new m365 SharePoint sites or teams with PowerShell and Graph API

Leave a reply

There are scenarios when you need to pull only newly created SharePoint sites, e.g. get sites created since yesterday or get last 100 created sites. Usually other articles and existing PowerShell scripts solve this by pulling all sites from tenant and then iterating through sites to get only new sites. That approach is not nice and simply does not work in large environments. How can we get only sites created recently, not all sites? Here is how I use Microsoft Graph API to get only new sites.

Update (6/28/2024): Microsoft announced updates to it’s delta API for SharePoint, so I added option 3 – see below.

Scenario

Let say you administer Teams, OneDrive and SharePoint Online in a Microsoft 365 tenant. You have a pretty big environment – ~10k or more sites and you want to quickly find just new SharePoint sites or teams (e.g. sites created recently – during last hour/day/week/month). This might be required for ad-hoc reports and for automation scenarios – like applying required configurations or assign some property value to all newly created sites.

With GUI it’s done easily: SharePoint Admin Center -> Active Sites -> sort based on “Date Created” – done.

With PowerShell – not so simple.
“Get-PnPTenantSite” cmdlet returns a site object but the object does not have “Created” field. It’s a web property (not site property). But to get a web object – you have to connect separately to each site and get root web object to check when the web was created. For small environments it is possible, for large environments it can take days… And still not nice.
“Get-PnPTenantSite” with “-Filter” option would help, but “…Currently, you can filter by these properties: Owner, Template, LockState, Url.”

Get-SPOSite – similar experience.

Teams + Exchange modules can help a little:

Get-Team | select GroupId | % { Get-UnifiedGroup $_.GroupId | select DisplayName, WhenCreated } | sort WhenCreated

but… 1) it’ll give you group-based sites only 2) it is not easy to automate 3) this might take long for large environments. I know much better solution:

Solution

Microsoft Graph API helps. It returns result in seconds and you can sort or filter results based on created date . Below are two methods: Option 1 is based on Search and filtering and Option 2 is based on Sites Search and sorting. So there are some pros and cons for each method.

Option #1: Microsoft Graph Search API.

Entry point: https://graph.microsoft.com/v1.0/search/query

Microsoft Graph Search API allows KQL in queries. So we can form a query with something like “created>=1/1/2021” and use entity type = ‘[“site”]’. Search should return only sites created after Jan 01, 2021.

Check PowerShell script sample here: Get-NewSites.ps1
https://github.com/VladilenK/PowerShell/blob/main/reports/SharePoint/Get-NewSites.ps1

If you are getting more than 500 results – think of paging.

Option #2: Microsoft Graph Sites API

Entry point: https://graph.microsoft.com/v1.0/sites

This option is also based on Microsoft Graph API, but sites entry point, which allows search too and sort results by property “createdDateTime”. So we will just search for everything and select how many results we need based on createdDateTime property.

Check PowerShell script sample here: Get-NewSites.ps1
https://github.com/VladilenK/PowerShell/blob/main/reports/SharePoint/Get-NewSites.ps1

Option #3: Microsoft SharePoint delta API

You can use “Get delta” under SharePoint Graph API – check for details here. It says “Get newly created, updated, or deleted sites without having to perform a full read of the entire sites collection”. I’ll do my own testing, but for now check this:
Video: Microsoft Graph Delta Capabilities in SharePoint API

References

Video tutorial:

Working with Yammer API from code

1 Reply

Post messages, read messages, get groups (communities) details and membership – etc. – all that you can do with Classic Yammer API. Here are steps:

  • register Yammer Application and generate access token
  • call API

Register Yammer Application

Navigate to the page: https://www.yammer.com/client_applications

Click “Register new application”:

Fill all the fields:

Client ID and Client secret will be generated automatically:

I’m not sure – how to get access token from Client ID and Client secret, so I use link “Generate a developer token for this application”. When you click this link, a token will be generated, and it says “Here is your personal token for testing. Please copy it somewhere safe as it will disappear when you leave the page:”

Calling Yammer API

Once you get the toke – you can use it in your code (consider vaulting or other save methods). Here is an example based on powershell, but surely you can do the same with programming language you comfortable with:

$baererToken = "Put your token here"

$headers = @{ Authorization = ("Bearer " + $baererToken) }

# get messages
$webRequest = Invoke-WebRequest –Uri "https://www.yammer.com/api/v1/messages.json" –Method Get -Headers $headers
$results = $webRequest.Content | ConvertFrom-Json

$results.messages | ForEach-Object {
    $message = $_ 
    Write-Host "Message Id:" $message.id
    Write-Host "Message body:" $message.body.parsed
}

# get users
Invoke-WebRequest –Uri "https://www.yammer.com/api/v1/users.json" –Method Get -Headers $headers | ConvertFrom-Json | select email

References

DepartmentId for Enterprise Intranet Portal – Home Site and Search Scope and Relevance

Leave a reply

Scenario

Your organization use Microsoft 365. You are implementing or configuring an Intranet Portal (Home Site). Search plays an important role here – you want search be relevant to the context – i.e.

  • Official Results – if a user searches something on a company’s intranet portal – user expect “official” results, not a something from somebody’s OneDrive or Yammer chat
  • Promoted Results – so information management team can adjust search with search answers – Bookmarks, Acronyms and Q&As

Problem

Microsoft Bookmarks are working only at tenant level search – i.e. if you want bookmarks work on site level search – you need to set up site search scope as tenant.

So if you configure the Intranet Portal site (Home site) search scope to “site” or “hub” to limit results with site/hub content – you will loose “answers” functionality.

Solution

The solution is very simple:

  • Keep site search scope as tenant-wide to use answers (boormarks), and
  • Configure search verticals and query to limit results to “official” sites only

Update Query field with KQL – e.g., with something like

(path:http//contoso.sharepoint.com/sites/IntranetPortal/ OR path:http//contoso.sharepoint.com/sites/CompanyNews/ OR path:http//contoso.sharepoint.com/sites/Onboarding)/)

to get results only from “Intranet Portal” and “Company News” sites.

Keep in mind that this will affect all other SharePoint search entry points – SharePoint landing page, Office.com etc. – so although you can configure All (and Files) verticals, but it’s not recommended. It will confuse users – they expect to search for everything under “All” vertical. Instead, consider custom vertical – e.g. “Official” scope.

After configuring – It might take 1-24 hours for the change to take effect, depending on tenant size.

Service vs Site search

If you configure that at the tenant level – i.e., Microsoft 365 Administration -> Settings -> Search and intelligence ->  Customizations -> Verticals
then search results will be trimmed everywhere  – SharePoint Landing Page, Office landing page (Office.com), Office App, Bing search.
Teams search will not be affected as from Teams you only search for teams content. Same for Onedrive and Yammer.

If you want the “official” search results only under Intranet Portal and leave other search entry points unaffected – then
you need to configure the same at Home (Intranet Portal) site level: Site Settings -> Microsoft Search -> Configure search settings -> Verticals
and configure site search scope to site or hub scope. But in this case you will loose answers functionality.

Global search settings – like acronyms, bookmarks and verticals – works only if site search scope is tenant.
If site search scope is site or hub – then site-level search verticals will apply (and no answers functionality will be possible).

Home site is a root site

There might be two problems with that:

  1. if a home site configured as a root site – you KQL will look like(path:http//contoso.sharepoint.com/ OR path:http//contoso.sharepoint.com/sites/CompanyNews)and that query will not work as any site Url will match the root site Url.
  2. if you need to mention many sites in KQL – like 50 sites with an Official Information – you might hit the “number of allowed character” limit

The solution is DepartmentId property:

DepartmentId

Use DepartmentId={<Hub Site ID>} in the KQL qury and your search results will be limited to your hub content while answers will still be working too. You can even combine DepartmentId with other conditions to add more sites (that are not in hub) to search scope:

(DepartmentId={4965d9be-929b-411a-9281-5662f5e09d49} OR path:http//contoso.sharepoint.com/sites/Onboarding OR path:http//contoso.sharepoint.com/sites/CompanyNews)

It worth to mention, that DepartmentId is the property that propagated from the root of the hub site to all associated sites and their content – list items, documents and pages.

Site Property Bag

Another possible option would be – site property bag…
The ultimate goal is to provide users with “Official” results only. But official sites might not be all part of one Home site hub. We can include in All search vertical query 10, 50, 100 sites, but what if we have 10k official sites in enterprise – e.g. operated by different departments – and all of them might want to be present in search results.
So, how about – if the site is considered official – we create an indexed site property, e.g. “SiteIsOfficial” with a value “Yes”. Then we map the crawled property to a managed property – e.g. RefinableString89 – and use this managed search property in query – e.g. (SiteIsOfficial=Yes).

This is actually clever idea, but this does not work… This query would only return sites, not sites content. E.g. what was indexed as site object – will be included (including home page). But site items – i.a. documents, lists, other pages – all site content – will not be included…

So let’s get back to DepartmentId…

Rename All Vertical

Again – the ultimate goal is to give users option to have “Official” results. But they still might want to be able to search through all content.
What if we rename the default “All” vertical to “Home Site” and configure query for official results only.
Then we can create a custom vertical called “Everything” or “All” with no query limitations to give users all reasults

Update: not a good idea either… If the home site search scope is tenant – so verticals are configured and be visible at tenant level – i.e. everywhere…

Separate Official Vertical

My personal preference is to keep All vertical as real All, and create a custom Vertical “Official” for official results only where we would use query trick.

In addition, it would be nice to highlight results from official sources by using custom result type – check “Manage result layouts for SharePoint results in Microsoft Search”

Update: Restricted SharePoint Search

There is a new feature in Microsoft 365 SharePoint Online – “Restricted SharePoint Search“. With Restricted SharePoint Search you can restrict organization-wide search to a curated set of “allowed” SharePoint sites – sites that you have checked the permissions and applied data governance for.

Resources:

Why we study foreign languages

Leave a reply

I know that my country – the country where I was born and grew up, the country where my parents gave me everything – from first sip of milk to education and cultural upbringing, the country where I got my friends – this country is the best country in the world. But wait… Did you say “the world”? What is the world? Is it something where other people live? Do those people live their own ways and speak their own languages? Do they love their countries and their culture?

Of course, we know that there are other countries (we use internet-connected computers and smartphones). We probably heard/use words like Gastarbeiter, automobile, opera. But did you know they all came from different other languages to our language. Now what are the other languages and why do we learn foreign languages? I would say there are two main reasons.

The first one is just a practical reason that came from the past – knowing a foreign language helps you make more money. International trade is an essential part of the world economy and impossible without communication. So the ability to communicate in different languages simply gives you an advantage – an ability to do thing not everybody can do – and an opportunity to earn more.

The second reason lies more in the cultural or cognitive field. Simple example: How do I know that The Hamburgeris is tasty? Only because I tried other food and I can compare. The same with languages. “Wer fremde Sprachen nicht kennt, weiß nichts von seiner eigenen” – which means “Those who do not know foreign languages do not know anything about their own” stated the famous German poet and scientist Johann Wolfgang von Goethe.

And that is so true. I just started understanding it because I just started studying foreign language. And the more I learn it – the more I like it. The world is diverse. And that’s the beauty of it. And that diversity is expressed via languages. Each language is unique and beautiful as it absorbs all historical and cultural legacy of a nation.

So, knowing other languages not only benefits us with more opportunities and perspectives, but makes us more tolerant, open-minded, intelligent and creative.

Essay: My future profession

Leave a reply

Change is live.  Static is death. Everything and everybody must change for live. The moment you stop developing – you are out. That’s true for everything starting from plants and animals. For human being that’s also true, but a little different. We are not competing for food any more. As a social creatures we want to be a part of the society, we need to be respected, recognized, useful. And that is done via job, via profession we choose.

In average we spent 8 hours a day at work. Some people might think that work time is distracted from our live because only after work you can have fun spending money hard earned during the work time when you have to do something unpleasant just to be able to do what you really want to do after work. Miserable and pitiful people they are. Eight hours a day is a decent part of our life. I do not want to waste that time. I want to enjoy that time. How do I enjoy that time? Choosing a profession that I love. 

In a modern world one of the biggest problem mankind facing is a human health. And that is where I want to work. I know, that’s not easy. In order to be successful in the healthcare field, I need to pose certain skills and abilities. One of the essential skills is an ability to love people. I must be able to treat and respect my clients in order to maintain their trust and support. As well as readiness and acceptance. I have to be precise and confident in my actions. I have to be able to accept the consequences if something goes wrong. There may be times when an emergency occurs and the environment gets chaotic. But, on the other hand, what could be better, what could be more demanded and rewarding then help people live healthier lives? 

Retrieve SharePoint Online system page html content programmatically (PowerShell)

Leave a reply

How can I get HTML content of a SharePoint online page from code, e.g. PowerShell?

Invoke-WebRequest returns “Sign in to your account” page, not a real page, even with -Token option.

Thanks to Denis Molodtsov, the solution is found. It turns out the “Invoke-PnPSPRestMethod” PnP cmdlet works not only against /api endpoints, but also against site pages and system pages.

But (as per my experience) it works only with PnP.PowerShell and with -UseWebLogin authentication option and with -raw parameter.

Connect-PnPOnline -url $siteUrl -UseWebLogin
Invoke-PnPSPRestMethod -url /_layouts/15/viewlsts.aspx -Raw

Other combination of authentication options ( -interactive, -clientId, -Token, -SPOManagementShell, -PnPManagementShell ) – worked well, but only for /_api endpoints, and gave me “401 UNAUTHORIZED” against system/site pages.
Unattended authentication (with clientId, clientSecret and certificate) – same.

Legacy PnP module SharePointPnPPowerShellOnline did not work at all: “EXCEPTION,PnP.PowerShell.Commands.Admin.InvokeSPRestMethod”.

I tested it with
– SharePointPnPPowerShellOnline v 3.29.2101.0 (under Windows PowerShell 5.1) and
– PnP.PowerShell 1.8.0. (both Windows PowerShell 5.1 and .net core PowerShell 7.1.5)

in the middle…

Leave a reply

It happened in the middle of 1990-x. I just started to work on software company in Almaty as a computer engineer. The company had a customer in the city of Kustanai. I was sent to Kustanay to solve some problem.
It was winter. Winters in the north-Kazakhstan are pretty cold. Business-trip turned-up a little longer than I expected. At the end, day of flight home came. By this time I had run-out of money and worm clothes.
A Little digress from the topic. I used to travel a lot – by plane, by train and even by ship. I used to be petty experienced traveler. What I do not like at all – is crowds and queues. But when you are travelling by plane, you have to be in queues and among the crowd. You have to stay in the row before check-in, then before security, customs, passport control etc. When boarding announces – everybody rush to the gate and stand at their feet half an hour in the row. Usually in such a situation I sit down somewhere near and wait until everybody is boarded, then, among the couple of the same calm as myself, I get on the plane.
But that time something went unusual. I had been waiting on the bench, and thinking of home. And when boarding was announced, I decided not to wait until the very end, but rushed among the others to the gate. There was no bus. All passengers had to walk across the take-off field, at night, at cold, when the wind knocked you down. Again, I was not first, but was not last one.
The plane was Yak-40, a little jet, designed for 50 seats. The entrance was at the back side of the plane. Convertible back door serves as a stairs when got back. I got on the plane, took my seat and then heard a noise from the back door. A flight attendant, strong woman, blocked access to the plane, standing in the doorway. I was heard something like “Stop, get out of plane. We have no free seats more.”
Passengers were trying to climb. There was almost a fight. A man in the uniform – a second pilot – stout man – hurried to help the flight-attendant. They managed to push people out and closed the door. The plane took off. I was sitting in my seat, in the warm cabin, on the way home, thinking of people who had stayed behind, along, in the middle of the airfield, in the cold night.