Make Sure that Mailbox Audit Configurations Capture Important Events

Following Microsoft’s announcement about the availability of the promised additional audit events for Purview Audit (standard) customers, some folks got in touch to ask if I had a script to report current mailbox audit configurations. As it happens, I didn’t, but cracking open Visual Studio Code and GitHub Copilot soon put that right.

How Not to Find Accounts with Purview Audit (Advanced) Licenses

My original plan was to find and report mailboxes owned by licensed user accounts. I wanted to know which accounts use Purview Audit standard and which use the advanced variant. This is more difficult than it seems because, as far as I can tell, there’s no Purview Audit standard service plan. At least, I can’t find one on the Microsoft page listing all the license and service plan identifiers.

There is a service plan called M365_ADVANCED_AUDITING (2f442157-a11c-46b9-ae5b-6e39ff4e5849), which seemed like a good candidate for Purview Audit (advanced). However, if you use the Get-MgUser cmdlet from the Microsoft Graph PowerShell SDK to find accounts with this service plan identifier in the assignedPlans property (see below), the service plan name returned for the identifier is “exchange.”

[guid]$PurviewAuditAdvancedPlanId = "f6de4823-28fa-440b-b886-4783fa86ddba"

[array]$Users = Get-MgUser -filter "assignedPlans/any(x:x/serviceplanid eq $PurviewAuditAdvancedPlanId)" -ConsistencyLevel eventual -CountVariable Test -Property Id, displayName, userprincipalName, assignedLicenses, assignedPlans

The service plan identifier appears in accounts that don’t have Office 365 E5 or Microsoft 365 E5 licenses, which are the products that include Purview Audit (advanced). This is because the service plan identifier has a disabled status in those accounts. To solve that problem, amend the filter to check for enabled service plans:

[array]$Users = Get-MgUser -filter "assignedPlans/any(x:x/serviceplanid eq $PurviewAuditAdvancedPlanId and capabilityStatus eq 'Enabled')" -ConsistencyLevel eventual -CountVariable Test -Property Id, displayName, userprincipalName, assignedLicenses, assignedPlans

But then I found that the resulting set of accounts only included those with Microsoft 365 E5 licenses. No trace existed of the Office 365 E5 accounts, even though Microsoft includes the Office 365 E5 license in the set with access to Purview Audit (advanced) in this useful comparison chart.

Microsoft documentation assures me that there is an app for Purview Audit (advanced). Usually, an app equates to a service plan. When I checked the Microsoft 365 admin center as directed, the app shows up under the moniker Microsoft 365 advanced auditing (Figure 1).

Disabling and enabling the app in the Microsoft 365 admin center disables and enables the 2f442157-a11c-46b9-ae5b-6e39ff4e5849 service plan behind the scenes. After all that, we know that a service plan called exchange controls an app called Microsoft 365 advanced auditing (aka the Microsoft Purview Audit (advanced) product) that only shows up in accounts with Microsoft 365 E5 licenses. It’s all very confusing, so I lost interest at this point.

Back to Scripting Mailbox Audit Configurations

After wasting too much time discovering the mess of service plans, product names, and SKUs, I went back to scripting and wrote some straightforward code to:

• Connect to Exchange Online.
• Run Get-ExoMailbox to find user and shared mailboxes.
• Define some critical audit events to check for in the owner and delegate audit sets.
• Check each mailbox to see if it uses the default audit configuration (maintained by Microsoft). Report the audit set defined in the configuration.
• Check that the critical audit events are present in the owner and delegate audit sets and flag any critical audit events (like MailItemsAccessed) found missing.
• Report what’s been found.
• If the ImportExcel PowerShell module is available, generate an Excel worksheet containing the results (Figure 2). If not, generate a CSV file.

You can download the full script from GitHub.

A Note About Enabling Audit with Set-Mailbox

The script checks if auditing is enabled for a mailbox, and if it is, the script runs Set-Mailbox to set AuditEnabled to true. Microsoft documentation says that if mailbox auditing is turned on by default for an organization, mailbox auditing ignores the AuditEnabled mailbox property.

But their May 20 announcement about the new audit events says that “Every standard user mailbox should have AuditEnabled set to true to ensure all audit records are uploaded to Purview Audit” and “Please note that this Set-Mailbox command must be run for every Standard license user regardless of its current value to correctly enable their mailbox to upload the new standard logs to Purview Audit.” Microsoft documentation is confusing on this point. I think the situation is that Microsoft manages mailbox auditing for accounts with Purview Audit advanced licenses while manual intervention is needed for mailboxes with Purview Audit standard, Whatever the reason, it’s always better to be safe than sorry when dealing with audit events, the script runs Set-Mailbox. You can certainly eliminate this section of the script to speed things up if you want to.

Feel free to improve and embellish the script to meet your needs. In the meantime, I need a headache tablet to recover from the trials of audit licensing.

Stay updated with developments like new events for mailbox audit configurations across the Microsoft 365 ecosystem by subscribing to the Office 365 for IT Pros eBook. We do the research to make sure that our readers understand the technology.

Mailbox Audit Events Problematic for Multi-Geo Audit Searches

At first glance, the title given to MC550039 (3 May 2023) is confusing. What exactly does “Configuration Change: Search-MailboxAuditLog cmdlet, multi-geo scenarios” mean?

Microsoft attempts to explain that in multi-geo tenants, a problem might arise when running the Search-MailboxAuditLog cmdlet to search mailbox audit records, leading to the message “An error occurred while trying to access the audit log.” The reason why this happens is that the administrator running the search uses a mailbox in a different region.

Remember that Exchange stores mailbox audit records in the Audits folder of Recoverable Items in user mailboxes. In a multi-geo organization, user mailbox can be in different regions. To make sure that searches work, administrators must “anchor” themselves in the target region by specifying a mailbox in the region when they run Connect-ExchangeOnline to connect PowerShell to Exchange Online. It’s a way of setting a scope for the mailbox audit log search.

The Unified Audit Log is a Better Search Target

Exchange Online generates mailbox audit events for mailboxes licensed with Exchange Online Plan 1 and 2. Enterprises running multi-geo organizations are likely to have Office 365 E3 or better licenses and therefore can use the unified audit log to gather data from multiple workloads in a single searchable store. In Office 365 E3 environments, administrators must enable mailboxes for auditing before mailbox audit events flow to the unified audit log. From March 2023, admin audit events from all regional locations end up in the unified audit log and can be searched there.

For instance, this command finds mailbox audit events recording the use of the Send As feature for a shared mailbox:

Search-MailboxAuditLog -Identity Customer.Communictions -LogonTypes Delegate -StartDate ((Get-Date).AddDays(-90)) -EndDate ((Get-Date).AddDays(+1)) -ShowDetails | Where-Object {$_.Operation -eq "SendAs"} | Select LogonUserDisplayName, LastAccessed

LogonUserDisplayName LastAccessed
-------------------- ------------
Tony Redmond         03/05/2023 20:29:39
Tony Redmond         03/05/2023 19:42:39
Tony Redmond         03/05/2023 19:13:41
Tony Redmond         20/04/2023 23:48:11
Tony Redmond         20/04/2023 17:44:10
Tony Redmond         20/04/2023 14:08:41

Within 15 minutes or so of creation, Exchange Online sends the mailbox audit events to the unified audit log. The events are immediately searchable through the Purview compliance portal (Figure 1) and PowerShell.

But, as MC550039 points out, unless you’re signed into an account in a satellite region, you won’t be able to see mailbox audit events from that region. Exchange Online does not transmit mailbox audit events from satellite regions to the unified audit log.

Admin Events

Exchange Online also generates admin audit events and transmits these events to the unified audit log. This process works for multi-geo environments, meaning that you can search for admin events using PowerShell or the audit search in the Microsoft Purview compliance portal.

Passing the Message

Microsoft hasn’t invested in Exchange mailbox audit logging for several years. Their focus is on the unified audit log. This is natural because the unified audit log offers better search options (even if the modern audit search GUI in the Purview compliance portal is very slow to retrieve audit data).

In general, it’s always best to use Search-UnifiedAuditLog or the Microsoft Purview compliance portal to search audit data. The exception, as appears to be in this case, is when searching for mailbox audit data from a satellite region, where you’re forced to run the Search-MailboxAuditLog cmdlet after connecting to an appropriate mailbox. It would be nice if Microsoft make it possible for mailbox audit events to flow from satellite regions to the unified audit log. Unified, after all, is the important word.

Learn about using Exchange Online and the rest of Office 365 by subscribing to the Office 365 for IT Pros eBook. Use our experience to understand what’s important and how best to protect your tenant.

If You Have Office 365 E3 Licenses, Check the Flow of Mailbox Audit Events

A reader contacted me to report some problems that a compliance exercise had thrown up when it was discovered that the Office 365 audit log did not contain mailbox audit records for some Exchange Online mailboxes. As it turned out, all the affected mailboxes belonged to Azure AD accounts with Office 365 E3 licenses. Not having some expected mailbox audit events available in the audit log is not a great situation for any compliance officer or tenant administrator to find themselves in.

When the customer organization contacted Microsoft support, they were told that a bug existed in Exchange Online that stops the AuditEnabled setting being accurately reported for mailboxes with Office 365 E3 licenses. When administrators examined the setting with the Get-Mailbox or Get-ExoMailbox cmdlets, Exchange Online reported $True, meaning that auditing is enabled for the mailbox:

Get-ExoMailbox -Identity Brian.Weakliam -Properties AuditEnabled | Format-Table DisplayName, AuditEnabled

DisplayName                 AuditEnabled
-----------                 ------------
Brian Weakliam (Operations)         True

However, the mailbox audit events remain in Exchange Online and never make it into the Office 365 audit log. Everything works perfectly for mailboxes assigned Office 365 E5 licenses.

What Microsoft Says

Microsoft’s documentation says:

Although mailbox audit logging on by default is enabled for all organizations, only users with E5 licenses will return mailbox audit log events in audit log searches in the Microsoft Purview compliance portal (Figure 1) or via the Office 365 Management Activity API by default.

The page also says:

Even when mailbox auditing on by default is turned on for your organization, you might notice that mailbox audit events for some users aren’t found in audit log searches by using the compliance center, the Search-UnifiedAuditLog cmdlet, or the Office 365 Management Activity API. The reason for this is that mailbox audit events will be returned only for users with E5 licenses when you [use] one of the previous methods to search the unified audit log.

Apparently, Microsoft doesn’t consider it a bug when audit data from mailboxes with Office 365 E3 licenses doesn’t reach the Office 365 audit log. The justification is that:

• The audit events are available in Exchange Online (in the Audits folder in the Recoverable Items part of each mailbox) and can be found there.
• If the organization wishes to ingest the audit events into the Office 365 audit log, all they need to do is run the Set-Mailbox cmdlet to update the AuditEnabled setting to $True. In other words, the $True value reported by Exchange Online is accurate because it reports that mailbox events are available. Running Set-Mailbox to update the setting to $True (again) doesn’t change anything (the setting is still $True) but Exchange Online takes the hint and will ingest mailbox events into the Office 365 audit log thereafter.

Clearly, this is a non-optimal situation. I wrote about the same issue in March 2020 when Microsoft rolled out mailbox auditing by default across Exchange Online. It seems like the same documentation is online and people are still running into problems. And that this Exchange Online functionality hasn’t progressed much since.

Audit Events Appear to Flow for New Mailboxes

Or has it? I created a couple of new Azure AD accounts and assigned them Office 365 E3 licenses and then signed into each mailbox to perform several actions that I knew should generate audit records. After waiting 30 minutes to allow ingestion to occur, I ran the Search-UnifiedAuditLog cmdlet to see if any events existed in the Office 365 audit log. They were!

[array]$records = Search-UnifiedAuditLog -StartDate 18-Aug-2022 -EndDate 19-Aug-2022 -Formatted -userids michael.king@office365itpros.com
$records | ft creationdate, operations
 
CreationDate        Operations
------------        ----------
18/08/2022 12:54:31 SoftDelete
18/08/2022 12:54:23 MoveToDeletedItems
18/08/2022 12:54:19 SoftDelete
18/08/2022 12:53:19 SoftDelete
18/08/2022 12:53:14 MoveToDeletedItems

This result is exactly what you’d hope to see. No administrator intervention was necessary to have events flow from Exchange Online to the Office 365 audit log. I obtained the same results with mailboxes in another tenant. I asked Microsoft about the issue and learned that mailbox auditing was enabled by default because of the configuration of the Exchange Online forest that hosts the mailboxes. For performance reasons (to reduce the number of audit events processed by Office 365), this is not the situation for every Exchange Online forest. This is why Microsoft’s documentation recommends that you should re-enable auditing for the E3 mailboxes to force ingestion of audit events for these mailboxes into the Office 365 audit log.

The conclusion is there might be many mailboxes with Office 365 E3 licenses running in tenants around the world that don’t feed data into the Office 365 audit log. The only solution is to run Set-Mailbox. And you might as well run the cmdlet for all mailboxes with Office 365 E3 licenses just to be sure.

The script included in my previous post will do the job of finding mailboxes with Office 365 E3 licenses and updating their audit settings. The script uses the Get-AzureADUser cmdlet to find the relevant accounts. If you’ve switched to the Microsoft Graph PowerShell SDK, the updated code is:

[array]$Mbx = Get-MgUser -filter "assignedLicenses/any(s:s/skuId eq $Office365E3)" -All

Good for the Future?

If you use Office 365 E3 licenses, be sure to check that mailbox auditing works as you expect. It’s odd that Microsoft forces customers to go through the additional step to enable auditing for mailboxes that appear to be already enabled for auditing, but that’s the way the system works.

Make sure that you’re not surprised about changes that appear inside Office 365 applications by subscribing to the Office 365 for IT Pros eBook. Our monthly updates make sure that our subscribers stay informed.