Copilot Audit Records Now Include Resources Used in Responses

In April 2024, I wrote about the appearance of audit events to capture details when Microsoft 365 applications call Copilot to process a user request (prompt). These events have an operation type of CopilotInteraction.

Since then, Microsoft announced progress in capturing records when people use Copilot in the Stream player to query video transcripts (MC720180, last updated 22 May 2024). It’s like MC720180 (also updated on 22 May 2024), which describes using Copilot to interact with meetings. In both cases, the important point is that the audit events generated for Copilot interactions capture details of resources accessed by Copilot when responding to user prompts (previously the AccessedResources property in the AuditData payload was empty).

Linked to the Change in Transcript Storage Location

Because Copilot depends on meeting transcripts to answer queries, meeting interactions are only possible when meetings are recorded with a transcript. As discussed last week, Teams is standardizing on OneDrive for Business storage for the MP4 files generated for meeting recordings and transcripts. Like many situations in Microsoft 365, developments reported in one message center notification are linked to what’s described in another, seemingly unconnected, update.

The change should be effective in most places now as Microsoft aims to complete worldwide deployment in early June 2024.

Updated Script to Handle Copilot Audit Records

To test the effectiveness of the change, I updated the script I wrote for the previous article (downloadable from GitHub) to support audit records generated by the Stream player and to pay more attention to the data recorded in the associated resources property. Figure 1 shows the output of the script as viewed through the Out-GridView cmdlet.

Please check out the updated script and let me know if it’s helpful or could be improved.

Copilot in Outlook Classic

Speaking of Copilot, for a long time Microsoft communicated the message that Copilot experiences would only be available in the new Outlook client (aka Monarch). This was no more than a thinly-disguised ploy to drive adoption for Monarch, which still isn’t close to ready for consumption by corporate users.

In any case, message center notification MC794816 (21 May 2025, Microsoft 365 roadmap item 388753) reports the availability of the Copilot for Microsoft 365 chat experience for Outlook classic (Win32). This feature joins “Summarize,” the Copilot option that extracts the major points from an email thread (my second favorite Copilot feature after meeting summarization), and the option to have Copilot draft or revise message drafts. Microsoft will roll out Copilot for Microsoft 365 chat to Outlook classic in the current channel in June 2024.

Before anyone gets too excited, let me say that Copilot for Microsoft 365 chat in Outlook is the same application as accessed as a web application and in Teams. The only difference is that Copilot has an icon in the Outlook application bar and runs in the Outlook window (Figure 2). In other words, if you’re used to Copilot chat elsewhere, you’ll find no difficulty using it in Outlook, providing you have the necessary Copilot for Microsoft 365 license.

As you can see from Figure 2, chats generated in other instances of the client are available in Outlook.

Change, Change, and More Change

Change is ongoing within Microsoft 365. Some changes are dependent on other changes, such as Copilot audit records capturing associated resources for the Stream player. Others are the delivery of incremental functionality within an application. The trick is to keep an eye on what’s happening and to recognize what kind of change each message center notification represents. That’s sometimes hard to do based on the way Microsoft describes a change. Oh well, into every life a little rain must fall…

So much change, all the time. It’s a challenge to stay abreast of all the updates Microsoft makes across the Microsoft 365 ecosystem. Subscribe to the Office 365 for IT Pros eBook to receive monthly insights into what happens, why it happens, and what new features and capabilities mean for your tenant.

Using Audit Events to Determine Level of Copilot Activity

Updated 9-Apr-2024

In an article reporting how Copilot usage had dropped off after the initial novelty of having a digital assistant, MVP Joe Stocker described how he used ChatGPT to analyze information about Copilot interactions using Copilot for Microsoft 365 audit events retrieved from the unified audit log. It’s an interesting technique that caused me to investigate the information captured for Copilot interactions.

Running an audit search to find Copilot interactions returns the audit events captured when Microsoft 365 apps use Copilot. Outlook is an exception. Neither OWA nor Outlook for Windows capture Copilot audit events when drafting or refining message text or summarizing a message thread. I wasn’t too surprised at this because Microsoft had said that the Monarch client would be the only supported email client for Copilot until recently.

Interestingly, despite Monarch or any other Outlook client failing to log audit events for Copilot interactions, usage data shows up for Copilot for Outlook in the usage reports section of the Microsoft 365 admin center. The usage data does not describe which Outlook client is used. I assume that the lack of audit events for Outlook interactions is an oversight that Microsoft will address soon.

Update: According to message center notification MC773621 (9 Apr 2024), Microsoft plans to ship support for eDiscovery of Copilot interactions in Outlook (all clients) in mid-April. This implies that audit events will become available.

What’s in the Copilot for Microsoft 365 Audit Events

The unified audit log is a terrific resource for Microsoft 365 administrators to learn about what really happens inside a tenant. Audit events have two parts. The first is fixed and is consistent across all workloads. This part contains properties like the record identifier, user identifier, and timestamp. The second part contains the AuditData multi-value property and is under the control of the workload developers. Audit events generated by SharePoint Online are different to those generated by Exchange Online or Teams. The inconsistency in audit content means that it takes time to understand and interpret the information contained in audit events.

Microsoft defines a schema for Copilot audit events. Regretfully, the formatting of the information written into the AuditData property is inconsistent across the different clients. Some differences are expected because of the variation in content processed by Copilot, but the inconsistencies make it more difficult to understand and analyze what happened than it should be.

Adjusting Audit Search Results for Accuracy

When reviewing audit events generated for Copilot interactions, the fundamental principle is that every prompt processed by Copilot results in an audit event. For instance, if a user asks Copilot for Microsoft Chat a question and follows up with a couple of clarifications, the audit log ingests three audit events. The overall total of Copilot interaction audit events is therefore not the total of discrete user conversations with Copilot. Rather, it is the count of Copilot prompts.

It’s also important to remove duplicates of audit events from the set returned by an audit search. To improve the accuracy of audit searches, Microsoft is introducing a high completeness mode for the Search-UnifiedAuditLog cmdlet. If you don’t use a high completeness search (which takes much longer to complete), make sure to remove duplicates before analyzing the search results. I usually sort the audit search results by unique values in the identity property to remove duplicates. The Purview audit search feature does the same when it downloads audit search results.

After some trial and error, I created a PowerShell script to extract Copilot audit events from the unified audit log and parse the information contained in the AuditData property. An example of the output is shown in Figure 1.

You can download the script I used to find and analyze Copilot for Microsoft 365 audit events from GitHub.

Using Copilot for Excel to Analyze Audit Events

Given that we’re exploring Copilot for Microsoft 365, it made sense to try to use Copilot for Excel to interpret the results. Copilot for Excel can only work with tables and the workbooks must be enabled for AutoSave. In other words, the workbooks must be stored in OneDrive for Business or SharePoint Online. I used the ImportExcel PowerShell module to generate a worksheet with a table of audit events with the following command:

$Report | Export-Excel -Path c:\Temp\Copilot.xlsx -WorksheetName "Copilot Interactions" -Title "Copilot for Microsoft 365 Interactions" -TitleBold -TableName "CopilotEvents"

Figure 2 shows some of the analysis generated by Copilot for Excel. It’s important to say that Copilot for Excel is still in preview, which accounts for the difficulties reported by many users when prompting Copilot for Excel to do something simple, like create a chart.

Some Gaps in Copilot Audit Coverage

Microsoft touts the data security and compliance coverage for Copilot in Purview solutions like auditing. Not capturing audit events for Outlook clients seems like a big gap that must be filled ASAP. Improving the consistency of the information included in the audit events to make it easier to understand and use the data. But I have been waiting for various Microsoft development groups to improve their audit events since 2016 and I suspect that I’ll be waiting before I see better Copilot events.

Learn how to exploit the data available to Microsoft 365 tenant administrators through the Office 365 for IT Pros eBook. We love figuring out how things work.