Copilot has entered the workplace. What's changed?

Microsoft has begun rolling out Copilot as part of Microsoft 365. Instead of using ChatGPT on the side, now workers will have chat embedded within the Microsoft applications that they use on a daily basis. For those that were on the sidelines because their organization didn’t allow AI use for data security reasons, enterprise data controls will likely result in holdout employers strongly encouraging the use of copilot. At $30 USD per month per license companies that do make the leap will let their workers know about it. A Microsoft 365 Copilot license (as opposed to Microsoft 365 Copilot Chat which is currently a free ad on) costs roughly the same as an E3 license, meaning an E3 customer would almost double their cost per user. They don’t want those licenses to go to waste^[1].

While this may look like a huge transformation on the surface, is it really? What changes when employees are suddenly given the ability to use unlimited chat prompts at work, and to feed excel documents and word documents into a box for a bot to do their bidding. A recent pre-print publication in arxiv titled “Early Impacts of M365 Copilot” by Microsoft researchers gives us some clues.

First, it is important to note that with a study conducted by researchers embedded within a company, while the data is likely to be trustworthy and accurate, the conclusions and forward looking statements usually have bias. That is certainly present here.

Second, this study relied on companies selecting a group of workers who would have a 50% chance of being assigned a copilot license, or have to forego a license for the duration of the study. As the authors note, this means that companies will not have selected their workers in a completely random way (if they thought a senior executive would save 50% of their time per week with copilot, there is no way they would risk not giving them a license).

The study revealed the following conclusions, which are, on the whole, underwhelming for how much money is being spent on the pivot to AI. I would argue that productivity gains similar in magnitude to what the workers in this study displayed could be observed in a regular interface update, or even on just changing policies and procedures for how a company operates. With that said, let’s get into the details.

Copilot use gradually decreased during the study period

Shortly after being granted a copilot license, workers quickly began to use the tool. This could have been for experimentation, or because they had been eagerly awaiting the point at which they could use AI in the workplace. However, as the study continues, the percentage of workers using Copilot falls quite drastically for all products except Microsoft Teams. By the end, less than 20% of workers were using Copilot for Word, Excel, Powerpoint, and Outlook, and only 38% of those given the license used the service at all.

Workers spent slightly less time on email

The headline result is that copilot users spent 18% less time reading email per week, equating to about half an hour of time savings. As this is the first instance in which we get a direct result for time savings, we can convert it into a rough estimate of dollars saved.

$17.75/hr * 0.5hr = $8.875 saved per week

Hourly wage taken from indeed.com average salary for “office worker” in the United States.

Those savings alone would justify a $30 per month subscription, however, the business doesn’t realize those savings unless they pay the employee less, or the employee was already at 100% utilization and they can now allocate those hours to something else. It also assumes that there was no information lost in the worker reading the summary (one of the hypotheses the authors introduce to explain the time savings). If workers are saving half an hour per week by reading summaries of emails, the company could also just work on improving succinctness of email communication.

The authors hypothesize that some of this time saved on email is attributable to getting the answer in a copilot chat, in which case, this is just measuring that the users spent time in a different app, not that they saved any time on net.

“We hypothesize that some combination of Outlook Summarize and searching for email content in M365 Copilot Chat allows users to spend less time with individual emails.”

Word documents are easier to produce

On average, copilot users initiated (created) 11.2% more documents. This effect is likely to extend beyond just word documents, and extend into things such as GitHub issues, Jira tickets etc. My hypothesis is that workers are going to produce more content as they no longer have to physically type things out, and thus there isn’t as much of a cost to verbosity. Whereas previously you may not have bothered to create something marginally useful, now it is effortless, and most managers will take more document creation as a positive indicator of productivity (even if it really isn’t).

No generalizable effect on meetings

Copilot usage was highest in the Microsoft Teams app, presumably for summarizing meetings into transcriptions or notes. However, the data for the effect that this has on the time that workers spend in meetings is murky at best. Some firms increased their time in meetings, while others decreased, there wasn’t a broad effect across the board.

Again, this is similar to the word document effect. Companies may feel more productive with more meetings that are transcribed (as they can show these transcripts to the higher ups), however this doesn’t always translate to actual efficiency gains. There is a reason why workers are always complaining about meetings, and it isn’t because they detest how productive they are.

Is it worth the price?

The current cost of $30 USD per month per person is not what it actually takes to turn a profit, as Edward Zitron recently explained in detail. This is especially true if enterprise users don’t face any quotas on things like chat. It is simply too much compute to offer at such a low price tag. The price will increase in the future, and copilot chat won’t always be a free add on, it is only being offered for free currently to try to shift the market.

These results suggest that even at $30 USD per month, it is barely worth it. Sure, there are perhaps some intangible benefits long-term to having meetings better documented, and more streamlined knowledge transfer between employees by increased documentation, but even those additional documents and transcripts are going to run up the costs for businesses in terms of cloud storage. Eventually the focus has to shift to a critical question; “do these tools really help better deliver on the objective of our business?”. If it doesn’t, then it’s just helping you spin in circles faster.