Microsoft’s new AI-powered Copilot instantly summed up my meeting yesterday (the meeting was with Microsoft to discuss Copilot, of course) before listing the questions I had asked seconds before. I’ve seen Microsoft demo the future of work for years with virtual assistant concepts, but Copilot is the closest I’ve ever seen them come to fruition.
“In our minds, this is the new way of computing, the new way of working with technology, and the most adaptive technology we’ve seen,” said Jon Friedman, corporate vice president of design and research at Microsoft, in an interview with The edge.
I spoke with Friedman on a Teams call when he activated Copilot midway through our meeting to work his AI-powered magic. Microsoft has a flashy marketing video showing Copilot’s potential, but watching Friedman demonstrate this in real time across Office apps and in Teams convinced me that it will forever change how we interact with software, create documents, and in ultimately how we work.
Copilot appears in Office apps as a helpful AI chatbot on the sidebar, but it’s much more than just that. You can be in the middle of a Word document and it will gently pop up when you highlight an entire paragraph — just like Word has UI prompts that highlight your typos. You can use it to rewrite your paragraphs with 10 new text suggestions that you can flip through and freely edit, or you can have Copilot generate entire documents for you.
Copilot can even teach you Office functions
This customizability is what sets it apart from Microsoft, which just shoves ChatGPT into a sidebar in Office. Copilot doesn’t just offer a chatbot interface – you can use it to control Office apps like Excel and PowerPoint. If you’re looking at a slideshow and wish every title were orange instead of blue, just ask Copilot instead of digging into PowerPoint features.
In Excel, you can have Copilot generate a pivot table, create a graph, or just help you understand the rows and columns of data in front of you. “One of the ways we’re starting with Copilot is to help analyze and understand data,” Friedman says. “You can ask Copilot what it’s doing with the data, you can get graphs from Copilot based on trends it sees in the data, and you can put those trends into a spreadsheet.” Excel even has a “show me” feature for Copilot that will let this AI teach you how it just completed a command so you can improve your Office knowledge.
It feels like Microsoft is slowly building on the vision it had for its Cortana assistant or even Clippy decades before. “I love that our heritage is full of products that try to adapt to people,” says Friedman. “Copilot shares some similarities with some things we’ve done in the past, but it’s far more capable, it’s humble, and it’s there to serve you things that help you save time.”
Microsoft has customized this Copilot system for each Office app, so there are different ways to command it. Friedman demonstrated to me how Copilot can help you write emails in Outlook by offering short or long message drafts with options to change the tone. It even works in the mobile version of Outlook, which got me thinking about how this could speed up work on the go.
“Outlook mobile is the first place we’re making a big push,” explains Friedman. Outlook can summarize all your emails on the fly, generate drafts and generally make it easier to triage your inbox. But imagine creating entire Word documents from your phone without having to type on a tiny on-screen keyboard. “We’ll have more to talk about mobile in the coming months,” Friedman says. But you can imagine where things will go.
As impressive as Copilot is, we’ve seen the countless ways large language models can fail, including inserting racial or gender bias into text and simply making things up. These characteristics are alarming enough in a search engine, but when you’re talking about Excel (which arguably drives the world economy) or your email inbox, it’s a whole other level of ethics, privacy and data issues.
“It gets things right a lot of the time, but not all the time,” admits Friedman. “In the user experience, we do things like put in affordances to not send something until you’ve read it, or to encourage you to try again, edit and discard.”
Microsoft also has a number of warnings inside Copilot that appear while you’re using it. In PowerPoint, you’ll see a message that says, “Content is generated by AI and may contain inaccuracies for sensitive material. Be sure to verify the information.” Elsewhere, there are messages that say, “AI-generated content may be incorrect.” Microsoft tries to design the system in a way that reminds you of that you are Is responsible for.
“We give you tools to report it when it’s wrong. We make prompt suggestions to help you write great prompts. Everything we do in the user experience is to make it conversational and empower you,” says Friedman.
We’ve seen what happens when things go wrong in Microsoft’s Bing search engine. The AI-powered chatbot has hallucinated on several occasions, and Microsoft has had to put limits in place to control its outbursts. In one conversation with the edge, Bing even claimed that it spied on Microsoft employees through the webcams on their laptops and manipulated them.
“Everything we learn from Bing in preview helps us mitigate those risks,” Friedman says. “We also apply all that learning and thinking to Copilot.” Microsoft is also starting small with its Copilot rollout. It will initially be available to just 20 companies before Microsoft opens it up to more when it’s ready. Microsoft is also starting with enterprise customers first before rolling out to consumers.
“We feel pretty good about what we have as a starting point, but we don’t know yet if it performs the way we want it to and really helps empower people to do their jobs,” Friedman says. “We’re going to iterate quickly, we’ve been building fast, very fast. But we’re pausing and we’re learning a lot and we’re updating really fast. Our plan is to move as fast as we possibly can to scale to more companies in a thoughtful and responsible way and ensure the experience is great.”
However, is Microsoft moving too fast? Google announced its own AI features for Gmail and Docs earlier this week, and the AI race has many experts worried that tech giants aren’t properly considering the impact of these new tools.
“In our minds, we are mind-bogglingly fast.”
“In our minds, we’re mind-bogglingly fast,” Friedman says. “We’re thinking of rolling it out to 20 customers and working side by side with them.” Despite reports of Microsoft firing its ethics and society team that taught employees how to make AI tools responsibly, Friedman says Microsoft is growing the people working on those concerns. “In terms of our investment in ethics and AI, we’ve got more and more ethics and AI experts in all the product teams working on this,” Friedman says. “We have to scale much bigger, so we’ve invested more heavily, and it continues to grow year on year.”
Microsoft knows that Copilot is not perfect and that it will take some time to get there. While it impressed me during the Teams meeting, it could have easily confused my voice with someone else’s if I had used a bad microphone or Outlook could pull the wrong summary in an email thread. There are big challenges ahead, but Microsoft hopes that work to make it easy to edit answers, correct sources, and issue feedback will ultimately improve the system.
“We know AI gets things wrong, we know it hallucinates, and we know it does it confidently,” Friedman admits. “We continue to work on making it better to make it smaller, but also that the user experience really empowers people and puts them in the driver’s seat.”
Despite all the challenges, the future of the Copilot system will not be just text-based generation either. Microsoft has a clear vision of using Copilot to generate images, video and more when large language models can handle these functions well.
Microsoft has already integrated OpenAI’s DALL-E model into its Designer app, which allows people to generate images based on text. Designer will also help PowerPoint choose the best images for AI-generated slides. “We’re going to further bake Designer into Copilot, so you can change things in Designer,” Friedman says. “The designer stuff you’ve seen today is just scratching the surface. I fully suspect we’ll be using Copilot to make great multimedia stuff.”
So where else could we see Copilot pop up? I asked Friedman about Windows integration. “We’re looking at all sorts of places and ways to expand (Copilot). I believe this is the next big wave of computing, and it’s going to change the way we work with all devices for years to come,” says Friedman.
Microsoft also has a multiplayer Copilot experience
The future will also include a multiplayer experience for Copilot. Loop components, one of the biggest changes to Office documents in decades, are available in Teams and Outlook. Loop components, the branding for Microsoft’s Fluid work, are blocks of collaborative text or content that can live independently and be freely copied, pasted and shared.
Now imagine copying a Loop component of text into an email and having multiple people edit it and interact with the Copilot. “In the component, as we edit, the conversation is a clickable story of the content being generated that you can go back and forth through,” says Friedman. “What’s so cool about it is that it feels like a whole new mental model of how to work with a Copilot with a group of people.”
All of these Copilot features for Office and Microsoft 365 feel like they will forever change the way we work and communicate, especially as these major language models evolve in the coming years. Microsoft’s push to embed this AI deeply into its products could have a lasting effect on the labor market.
“Every time there’s a new technological advance, there are both opportunities and things we have to consider,” Friedman says. “We believe that the infusion of this AI will create new job opportunities in the long term and increase job satisfaction in the short term. We expect that it will change the nature of a lot of jobs and create new jobs that did not exist before. Therefore, it’s so important to us to empower people and build this shared design system.”