Microsoft Copilot can now ‘see what you see’ on the internet

You might want to start treating your web browser like you’re always at work, at least if you want to use Microsoft’s new Copilot Vision feature. The feature, which is natively built into Microsoft Edge, is able to “see what you see, and hear what you hear” as you navigate your browser, according to Microsoft’s Executive Vice President Yusuf Mehdi.

All of this AI snooping isn’t for nothing. Copilot Vision looks at what you’re doing online to answer questions, provide recommendations, and summarize content. It can work with the new Copilot Voice feature, for example. Microsoft demoed the capabilities on Rotten Tomatoes, showing a user chatting with Copilot while browsing the website and looking for movie recommendations. Ultimately, Copilot settled on an Australian comedy for the Australian speaker, saying it made the choice because, “well, you’re Australian.” I guess that’s taking personal context into account.

Understandably, the idea of an AI looking at everything you do online might rub some people the wrong way. Microsoft says the feature is opt-in only, meaning you have to manually turn it on, and that there will be an indicator onscreen showing that Copilot is watching what you’re doing. Microsoft also says that data from your session is deleted after you close the session, and that the data won’t be used for training.

In addition, Microsoft says that Copilot Vision will only initially work with a limited number of websites. We don’t have the full list yet, but at the very least, the feature won’t work on “paywalled or sensitive content.” Microsoft also points out that the feature doesn’t process anything on a webpage — it just sees the text and images on a page and analyzes them.

Copilot Voice works in conjunction with Copilot Vision, but it also works on its own. Across desktop and mobile devices, Microsoft says you’ll now be able to hold a conversation with Copilot using just your voice.

Building on the new voice capabilities, Microsoft is releasing Copilot Daily. It’s almost like a personalized morning news show, rounding up stories and giving you daily details like traffic and weather. This, too, is built with a generative AI voice that walks you through your morning news every day. It’s generated based around the content you interact with.

Copilot Voice and Copilot Daily are rolling out to Windows users now in the U.S. and UK. Voice is also available in Australia, Canada, and New Zealand, but only in English. Microsoft says it will bring the feature to more countries and languages soon.

Copilot Vision is starting to roll out today in the U.S., but only for Copilot Pro subscribers. The feature is part of Microsoft’s new Copilot Labs program, which gives Copilot Pro subscribers early access to Microsoft’s latest AI features that are still being tested.