Microsoft details how apps can integrate with Windows 11's new AI Recall feature

What you need to know

• Microsoft has detailed its new Windows Copilot Runtime and Semantic Index technology at Build 2024.

• These technologies are what power many of the new AI experiences that were announced for Windows 11 yesterday, including Recall.

• Windows Recall is the highlight AI feature, which uses on-device AI models to function.

• App developers can tap into Windows Recall to allow users to jump back into their apps based on what Recall has stored.

Microsoft has today expanded on how the newly announced Windows Recall feature is going to work on Copilot+ PCs, leveraging the Windows Semantic Index and Windows Copilot Runtime. It has also detailed how app developers can enhance their apps to better work with Windows Recall, with support for creating a more seamless flow between snapshots captured by Recall and third-party app.

The new Windows Copilot Library has a set of APIs that are powered by 40+ on-device AI models and algorithms, which are now built directly into Windows. This library is what powers many of the new AI features in Windows, including Studio Effects, Live Captions, OCR in Snipping Tool, and now the new Windows Recall feature.

Microsoft says Recall should work automatically across many apps, but developers are able to enhance the experience by adding contextual information to the underlying vector database via the User Activity API. Doing this will enable the Recall app to be able to jump back into the same part of an app that Recall has picked up, which creates a more seamless experience.

Microsoft PowerPoint and Teams are the first Microsoft apps to support this flow, and third-party sketching app Concepts is also supporting this. Additionally, Microsoft has outlined how the Windows Recall feature actually works:

"Recall database is powered by Windows Semantic Index, a new OS capability that redefines search on Windows. Recall is grounded in several state-of-the-art AI models, including multi-modal SLMs, running concurrently and integrated into the OS itself. 

These models understand different kinds of content and work across several languages, to organize a vast sea of information from text to image to videos, in Windows. This data is transformed and stored in a vector store called Windows Semantic Index. The semantic index is stored entirely on the user’s local device and accessible through natural language search. This deep integration allows a uniquely robust approach to privacy as the data does not leave the local device."

Microsoft is hammering home the point that this experience is handled completely on-device. None of the data that Recall processes is sent to the cloud, and Microsoft does not train any AI models on data generated from Recall.

Windows Recall is available only on Copilot+ PCs, the first of which will begin shipping next month.