Copilot Studio adds 'Computer Use' tool to let AI agents operate websites and desktop apps

Microsoft has introduced a new research preview tool for Copilot Studio called "Computer Use," aimed at improving AI agents' interactions with websites and desktop applications. This tool enables Copilot agents created in Studio to autonomously perform tasks such as clicking buttons, selecting menus, and typing into fields, even in environments lacking APIs—essentially allowing Copilot to use your computer like a human would.

This broadens automation possibilities across various software, enhancing Robotic Process Automation (RPA) by offering real-time responsiveness to UI changes and intelligent decision-making. Users can issue commands in natural language without coding and test these commands with video previews of the agent's reasoning and actions, including screenshots and decision chains. Operating on Microsoft-hosted infrastructure, "Computer Use" ensures customer data remains within Microsoft Cloud boundaries and is not used for training language models.

This development follows the introduction of a similar feature, "Actions," in Microsoft's consumer Copilot, which allows background task automation via Microsoft Edge. The technology may share similarities with OpenAI's "Operator," which uses GPT-4o's vision capabilities and reinforcement learning for computer interaction.