Universal Agentic AI with MATLAB

Support package for a Universal macOS AI desktop agent

A desktop AI app can be superpowered to serve as a agentic AI assistant for your interactions with all local applications and with remote websites and data sources as described in A universal agentic AI for your laptop and beyond.

This support package enables AI assistants like Claude Desktop to automate macOS applications via the MATLAB MCP server and other MCP services. Using MATLAB as a trusted bridge for macOS accessibility permissions, the system provides capabilities including but not limited to: (1) AI-to-AI delegation where Claude sends natural language commands to for example Perplexity AI running in Comet browser, has Perplexity execute multi-step browser tasks, captures window-specific screenshots for verification, and iterates without manual intervention; (2) messaging automation to send text messages to any phone number via AppleScript and images to pre-configured recipients via the Shortcuts app; and (3) general application control through keystrokes, mouse clicks, and screenshots.

MATLAB, once granted Accessibility permission in System Settings, can execute AppleScript commands that are blocked when called from other processes, and execute helper codes making it an ideal intermediary for AI-driven desktop automation. The package also documents discoveries and workarounds. For example, while AppleScript can reliably send text messages, its `send POSIX file` command fails for image attachments (images appear in conversation but show "Not Delivered"). The workaround uses the macOS Shortcuts app with pre-configured recipients.

The package solves several non-trivial macOS automation challenges: using `cliclick` for reliable HID-level mouse clicks when System Events fails, capturing specific windows by CGWindowID (via a Swift helper script) to ignore overlapping windows, and leveraging the built-in `sips` command for fast (~90ms) image compression. Image compression is essential for agentic feedback to the host AI in the cloud.

Complete documentation includes step-by-step installation for both Intel and Apple Silicon Macs, detailed instructions for AI assistants, helper functions for messaging (sendTextMessage.m, sendImageMessage.m), and a comprehensive record of problems solved. Requirements include Claude Desktop with MCP server support, MATLAB, Homebrew, cliclick, and Xcode Command Line Tools.

This system was developed and documented with Claude AI assistance. The documentation can guide you and your AI in recreating and then expanding universal agent capabilities. Several included support functions make be used by your agent to effect operations. The package may serve as a model for constructing such an agent for other operating systems.

In 2025, Anthropic introduced "skills," a standard markdown format to share AI process information developed interactively, that is being adopted in various forms throughout the industry. This package includes a macOS-automation-installation guide and skill package.