Anthropic has introduced a new capability called “computer use” for its Claude 3.5 Sonnet language model, allowing it to operate computers like a human. This feature, currently in beta, enables Claude to perform tasks such as typing, clicking, and navigating software, paving the way for advancements in robotic process automation (RPA).
By leveraging the new capability, Claude can interact with software environments seamlessly, starting from a goal and working through tasks by analyzing screenshots. This innovation could potentially disrupt the RPA market by reducing the need for hard-coded scripts, adapting to various user interfaces, and improving through feedback.
However, limitations remain, including struggles with high-resolution screens and risks of prompt-injection attacks. Anthropic recommends strict security measures and human supervision to mitigate these challenges. Despite the hurdles, the computer use ability marks a significant shift toward AI-driven automation.