In recent years, the rise of Large Language Models (LLMs) with advanced general capabilities has accelerated progress toward building language-guided agents capable of performing complex, multi-step tasks, much like human assistants. Developing agents that can perceive, plan, and act autonomously has long been a central goal of artificial intelligence research.

In this talk I will introduce Multimodal AI agents capable of planning, reasoning, and executing actions on the web, that can not only comprehend textual information but also effectively navigate and interact with visual settings. I will present VisualWebArena, a novel framework for evaluating multimodal autonomous language agents, along with an inference-time search algorithm that enables explicit exploration and multi-step planning in interactive web environments. Next, I will demonstrate how an automated data pipeline can facilitate Internet-scale web-agent training by generating web navigation tasks across 150,000 live websites, deploying LLM agents, and assessing their performance. Finally, I will discuss some insights for developing more capable autonomous agents in both digital and physical environments.