Why the LLM in my automation agent never clicks anything

Description and use

<p>I'm a computer engineering student in Manaus, Brazil, and for my research project I've been building <strong>ANCHOR</strong>, a web automation agent: you describe a task in plain language ("Register Maria Silva, IT department, contractor, accept the terms and save"), and it runs the task in the browser. Everything runs locally, with small models (Qwen 3.5, 4B and 9B, through Ollama) on a GTX 1050 Ti with 4 GB.</p> <p>This post is about the one design decision that made the biggest difference, and a few things I learned by measuring everything.</p> <h2> The problem </h2> <p>Classic RPA depen

Problem

<p>I'm a computer engineering student in Manaus, Brazil, and for my research project I've been building <strong>ANCHOR</strong>, a web automation agent: you describe a task in plain language ("Register Maria Silva, IT department, contractor, accept the terms and save"), and it runs the task in the browser. Everything runs locally, with small models (Qwen 3.5, 4B and 9B, through Ollama) on a GTX 1050 Ti with 4 GB.</p> <p>This post is about the one design decision that made the biggest difference, and a few things I learned by measuring everything.</p> <h2> The problem </h2> <p>Classic RPA depen

Outcome

<p>I'm a computer engineering student in Manaus, Brazil, and for my research project I've been building <strong>ANCHOR</strong>, a web automation agent: you describe a task in plain language ("Register Maria Silva, IT department, contractor, accept the terms and save"), and it runs the task in the browser. Everything runs locally, with small models (Qwen 3.5, 4B and 9B, through Ollama) on a GTX 1050 Ti with 4 GB.</p> <p>This post is about the one design decision that made the biggest difference, and a few things I learned by measuring everything.</p> <h2> The problem </h2> <p>Classic RPA depen

Evidence

Why the LLM in my automation agent never clicks anything

Source