Armature, a studio specializing in growth services for programming tools, has published a large-scale study decoding how coding agents select tools. They tracked 16893 working sessions of the three most popular agents today: Claude Code, Codex, and Cursor.
The experiment was deployed across 75 self-built repositories spanning 10 programming languages, accompanied by 1163 prompt variants representing 4 user groups (ranging from beginners to enterprise compliance and procurement engineers). After the filtering process, 5292 qualifying working sessions were analyzed, spread across 51 codebases and 18 different domains.
Research Methodology
The repositories were set up with simulated company names and git histories while keeping actual lockfiles intact. The sandbox environment rotated continuously among 3 providers, integrating an additional “simulated human” element in the loop to increase naturalness for the scenarios. Notably, both the orchestration role and the referee evaluating the sessions were handled by Gemini 3.7 Flash.
Some key findings from the report:
- Disagreement among agents: Only in 42% of cases did all three tools make the same service choice for a given need. A prime example is in the voice API group: Claude Code prioritized Twilio, Codex chose OpenAI Realtime API, while Cursor named Vapi. This difference stems from data collection strategies: Codex performed web searches in 94% of sessions (with 90% of queries using site operators to limit trusted sources). Claude Code relied mostly on pre-trained knowledge (searching the web only about 30% of the duration, except for new areas like sandboxes where this rate jumped to 80%), whereas Cursor looked up the web in about 2 out of 3 total sessions.
- Repository context overwhelms brand: With the same email-sending task, 4 repos across 4 different languages led to 4 separate results: Resend (TypeScript), SendGrid (Python), Postmark (Go), and Azure ACS (Java). Vercel dominated completely on the NextJS repo but was absent on the Python repo, where Render prevailed.
- Being mentioned does not mean being chosen: In the payment sector, PayPal appeared 139 times but received 0 choices, losing out to Stripe which won 124 sessions. LangChain was mentioned the most in the framework group (194 times) but was only installed 4 times. Netlify was mentioned 152 times but only chosen 6 times. In the database sector, Supabase led in mention frequency (242 times) but Neon was the one that captured the majority of results. Sometimes decisions were influenced by very small details, such as Mailgun being outperformed by Postmark simply because its pricing page stated that the free tier only stored data for 1 day.
The report opens up an entirely new distribution channel. Vercel once recorded over 30% of deployments on their system coming from coding agents, a 1000% growth in just half a year. When agents proactively read documentation, analyze pricing tables, and decide on integrations, website content no longer serves purely as marketing for end users, but becomes data for AI to filter. The concept of “Agentic SEO”—optimizing so that products are preferentially chosen by AI agents—is becoming an inevitable trend.
Practical Lessons
-
For agent users: You should ask critical questions about the rationale behind recommendations and demand alternative choices for comparison, especially for long-term and hard-to-change technology solutions.
-
For product developers: Technical documentation systems and pricing policies serve as a sales and marketing team targeting an entirely new customer group—working continuously without a break, unaffected by brand reputation or partners, and examining every single detail with extreme care.
The most notable core point distilled from the 5292 valid trial sessions is: Humans almost no longer retain the role of making the final decision in most scenarios.