Products, papers, and patents

Product

How do you generate correct code in a proprietary language the model barely knows?

DataWeave appears in most Mule applications and has a steep learning curve, and it is low-resource enough that general models have little to go on. Rather than asking users to describe a transformation, the feature infers intent from sample input and output pairs, aligns nested schemas, and then checks itself the only way that really counts: compile the script, run it, compare what came out against what was expected. The design decision I care about most is the restraint — when a mapping is genuinely uncertain, it is left blank for review instead of guessed.

• Article
• Patent application pending

Product · Benchmark

Does a quality-first pipeline actually beat just calling the model?

I owned the benchmarking that answered this: the metric definitions, the LLM-as-a-judge design, and the comparison across nine widely used models. The distinction that made it useful was holding validity apart from correctness — whether the code is syntactically sound and uses operations that exist, versus whether it does what the user actually asked for. A flow can be perfectly valid and still miss the business logic entirely. Grounded generation with validation and error correction reached roughly 90% validity and 80% correctness, against roughly 20% and 17% for direct prompting. The comparison chart on MuleSoft's Vibes page is mine.

• Article
• MuleSoft Vibes
• Patent application pending

Research · Benchmark · ICSE 2026 · DL4C at NeurIPS 2025

Can an agent modernize a real codebase without quietly cheating?

A benchmark of 228 Java repositories, deliberately chosen for high test coverage so that semantic preservation is measurable at all, with a coverage guard that catches the failure mode nobody wants to talk about: an agent deleting or weakening tests to make a migration look successful. I worked on dataset cleaning and validation, agent evaluation, and comparing agentic frameworks against established rule-based tools. The best model migrated 52.3% of projects to JDK 17 — a useful number precisely because it is not a flattering one.

• Paper
• Code
• ICSE 2026

Research · Benchmark · ICLR 2026

Do computer-use agents survive real enterprise software?

300 tasks drawn from real user interviews across three personas — platform administrators, sales representatives, service agents — executed in live Salesforce sandboxes with milestone-level scoring so partial progress counts for something. The headline result is a warning about benchmark transfer: agents that perform well on general desktop benchmarks fall below 5% here. Closed-source models reach 39%, and 50% when given demonstrations, while cutting time and cost by 13% and 16%.

• Paper
• Project
• Salesforce blog
• ICLR 2026

Research · ICLR 2026

Should agents click through interfaces, or use the automations the site already has?

Websites already implement search, filtering, sorting, posting, and content management. WALT reverse-engineers that latent functionality into deterministic, callable tools, so an agent invokes search(query) instead of reasoning its way through every click. That moves the burden from fragile step-by-step reasoning to reliable tool invocation: 52.9% on VisualWebArena and 50.1% on WebArena with fewer steps, and on 139 real-world sites it discovered 252 tools on its own and improved success by 20.5% over a tool-free baseline.

• Paper
• Code
• ICLR 2026
• Patent application pending

Research · TMLR 2026, Featured certification · ICML 2026 · DATA-FM at ICLR 2026

Can you train a competitive model without indiscriminate web scraping?

An open pretraining corpus sourced permissive-first — public-domain and permissively licensed text, plus carefully justified low-risk additions like government works — with every source sorted into one of three risk tiers and shard-level provenance metadata attached, so downstream users can make their own call about legal exposure instead of inheriting ours. The recipe also folds a large share of permissive synthetic instruction and reasoning data into a single pretraining stage, rather than saving it for post-training. At 1.7B parameters and 300B tokens it surpasses FineWeb-Edu and matches a strong instruction-tuned baseline on GSM8K, HumanEval and MBPP using roughly 36 times fewer tokens, with a decontamination analysis to back the claim up.

• Paper
• Dataset
• Code
• ICML 2026

Research

What should a retrieval system actually retrieve?

Attention costs grow quadratically with context length, so the choice of what to put in the window is forced rather than optional. The standard move is to take whatever scores highest for similarity to the query, which quietly fills the window with near-duplicates and crowds out passages that were relevant but phrased differently. Borrowing from Maximal Marginal Relevance and Farthest Point Sampling, we select for diversity as well as relevance, which substantially improves recall of the sentences and chunks that actually matter before question answering and summarization.

• Paper