Skip to content

Yanqi Luo

Senior Machine Learning Scientist at Salesforce

AI should amplify humanity — not diminish it.

I've spent the past several years making generative AI and agents more reliable: building evaluation systems, benchmarks, validation pipelines, and recovery mechanisms that test whether AI actually works.

But the question that keeps pulling me back is more human.

At nineteen, I stood outside a boba shop counting who actually walked in because I trusted behavior more than survey answers. Since then, I've kept returning to the same belief: people are messy, diverse, contextual — and far more interesting than an average.

For thousands of years, we've tried to better understand ourselves and one another. AI turns that ancient challenge into a technical one: can we build intelligence that understands humanity without flattening it?

I'm interested in AI that adapts to the person in front of it — how they communicate, collaborate, decide, delegate, and exercise control.

Human diversity is not noise to be averaged away.

The future I want to help build is one where AI expands human agency, creativity, and possibility while preserving what makes each of us distinct.

Yanqi Luo

The theme I'm interested in

Reliable for whom?

Ask two people to describe a great assistant and you get two incompatible answers. One wants it to just handle things — book the flight, file the expense, don't ask. The other wants to be consulted before anything is spent. Same agent, same action: one person delighted, the other's trust permanently damaged.

Almost all the preference data being collected today measures what an agent produces: which output is better. That is outcome taste, and it converges — ask enough people which summary is cleaner and they largely agree. Procedural preference is a different axis: how the work should be done. When to ask versus act, how much to verify, what to do when something is ambiguous. Unlike outcome taste it diverges by person, which is exactly why it is what agents acting on your behalf need to know, and exactly why averaging destroys it.

If procedural preference is real and learnable, an agent should be able to observe how a particular person works and behave accordingly: asking when they want to be asked, acting when they want it handled, verifying about as much as they would have verified themselves.

The signal is already there and mostly thrown away. Every override, every edit, every takeover halfway through, every quiet decision to abandon the agent and do it by hand is a preference being expressed. Analytics counts it as a click and observability keeps it as a trace to debug. Almost nobody reads it as this is how this person wants the work done.

It is a kind of segmentation, but not the kind we know how to do. The old version was demographic and static. What is observable now is procedural: not enterprise customer in financial services, but verifies before committing, wants sources shown, never delegates payment authority.

Open questions

It would be easy to assert that everyone is unique and stop there. The interesting version is falsifiable, and I don't think anyone has answered it:

  • Are one person's procedural preferences actually stable across different tasks, or do we behave more randomly than we would like to believe?
  • Do they cluster into a modest number of learnable patterns, or is everyone their own irreducible case?
  • Does conditioning an agent on them measurably change how people rate its behavior, or is the effect too small to matter?

The full argument

August 2026

5 min read

Reliable For Whom?

Read the essay →

What the pilot found

August 2026

10 min read

I Was Wrong About What Makes People Want an AI to Ask

Read the essay →
All writing →

What I believe

  • 01

    Behavior is the signal

    What people actually do — correct, override, take control, redo, abandon — says more about how they want to be served than anything they would put in a settings panel.

  • 02

    Difference is not noise

    The variance between people is the most valuable thing in the data. Averaging it produces an agent that is acceptable to everyone and right for no one.

  • 03

    Preferences belong to you

    I'd like mine picked up from how I actually work, rather than a settings panel I have to fill out in advance — and portable, so a new tool knows how to work with me on day one instead of the knowledge staying locked inside whichever platform saw me first.

Featured work

Three acts, in the order I lived them

Papers, patents, articles and shipped features are usually the same project seen from different angles, so they sit together here. The acts move outward from the thing itself: build it so it is right, prove it stays right, then look at the data underneath that decided what it could do at all.

Act I

Make it correct

Correctness has one right answer, which makes it a satisfying engineering problem. Take a system that produces the right result under 20% of the time and get it above 90%.

Einstein for Anypoint Code Builder: Generative Flow

2023 – 2024

Product

Can a developer describe an integration in plain English and get something that actually runs?

MuleSoft's first generative AI capability, taken from foundational research through General Availability. I designed the domain-specific retrieval system covering more than 7,000 connector operations and built the curation pipeline behind it — filtering, deduplication, sensitive-data redaction, LLM-based labeling. The part I find most interesting is not the generation but the machinery that decides whether to trust it: IDE-integrated validation, an LLM-as-a-judge correctness evaluator, and multi-step error recovery that reads the validator's complaints and tries again.

DataWeave Generative Transformation

2024 – 2025

Product

How do you generate correct code in a proprietary language the model barely knows?

DataWeave appears in most Mule applications and has a steep learning curve, and it is low-resource enough that general models have little to go on. Rather than asking users to describe a transformation, the feature infers intent from sample input and output pairs, aligns nested schemas, and then checks itself the only way that really counts: compile the script, run it, compare what came out against what was expected. The design decision I care about most is the restraint — when a mapping is genuinely uncertain, it is left blank for review instead of guessed.

Act II

Prove it is reliable

A demo proves nothing about a stochastic system. Reliability is a claim, and claims need evidence: benchmarks in realistic environments, traces rather than final answers, and enough repeated runs to tell a real improvement from luck.

AI Quality Pipeline benchmarking

2025

Product · Benchmark

Does a quality-first pipeline actually beat just calling the model?

I owned the benchmarking that answered this: the metric definitions, the LLM-as-a-judge design, and the comparison across nine widely used models. The distinction that made it useful was holding validity apart from correctness — whether the code is syntactically sound and uses operations that exist, versus whether it does what the user actually asked for. A flow can be perfectly valid and still miss the business logic entirely. Grounded generation with validation and error correction reached roughly 90% validity and 80% correctness, against roughly 20% and 17% for direct prompting. The comparison chart on MuleSoft's Vibes page is mine.

MuleSoft Vibes agent evaluation framework

2026

Product · Benchmark

How do you know an autonomous agent improved rather than got lucky?

An end-to-end framework that runs the agent inside a production-aligned IDE runtime through simulated multi-turn developer sessions, captures the entire execution trace — conversation, tool calls, workspace changes, where it gave up — and scores it against a declarative evaluation contract covering planning quality, tool use, output correctness, and task alignment. Results aggregate across repeated runs under identical conditions, because one run of a non-deterministic system tells you almost nothing. It now backs regression detection and release gating.

FreshBrew: evaluating AI agents on Java code migration

2025 – 2026

Research · Benchmark · ICSE 2026 · DL4C at NeurIPS 2025

Can an agent modernize a real codebase without quietly cheating?

A benchmark of 228 Java repositories, deliberately chosen for high test coverage so that semantic preservation is measurable at all, with a coverage guard that catches the failure mode nobody wants to talk about: an agent deleting or weakening tests to make a migration look successful. I worked on dataset cleaning and validation, agent evaluation, and comparing agentic frameworks against established rule-based tools. The best model migrated 52.3% of projects to JDK 17 — a useful number precisely because it is not a flattering one.

The FreshBrew pipeline: dataset curation, migration agent, evaluation protocol.

SCUBA: Salesforce Computer Use Benchmark

2025

Research · Benchmark · ICLR 2026

Do computer-use agents survive real enterprise software?

300 tasks drawn from real user interviews across three personas — platform administrators, sales representatives, service agents — executed in live Salesforce sandboxes with milestone-level scoring so partial progress counts for something. The headline result is a warning about benchmark transfer: agents that perform well on general desktop benchmarks fall below 5% here. Closed-source models reach 39%, and 50% when given demonstrations, while cutting time and cost by 13% and 16%.

ICLR 2026 conference poster for SCUBA: Salesforce Computer Use Benchmark.

WALT: Web Agents that Learn Tools

2025

Research · ICLR 2026

Should agents click through interfaces, or use the automations the site already has?

Websites already implement search, filtering, sorting, posting, and content management. WALT reverse-engineers that latent functionality into deterministic, callable tools, so an agent invokes search(query) instead of reasoning its way through every click. That moves the burden from fragile step-by-step reasoning to reliable tool invocation: 52.9% on VisualWebArena and 50.1% on WebArena with fewer steps, and on 139 real-world sites it discovered 252 tools on its own and improved success by 20.5% over a tool-free baseline.

Task success rate and average number of steps, WALT against baseline.

Act III

What goes into the model

Before a system can be correct or reliable, something already decided what it knows and what it gets to see. These two ask what belongs in that input: one for the pretraining corpus, one for the context window.

MixtureVitae: permissive-first pretraining data

2025 – 2026

Research · TMLR 2026, Featured certification · ICML 2026 · DATA-FM at ICLR 2026

Can you train a competitive model without indiscriminate web scraping?

An open pretraining corpus sourced permissive-first — public-domain and permissively licensed text, plus carefully justified low-risk additions like government works — with every source sorted into one of three risk tiers and shard-level provenance metadata attached, so downstream users can make their own call about legal exposure instead of inheriting ours. The recipe also folds a large share of permissive synthetic instruction and reasoning data into a single pretraining stage, rather than saving it for post-training. At 1.7B parameters and 300B tokens it surpasses FineWeb-Edu and matches a strong instruction-tuned baseline on GSM8K, HumanEval and MBPP using roughly 36 times fewer tokens, with a decontamination analysis to back the claim up.

ICML 2026 conference poster for MixtureVitae, a permissive-first pretraining corpus.

Diversity Enhances an LLM's Performance in RAG and Long-context Tasks

2025

Research

What should a retrieval system actually retrieve?

Attention costs grow quadratically with context length, so the choice of what to put in the window is forced rather than optional. The standard move is to take whatever scores highest for similarity to the query, which quietly fills the window with near-duplicates and crowds out passages that were relevant but phrased differently. Borrowing from Maximal Marginal Relevance and Farthest Point Sampling, we select for diversity as well as relevance, which substantially improves recall of the sentences and chunks that actually matter before question answering and summarization.

Retrieval with and without a diversity step before question answering.

Patents

Inventions

Additional applications are pending, covering data transformation with large language models, LLM-as-a-judge quality assurance, and tool-learning web agents.

Co-invented with Hadi Minooei, Yazdan Jamshidi, Santiago Bassani, Mofeyifoluwa Oluwalana, and Sriram Ranganathan.

All filings on Google Patents →

Beyond the desk

Convening, judging, and building in public

The parts of the work that happen in rooms rather than repositories.

  • Yanqi Luo with the Agentic AI panelists in front of the Harvard College China Forum backdrop

    Harvard College China Forum

    Programming Advisor, 28th Forum

    I proposed and led an AI panel called Agentic AI: The Rise and Future — set the framing, wrote the questions, and recruited the speakers myself: Salesforce COO Vivienne Wei, Liquid AI founder Ramin Hasani, former Alibaba VP Tu Zipei, and Harvard professor Boaz Barak. Several hundred people came. I also coordinated a technology and business panel with leaders from BrainCo and Cheetah Mobile.

  • Yanqi Luo in front of the Y Combinator Startup School sign

    YC Startup School

    2026

    A weekend among people building things from nothing, mostly to hear how they decide what is worth building. The same pull turned up much earlier: awards at the Shanghai Internet+ and Lin Gang Bei innovation and entrepreneurship competitions.

  • UC Berkeley AI Hackathon

    Judge and panelist, 2025

    Judged the hackathon and spoke on the panel for underrepresented builders, mostly about what the early years of a research-and-engineering career actually look like from the inside.

  • MuleSoft Agentforce Hackathon

    Award winner, March 2025

    Our team built Sublime Force, an agent for insurance eligibility verification — an unglamorous healthcare bottleneck that turns out to be a good fit for agents precisely because the rules are fussy and the verification is mechanical.

Path

How I got here

  1. 2024 —

    Senior Machine Learning Scientist, Einstein AI — Salesforce

    Agent evaluation and generative AI quality for enterprise integration, behind MuleSoft Vibes.

  2. 2023 – 2024

    ML Engineer Senior Associate, Advanced AI Solutions — Salesforce

    MuleSoft's first generative AI features, from foundational research to General Availability.

  3. 2022 – 2023

    Teaching Fellow — Harvard SEAS

    CS109a and CS109b, Data Science I and II.

  4. Sep – Dec 2022

    Researcher — Rainforest Connection

    Bioacoustic classification of rare tropical species from field recordings, where a wrong label is a species you stop looking for.

  5. Summer 2022

    Machine Learning Engineer intern — Salesforce

    Classifying integration patterns across Mule applications: the first look at the codebase the generative work would later have to reason about.

  6. 2021 – 2023

    MS Data Science — Harvard University

    Cross-registered at MIT for computer vision and natural language processing. Applied ML for transportation and conservation.

  7. 2020

    Data Analyst — eBay

    Shanghai. Measuring what shoppers did rather than what they said they wanted.

  8. 2017 – 2021

    BS Statistics — Fudan University

    School of Management, with an exchange year at UC Santa Barbara. Designed learning software for older adults — the project that first made the problem legible, since mainstream interfaces so plainly assumed users who looked nothing like them.

Yanqi Luo in a Salesforce office lobby, beside the Salesforce cloud logo
Salesforce, since 2023 — the chapter this list is still in.
Yanqi Luo at Harvard commencement, holding her mortarboard and wearing a Pan-Asian graduation stole
Harvard commencement, 2023 — the MS, and teaching CS109 alongside it.