The challenge
You’re building an AI contract tool for a manufacturer’s procurement team. In week one they wanted clause extraction, by week three it was risk scoring, and after their CFO saw a competitor’s demo, now it’s Tamil summaries for plant managers.
- W1Clause extraction
- W3Risk scoring
- W5Tamil summaries
after the CFO saw a competitor’s demo - W11Pilot ends
6 weeks left
Nothing is finished, and the plant managers it’s meant for have barely used it. The pilot ends in 6 weeks, nobody has agreed what success looks like, and your leadership has already presented it as a flagship AI win. Stepping back now would look like failure.
What is really going on
So your partner keeps changing what they want. Every week there’s something new, and your team is rebuilding again. It’s frustrating, and it’s easy to think they just can’t make up their mind.
But if you go back and look at each request on its own, they’re usually not the same kind of request at all. Some are about quality, some came from someone else, and some are just curiosity. Each kind needs a different answer from you.
The simplest way to tell them apart? Next time a request comes in, ask what made them ask for it. Whatever they point to tells you a lot.
If they show you an output you got wrong, it’s really a complaint. “Can we add a screen where my team can correct the extracted clauses?” is them telling you, politely, that the extraction isn’t good enough yet. Fix that first, because anything you build on top will carry the same errors.
If they show you an output they didn’t like, their bar is going up. “Can the summaries be shorter and more formal?” means they’re figuring out what good looks like by reading what you produce [1]. That’s a good sign, honestly. Save those examples so your team tests against what they want now.
If they mention a meeting or a demo, it’s probably someone else’s idea. “Our CFO wants Tamil summaries” came from a competitor’s demo, not from the plant managers. Before anyone builds it, ask two plant managers whether the English summaries actually give them trouble.
If they point to something in their daily work, you may have found a real new problem. “Our buyers want an alert before a supplier contract renews” is that kind. Write it down. It’s worth building when a second team asks for it too, or when someone is ready to fund it.
And if they can’t point to anything, it’s mostly curiosity. “Could it also do something with our emails?” is excitement about AI more than a real need. Ask what they’d put in to try it: some time, some data, a bit of budget. If the answer is nothing, it can wait.
What do you want?
What did you see that made you want this?
The mistake most teams make
The natural move is to freeze the feature list and get it signed. But nobody had judged a real output when that list was written, so they’ve outgrown it within weeks. The partner learns feedback doesn’t change anything, goes quiet, and the pilot just fizzles out.
Get the problem signed off instead, along with a fixed test set, and let the features keep changing.
The options you have
Most teams pick two or three of these.
| Option | Best when | What it costs you | Watch out |
|---|---|---|---|
| Widen to more partners | One partner is driving most of your backlog. | Several weeks of finding and onboarding partners, with little new built in that time. | The new partners need the same problem, or they’ll pull the product apart. |
| Ask for a real commitment | They feel the problem and someone on their side has a budget. | Some partners will walk away. | It fails if the ask never reaches the person with the budget. |
| Test new ideas with a quick prototype first | Each new request costs you weeks of rebuilding. | A few days of one person’s time per idea, and you’ll throw most of them away. | Label it as a test, or it’ll be treated as a promise. |
| Embed an engineer with the partner | Several partners keep asking for similar things. | One engineer’s time goes to one partner, who may start treating them as their own. | It turns into consulting if most requests are one-offs. |
| Scale back to a lighter role | They won’t commit anything and the problem doesn’t hurt them much. | The executive who championed the pilot may feel dropped. | Agree it with your own leadership first, and don’t do it if this partner is your only way into a market that matters. |
If you’re asking a customer for a commitment, it can be a fee, and Sierra’s first go-to-market hire suggests 10–20% of contract value [2]. In India, founders and investors describe repeated requests for free proofs of concept [3], so ask for data, named expert time and a signed decision date instead. If they say no to all three, that’s your answer.
PM tools to put in place
Keep a request log. Each request gets a row with what they’d just seen, about 10 real examples, and which kind it is. Ideas from someone else’s demo rarely come with examples, so they filter themselves out.
| Request | What they had just seen | Examples | Kind |
|---|---|---|---|
| Can we add a screen to correct the clauses? | Clauses the tool extracted wrongly | attached | A complaint in disguise |
| Can the summaries be shorter and more formal? | A summary they didn’t like | attached | Their standard is sharpening |
| Our CFO wants Tamil summaries for plant managers. | A competitor’s demo | none | Someone else’s idea |
| An alert before a contract renews? | A renewal in their daily work | attached | A genuinely new problem |
| Could it do something with our emails? | Nothing in particular | none | Curiosity, not need |
You’ll also want two test sets. The fixed one, about 100 examples, stays locked and decides go or no-go. The working one grows every week with what they want now.
Fixed set
Working set
And get one page signed off: the problem, the fixed test set, who judges good, and the decision date.
- The problem to solve
- One sentence in the partner’s own words
- The fixed test set
- About 100 real examples with agreed good answers
- Who judges good
- One named person
- Decision date
- The day the budget owner decides go or no-go
Your first three steps
- Put every request so far into the log and mark its kind. Once you see them together, you’ll know whether you’re mainly dealing with quality, with who decides and commits, or with a real new need.
- Ask the partner to send examples with every new request, so you build what their team actually needs.
- Get 30 minutes with the budget owner this week to agree who judges good, the decision date, and what they’ll commit if it works. With 6 weeks left, anything you don’t agree now gets decided by default.
What to measure, and how
Look at these every week, in this order. The first one matters most.
Are the people who do the work actually using it?
Count the contracts that go through it each week and how many outputs are used without rewriting. Buyers rank this above everything else when deciding to scale [4]. If it’s flat, spend a day with the users before building anything new.
Is the partner committing more over time?
At every review, mark which step they’re on. If they’re stuck on the same step for two reviews, ask the budget owner for the next one by a set date.
Is it getting better at what they already asked for?
If it still isn’t reliable at the job it already has, they’ll keep sending new requests to work around it. Run both test sets after every change. If the fixed set goes up, it only counts when the working set goes up too. Otherwise your team is tuning to the test.
If you see the right-hand picture, or someone asks to change the fixed set, pause new features and fix quality on what they already asked for.
Is the rebuilding under control?
Each week, track how much of your team’s time goes into redoing things. It should fall after the first month. If it jumps after a senior person sees a demo, show that number to the budget owner and agree which requests to stop.
Real examples
Sierra
Sierra builds AI agents for customer service. In 2023 their first go-to-market hire, Logan Randolph, set out to find their first design partners, and two of the partners he signed tell this whole lesson.
The first one kept changing what they wanted. Looking back, Logan put it down to how they’d picked that partner more than to the requests themselves. This partner didn’t feel the pain the way the others did. They wanted to use AI agents and went looking for a problem to put one on.
SiriusXM pushed just as hard, but for something specific. They wanted voice. SiriusXM became the voice design partner and launched first. They also wanted a proper way to give feedback on their agent, instead of the CSV exports Sierra was sending. That push is where Agent Studio came from [2].
Both partners asked for a lot. What made the difference was why they were asking. SiriusXM’s requests gave Sierra a product, and the first partner’s just kept moving.
That’s what Logan screened for. He looked for partners with real pain, at real scale. His advice is to charge partners 10–20% of the contract value. A boss has to sign that off, and it filters out teams that are only curious about AI [2].
The lessonWhen a partner keeps changing what they want, ask what made them ask. If they have a real problem, their requests make your product better. If they’re just looking for somewhere to use AI, the target keeps moving.
Sources
- Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences — Shankar et al., arXiv, Apr 2024. https://arxiv.org/abs/2404.12272
- The Hard Way Pays Off: Inside Sierra's Design Partner Strategy — First Round Review, Sep 2025. https://review.firstround.com/sierra-design-partnership/
- Is India Really That Hard to Sell To? Rethinking AI Start-Ups' 'Skip India' Narrative — Outlook Business, Jun 2025. https://www.outlookbusiness.com/in-depth/is-india-really-that-hard-to-sell-to-rethinking-ai-start-ups-skip-india-narrative
- Harnessing Enterprise Value: The ROI of AI — Madrona, Aug 2026. https://www.madrona.com/harnessing-enterprise-value-roi-of-ai/