I've run an ecom brand. I've hired agencies. I've fired more agencies. Now I do CRO consulting. This is the breakdown nobody wants to write because every side of it is uncomfortable.
80% of agency retainers don't work. Not because the agencies are bad. Because the incentive structures are broken from day one.
Problem 1: The Klaviyo Attribution Lie
Every email agency will send you a monthly report that says "Klaviyo drove $87,000 in attributed revenue." They'll use that number to justify the retainer.
The number is a lie. Klaviyo attributes any sale within a 5-day click window AND any sale within 24 hours of open. So if a customer clicks your welcome email, buys 3 days later after seeing an Instagram ad, Klaviyo takes full credit.
Run a holdout test. Take 10% of your list out of all email sends for 30 days. Compare their purchase rate to the 90% who got emails. The delta is your real email lift. Usually it's 30-50% of what Klaviyo reports.
Agencies don't run holdout tests because the math would make their retainer look like 40% less value.
Problem 2: The Web Dev Who Killed Conversion
Classic pattern. Agency redesigns the site. Adds fancy animations, hero carousel, and a "premium" checkout flow. Site "looks great." Conversion drops 15%. Agency blames traffic quality.
Web dev agencies get paid for deliverables, not outcomes. A homepage with 14 sections ships. A homepage that converts doesn't. Guess which one agencies ship.
Red flag questions to ask before hiring any web agency:
- "What was the conversion rate before and after your last 3 redesigns?"
- "Show me session replays from after the launch."
- "What metric do you guarantee?"
If they dodge, walk.
Problem 3: The Ad Team Incentive Misalignment
Your paid media agency gets paid as a percent of ad spend. 10-15% is standard. So their incentive is to spend more, not spend better.
You'll see this in the monthly review. "We scaled Meta to $45K last month, ROAS held at 2.8x." Sounds good. What they're not saying: contribution margin dropped from 28% to 17% because the new audience converts at lower LTV.
Want Your Ad Spend Audited Fresh?
Get a free funnel audit. I'll map the fixes that move revenue fastest.
Ask for contribution margin, not ROAS. Ask for first-order profitability, not attributed revenue. The agency's monthly report won't have either because they don't pay agencies to calculate it.
Problem 4: The CRO Retainer Trap
"$8K a month and we'll run CRO tests on your site." Sounds great until month 4.
By month 4, you've had 2 tests reach significance. Both were inconclusive. The other 6 tests are "still running." You're $32K in and your conversion rate is identical.
The math: most stores don't have enough traffic to run CRO tests with statistical rigor. You need 10K+ PDP sessions per variant per month to call a 10% lift inside 30 days. Most sub-$5M stores don't have that.
Agencies take the retainer anyway because it's their business model. The better answer is direct-to-conclusion fixes based on VoC, session replay, and heatmap evidence. Not a 6-month test queue.
Problem 5: The Revenue Share Deal
"We'll take 10% of revenue above baseline." Sounds aligned. Isn't.
What happens: agency pushes top-of-funnel paid spend because it drives revenue growth, even if it's unprofitable. You pay 10% revenue share on top of already-thin margins. Your P&L bleeds while their invoice grows.
Real alignment is 10% of incremental profit, not revenue. Propose that and watch the agency backpedal.
Problem 6: The "Full Service" Lie
"We do paid, email, SEO, and CRO." Marketed as a one-stop shop. Reality: they have one good paid media lead, one junior email marketer, a subcontracted SEO team in Pakistan, and a CRO person who runs generic A/B tests.
Full service means one thing done well and four things done badly. Pick the agency that does one thing world-class. Outsource the other four to specialists or do them in-house.
What Actually Works
Small, accountable, outcome-priced engagements.
- One-time audit with implementation roadmap. Fixed price. Deliverable is a plan. You execute.
- Fractional expert on a specific channel. Email operator for 10 hours a month. CRO consultant for a sprint. Defined scope.
- Performance-only on incremental profit. Only pay on profit you didn't have. Rare but real.
Skip the 12-month retainer. Skip the "full service" pitch. Skip the percent-of-ad-spend deal. Buy specific outcomes, not generic hours.
The One Question to Ask
Before signing any agency contract: "What's the specific dollar outcome you guarantee, and what happens if we don't hit it?"
If the answer is "we don't guarantee outcomes," you know what you're paying for. Hours. Not results.