Stop Arguing About Email Length: Run a Long‑vs‑Short Test Instead

Free

This playbook walks through a long‑vs‑short onboarding email experiment in Spreeflo for Shopify and e‑commerce apps, using behavioural triggers, Random Split, tags, and events so you can measure real activation instead of arguing about copy length.

Industry

Niche

Pattern

Loading sequence...

The founder wanted a story.

Their marketing lead wanted a punch in the face.

At CartWizard, a Shopify cart recovery app doing around $90k MRR, the team had been going back and forth for weeks over one email: the “set up your first recovery flow” message that goes out right after install. One camp swore a long narrative about recovered revenue would convert best. The other camp wanted a three‑line email and a big “Create my first flow” button.

They were both guessing.

For a small team, that guessing game is expensive. Every week you wait to “just decide” is a week of trial users quietly churning out without ever activating.

The sequence at the top of this page is the whole journey, end to end. It shows a simple way out of that debate: a content variation test where half your users get the long version, half get the short one, and Spreeflo records who actually clicks and activates.

In this article, we’ll walk through that journey node by node, explain why it’s built the way it is, and how you can adapt it for any Shopify app or e‑commerce tool.

The point of this test (and why it suits Shopify apps)

For e‑commerce apps, onboarding email is not “nice to have.” It’s where you rescue all those “installed but never configured” users.

Two constraints shape this pattern:

You care about a single offer
“Create your first recovery flow,” “Turn on performance alerts,” “Connect your ad account.” The call to action is fixed; only the content framing changes.
You don’t have a growth team sitting in spreadsheets all day
You need an experiment you can set up once, then let run while you ship the product.

That’s exactly what this long‑vs‑short test does. It uses:

A behavioural trigger (like app_installed)
A Random Split node to assign contacts to “long” or “short”
One email variant per branch
Shared follow‑up logic and measurement

You build it as a journey with Spreeflo’s visual editor, the same place you build a journey for any lifecycle automation.

Let’s break it down.

Step 1: Let behaviour, not a list, start the test

CartWizard wants to know which email gets more merchants to create their first recovery campaign.

The cleanest trigger is a Custom Event:

Event name: app_installed
Source: your app or backend sending events into Spreeflo via the Spreeflo API
Re-enrollment: off

Why this trigger?

It’s a high‑intent behavioural event. You’re testing copy on people who just installed, not a stale list.
Re-enrollment stays off so a contact only goes through this journey once. If a store uninstalls and re‑installs six months later, you’ll likely want a fresh journey anyway, not a polluted experiment.

If your app has a more precise activation point, you can substitute:

trial_started
store_connected
analytics_installed
A custom “activation milestone reached” event if you want to test a later stage

The pattern is identical. The journey just starts somewhere else.

Optional but useful: insert a Time Delay right after the trigger.

Time Delay: 2 hours, unit “Hour(s)”

This gives new installers a bit of in‑app time before your email lands, and it ensures both variants go out on a similar timeline.

Step 2: Randomly assign long vs short with Random Split

Next comes the heart of the experiment: the Random Split node.

Configuration:

Branch A: “long”
Branch B: “short”
Percentage: 50/50

Every contact who reaches this step is randomly assigned to one branch. Because Random Split is coded at the platform level, you avoid subtle bias you’d get trying to do this manually (day‑based rules, alphabet slices, etc.).

Why 50/50?

You want each variant to see enough volume to make comparisons meaningful.
You’re not holding back a control; both variants are “real” attempts to win the conversion.

Could you run 70/30 or keep 10% as a no‑email holdout? Sure. But for the basic “which email style works better?” question, equal traffic to both is the simplest and fastest read.

Step 3: Tag each variant for clean reporting

Random Split routes people, but by itself it doesn’t label them.

You need each contact permanently marked as “saw long” or “saw short” so you can pull stats later. That’s where Add Tag comes in.

On the “long” branch, add:

Add Tag
Tag: exp_onboard_long

On the “short” branch, add:

Add Tag
Tag: exp_onboard_short

These tags mean you can later build segments like:

“Everyone who got the long email and created a campaign”
“Everyone who got the short email and unsubscribed”

It’s a tiny step that makes analysis trivial. If you’re new to this way of working, the doc on getting started with tags is worth a skim.

Step 4: Send the actual emails (and keep variables tight)

Now each branch sends its version of the email using a Send Email node.

Long‑variant email:

Subject: identical to the short version
From: same sender identity
Content: long narrative body with social proof, examples, FAQs, maybe a short “how‑it‑works” section
Single primary CTA button that deep links into your app’s “create first campaign” flow

Short‑variant email:

Subject: same as long variant
From: same sender
Content: stripped down copy with a few bullets and the same primary CTA
Single primary CTA button that deep links into your app’s “create first campaign” flow

Both emails are created in the email builder, and you leave “Send only once” on so the same contact can’t be hit twice by the same node.

The discipline here is important:

Same subject line
Otherwise you don’t know whether it was the subject or the length that moved click rate.
Same offer and CTA URL
You’re not testing different destinations, just framing.
Same send time
Both variants are triggered by the same behaviour and Time Delay, so timing cancels out.

Everything except the body copy stays constant. That’s how you get a clean read.

To avoid spammy pacing, each branch flows:

Trigger → (optional delay) → Random Split → Add Tag → Send Email

There’s no second email yet, so the “no back‑to‑back emails” rule is satisfied naturally.

Step 5: Merge branches and wait for activation

After each branch sends its email, you want contacts to rejoin a single path so everyone shares the same downstream logic.

That’s the job of the Merge node. Both branches connect their Send Email nodes into one Merge. From the contact’s perspective, nothing happens at this step; it just keeps the canvas tidy.

From Merge, add a Wait Condition:

Condition: Custom Event recovery_campaign_created triggered at least 1 time (over all time)
Timeout: 3 days

This does two things:

It gives merchants time to act on the email. If they create a campaign in 30 minutes, the journey wakes up and moves on. If they don’t, it waits a maximum of three days.
It avoids hammering people with another message too quickly. For non‑converters, you get at least three days between the first variant email and any follow‑up.

In CartWizard’s case, recovery_campaign_created fires from their app backend whenever a user saves their first recovery workflow, again via the Spreeflo API. You’ll substitute your own activation event:

report_created
alert_enabled
feed_synced
automation_published

Whatever “first value” looks like for your app.

Step 6: Branch on who activated vs who stalled

Once the Wait Condition releases the contact, you want different treatment for:

Users who did the thing (they created a campaign)
Users who didn’t (they installed and then ghosted)

Use an If/Else process node for this.

Condition (Yes branch):

Custom Events
Event: your activation event (e.g. recovery_campaign_created)
Frequency: at least 1 time
Time window: over all time

If the contact has triggered that event, they go down the “Yes” branch. If not, they go down “Else.”

On the Yes branch, you might:

Add Tag: activated_via_email
Maybe send an internal email to your team for high‑value installs

On the Else branch, you’ll usually send one more nudge:

(Optional) Time Delay: 0 days, 0 hours is not allowed, so if you already waited 3 days in the Wait Condition, you can safely skip an extra delay here. The gap is baked in.
Send Email: a short “Need a hand setting up?” message with:
A quick primer on why activation matters
Links to docs or a 5‑minute setup video
Contact to support if they’re stuck

Because there’s at least a three‑day gap created by the Wait Condition, sending another email immediately on the Else path still respects healthy pacing.

You can end the journey here or add another Wait Condition + tag update if you want to mark “still inactive after help email.”

Step 7: How to actually read the results

The experiment isn’t finished when you place the nodes. It’s finished when you look at the numbers in a way that stays honest.

You care about three things:

Variant click‑through rate (CTR)
Among people who received each email, who clicked the CTA?
Variant conversion rate
Among people who received each email, who fired the activation event?
Variant unsubscribe rate
Among people who received each email, who unsubscribed from marketing email soon after?

Because you tagged each variant, you now create two saved segments using the segment builder:

Segment A: has tag exp_onboard_long
Segment B: has tag exp_onboard_short

For each segment, you can:

Use Email Activity rules to see:
Email “Onboarding – Long” clicked at least 1 time in the last X days
Email “Onboarding – Short” clicked at least 1 time, etc.
Use Custom Events rules to see:
Activation event triggered at least 1 time in the last X days
Use Email Subscription Status rules to see:
Status is “Unsubscribed” after getting that email

The comparisons you want:

CTR_long vs CTR_short
Which body style gets more people to the app?
Activation_long vs Activation_short
Which actually drives setup, not just curiosity?
Unsub_long vs Unsub_short
Is the long email annoying people into quitting your list?

For smaller apps, you won’t hit massive sample sizes quickly. That’s fine. Let the journey run until both segments have, say, a few hundred sends before you call a winner.

Once you’re confident one version is clearly better, you can:

Swap the underperforming email out, so both branches send the winner, or
Remove the Random Split and tag steps entirely and keep only the winning email path

Either way, your “default” onboarding is now backed by actual user behaviour, not gut feel.

Variations on the same pattern

While this walk‑through used CartWizard’s first‑time activation email, the long‑vs‑short test works anywhere you have:

A clear behavioural trigger
e.g. payment_failed, plan_downgraded, feature_used, or hitting your pricing page repeatedly.
A single outcome you care about
e.g. card updated, plan upgraded, key feature activated, demo booked.
Enough volume over a few weeks to compare variants.

Some concrete examples for Shopify‑focused apps:

A billing recovery test
Trigger on payment_failed, split long “here’s why staying active matters” vs short “update your card” messages, measure successful charge events.
A feature adoption test
Trigger when someone installs but hasn’t used a flagship feature in 7 days. Test a long educational walkthrough vs a tight “enable this in 60 seconds” email.
A pricing nudge test
Trigger on repeated visits to your own pricing page (via web tracking), split narrative “how merchants use our Pro plan” vs concise comparison chart.

In every case, the skeleton is the same: behavioural trigger → Random Split → variant tags → variant emails → shared follow‑up → measurement via segments and events.

Why this kind of test is perfect for small teams

The real win here isn’t just knowing whether long or short copy wins for one email.

It’s that you’ve built a pattern you can clone in minutes:

Duplicate the journey
Swap in a different trigger event
Change the tags to a new experiment name
Draft two new emails

You spent real thought once, and now the system runs quietly in the background while you ship features. That’s the essence of the second core belief behind Spreeflo: founder‑led businesses win on leverage, not headcount. The journey editor, Random Split, and behavioural triggers do the grunt work a growth team would usually handle.

If you’re already sending onboarding email from somewhere like Mailchimp or Klaviyo, but you’re not sure which version is pulling its weight, recreating that flow as a Spreeflo journey is almost always worth the hour. You’ll have better targeting, cleaner experiment design, and you won’t pay extra for all the non‑marketing contacts your app naturally accumulates.

The long‑vs‑short argument never really goes away. Different audiences respond differently. New features change what matters. But once you’ve wired this pattern into your stack, those arguments stop blocking decisions. You draft two versions, plug them into the sequence at the top of this page, and let the numbers settle it.