Skip to content
This repository has been archived by the owner on Jul 14, 2021. It is now read-only.

Multivariate Testing: final call to action #141

Open
livlab opened this issue Feb 18, 2015 · 12 comments
Open

Multivariate Testing: final call to action #141

livlab opened this issue Feb 18, 2015 · 12 comments

Comments

@livlab
Copy link

livlab commented Feb 18, 2015

Purpose

Based on lessons learned from nprapps/lookatthis#717, we know that asking users a "care question", leads to greater conversion of the subsequent call to action. For this story, we will explore nuances in language used in asking the question (with support public radio action).

Recipes

  • Recipe A (control): Support Public Radio
  • Recipe B: Care Question version 1
  • Recipe C: Care Question version 2
  • Recipe D: Care Question version 3

Paths

Final screen > Support Public Radio
Final screen > Yes > Support Public Radio
Final screen > No > Feedback Email

Traffic

  • Estimated total traffic in the 100k unique users range
  • Full split between all recipes (25% each).
  • Start 2/20, end date TBD

Note: eliminate share options as well as "next story" image from this piece.

@lindamood
Copy link
Contributor

I threw together a quick sketch to share with the rest of the team to help explain the proposed change. I'll check with @beckylettenberger to see if we want to check in with the team today or wait until @TylerFisher has a version of this on a branch to review.
support-flows

@livlab
Copy link
Author

livlab commented Feb 18, 2015

Care Question language exploration

(Note: Attempting to write lots of bad ones so good ones pop out more easily)

Did you like this story? << (control)
Did you like our ebola coverage?
Did you enjoy this story?
Did you enjoy our ebola coverage?
Did you appreciate this story?

Do you like this story?
Do you like stories like this?
Do you support this story?
Do you care about ebola?

Does this story trouble you?
Does this story concern you?
Does this story worry you?

Was this story helpful?
Was this story helpful to you?
Was this story relevant?
Was this story relevant to you?
Was this story meaningful?
Was this story meaningful to you?
Was this story interesting?
Was this story interesting to you?
Was this story useful?
Was this story useful to you?
Was this story ok?
Was this story any good?

Will you remember ebola?
Will you forget ebola?

How do you like this story?

Were you aware of this story?

Note: The term story was used in the previous experiment. Piece or essay could be used as well.

YES
Show us your love << change to align with tone after care question is figured out
[Support public radio]

NO
We'd love to know why
[Email us your thoughts]

@livlab
Copy link
Author

livlab commented Feb 18, 2015

"Did this story make you care?" is really what we want to uncover with this question (which is an awkward question to ask explicitly. The question then needs to be a proxy but as close as possible to that sentiment while in keeping with the tone of the specific story we are testing with (i.e.: more sober and serious for this one).

I reviewed the above list with Brian, these are the current contenders for what to test (not final wording):

  • Var 1: Did you like this story? << control to assess against previous test
  • Var 2: Did you like this story? (it helps us to know!)
  • Var 3: Did this story make you care about Liberia?
  • Var 4: Did this story make you care about Liberia? (it helps us to know!)

Refinement options:

  • Did this story make you care about Liberia?
  • Did this story make you care about the people of Liberia?
  • Does this story make you care about Liberia?
  • Does this story make you care about the people of Liberia?
  • Did this story make you care about Barkedu?
  • Did this story make you care about the people of Barkedu?
  • Does this story make you care about Barkedu?
  • Does this story make you care about the people of Barkedu?
  • (it helps us to know!)
  • (It helps us to know)
  • (it helps us to know)

@livlab
Copy link
Author

livlab commented Feb 18, 2015

I vote for these variations:

  • Did you like this story?
  • Did you like this story? (It helps us to know)
  • Does this story make you care about the people of Liberia?
  • Does this story make you care about the people of Liberia? (It helps us to know)

@TylerFisher
Copy link
Contributor

I think I'd rather say "Does this story make you care about the people of Barkedu?", since this story really is just about Barkedu and hardly about Liberia as a whole.

@livlab
Copy link
Author

livlab commented Feb 18, 2015

Wes just suggested something that I think is even stronger, to align with this particular kind of storytelling and encourage support for this type of storytelling in particular.

"Does this kind of reporting matter to you?"
"Does this kind of reporting matter to you? (It helps us to know)"

(language can be further tweaked)

Thoughts?

@TylerFisher
Copy link
Contributor

I like that.

@livlab
Copy link
Author

livlab commented Feb 18, 2015

Alright, last pass, confirming our test options:

  • Did you like this story?
  • Did you like this story? (It helps us to know)
  • Does this kind of reporting matter to you?
  • Does this kind of reporting matter to you? (It helps us to know)

(Now that we've mulled over this, I specially like the term matter because you CARE about things that MATTER; care is what you do (personal), matter is an attribute of the thing you care about that makes it worth it)

@lindamood
Copy link
Contributor

@brianboyer Here's the proposed language and design of screen if a user selects the 'yes' button. Thoughts?

life_after_death__npr

@lindamood
Copy link
Contributor

@brianboyer just pushed to stage. here's the latest. @livlab is working on alternate labels for 'get involved'

screen shot 2015-02-19 at 11 01 24 am

@livlab
Copy link
Author

livlab commented Feb 19, 2015

I think the "get involved" label is a great way to unify the Our Coverage sentence and the main action. While the label itself seems not 100% to me, I can't think of something better.

Increased font size on Our coverage I think really helped. If there are any other changes, it might be making the Get Involved header smaller than the Care Question. I know it's not really the right hierarchy for a title, but it's be more like a tag. just something to consider, otherwise, I think this approach is right for the test.

@livlab
Copy link
Author

livlab commented Feb 19, 2015

Recap Recipes

  • Recipe A (control): Support Public Radio
  • Recipe B: Care Question version 1 - Did you like this story?
  • Recipe C: Care Question version 2 - Did you like this story? (It helps us to know)
  • Recipe D: Care Question version 3 - Does this kind of reporting matter to you?
  • Recipe E: Care Question version 4 - Does this kind of reporting matter to you? (It helps us to know)

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants