Comments on: Split Testing Adwords: You're Doing It Wrong

By: Incorporate Online

Incorporate Online — Wed, 30 Mar 2011 21:13:29 +0000

I implemented a few of the things you talked about.. and wow! Great results!

Would you be interested in checking out my campaign, and optimizing for me? What is your consulting fee.

Best Regards

By: Jon Baily

Jon Baily — Tue, 21 Sep 2010 16:09:00 +0000

Hi Dan,

Thank you for the excellent advice. I have one question; when you are testing do you use the Google optimiser?

All the best Jon

By: Joel McDonald

Joel McDonald — Sun, 11 Jul 2010 13:56:11 +0000

Hey Dan,

I just referenced this post when explaining my strategy to a client who wanted to test a new ad that they were confident would "kill it". (I was probably one of the head slaps you heard when you presented this concept at a Stompernet event, and have used that strategy ever since. .

Another great use of the additional copies is that they are built in validators, so you don't even need outside tools like splittester. For example, if you're running 3 copies of your control ad, you'll know you don't have statistical significance until those 3 ads' performance stats are in alignment. Using that logic, you're less likely to determine a winner or loser prematurely.

Example of "Not enough testing": Copy ad 1: CTR: 2.3%, Conversions: 25 Copy ad 2: CTR: 1.5%, Conversions: 18 Copy ad 3: CTR: 3.1%, Conversions: 41 Test ad: CTR: 1.6%, Conversions: 17

In the above example, even though you might be tempted to "pull the plug" on the test ad, you'll know that you haven't done enough testing solely because 3 other ads that should be performing identically – aren't.

Example of a statistically valid data set: Copy ad 1: CTR: 2.5%, Conversions: 86 Copy ad 2: CTR: 2.4%, Conversions: 80 Copy ad 3: CTR: 2.7%, Conversions: 87 Test ad: CTR: 3.0%, Conversions: 95

In the above example, if you were only running an A/B test, you might need to run the test a lot longer, but considering it's an A/A/A/B test, and the fact that all 3 A's are in alignment, you know you've been running the test for long enough to get reliable results.

As you pointed out, you have to ignore the control ad because it may be getting preferential treatment due to performance history.

By: Dan Thies

Dan Thies — Wed, 23 Jun 2010 22:27:51 +0000

Hi Paul – Stompernet has a version of that here: http://www.stomperblog.com/marketing/stompernet-going-natural-30/

I'll be releasing an update some time this summer.

By: Paul Wright

Paul Wright — Wed, 23 Jun 2010 21:48:17 +0000

Hi Dan, I have to say that your information is priceless and thanks for that. I just wanted to ask you if you dont mind to see if you can take a look at the link for the 50 minute video on this site about split testing adwords. The link seems to be broken

regards and thanks heaps

Paul

By: Renford Nelson

Renford Nelson — Thu, 17 Jun 2010 20:40:36 +0000

I have now setup my split test following your advice. One thing I would like to know; is there any pitfall to this method?

By: Dan Thies

Dan Thies — Mon, 05 Apr 2010 23:12:14 +0000

They've moved it – I'm working on a new version anyway, which will be hosted here.

By: Rennell

Rennell — Sun, 14 Mar 2010 23:13:57 +0000

This is obviously an interesting and different way to look at testing. Most people just do ab testing without looking into the true science of testing.

By: J. Mark Bangerter

J. Mark Bangerter — Sun, 14 Mar 2010 21:42:46 +0000

Has the video been removed? The link is no longer working.

By: Rex Dixon

Rex Dixon — Thu, 07 Jan 2010 22:29:13 +0000

With all the different services available out there, wouldn't using a service such as http://www.performable.com/ be a better alternative? Might simplify the process? Or are you against using services to do your testing?