>> Home > Mach5 Resources, Support, and Idea Bank

Home | My Mach5 | | About Us | Purchasing

Home
E-mail Tips
Website Metrics
  Scenarios
  A/B Testing
  Search Engines
  Layout/Design Tips
Book Reviews
Web Resources

Explore M5 Subscriber
Web-based list management made simple!

Simple subscribe tool
Opt-in confirmed
Track reads and clicks
Custom matching look
Free for up to
5000 contacts

Explore M5 Mailer
Take the bulk out of email! ^TM

Use your own data
Personal email merge
Handy email templates
Scriptable with
JavaScript, PHP, VBasic

Explore M5 Analyzer

Get great site statistics directly from your web server log files.

Optimize site design
Track campaigns
Performance Metrics

A/B Split Testing

Whenever you make a change to your web site, presumably it's to make things better. You certainly don't want to make things worse! But what does "better" mean?

You really need to know what "better" means in some operative sense before you make a change to a page on your site. At every page on your site, you should have a general idea of what else you may want a visitor to do there, even if it's to leave and go somewhere else.

Once you know what "better" means, you can devise a test to see if the change you made to a page actually improves your site performance. This test is called an A/B Split Test. Imagine if you set up a scenario already and knew that, with your current site, 20% of all of your visitors went from your "real estate info" page to your "contact me" page. (Situation A) Then, after you make a change to your site, you find that 18% of your visitors ended up there (Situation B). If your goal is to have your visitors contact you after inquiring about your real estate offerings, you might want to back out that change! The A/B test measurement tells you the effectiveness of A vs. B.

Statistical Significance: In the above situation, what if you base your percentage on, say, 10 visitors? That's not going to be very accurate. More precisely, given a certain number of visitors, your measurement will have a certain statistical significance. In general, if you have about 100 visitors you can assume your metrics are signifigant to within 2% or so. In this case, there may not be much point in using the above numbers of 20% and 18% to make a decision about the effectiveness of your site change.

Timing: Making a change to your website on Friday, which is on the border between weekday visitors and weekend visitors, is likely going to skew your measurement.

Two types of A/B Split Testing:

Randomized - in a randomized test, for a period of time your site randomly serves either page A or page B, and in such a way that you can tell which is which. Randomized tests are good,but generally require some form of web programming to dynamically change the link or query string variable on a page.

Sequential, Date Based - If you know when you made a change, you can measure before the change and after. Time your change well if you are going to do that!

Any change you make to your website should be backed with solid metrics. Check out Analyzer, which can do A/B testing with its built in Scenario Analysis!