Mobile Testing Tips – Use a Fixed Location


Just returned from a trip to the Pacific Northwest and had a chance to run some side-by-side mobile Web performance tests using the AT&T Motorola Atrix and the T-Mobile MyTouch (HTC).  I compared the same 3 pages throughout my trip to get some consistency, and wanted to share just how complex the mobile testing matrix can be.

Test from a Fixed Location

I must not have applied equal pressure to my touch screens, as the tests show a 1/2 second difference in start time.  Why is that important?  I was in motion.  A half-second translated into different locations (see the two images below).

Because of that, I cannot say with certainty if the page performance variance (4.61 seconds) was due to the device, OS version or carrier – or if maybe a hill momentarily blocked the signal during one of the tests.  (look the left of both “drive by” shots).

T-Mobile MyTouch:

AT&T Atrix:

I do know from test frequency that it is probably the carrier and/or device (in the locations tested, AT&T/Atrix was consistently faster than T-Mobile/MyTouch), but this brings up a great point to remember when building your mobile Web performance test matrix:

Always fix your location when comparing other mobile context attributes.

While using automated testing in motion to determine a traffic corridor’s performance is quite effective (and makes for pretty pictures and a “Map my Ride” visual experience) , it does not work for apples to apples performance comparison.

A little bit of up front planning will go a long way towards provide accurate results and comparisons.

(As a safety note, I was not driving while conducting these tests)

Posted in: Uncategorized

Email Subscription