Mobile Testing Tips – Use a Fixed Location
by Elizabeth
Just returned from a trip to the Pacific Northwest and had a chance to run some side-by-side mobile Web performance tests using the AT&T Motorola Atrix and the T-Mobile MyTouch (HTC). I compared the same 3 pages throughout my trip to get some consistency, and wanted to share just how complex the mobile testing matrix can be.
Test from a Fixed Location
I must not have applied equal pressure to my touch screens, as the tests show a 1/2 second difference in start time. Why is that important? I was in motion. A half-second translated into different locations (see the two images below).
Because of that, I cannot say with certainty if the page performance variance (4.61 seconds) was due to the device, OS version or carrier – or if maybe a hill momentarily blocked the signal during one of the tests. (look the left of both “drive by” shots).
T-Mobile MyTouch:
AT&T Atrix:
I do know from test frequency that it is probably the carrier and/or device (in the locations tested, AT&T/Atrix was consistently faster than T-Mobile/MyTouch), but this brings up a great point to remember when building your mobile Web performance test matrix:
Always fix your location when comparing other mobile context attributes.
While using automated testing in motion to determine a traffic corridor’s performance is quite effective (and makes for pretty pictures and a “Map my Ride” visual experience) , it does not work for apples to apples performance comparison.
A little bit of up front planning will go a long way towards provide accurate results and comparisons.
(As a safety note, I was not driving while conducting these tests)
Posted in: Uncategorized