3PHealth Blog

Mobile Testing Tips – Use a Fixed Location

Tuesday, July 5th, 2011

Just returned from a trip to the Pacific Northwest and had a chance to run some side-by-side mobile Web performance tests using the AT&T Motorola Atrix and the T-Mobile MyTouch (HTC). I compared the same 3 pages throughout my trip to get some consistency, and wanted to share just how complex the mobile testing matrix can be.

Test from a Fixed Location

I must not have applied equal pressure to my touch screens, as the tests show a 1/2 second difference in start time. Why is that important? I was in motion. A half-second translated into different locations (see the two images below).

Because of that, I cannot say with certainty if the page performance variance (4.61 seconds) was due to the device, OS version or carrier – or if maybe a hill momentarily blocked the signal during one of the tests. (look the left of both “drive by” shots).

T-Mobile MyTouch:

AT&T Atrix:

I do know from test frequency that it is probably the carrier and/or device (in the locations tested, AT&T/Atrix was consistently faster than T-Mobile/MyTouch), but this brings up a great point to remember when building your mobile Web performance test matrix:

Always fix your location when comparing other mobile context attributes.

While using automated testing in motion to determine a traffic corridor’s performance is quite effective (and makes for pretty pictures and a “Map my Ride” visual experience) , it does not work for apples to apples performance comparison.

A little bit of up front planning will go a long way towards provide accurate results and comparisons.

(As a safety note, I was not driving while conducting these tests)

Why the Browser Matters

Thursday, June 30th, 2011

I borrowed the title from “Ben’s Blog”, however the content is going to be a little different.

Why does the browser matter?

It’s simple to use. Ask anyone if they know A) What a browser is & B) how to use it and the answer will be “Yes”
It’s cross platform, meaning that no matter what device you’re on, the browser works the same way
It connects to the Web in a way that is universally understood

In short, you have an “app” that is universally understood, works the same way on every platform and delivers content in a consistent and easy to view fashion. No other app can make that claim and that’s why the “Browser Matters”.

But let’s not stop there. Let’s look at the other side of the coin:

Why does the Web Matter?

The Web is simple.
The Web is flexible and forgiving. (The browser ignores things that it doesn’t understand).
The Web is heterogeneous, which means it works on all platforms. (Not just Windows)
The Web is loosely coupled. Most previous computing architectures required tight integration between the “server” program that stores the data and the “Client” program which manipulates it. In contrast there is no need to upgrade the Web browser every time a Web publisher changes a site. Server and Client are loosely coupled.

No other platform can make that claim and that’s why the “Web matters”.

So what does the Enterprise want?

One Interface – the Browser (see Why the Browser Matters)
One Platform – the Web (see Why the Web Matters)
Access to Multiple Data Sets – the Context (because that’s where all the customers data is)

It’s the last one that is the driving factor behind new revenue opportunities and ultimately why the Browser (& the Web is becoming more important and valuable every day).

Ben’s article talks about the importance of the Browser and about a company called Rockmelt which just raised another $30m dollars to improve their browser. He goes on to talk about how Rockmelt is focusing on 4 major items:

People – it’s all about “social”
Information Flow – it’s all about “feeds”
Search – it’s all about “better search”
Multiple Computing devices – It’s all about the cloud (storing my bookmarks, history, configuration in the cloud)

Totally agree, and everyone of those features is already available in the current browsers – well you might have to open up another tab but that’s about it. So if Rockmelt got $30m to improve the browser (because it matters)… what else might need improving while you’re in the code.

Well how about looking back at what the Enterprise wants, One Interface, One Platform and access to Multiple Data Sets (databases). Why is this so important – a single word sums it up – Money

They’re looking to leverage all the data that’s sitting in those databases. They know that customers the world over all know how to use a browser, and because it ships on every device there’s always a way to reach out and touch your customer.

So if all this matters so much what’s missing from the browser?

Well we asked a lot of users this very question and it all boiled down to three things:

Convenience
Privacy
Control

Summarized – Give me a better user experience and don’t abuse my privacy, (let me control it).

So the goal now is to align those three things with what the Enterprise wants. And therein lies the things you need to do to really improve the browser.

So what’s missing?

How about a secure database where I can store my data (like a wallet)
How about a way to integrate this wallet with the browser so I can send my data to trusted Web sites
How about a way that I can “control” what gets sent to whom

Seems so simple, but there’s currently no way to do it. Microsoft has started the ball rolling with IE9 and “whitelisted” Web sites but still no real way to control my private data. And while we’re here, lets talk about Mobile for a moment. To me this is where the biggest opportunity lies. We keep these devices with us 24*7. We use them constantly and surprise, surprise – they’re mobile.

And yet to this day the Web really has no idea what the connecting device is really capable of doing.

So if someone offered me $30m to improve the browser I’d focus on the above. Doing so opens up net new revenue opportunities, it offers a way to improve the customers experience, and it offers a way to improve customers privacy.

And I’d put the other $29m in the bank for a rainy day.

Why Mobile Performance Matters to “Travel Agents”.

Wednesday, June 29th, 2011

Let’s start with some images…

Here’s a Mobile tester in Kauai using his Mobile Browser to test carrier/web page performance. In this case he was using Verizon. Yellow dots are “middle of the road”, green is good and red is “not so much”. Overall Verizon barely makes the grade

Now lets switch over to Washington State. Lots of green flags. That’s good.

But what happens when we widen the test area? Well overall still pretty favorable. So whose the carrier? Clicking on the flags shows us that it’s AT&T

So why would Travel Agents be interested in this data?

Simple – people want to use their phones on vacation – this maps tells you where you get the best performance.

Mobile Performance, Real Time, All the Time–On the West Coast

Wednesday, June 29th, 2011

Now that’s what I’m talking about! That’s AT&T in real time, not too shabby (Green is good, Red isn’t)

There’s ONLY 2 ways to Measure Mobile Performance

Tuesday, June 28th, 2011

Believe it or not there’s only two ways to measure Mobile Performance. And the choices are:

The Wire Only – Look at what is ‘on the wire’ and then start scratching your head about why it doesn’t seem to match what you can even see with your own eyes watching the page get displayed.
What is WebKit actually doing? – Look at what ‘WebKit’ is actually doing and then start scratching your head about why that doesn’t seem to match what is happening ‘on the wire’.

Either way there’s going to be some head scratching going on. Mobile is different, there’s no silver bullet and you have to carefully define exactly how you’re measuring the content.

If you’re measuring “On the Wire” then the numbers tell you how long it takes the content to get to the device. (Doesn’t tell you a thing about how it processed the content)

If you’re measuring “What is Webkit actually doing” (the User Experience) then you’re going to see what happens “after” the data has been processed. So what’s doing the processing? Well HTTP Connect is doing some (post data arriving operations) and then Webkit is doing the rest.

Welcome to Mobile – it’s NOT the same as the desktop world. Don’t believe me? Here’s some light reading…

Android HttpClient can hang on transfer encoding – link
The WebKit ‘onLoadResource’ event is called even before the resource “starts loading” – link
Webkit – Provisional vs. Committed load – link and link
How Webkit loads a page – link

Enjoy!