Measuring Mobile Web Performance – Simulated Vs. Real Time? (That is the Question)
I’m going to start this blog with a Quote from Shakespeare…
To be, or not to be, that is the question:
Whether ’tis nobler in the mind to suffer
The slings and arrows of outrageous fortune,
Or to take arms against a sea of troubles
And by opposing end them.
Taking a “little” liberty with this famous soliloquy I’m going to look at measuring Mobile Web performance from a 50,000’ overview and address the thorny question of simulated vs. real time measurement.
The story has to begin with “measure what”? As in, what do you want to measure? Let’s take a quick peak at the OSI model
In a previous blog post “Measuring Mobile Web Performance 101” I talked about “Over the Wire” (the transport layer) vs. the “Application Presentation” layer (the browser). So the question has to be – do you want to measure the network performance or the user-experience?
Up until Smartphones came along I would happily agree (rather “than suffer the slings and arrows) with the simulation experts, that there’s virtually no difference between the packets arriving on the desktop and the browser processing them and displaying the content. If there is a difference, it’s not enough to materially impact the quality of the experience or most company’s bottom lines. Running simulated network tests is good enough for the desktop.
Well that’s all changed now. (Time to suffer the slings and arrows).
When it comes to the Mobile Web there is a difference. And in some cases a pretty big one. Our focus was to look at the user experience and apply our measurement tools squarely from the mobile user’s perspective – inside the browser (The Application/Presentation layer).
In my opinion, as we move to Mobile, not measuring the real user experience is a costly mistake. Mobile Web experience measurement is pushing us to transition from our current Quality of Service(QoS) thinking (simulated networks) to consider the total Quality of Experience (QoE) (real time networks).
What exactly is QoE? Well in short, it’s where the “user consumes IT services”. It’s also where IT and Marketing objectives intersect. That means not only does your network performance matter, but you also have to consider everything else that happens from the time the content leaves your Web Server to when it is displayed on the smartphone screen. The transition from QoS to QoE requires the alignment of IT and Marketing strategies to deliver a compelling mobile Web experience to mobile customers and workers. QoE measurement for the mobile Web is about extending the QoS concept out of the data center and into the real world.
So I ask you – how can a simulation of this possibly work?
Well in short it can’t (more arrows). And here’s why – until you’re measuring inside the browser on a real device, on a real carrier network, anywhere you can connect to the network, then you’ll never know the real experience. There’s simply not enough statistical data out there (as there is for the desktop) to build adequate simulations spanning the range of mobile user experiences.
In the old days, (pre-Mobile), approximation was “good enough”. Not anymore. There are more variables in play, you can’t simulate a real carrier network, you can’t simulate every location or connection and you can’t simulate every device. It will always fall short.
What really counts and what we should all be measuring is what the user – the Consumer – gets to experience/consume the results of what IT and Marketing put together.
And finally what of the browser? Does this factor into Simulated or Real? Fortunately another line from Hamlet comes to the rescue – “ay, there’s the rub”
And the “rub” is a big one. We’ve had years to perfect the browser on the desktop. Routines have been re-written and optimized for new processors, more memory is allocated to load pages faster. This is NOT the case on mobile. It’s still in the opening act.
Compare my desktop activity monitor (525MB allocated to Safari)
To my iPhone… total active memory is 38MB! (and Mobile Safari looks like it’s using about 4.5MB)
To my HTC Android phone (1.72MB – sorry no picture).
The delta between the desktop and Mobile device is huge. The processors are different, the memory is different – in short everything is different.
So desktop simulation now has to give way to a “more precise” measurement. One that includes the Carrier Network, the actual device location, the device’s Operating System version (yes we’ve found that it makes a huge difference).
In my opinion it would be pure folly to insist that simulation of Mobile performance will come anywhere close to really showing us what the Quality of the User Experience (QoE) really is.
If you have a different notion – be sure to send me an email and please include your data. When it comes to your mobile Web customers, the only thing that matters is the experience. If we can’t accurately measure it, we can’t improve it. .