Quote:
Originally Posted by schettj
The problem with your data is statistical in nature.
You cannot control for variations in the aether, as they used to call it. You're testing a radio, and radios are subject to interference, and even (in the case of our beloved phones) contention with the tower.
Couple that with the lack of control over the internet itself (who knows if you had more or less retransmitted packets on one trial or another) and a lack of control over what was happening on the phone (WinMo does decide to do its own thing from time to time) and you realize that even a 10% difference between two radios *might* not be significant.
That's why you would need many many trials with each radio. Tossing out the high and low 5%, averaging the rest will give you a more significant result.
It's a big leap to just run one trial and say "yep, that's it!"
As a test, just run the same test 5 times on the Telus ROM. Look at the results. Tell me if they are identical each time. My money is on "No". They may be close, but its very unlikely they are identical. Pick the WORST result, and then re-rank the radios. Is Telus still #1?
|
I understand what you're saying. However, I just do not have the time to run a ~26 minute benchmark routine multiple times (Say 5) on multiple Radios. (6) That would take
13 hours of being unable to use my phone for anything. Will I perhaps do it a bit at a time, running more tests on each Radio? Maybe, if time permits and interest holds. But for now, at least I benchmarked all the available Radio's at least once, providing a starting point backed up by empirical evidence for which is better and which is not. Can you show me someone else who has done this?