Newer radio? (Telus?)

[SweetBearCub]

Originally Posted by schettj

Wow... that's a mighty BIG leap. I would have said

* any but #1 (Power Log #1 - Radio version 1.03.15F)

If you look at the data, #1 is the outlier. In all other cases the variation is less then 5% +/-, in many cases under 1%. Given you did single trials (you did single trials, right?) those differences are meaningless.

My guess is if you repeated each test 100 times, tossed out the 5 highest and 5 lowest scores, and averaged the remaining 90 (likely to be a statistically accurate average) these variations would even out to essentially 0.

So, don't use that first radio rom, its a dog. Use any of the other ones It seems pretty accurate since #1 is the shipped sprint rom. I switched from that to #2 and it sure seems better overall.

It is true that the variations in the other ROMs than #1 are in most cases small. However - I am satisfied that the benchmarking program put each ROM through a significant enough test - It downloaded a large file for about 6 minutes and placed a 20 minute phone call for each Radio, monitoring all sorts of variables the entire time. As my conclusions show, other Radio's were better than the stock shipping Sprint Radio, but the Telus ROM was better than any of them. Do you disagree with my data? If so, why? I'm open to discussion here.

[schettj]

Originally Posted by SweetBearCub

Do you disagree with my data? If so, why? I'm open to discussion here.

The problem with your data is statistical in nature.

You cannot control for variations in the aether, as they used to call it. You're testing a radio, and radios are subject to interference, and even (in the case of our beloved phones) contention with the tower.

Couple that with the lack of control over the internet itself (who knows if you had more or less retransmitted packets on one trial or another) and a lack of control over what was happening on the phone (WinMo does decide to do its own thing from time to time) and you realize that even a 10% difference between two radios *might* not be significant.

That's why you would need many many trials with each radio. Tossing out the high and low 5%, averaging the rest will give you a more significant result.

It's a big leap to just run one trial and say "yep, that's it!"

As a test, just run the same test 5 times on the Telus ROM. Look at the results. Tell me if they are identical each time. My money is on "No". They may be close, but its very unlikely they are identical. Pick the WORST result, and then re-rank the radios. Is Telus still #1?

[SweetBearCub]

Originally Posted by schettj

The problem with your data is statistical in nature.

You cannot control for variations in the aether, as they used to call it. You're testing a radio, and radios are subject to interference, and even (in the case of our beloved phones) contention with the tower.

Couple that with the lack of control over the internet itself (who knows if you had more or less retransmitted packets on one trial or another) and a lack of control over what was happening on the phone (WinMo does decide to do its own thing from time to time) and you realize that even a 10% difference between two radios *might* not be significant.

That's why you would need many many trials with each radio. Tossing out the high and low 5%, averaging the rest will give you a more significant result.

It's a big leap to just run one trial and say "yep, that's it!"

As a test, just run the same test 5 times on the Telus ROM. Look at the results. Tell me if they are identical each time. My money is on "No". They may be close, but its very unlikely they are identical. Pick the WORST result, and then re-rank the radios. Is Telus still #1?

I understand what you're saying. However, I just do not have the time to run a ~26 minute benchmark routine multiple times (Say 5) on multiple Radios. (6) That would take 13 hours of being unable to use my phone for anything. Will I perhaps do it a bit at a time, running more tests on each Radio? Maybe, if time permits and interest holds. But for now, at least I benchmarked all the available Radio's at least once, providing a starting point backed up by empirical evidence for which is better and which is not. Can you show me someone else who has done this?