Newer radio? (Telus?) - Page 3

SweetBearCub · #21 (**permalink**) 01-31-2009, 07:39 PM

Power Log #6 - Radio version 1.03.03V (VZW Radio)

- Stock Setup (Some programs loaded, but nothing major.)
- Charging via USB
- Power CPL set to not turn off during test
- LCD Auto-dim off, LCD at maximum brightness
- Keyboard open, sitting untouched on desk during test

Logs enclosed as attachments. Part 1 zipped due to exceeding file size limit.

Final radio ROM tested. Also, forgot to mention that ActiveSync was connected during all the tests (but unused until the end of the tests to retrieve the logs). Also, BT was off at the start of each test. The benchmark turned it on, I turned it off after each test.

SweetBearCub · #22 (**permalink**) 01-31-2009, 10:40 PM

Ok, after spending a bunch of time and getting cross-eyed from working with so much text, I've come to the conclusion that Radio ROM 1.04.05V (Telus) is the best out of all 6 available choices.

I figured this by finding the best high or low score for each category (17 in all) and assigning one point per ROM in each category. The Telus ROM scored the highest, 10 out of 17 points.

My files used to draw the conclusions are attached to this post.

Soprano · #23 (**permalink**) 02-01-2009, 02:37 AM

Hey just Curious is Any one Else is running the Telus 1.07.01V, I picked up a touch pro to replace my P4000 and this is what was running out of the box. Is this new? If not should I be upgrading...or downgrading?..to the recommended 1.04.05V? Thanks

Soprano

stevedusa · #24 (**permalink**) 02-01-2009, 03:00 AM

Quote:

Originally Posted by Soprano

Hey just Curious is Any one Else is running the Telus 1.07.01V, I picked up a touch pro to replace my P4000 and this is what was running out of the box. Is this new? If not should I be upgrading...or downgrading?..to the recommended 1.04.05V? Thanks

Soprano

Stay with what you get, because so far no one has a RUU-ized for that version so... don't flash unless you don't mind can't going back.

**schettj** · #25 (**permalink**) 02-01-2009, 10:27 AM

Quote:

Originally Posted by SweetBearCub

Ok, after spending a bunch of time and getting cross-eyed from working with so much text, I've come to the conclusion that Radio ROM 1.04.05V (Telus) is the best out of all 6 available choices.

Wow... that's a mighty BIG leap. I would have said

* any but #1 (Power Log #1 - Radio version 1.03.15F)

If you look at the data, #1 is the outlier. In all other cases the variation is less then 5% +/-, in many cases under 1%. Given you did single trials (you did single trials, right?) those differences are meaningless.

My guess is if you repeated each test 100 times, tossed out the 5 highest and 5 lowest scores, and averaged the remaining 90 (likely to be a statistically accurate average) these variations would even out to essentially 0.

So, don't use that first radio rom, its a dog. Use any of the other ones

It seems pretty accurate since #1 is the shipped sprint rom. I switched from that to #2 and it sure seems better overall.

SweetBearCub · #26 (**permalink**) 02-01-2009, 11:10 AM

Quote:

Originally Posted by schettj

Wow... that's a mighty BIG leap. I would have said

* any but #1 (Power Log #1 - Radio version 1.03.15F)

If you look at the data, #1 is the outlier. In all other cases the variation is less then 5% +/-, in many cases under 1%. Given you did single trials (you did single trials, right?) those differences are meaningless.

My guess is if you repeated each test 100 times, tossed out the 5 highest and 5 lowest scores, and averaged the remaining 90 (likely to be a statistically accurate average) these variations would even out to essentially 0.

So, don't use that first radio rom, its a dog. Use any of the other ones

It seems pretty accurate since #1 is the shipped sprint rom. I switched from that to #2 and it sure seems better overall.

It is true that the variations in the other ROMs than #1 are in most cases small. However - I am satisfied that the benchmarking program put each ROM through a significant enough test - It downloaded a large file for about 6 minutes and placed a 20 minute phone call for each Radio, monitoring all sorts of variables the entire time. As my conclusions show, other Radio's were better than the stock shipping Sprint Radio, but the Telus ROM was better than any of them. Do you disagree with my data? If so, why? I'm open to discussion here.

**schettj** · #27 (**permalink**) 02-01-2009, 11:29 AM

Quote:

Originally Posted by SweetBearCub

Do you disagree with my data? If so, why? I'm open to discussion here.

The problem with your data is statistical in nature.

You cannot control for variations in the aether, as they used to call it. You're testing a radio, and radios are subject to interference, and even (in the case of our beloved phones) contention with the tower.

Couple that with the lack of control over the internet itself (who knows if you had more or less retransmitted packets on one trial or another) and a lack of control over what was happening on the phone (WinMo does decide to do its own thing from time to time) and you realize that even a 10% difference between two radios *might* not be significant.

That's why you would need many many trials with each radio. Tossing out the high and low 5%, averaging the rest will give you a more significant result.

It's a big leap to just run one trial and say "yep, that's it!"

As a test, just run the same test 5 times on the Telus ROM. Look at the results. Tell me if they are identical each time. My money is on "No". They may be close, but its very unlikely they are identical. Pick the WORST result, and then re-rank the radios. Is Telus still #1?

SweetBearCub · #28 (**permalink**) 02-01-2009, 11:46 AM

Quote:

Originally Posted by schettj

The problem with your data is statistical in nature.

You cannot control for variations in the aether, as they used to call it. You're testing a radio, and radios are subject to interference, and even (in the case of our beloved phones) contention with the tower.

Couple that with the lack of control over the internet itself (who knows if you had more or less retransmitted packets on one trial or another) and a lack of control over what was happening on the phone (WinMo does decide to do its own thing from time to time) and you realize that even a 10% difference between two radios *might* not be significant.

That's why you would need many many trials with each radio. Tossing out the high and low 5%, averaging the rest will give you a more significant result.

It's a big leap to just run one trial and say "yep, that's it!"

As a test, just run the same test 5 times on the Telus ROM. Look at the results. Tell me if they are identical each time. My money is on "No". They may be close, but its very unlikely they are identical. Pick the WORST result, and then re-rank the radios. Is Telus still #1?

I understand what you're saying. However, I just do not have the time to run a ~26 minute benchmark routine multiple times (Say 5) on multiple Radios. (6) That would take 13 hours of being unable to use my phone for anything. Will I perhaps do it a bit at a time, running more tests on each Radio? Maybe, if time permits and interest holds. But for now, at least I benchmarked all the available Radio's at least once, providing a starting point backed up by empirical evidence for which is better and which is not. Can you show me someone else who has done this?

**schettj** · #29 (**permalink**) 02-01-2009, 04:15 PM

You are totally misconstruing me...

Its wonderful you actually bothered to test them all, and you confirmed my suspicions that the stock radio sucks turds, and all the other ones are pretty much the same.

My ONY issue is declaring a winner among all the others. You can clearly say the stock sprint rom is the loser, but beyond that, not so much. The way you've scored points is not unlike the way the electoral college works - "winning" a category by 1% earns you 100% of the point, so it "looks" like the Telus is a HUGELY better radio. Even if the results were statistically valid, it would be at best slightly better then any other radio that wasn't the crappy one. So slightly that it would be impossible to notice (as in, an extra few minutes of run time in a day.)

That's all.

SweetBearCub · #30 (**permalink**) 02-01-2009, 05:30 PM

Quote:

Originally Posted by schettj

You are totally misconstruing me...

Its wonderful you actually bothered to test them all, and you confirmed my suspicions that the stock radio sucks turds, and all the other ones are pretty much the same.

My ONY issue is declaring a winner among all the others. You can clearly say the stock sprint rom is the loser, but beyond that, not so much. The way you've scored points is not unlike the way the electoral college works - "winning" a category by 1% earns you 100% of the point, so it "looks" like the Telus is a HUGELY better radio. Even if the results were statistically valid, it would be at best slightly better then any other radio that wasn't the crappy one. So slightly that it would be impossible to notice (as in, an extra few minutes of run time in a day.)

That's all.

Ok. Can you suggest a better method of scoring? I am aware that the variances in the radios other than the stock Sprint radio are slight but they are there, nonetheless. Even if the difference is slight, I'd want the radio that had the edge in whatever category I desire. (Signal, temp, battery life, etc..)