[ROM SHOWDOWN] [UPDATE 5/15] (RAM-Benchmarks-ROM) **Objective Analysis**
 

I am writing this thread to offer an objective comparative analysis of some of the most popular ROMs available at this site. That being said, there are a lot of ROMs available on this site and I know that I have not included all of them, though I will be adding more in the coming days and weeks. If you have a ROM that you would like tested, please send me a link to your ROM via a PM and it will be included in the testing. I have tested ROMs in four different categories: WM 6.5 + Sense 2.1, WM 6.5 + Sense 2.5, WM 6.5.5 + Sense 2.1, and WM 6.5.5 + Sense 2.5. I have also included information for the stock Sprint ROM (WM 6.1 + Manila 2.1). I hope to add some Titanium only builds to the mix in the near future.

Testing Methods

All ROMs were tested in the same manner which includes the following: Each ROM was flashed from Bootloader and allowed to run through customizations while still connected to the computer. Upon reboot, the phone was disconnected from the computer and available RAM and ROM were recorded as well as OS and Manila versions. Note that the available RAM was documented while Manila/Sense was active. Next the device was connected back to the computer and SPB Benchmark was installed. There were a few ROMs in which I was unable to install SPB Benchmark. For most of these ROMs, I received ActiveSync errors which is denoted by “No AS Connection,” and one ROM failed while running the benchmark test. The benchmark tests included everything but the “Built-in Applications” because the tests kept freezing while running the MS Word and PIE analysis. Thus, the test do not include everything that was tested on the IPAQ which is SPB’s standard device. Nonetheless, all the ROMs here were tested in the same manner so the benchmark scores are relevant. Finally, after the benchmark test were complete, the device was shut off so that I could document the boot time. On each trial, with the phone turned off and unplugged from a power source, I turned the phone on while at the same time starting a stopwatch (G-Watch). Time stopped when Manila/Sense finished loading.

*Note: the pagepool settings for each of the ROMs are the default that is set by the chef, i.e. I did not set the pagepool settings to be equal.

NEW RESULTS Attachment 69873

I am sure you will notice that there are fewer categories this time around. I have noticed that most chefs have switched to building primarily Sense 2.5 ROMs. Also, other chefs have left to other phones or platforms so there are fewer ROMs for that reason. This update is meant to include only ROMs that are currently in development. I realize that I have missed a few (including the ROM I cam currently using) and will be doing a few more to add to this list this weekend.

WM 6.5.5/M 2.5
Most RAM- MightyROM Full Force
Most ROM- Zone Chief's Choice
SPB Benchmark- MightyROM Full Force
Processor Benchmark- MightyROM Full Force
Shortest Boot Time- Icey ROM

WM 6.5/M 2.5
Most RAM- Lead ROM
Most ROM- JD's
SPB Benchmark- MightyROM Official
Processor Benchmark- MightyROM SE
Shortest Boot Time- Lead ROM

BEST OVERALL
Most RAM- Lead ROM
Most ROM- SuperROM
SPB Benchmark- SuperROM
Processor Benchmark- SuperROM
Shortest Boot Time- Blackout ROM


OLD RESULTS
Results (see attached .pdf for full table of results) Attachment 63314

WM 6.5.5/M 2.1
Most RAM- Mighty ROM Love
Most ROM- Mighty ROM Love
SPB Benchmark- MalROM v1.5b6
Processor Benchmark- MalROM v1.5b6
Shortest Boot Time- Mighty ROM Love

WM 6.5.5/ M 2.5
Most RAM- Baseline 002 RC
Most ROM- Baseline 002 RC
SPB Benchmark- MalROM Beta v1.6
Processor Benchmark- Baseline 002 RC
Shortest Boot Time- Baseline 002 RC

WM 6.5/ M 2.1
Most RAM- CFAY 2.0 RTM
Most ROM- MalROM v1.5b6
SPB Benchmark- MalROM v1.5b6
Processor Benchmark- MalROM v1.5b6
Shortest Boot Time- Silence_12

WM 6.5/ M 2.5
Most RAM- Medium Hill 2.5
Most ROM- MR. X 2.5
SPB Benchmark- MR. X 2.5
Processor Benchmark- MR. X 2.5
Shortest Boot Time- Mighty ROM War

Titanium
Most RAM- Baseline001
Most ROM- JD's Clean 6.5
SPB Benchmark- MalROM 6.5
Processor Benchmark- MalROM 6.5
Shortest Boot Time- JD's Clean 6.5


On a final note, I am able to run these test right now because I am not using my TP2 as my primary device though I will be again in a couple of weeks. That being said, I will do what I can to keep this updated until I start using my TP2 on a daily basis. When I switch again to another phone I will resume testing with another round of updated ROMs. I hope you enjoy the results and find the information useful. If you have any suggestions of things to add, remove, or do differently, let me know.

Here is an example of what the benchmark tests consist of:
Attachment 62657

Re: CDMA TP2 ROM Comparisons
 

good info

Re: CDMA TP2 ROM Comparisons
 

where did u find the 1st 3 NRG under 6.5.5 M2.5


found it

Re: CDMA TP2 ROM Comparisons
 

Very nice thanks!

Re: CDMA TP2 ROM Comparisons
 

mmm.... i love that available RAM :D

Re: CDMA TP2 ROM Comparisons
 

:occasion5:

Re: CDMA TP2 ROM Comparisons
 

I'm glad you are finding this information useful. I'll try my best to keep the list of tested ROMs updated and perhaps add a few other measures of comparison in the future.

Re: CDMA TP2 ROM Comparisons
 

A battery life comparison would be nice too, though I guess that would take a bit of time and would depend on a lot of factors.

Re: CDMA TP2 ROM Comparisons
 

This would be a nice comparison though I don't know how feasibly I could get it done. I could run benchmark battery test though that would take a lot of time.

Re: CDMA TP2 ROM Comparisons
 

cool. thanks.

Re: CDMA TP2 ROM Comparisons
 

now you post this after i'm on my fourth ROM in 24hours.
using mal's rom now, titanium only, over 100mb free on start up. will be adding spb ms soon.

Re: CDMA TP2 ROM Comparisons
 

any updated info?

Re: CDMA TP2 ROM Comparisons
 

Wow,Mighty 2.1 is the best.
Seems like nobody had a good luck with 6.5.5...
Thanks for this tread.

Re: CDMA TP2 ROM Comparisons
 

6.5.5 and sense2.5 is the toughest combination to build!;)

Re: CDMA TP2 ROM Comparisons
 

it honestly is...mighty has got a great balance though and I cannot replicate that to save my life.
I swear whatever mightymagic he puts on his roms...

Re: CDMA TP2 ROM Comparisons
 

Best bet is to stick with known working combination. But the problem is coming up with that combination! Takes a lot of flashing at times! LOL

Re: CDMA TP2 ROM Comparisons
 

There will be an update soon. I have tested several more ROMs as well as added some benchmark testing. I am not yet finished but will have the results up within the next few days.

Re: CDMA TP2 ROM Comparisons
 

Mlin,your attached image looked better,I think
Now I dont really understand meaning of those numbers.

Re: CDMA TP2 ROM Comparisons
 

paste into .txt file and attach?

Re: CDMA TP2 ROM Comparisons
 

You and me both, I couldn't even figure it out, and I knew what the table looked like. ](*,)

Thanks for the suggestion, I tried that earlier but it didn't work. Anyhow, I have found a solution, the full table of results is available in an attached pdf, though the category leaders are named in the first post as well.

Re: [ROM SHOWDOWN] (RAM-Benchmarks-ROM) **Mighty v NRG v Juicy and Many More**
 

nice write up!

are all these tests being done with the same pagepool setting?

Re: [ROM SHOWDOWN] (RAM-Benchmarks-ROM) **Mighty v NRG v Juicy and Many More**
 

Thanks! No, the pagepool settings are the default of each cook. I meant to mention that in the first post as I understand changing the PP will effect the available RAM amongst other things. I figured I would test each ROM exactly as it is delivered by the chef. Perhaps in the next round of testing I will set each PP to be the same and see how the results compare.

Edit- I will update the first post with the pp settings for each ROM within the next few days.

Re: [ROM SHOWDOWN] (RAM-Benchmarks-ROM) **Mighty v NRG v Juicy and Many More**
 

Not that I personally would want to spend the time doing it, but you could create a somewhat controlled test and measure the total discharge rate using this tool:

Measure and compare your battery capacity - easy, foolproof, comparable - any WM dev
http://forum.xda-developers.com/showthread.php?t=629348

Re: [ROM SHOWDOWN] (RAM-Benchmarks-ROM) **Mighty v NRG v Juicy and Many More**
 

Great idea, I will take a look into it, but no promises that I will incorporate it into the ROM comparisons.

Re: [ROM SHOWDOWN] (RAM-Benchmarks-ROM) **Mighty v NRG v Juicy and Many More**
 

This might be a noobish question, but what is that SPB score? Some people get 400 and then one is in the 1000's lol, I'm lost

Re: [ROM SHOWDOWN] (RAM-Benchmarks-ROM) **Mighty v NRG v Juicy and Many More**
 

Oh,yeah,looks good.Can you please replace my nd-8.5 results with Nd-V9 results

Re: [ROM SHOWDOWN] (RAM-Benchmarks-ROM) **Mighty v NRG v Juicy and Many More**
 

Benchmark scores are calculated based on the time the device takes to complete a variety of tasks. The numbers by themselves do not mean much and are really only relevant when compared to another device, in the case of SPB, that standard device is the iPAQ 3650.

Glad it looks better. I have updated the results with Nd-V9.

Re: [ROM SHOWDOWN] (RAM-Benchmarks-ROM) **Mighty v NRG v Juicy and Many More**
 

I would like to see a stock rom added for Verizon.....A question tho, do carrier provision's have any effect on rom performance? Say Verizon vs. Sprint? Or even once you get MMS working?

Re: [ROM SHOWDOWN] (RAM-Benchmarks-ROM) **Mighty v NRG v Juicy and Many More**
 

Ahh I see thanks, I kinda get it now lol but great tests, I might try out some other ROMs now based on this!

Re: [ROM SHOWDOWN] (RAM-Benchmarks-ROM) **Mighty v NRG v Juicy and Many More**
 

Can we PM you get our ROMs run through the gauntlet and posted for the world? :)

Re: [ROM SHOWDOWN] (RAM-Benchmarks-ROM) **Mighty v NRG v Juicy and Many More**
 

A comparison of the stock ROM's against the cooked ones would be interesting to see. I'd especially be interested in seeing some stock 6.5 w/ 2.1's against the cooked 6.5's w/ 2.5.

Re: [ROM SHOWDOWN] (RAM-Benchmarks-ROM) **Mighty v NRG v Juicy and Many More**
 

Very deserving of many thanks.

Re: [ROM SHOWDOWN] (RAM-Benchmarks-ROM) **Mighty v NRG v Juicy and Many More**
 

Please do. I am going to try and test several new ROMs this weekend as well as update some of the ROMs I've already tested with newer versions. I will also try to post some Titanium only builds as well.

Re: [ROM SHOWDOWN] (RAM-Benchmarks-ROM) **Mighty v NRG v Juicy and Many More**
 

what about stability?

Re: [ROM SHOWDOWN] (RAM-Benchmarks-ROM) **Mighty v NRG v Juicy and Many More**
 

Stability isn't so easy to test in an objective manner. I can say that I have used NRG, MalROM, and Mighty extensively and stability amongst those three is great.

Didn't have a lot of free time so I did not do much additional testing this weekend.

Re: [ROM SHOWDOWN] (RAM-Benchmarks-ROM) **Mighty v NRG v Juicy and Many More**
 

my test of stability:

how long before a restart is needed to correct ANY kind of issue

Re: [ROM SHOWDOWN] [3/14] (RAM-Benchmarks-ROM) **Mighty v NRG v Juicy and Many More**
 

With each of the ROMs that I mentioned that I used regularly, I only soft reset upon installing new software that required it. Again, its a difficult test to perform with little bias. The test I have done are very controlled and all done in the same way and do not require a lot of time spent with each ROM so I can churn out results quickly.

Re: [ROM SHOWDOWN] [3/14] (RAM-Benchmarks-ROM) **Mighty v NRG v Juicy and Many More**
 

either way though, this puts things into perspective. Mr X's 2.1 rom is great, and after seeing this chart i see that not only is it NOT my imagination, but perhaps I should give Mr. X 2.5 a try.

Listen man, even if people want stability tests, what you did here is a nice step in the right direction. If you want to take time to test such a variable characteristic, by all means do so, but from what you've done here alone, you've more than proven yourself valuable to this community.

Re: [ROM SHOWDOWN] [3/14] (RAM-Benchmarks-ROM) **Mighty v NRG v Juicy and Many More**
 

Thanks mlin! :) its cool to see how your ROM stack up against others from an unbiased perspective. the question now is, what exactly is SPB benchmark score (I know was SPB benchmark is, I have run it myself..) and why is malrom like twice as high as all the others? seems like when there is one piece of data that is totally inconsistent with the others it should be reconsidered?

I'd like to improve by score in the benchmark category but to get nearly TWICE as high seems impossible. I was thinking maybe a few percent...

edit: it will be impossible to test stability

Re: [ROM SHOWDOWN] (RAM-Benchmarks-ROM) **Mighty v NRG v Juicy and Many More**
 

Thanks, I'd be curious about the titanium ROMs as well.