Thread: VS Benchmark
View Single Post
  #5 (permalink)  
Old 02-03-2009, 09:44 AM
schettj's Avatar
schettj
morsus mihi
Offline
Location: Not from around these parts
 
Join Date: Oct 2006
Posts: 3,017
Reputation: 3616
schettj is still contributing even after becoming a VIPschettj is still contributing even after becoming a VIPschettj is still contributing even after becoming a VIPschettj is still contributing even after becoming a VIPschettj is still contributing even after becoming a VIPschettj is still contributing even after becoming a VIPschettj is still contributing even after becoming a VIPschettj is still contributing even after becoming a VIPschettj is still contributing even after becoming a VIPschettj is still contributing even after becoming a VIPschettj is still contributing even after becoming a VIP
Mentioned: 2 Post(s)
Tagged: 0 Thread(s)
Re: VS Benchmark

Guys, all this shows is that the benchmark isn't valid

Post #31 at least tried multiple runs on the same rom/hardware, and various roms on the same hardware.

The only way to make the benchmark meaningful is to control for every variable (ie, every reg setting that changes the score)

Once we have a set of reg settings that we all agree on, and a test procedure we all agree on, then we can generate meaningful numbers (looks like +/- 1% across trials for normal variance) and compare them. That will tell you which rom OS is performing better or worse - likely the results will vary just above the noise (maybe 5% spread.)

Me personally, I don't care But seriously if you want to do this that's the way, and VreebieZ has generated the most useful data so far. Note how all his "good" results were in the ballpark of 2500 (+/- 50, or 2%) - you'd be REALLY hard pressed to notice one device being 2% faster then another as a user.
__________________
Reply With Quote
This post has been thanked 1 times.