One thing I noticed just in reading this thread is that many chefs and "vets" are getting defensive. The first step of progress is to realize nothing is perfect and there is always room for improvement, regardless of if that source of improvement lies in community feedback or your own doing. This guy wants to improve the ROMs out there, and I think everyone else does too for the greater good. Of course, making a community ROM is not feasible (everyone has different tastes!), but there seem to be some things that could be improved upon.
That being said, here are a few of the problems I've noticed in my extensive lurking, some of which may not be detailed in the OP (but felt by many):
1. Organization of ROMs. Everyone has their own spin on how to display a ROM's details in a thread, show screenshots/features/etc., and the threads are generally big cluster****s. Like many have mentioned, there is competition present, so some chefs organize their thread posts better than others, but some of the spatially-challenged chefs make it difficult to evaluate. Perhaps a post standard / structure can aid this process for users to evaluate which features are present in the ROM, resource details, screenshots, etc. No information should go in a ROM topic other than ROM-specific details; i.e., CABs should stay in a separate thread/forum because they can really make a thread chaotic.
2. Performance statistics. Obviously there are reference points like pagepool, RAM consumption, etc., but these ROMs don't have benchmarks or tools to really TEST speed. I don't know enough about the technology to know what is feasible here, but I think this is an important datapoint missing from the decision-making process for new users. Saying your ROM is fast but still having different sections of the OS bog down does not mean it is fast... In fact, I'm currently using a supposedly complete ROM right now that takes SECONDS to navigate between certain menu items in the contacts manager (all the while the chef claims everything on it is super fast). Know that I'm providing constructive criticism when I say this, but that claim seems unacceptable and is misleading.
3. Community feedback. There's currently no way to know how highly rated a ROM is by the community. You can attempt to gauge by number of posts in a ROM's thread, but this is by no means an accurate or easy-to-use reference point for ratings/reviews of a ROM. I'm not sure how to rectify this aspect of the ROM scene, but perhaps this can only be solved by a third-party ROM collaboration site or a survey monkey review link in each ROM's OP or something (sorry, I have a market research background
).
The bottom line is, not everyone has time to go in and test all these ROMs out. Everyone GREATLY appreciates the time these chefs put into building custom ROMs (and most reasonable people should show support through donations), but there is always room to improve the ROM distro/eval process for newcomers.
One last note--obviously, some chefs are better at some of these than others are, but the focus should be a standardization of the presentation of ROMs (not the ROMs themselves)...