Use the example for the Ipod from this link (
http://arstechnica.com/guides/tweaks/ipod-video.ars/5) and change the screen size to 240 X 320 (step eight).
For the file size, try around 350MB, make a couple of sample clips using different file size and see it on the phone. See which will give you a clear picture without too much pixelation.
Too high of file size will make the player un-stable when watching a movie. FYI.. if its a animation, use less (200 MB).
For removing encryption, I use AnyDVD from SlySoft
http://www.slysoft.com/en/anydvd.html. Its not cheap but its always updated. Search for other de-cryption program.
For the Nero Recode, its part of Nero Ultra Suite 6,7,8, and the new 9. If you have it, great.
If not, buy OEM from E-bay or mom & pop computer store (Got Nero 7 for $12 from computer store).
B4 you buy, make sure it has Nero Recode!!