Are you sure the folders in question are empty? Do they exist elsewhere on your device or storage card? Remember, it scans for pictures everywhere.
On mine, the folders appear/disappear as soon as I create/delete them. In fact, I removed only the pictures from a folder and left the album.cache file. It still removed it from the albums list.
Try creating a new folder, copy a picture or two into it and see if it appears in the list. If so, remove the folder and see if it leaves the list. If this works, the other folders in question probably exist somewhere on your device or storage card. If it doesn't, you have something else interfering with Album.
If you can't get it, you can always tell Album to ignore the folders. Follow
these instructions on XDA and they will disappear. I didn't even have to reset my phone.