In the end it is about acceptable risk. What is the amount of risk you are willing to accept for the convenience?

With the multitude and speed of the port scanners out there I do not think I would feel comfortable entrusting my system to a random number...at least not without some other layers of protection.