From Wikipedia's entry on USB power:
Power
The USB 1.x and 2.0 specifications provide a 5
V supply on a single wire from which connected USB devices may draw power. The specification provides for no more than 5.25 V and no less than 4.75 V (5 V±5%) between the positive and negative bus power lines. For USB 2.0 the voltage supplied by low-powered hub ports is 4.4 V to 5.25 V.
[33]
A unit load is defined as 100 mA in USB 2.0, and was raised to 150 mA in USB 3.0. A maximum of 5 unit loads (500 mA) can be drawn from a port in USB 2.0, which was raised to 6 (900 mA) in USB 3.0. There are two types of devices: low-power and high-power. Low-power devices draw at most 1 unit load, with minimum operating voltage of 4.4 V in USB 2.0, and 4 V in USB 3.0. High-power devices draw the maximum number of unit loads supported by the standard. All devices default as low-power but the device's software may request high-power as long as the power is available on the providing bus.
[34]
A bus-powered hub is initialized at 1 unit load and transitions to maximum unit loads after hub configuration is obtained. Any device connected to the hub will draw 1 unit load regardless of the current draw of devices connected to other ports of the hub (i.e. one device connected on a four-port hub will only draw 1 unit load despite the fact that all unit loads are being supplied to the hub).
[34]
What they are saying is that you may only have 100ma available on each USB port (not enough to charge the phone). If you are only using one port per hub, then you may have the full 500ma available. Keep in mind there are internal hubs in the PC (check out your device manager if you doubt this).
So, if you are using one of your front-panel USB ports for something else, don't use another for charging the phone. Same with the back-panel ports. I usually use all back-panel ports for everything else, and only one front-panel port for charging devices. Even then, only one specific port up front will actually charge the device. You can sometimes see a special USB logo showing the USB icon with a solid/dashed line below it. This indicates a powered port.
This is also why some 2.5" external hard drives require a separate power adapter on some computers/ports but not others.