Along with being aware of what's using battery, and battery temp, consider using your OWM wall charger as it has a higher charging rate. USB chargers (don't know about car chargers) max at 500mAh. The wall charger has a higher rating.
As for why usage can affect charging time, heavy usage can bump the temp, inhibiting charging efficiency. And I'm fairly sure that all power to the unit is from the battery, not like a bypass while it's charging. So, if yer drawing power while you're charging power... you do the math.
Either way, the power off the charger (wall or usb) is finite, so it's either being used or stored.
When I want to maximize charging time w/o powering off, I doublecheck any unneeded apps are exited, turn off WfFi and bluetooth, dim the screen as low as it goes, and turn the screen off. (screen dimmed so if an event triggers, it's not burning power as a nightlight.) I don't enable the auto-power off on AC feature, as this was causing me lockups when docked to the PC and it powered down.