Well, I figured out my problem.
My PRI was version 1.33_003 and after reading a bunch of different threads about PRI, it needs to be 2.x in order for AGPS to work.
Without AGPS I don't even get a sat lock after waiting 10-15 minutes, don't know what's up with that.
After updating to PRI 2.03_003 I can get a GPS lock in under 10 seconds.
I believe, though am unsure, that if you have AGPS enabled and don't have access to the AGPS servers (via a proper PRI) than the GPS doesn't work at all.
Hopefully this post is helpful to someone else out there.
Source of fix:
http://forum.ppcgeeks.com/showthread.php?t=29526