The card takes a while to "power up" - coupled with the generally slow response from the phone (couple of reasons for that) it sure can happen that your phone gets the wakeup call from the tower and...
tries to power on, lumbers along for at least 2 rings, or if you are unlucky 3 rings on the other side ( you might have missed 1 ring just getting to the cellphone tower!! )... then finally does get powered on... only to have to wait...
up to 2-3 seconds for the SD card to power up, so it can open the sound file which, if it's a large file (say, 100KB or more...)
reads the whole sound file into memory (yes, we could be talking another second or two, and finally....
Oh hey, the call went to voice mail - never mind
You want to keep your phone lean and mean - your phone should just "pop" on when you press the power button - if it does not, start by seeing what you can do to speed that up.
Next, keep your ring tones on internal memory, and keep em small - a few KB, a few 10s of KB is fine. 100s of KB bad.
Lastly, test it - call your phone from a landline, count the rings until your phone actually rings in your other hand
If its very slow, try a soft reset and test again - if its faster right after a soft reset then you're probably low on program memory too.