I defrag my card at least once every month or so.
I simply stick it into my mini-SD into the SD converter, put into my SD card reader and use my defrag program (I use Diskeeper). Sure enough, my card was over 70% fragmented.
For more information, read this article:
How do I defragment my Pocket PC memory cards? Do I need this at all?