3 week old, does not sleep much during day, related to BF?

Hi,

I just want to make sure I'm not crazy.  Our 3 week old (March 16) EBF baby does not really sleep for any "stretches" during the day.  He'll sleep at the breast for some time (basically as long as I can be still), then wake up as I move about.  He will fall asleep at the breast and then I can sneak him into a bouncer or swing and he'll stay there maybe 40 minutes to an hour.  He does sleep well at night, 3 hour stretches or so, I think  (I don't really pay attention because we co-sleep and I just feed him and we both collapse again). 

He will take one really long nap in the afternoon, say 3 hours, usually in the swing.

Mostly I spend all day carrying him and nursing him.  I don't mind this terribly, but it would be nice to know there are some nap times coming. And my back hurts.  

I think that I rarely have any quantity of milk--when he did take a long nap one afternoon I pumped and got just shy of an ounce (total, both sides).  So I am thinking he has to just nurse and nurse and nurse to get what he needs.  He has lots of wet diapers and at least a few dirty ones every day, and he's gained back his birthweight and is visibly chunking up. 

I know there are growth spurts at around 3 weeks, but he's been like this since he got here.  I don't really know my question, but thought maybe you folks would have some encouragement/advice/shoulder patting. : )

TIA