Probably no one cares but I felt compelled to share anyways.
@NandaB I took your advice and moved the swing to our room so I could put him to sleep in it. He slept TONS better. I don't even turn it on. I have been nursing him several times in the hour before bed to fill him up. Then nurse him to sleep, wake him up slightly to burp, then put him in the swing a little drowsy. Night before last he woke up 3 times. Last night was 3 or 4, I can't really keep track.
I know that sounds like a lot but he is sleeping about 4 hours straight before his first wake-up and that is huge. After his first wake-up I put him in bed with me, where he nurses a couple more times throughout the night. I think the swing holds him tight so he can't move around much. He is a strange baby and hates sleeping on his tummy, so I think the constant rolling to his belly in his sleep was disrupting it. I think the straight stretch at the beginning on the night helps him to sleep better the rest of the night, which is why he isn't waking when he is in bed with me. All of this is just my theory of course...
@Stefibeth your response prompted me to do even more research on baby sleep, so I really feel like I have a plan of action now. I am feeling optimistic!
Now I just having to worry about weaning him out of the swing...
ETA: Fixed words
Re: Follow Up: Baby sleep help