This is driving me nuts!! Mommy time?

I'm a WAHM, and DH takes care of DS for 5 hours every afternoon while I do some hourly work for one of our clients. DS is in the office with me during some of that time (DH and I share an office), maybe 2 hours a day but DH is watching him.   I generally feed him around 3pm at my desk and he is happy until 4:00 when things start to go downhill.  DS starts getting grumpy around this time... by 4:30pm DS will be in all out TEARS unless I am holding him.  He wants nothing to do with daddy, with his bouncy chair, with his toys, with anything!  His tears will escalate to screaming unless I go get him.  If I pick him up or hold him around this time he will immediately stop crying.  He requires being help by mommy until 5pm when work is over.  Sometimes he'll comfort nurse / doze (doesn't have a real "meal" until 5:30).)  I guess this is just "mommy time?"  Everyday it is a repeat of the same scenario.  It is very stressful all around, as I am trying to finish up projects at this time and DH starts getting really frustrated that he cant make DS happy.  I give DS TONS of attention all morning until 12...and give him as much as possible after work as well. I even make time to go give him hugs/kisses during my work hours. It is very sweet that he wants me but it is very stressful. Does anyone have any tips?