Irrationally annoyed at contractor

We are getting quotes to have some home improvements done so I've had a lot of contractors in and out.    One of them had wandering eyes and noticed an envelope and brochure from a special needs child development center that was on our kitchen island.    My son wasn't particularly well behaved while the contractor was here so he made the assumption that my son has autism.  (He does not but he does have SPD and probably ADH, too early to tell on that one)

So rather than picking up on some social cues and getting the hell out of my house so I can deal with my tantruming 3 yo, he initiates a conversation with the opening line, "Hey, you want to hear something fantastic?  My DD is going back to school to get her master's so she can teach autistic kids.  Yeah, she was having a hard time getting hired so now she's back in school doing that.  She's student teaching right now."

So while he clearly thinks I should be impressed that his daughter is getting her AI certification, I'm secretly thinking, "Great, another teacher forced into special Ed due to the crappy economy, aren't her students lucky?" (sarcasm)

My mom is a retired teacher who taught at a very large school for kids with autism so I replied with, "Really, where is she doing her student teaching?". He replies, "Well, I'm embarrassed to tell you it's xyz town." It wasn't my mom's school but it was a less affluent school district. It was also the town I grew up in.   So now I'm pissed and not caring about being polite.  I tell him, "I don't know why that embarrasses you.  Don't kids in xyz town deserve a good education like anywhere else?"

I then told him I really needed to care for my son and I would be in touch if we decided to use his company. 

He REALLY rubbed me the wrong way.  Unfortunately, we really liked his company (he's not the owner, he's the sales rep). His product really did have some advantages and DH is sold on it so now I have to call him back and tell him he has the job.  I'm going to have to force myself not to call him an assh0le to his face.