outed at work

I haven't told most of my coworkers that I'm pregnant yet. . . . too much drama in the office that i'm in and I prefer to stay under the radar.  One guy that I work closely with knows because we occasionally have to carry heavy things, and I don't feel comfortable doing that at the moment.  I told him, and explained that I wasn't ready to tell everyone yet.  I was sitting in another coworker's office this morning with a few people. He came in and asked how I was feeling, I blew it off.  He then announced "you know, because you are 3 months pregnant."  In theory, it's not a big deal but I'm still kind of annoyed!