Depends on the definition of "looking hot". Some guys want Miss
Supermodel, some are contented with the girl-next-door.
Remember what I said about really stunning girls, some guys are
fearful of them no matter how much they admire them in the company
of fellow guys. When it's one on one, I've seen friends shrink
away. 
Nothing wrong with a book in hand what. Guess maybe you look
quite serious even at the bus stop or while walking around your
dept area, that's why they tend to avoid you?