Yah, the cold might be uncomfortable for some. For me, the trick
is to keep walking, don't just stand doing nothing. When you are
engrossed in something, you will not think of the cold.
Yah, it is always not enough time. Too many places to go. 
Did you go up the Peak? IMHO, the view is very spectacular at
night. Feels like Genting but with a spectacular view of the Hong
Kong skyscrapers.