I keep mine in, I don't get too warm. I don't wear a balaclava when it warm though. Sometimes I wear a thin skull cap. I've tried it without and didn't like it.
I leave mine out. Personally I feel to claustrophobic when I have it installed. Never have had any issues getting cold. On the other and my husband has his in his helmet all the time.