True, when it's cold you can always get warmer... it's not so easy to cool down.
Drench yourself. Water evaporates => uses energy => you get colder. Recommended if your clothes can stand it.
And stacking layers when you are cold doesn't work too well in practise. First of all, once you start sweating, you're done for. Your inner garments keep the moisture, which sap heat from your body so it can evaporate. Second, it's not your clothes that keep the warmth, it's the layers of air between them. If your body can't warm up the inner layers, there's no help in adding outer layers. Once your inner garment gets wet, you'll have to swap it, or stay cold/freeze to death. If you don't move, so you don't start sweating, you don't produce as much heat either. Enter the windchill, and you're out. By the way, your feet produce sweat at any temperature anyway, so you're doomed to be cold on your feet if you don't move.
Then there's the "hate" factor. You see, once you become cold enough, you become sluggish. You move slowly, and act like you're half asleep (usually called "hate mode" in the Norwegian military). Even the simplest of tasks seems like a huge chore, and you stop caring about anything. Enter that mode, and you've got a foot inside the door to a downward spiral which ends in severe hypothermia (usually followed by death). It's a bit like dehydration in the heat, but a lot colder.
Frostbites are other nasty things. At -40 and windchill, bare skin will be frozen within two minutes. If you touch metal with your bare skin, it will only take seconds (yes, you can see the frostbites spreading, and no, it's not pleasant). Fun fact: frostbites never heal completely.
In short, I'd take the heat any day.