Haha, I'm pretty sure that's like, a level or two (or three or four) before what we're talking about here. You're talking about how to determine the key based on the number of flats out sharps in the key signature, which, in this setting, would mean you already know everything about the key except for the explicit letter name, which, at that point, is kind of irrelevant. However, if you do want to know those two rules (or, at least, the two I'm familiar with) for determining the letter name of the key signature based on the number of sharps or flats, here they are (for major keys).
Flats: The tonic (first note of the major scale) is the second to last flatted note in the key signature (order of flats = Bb, Eb, Ab, Db, Gb, Cb, Fb, or the circle of fourths starting on Bb). So I guess you could also define the letter name as "a perfect fifth above the last flatted note in the key signature".
Sharps: The tonic is a half step above the last sharped note in the key signature (order of sharps: F#, C#, G#, D#, A#, E#, B#). In other words, the last sharp in the key is the seventh scale degree, or leading tone, of the scale.
As for being able to determine the key signature, I feel like at a certain level (or maybe not, I never actually started transcribing until I'd already had a decent amount of theory knowledge and ear training, so I've always kinda done this), it doesn't have to be so abstract as "hum what you think is the tonic" (although, still kinda abstract). The first thing I do when I listen to a piece is I determine what the intervals in the melody are. Then, I see how those intervals conform to a major or minor scale (i.e. where are the half steps, how are the chord tones (1, 3, 5) emphasized, and whatnot). From there, I determine the tonic, and then I already have the intervals figured out, so once I check that note with a reference pitch, I already have the intervals sorted out, and at least the beginning of the melody/whatever line I'm working on is already finished.
...Okay, after typing that out, that sounds overly complicated. But it works for me, haha.