This is an old chestnut. The current thinking on health (as opposed to fitness) is the waist measurement.
Anyway, my measure on fitness would be the time it takes your heart to return to its resting levels after vigorous exertion. That to me defines how fit you are.
The reason I don't consider weight is that a lean-looking person can still have a large amount of 'visceral fat' - this is fat around the organs in the abdominal cavity, not fat under the skin (pot belly fat). Hence, someone can still look thin but if they don't exercise then they may have a large amount of visceral fat that will cause health problems.
Any other definitions of 'fitness' (cue the comments about good-looking women...)?