(Based on something somebody told me about the Halo damage/shield system, and some mud discussion.)
This isn't actually one system, it's a couple ideas that can work together.
A lot of RPG health systems are based on some variant of the d&d model, where you have X hit points which gradually get worn down, and occasionally you can use a power to regenerate them, but generally speaking they trend steadily downwards until death or the end of the adventure. This is paralleled by FPS design, where you run around Castle Wolfenstein shooting Nazis, losing health when they shoot you and picking up health packs when your health gets too low. If at some point you screw up and get hit a lot, then you're probably fucked — health packs are kept rare to keep things in balance, so you're going to be running around with low hit points (and hence in danger of getting instantly killed by one good hit) for a long time.
A bunch of modern games, on the other hand, have switched to the Halo model. There you have a much smaller health pool, so you never have a huge buffer between you and death, but you also have regeneration that kicks in when you stop being hit. So the result is that getting hit a lot doesn't mean "you're screwed", it means "stop fighting for a while and go hide" — in other words "that didn't work, try something else" without punishing you long-term for it.
Several RPGs have already adopted some version of this. 4e's healing surges mean your hit points are restored to full after each encounter; 7th Sea's flesh wounds go away at the end of the encounter; FATE has similar per-encounter and per-scene wounds. I don't think there's much of anything that works at a level smaller than the encounter, though, and that seems like it has some unexplored potential. Like, the question when modelling anime series is usually "why don't people use their big super-attack at the beginning of combat instead of the end?" This provides a system explanation — there's little point in using it at the beginning because they'll just regenerate the damage; much better to beat them down with smaller attacks, then use the big attack to finish them off. Alternately, you might want to use the big attack as a defense: if you're hurting bad, you can hit them with it, and force them to stop attacking and start regenerating, which buys you some time as well.
Another interesting thing to do with hit points is change the definition. A number of RPGs have "flesh wounds" and "dramatic wounds" where the former regenerate quickly and the latter regenerate slowly, and generally you don't lose the latter unless the attack exhausts or bypasses the former. That's pretty good, but what if you expand it beyond two layers? And what if you touch on other things besides just damage? Like, if you're a wizard, maybe you have a power pool along with your health pool, and as long as you have a power pool you can use your magic to deflect damage that would otherwise hurt your health. Power is easier to regenerate than health (say, you just have to meditate or tap into ley lines or something) but you also spend your power to fuel your attack spells, so you have to decide how much to spend when.
Ok, here's a little more concrete version of those. This system is designed to model James-Bond-like spies (books more than movies, but both). Generally a James Bond adventure has a bit where he's infiltrating a base either by stealth or deception, and occasionally fighting dudes, and then he's captured and beaten up, and then he escapes and fights his way out and wins. Ok, so, for this setup spy characters have three pools: Stealth, Fight, and Health.
When a player wants to do something, the GM lets them know what kind of pool is at stake: if they want to sneak by a guard, it's Stealth; if they want to punch the guard on the nose, it's Fight; if they want to charge the guard with a machine gun, it's Health. If they lose the (say) sneaking roll, that doesn't mean they're discovered — it just means they take some Stealth damage. If they run out of Stealth points, then they're discovered. Similarly if the character runs out of Fight they're disarmed, and if they run out of Health they're dead.
The character can regenerate Stealth when they're unobserved, can regenerate Fight when they switch to a new weapon, and can regenerate Health when they're in a hospital. So the overall effect here is to encourage games to follow the model: you start out sneaking around, but eventually run out of Stealth and get discovered; then you fight people until you get disarmed, and finally get captured and tortured to lose some Health. Then the bad guy leaves you alone and you start regenerating Stealth, which lets you find a sneaky way out of the death trap and then find new weapons to regenerate Fight, which lets you protect your wounded Health long enough to bust up the whole operation.
There might also be additional ways to regenerate these. For instance, maybe you can regenerate Stealth by making a bluff check when observed (you fast-talk the guard, or make them think the noise was only a squirrel, or whatever), but if you fail you lose even more Stealth. Similarly you could allow regenerating Fight by stopping to reload or just taking a breather (though this last makes Fight as easy as Stealth to regenerate, which is probably a mistake).
Oh, and I didn't mention it explicitly, but presumably the GM can't force you to get into conflicts where the stakes are Fight or Health if you have Stealth remaining — you're not discovered so they don't know where to send out the ninjas. I guess this means if you fight a guy when you do have Stealth points remaining, you might take Stealth damage even if you win.