The situation is not quite so simple as you seem to suggest. Hyper-threading works by using a single execution unit but two parallel instruction decode pipes. This means that if one of the pipes stalls, the other can still feed the execution unit. The overall effect is a small but useful performance increase. It is true that the caches can thrash somewhat, but that's what good design is all about and you should see a performance increase even so. Personally I have never switched hyper-threading off.

The operating system does need to know how to handle scheduling on hyper-threaded CPUs, especially if your chip has more than one execution unit, but I make the assumption that any OS more modern that the CPU will be able to handle that correctly.

The hyper threading efficiency bump is closer to 30% on newer processors starting with the 9XX series released late 2008.

That sounds correct. Possibly up to 15% increase with HT with that CPU.
I too have a 3.4GHz P4 but I shut it down when HCC ended. It takes way too long to run the work units that are available now on a P4. It took ~ 6 hours to run HCC work units! If the current or future work units start running any longer than they are now on my Q6700s I'll shut them down too and just drop out of WCG.