Speed 4 bypasses the resistor entirely.
The fact you're getting *any* slower speed with the resistor removed hints at a damaged cable harness.
(or broken switch, or weirdo ground path)
Depending upon your year, your resistor could be of either of these two hookups (usually they're the one on the left):
Where it sits in the overall circuit:
Where current goes from the resistor area itself:
In that last diagram, C220 is the feed from the REST and AuxHeater systems.. both of which will turn on the fan at (at least) the slowest setting.
--dick