Sounds like your battery is delivering the power. Some heat is inevitable when you draw lots of amps, provided its not too hot to pick up I would consider this normal. if by hot you mean ouch then you drawing too much current. Have you tried a smaller prop 40mm is quite big for a 500 size motor, adding a gearbox reduction will reduce the amps but also the prop speed so it may not help. A 35mm may be better or even a 30mm, if you are in a club perhaps you can borrow some different sizes and pitches to find the best combination.

Dave,
motor and esc water-cooled but battery hot to touch.

HI phishedin
Welcome to the site.
A smaller prop will reduce the load on the motor and allow it to reach peak performance. is everything very hot when you finish a fast run? Assuming your battery and ESC can provide the maximum power quoted 9.6v x 20 amps then your motor would need to be able to stand 192 watt. I think it unlikely that the ESC will deliver much over half its rated output for any length of time so 100 watts is more likely. if you have or can borrow a wattmeter you will be able to verify the actual values with different props. it may be that you need to upgrade your motor to achieve greater speed. GiantShark (ex GiantCod) sell suitable wattmeters. For ultimate speed then a brushless set up will deliver, at a cost.
Good luck with your rebuild, please post a follow up with your progress. 😀