there are energy audit programs available, masssave or something like that.
The auditor should be able to tell you what to do and what your best bang for the buck is.
Typically testing consists of blower door testing, thermal imaging, heating system testing and an inspection of the house as a system.
Typically the biggest bang for the buck is cutting down on air infiltration at the top (attic) and the bottom(cellar) of the house, lots more effect and way less expensive than new windows (shhh, I just saved you a bunch of money)
Next after you airseal the attic is increasing the thermal value of the insulation in the attic. R50 in cellulose is a good starting point, but airseal and do a follow up blower door first.
Now walls as dense pak cellulose.
Then windows and doors.
The heating system is a somewhat seperate issue, but if it's over 15 years old it probably should go.
Why are windows not worth doing? A little google research on "stack effect" will help you understand this. For the most part they are in the center of the height of the house and don't have a lot of pressure on them. This does'nt mean you can leave them open, but for the most part a old single glazed window with storm is only a little worse than a new window as far as heat loss and is a small portion of the envelope. For the price of a new window you can dump alot of insulation in your attic and walls. New windows are the easiest way for a home renovator to make a good profit and they have replaced tons of good windows and not saved people much for the $$$ spent.
Don't get me started on fiberglass insulation
http://infrared-energy.com/files/Spe...onProblems.pdf