I was watching a Phillies game from last season on MLB TV and the Philadelphia announcer said that the extremely rude treatment his wife and members of the Texas Rangers wives and families received in their section at the 2009 ALCS factored greatly in his decision to turn down the Yankee offer.
So then I looked it up and on the ESPN website, and it contradicted what the announcer said. Soon after the incident, Lee really downplayed it.
I think at the time he just wanted to keep the focus on the job at hand, but inside he was seething. He was also pissed off at the management and ballpark security for allowing this to go on. All security had to do was crack one guy over the head with a billy club and make a dozen ejections and that would have put an end to it. How this could happen to relatives, women and children of professional baseball players, with amount of security they have at Yankee Stadium, is beyond belief.
ESPN Article (http://sports.espn.go.com/new-york/mlb/news/story?id=5729471)

That, and the fact that his wife loved Philidelphia....she wanted back there in the worse way.