A word of caution. While it's inviting, a pima cotton sweater vest is no attire for tree stump removal. I'd stick with double stiched cotton duck for breathability and durability.
I don't think it will be an issue with the current neighbors. My only fear is that a house could sell and the new owners would have it surveyed. But I'm not going to worry about it one bit unless something happened.
id have it surveyed . but first go and talk to your neighbors . its the right and neighborly thing to do . even though they didnt with you . If they give you crap then the gloves come off and do what ever legal action you find most pleasing .