Nope. You did right. And it is your call. No one else's. Consider this; they're 5 and 2 now. You hike, fish, spend quality time with them, and the lovely Starfish as well. You will miss these times when they are 11 and 8, and you are driving here and there from field to rink, or whatever. Even ith the older son; you are not putting him at a disadvantage by not having him skate/play now. (unless you want to be one of these idiot parents that project their offspring for Division 1 or even pro material from the age of 4.....
)