Like anything its all relative. "Extreme sports" is just a buzzword created by the media to sensationalize things, and the X-games are an extension of that. One man's extremes are another's comfort zone. I spent many years in Vermont and Colorado skiing everyday possible and doing it in a way that some might consider extreme, but no one who participates in what ESPN would call an extreme sport ever refers to it as that, unless they're a wanker or full of crap. I certainly wouldn't consider skateboarding extreme, especially compared to something like climbing K2. And when I hear about guys donning wetsuits and swimming to rocks to surfcast, that is definitely something that is out of my comfort zone and that I consider a little crazy, but to the guys that do that sort of thing regularly, its just fishing. It's all relative to your personal experience, so why worry about giving it a label. Just my $.02.
