Fewer fans are attending games for a fourth year in a row, and the reason might be more than a weather-induced blip.