Apart from the fact that snow tires have deeper tread than all season tires, the other big difference is the tire compound, snow tires are softer, therefore they stick more to the road. And for that reason, you shouldn't be driving with snow tires in warm weather (snow tires perform best with temperature of 5 deg. or lower), or the tread will wear out before next winter comes..
As to whether to change or not, it's really up to your driving habit and preference. If your all season tires are new, you are probably ok for a year or two, but as the tread wear down, it loses the traction. Think of it this way, with all season tires in winter driving, they'll wear out twice as fast as combining all season/winter tires, so you are not really saving much more money driving with all season tires year round. Besides, how much is one's life worth?