I read an artical from Today's Parents. It said the children experts did a study on children and swimming, they found kids under the age of 4 somehow did not have the capacity to learn swimming(they meant really swimming), they said somehow the kids don't have the memory on how to swim if they were taught how to swim before that age. But parents can sent kids for lessons so they can get familar w/ the water and they won't panic when it comes to learn swimming later on.
I think kids should start learning to swim early. While in general kids may not be fully capable of swimming before the age of 4, they could be comfortable and familiar with being in water. Some kids do learn earlier, I have seen kids as young as 3 being able to float and do basic strokes.
If you are concerned with chlorine, you can take your kids to the Soloway Jewish centre, where they have a salt water pool. As for adults, you can also take private swimming lessons, all public pools in the city should offer this.