Go to check your deed. it shows how far from your house is your land. For example:
front side of house, 10', end of your house, 8'6".
If your neighbour's fence pass this line. Tell him and ask him to remove by himself. You just sitting there, drinking tea and watching him to remove it. I think that's the best.
But if not, you can do nothing. Just be careful next time.