When I build my fence, the worker told me if we share with our neighbour then the fence can build on the property line of two house, otherwise, the fence need to build 6 inch behind the line, that is because if the other house want to build their fence then they still have enough space to have their fence on their property line. And the situation as what happened in your house is not special at all, sometime some people is so cheap. The worst is if they ask you to rebuild your fence back to your own property. Good luck.