A 3-bedroom townhouse in a good location costs about 250 - 270K so you can buy it without mortgage. That might help ease your worry.
The house may not be your dream house, but it is your first house. A good starting point. It should be one that can be used now, and can be sold easily if you have to, and can be rented out if you don't want to sell. Most first time home buyer often buy at this level and consider "use - sell - rent" when they buy.