I guess there is not any wonderful horseback riding site around Ottawa area like what you want, I once checked a site by myself, there you can only allowed riding in a very small fenced square and have to be guided by their people, it would be so bothering.
There is one. It is around Russel area east of Ottawa, go 417 will take about 35-40 min. I went there once about 5 years ago but I really can't remember the name of the farm and the exact route to get there
If you don't mind waiting, I can send an email to some of my ex-ex coworkers to find out.
There are tons of stables around Ottawa Valley. You can look in the yellowpages or for God sake, use Google. Stable that we went is at the bottom of the Eardley Escarpment at Luskville Falls of Parc Gatineau.