Not necessarily. I was out of Canada for a couple of weeks during maternity leave. I called the Service Canada EI Office, told them the reason why I was out of Canada, and the date I left, the date I came back to Canada, and they told me that my EI during these 3 weeks are not impacted at all. So if you have a valid reason that Service Canada will accept, your EI will not be impacted so you do not need to pay back the EI for that period.
So you might want to call Service Canada and ask them directly