If you have a wood-burning fireplace, close the damper, that should stop the draft from coming into your house. But I'm not sure what can be done if you have a zero-clearance gas fireplace.
If you don't plan to use the fireplace very often. Why not use plastic film to cover the glass face, and use duct tape to tape the edge. Plastic film is the best insulation for this application. Please turn off the pilot light before applying this method.