Interesting point, but this isn’t about your gender but about your doctor’s. You’d be potentially misgendering a third party in the eyes of the person to whom you’re speaking. To most people, your doctor included, their gender is relevant to who they are!