Humans are lazy. Not only physically, but more so mentally.
Ok, it's also that we like to know things and what to expect. People like to know what you are known for to get shortcuts, like "I can always go to person X for advice on Y".
Never get put into a box. You are you and only because people have a certain image of you, your value is more than that image. More importantly, don't give yourself that image. Nothing exists, we are all just a bit of dust that's gone soon anyway.
Without sounding too depressing, rather take it from a liberating point of view. Do you. Do what feels right for you. Don't be completely impulsive - or be (I like people being reliable and straight, but that's a different story).
Only because you are known of being good in finance doesn't mean you can't switch to a bouldering career.
Do you. There is no box. Only the mental box and we don't play that game.