I think people tend to get nervous about differences between genders because they believe that equality is the same thing as sameness. It's really not. Equality (at least in a social sense) means that one has the same standing as another instead of being lower on the totem pole. Sameness is not that, but rather making everyone exactly like each other.
I do not have the potential in general to be as strong as a man who is 6'6" and outweighs me by 60 pounds. In the same way I will in general not have the same potential for flexibility in my movements as a woman. Differences make the world an interesting, nuanced place. I say go for it and throw in some exceptions here and there in the game world to make it really interesting. I probably don't have the potential to beat, say, Holly Holm in a fight, but she is an example of an exceptional talent in fighting and strength. That doesn't mean more women than her can't beat me up, but the differences between us would most likely be an advantage due to our differences rather than a weakness due to them.
Take for example sword fighting. In sword fighting you want speed much more than you want strength. It doesn't take much force to cut flesh, but you have to be able to get to it first. Same thing with guns. Let weaponry and equipment be the great equalizer, but let the differences between gender and people make the world an interesting place instead of a cookie cutter bland pile of nothing.
Sorry, that got kind of long, but I hope it gives encouragement to go for what makes something real and interesting instead of bowing to cultural pressures of sameness. :-)