EX-tra Confusing

So I was just wondering. Wondering about the word "EX".
Why is it spelt with an E-x? When using it in the phrase of Ex-boyfriend or Ex-lover wouldn't it be more appropriate without the e? That way of spelling it makes it seem to me like it would be short for something else and not a stand alone word. Wouldnt it be more efficient if one was to just use the letter X in its place? That simple X would, in my opinion, better imply the fact that the subsequent word will be what is being "X"d out, what would be no longer viable, no longer significant, no longer included in the sentence or the speakers life. In a sense, that X would be implied to be writen over the following word, to show that it was once there but now is not.
Maybe people don't like complexities (and messy papers) so they put it in front of the word and added the "e" so it would be a proper english word being that it would then include a vowel. We don't want to have to say a-e-i-o-u and sometimes y and on one occasion x.
Makes sense to me.

You People

I was asked to define what it was I meant when I said “you people”.
Here is the definition according to Rebecca's Dictionary (Abridged Edition)

“you people” – lying, cheating, bastards perfectly capable of not only living a double life without remorse, guilt or shame but justifying it at the same time.

Thought I would clear up any confusion. Next time you are talking to me and I use that term now you know what I am referring to.