You did not waste your life. You are labeling. Thinking errors such as this one could well be the cause of your depression.
Maybe you hate math because you were hoping to get the Fields medal and since you didn't get it now you are rejecting math, instead of math rejecting you?
When you get past your depression you'll figure out that the world is full of little pleasures, and that there is no need to qualify for your right to exist. You are free to enjoy life even if you are only average.
I heard that 80% of US mathematicians work for the NSA. Can you work for them?