I wouldn't worry about what someone else says about your accent - we all sound different to each other.
I moved across the country and I never thought I had an accent. (I thought a lot of the people where I moved TO had an accent) anyway, we were talking about accents one time at work and someone jokingly said to me "I got used to your accent"
Moi?! An Accent?! I just moved a few states to the left!

I've been called "dam# Yankee" quite a few times since I've been in Texas!