This is a fascinating short video from Numberphile interviewing a young woman who claims to have number-related synesthesia–when she sees numbers they are intrinsically linked in her mind to specific colors.
One of the questions the woman is asked is how the rest of us can know she is really experiencing this. According to the Wikipedia entry on synesthesia,
Although synesthesia was the topic of intensive scientific investigation in the late 19th century and early 20th century, it was largely abandoned by scientific research in the mid-20th century, and has only recently been rediscovered by modern researchers.[20] Psychological research has demonstrated that synesthetic experiences can have measurable behavioral consequences, while functional neuroimaging studies have identified differences in patterns of brain activation.
In college I knew someone who had a similar synthesia only he associated certain colors with musical tones.