In 1850 and after census records, have you compared your ancestor to his neighbors? Were they from the same place, about the same age, similar occupations, etc.? Or was your ancestor significantly different from his or her neighbors? It might be a clue.