Is it possible that your ancestor never moved, but where he lived changed? In early days of settlement county boundaries were sometimes in flux. And in urban areas, street names or numbers sometimes changed as occasionally did city boundaries. Was your ancestor annexed?
Just something to think about.