Take a good look at that tax list or pre-1850 census on which you found your ancestor. Are the names written in rough alphabetical order? Those people didn't only have neighbors whose last name started with the same letter as theirs; the enumerator was attempting to organize the information. The end result is that some of the sense of neighborhood one gets from a tax record or census is lost.