A Loyalist from the Mohawk Valley, New York, Nelles came to Canada during the American Revolution and from 1780 to 1784 served in the Indian Department. Following hostilities he settled near the Grand River but by 1792 had moved to this area. Nelles built mills and a store on nearby Forty-Mile Creek and soon became a prominent merchant. About 1798 he constructed a Georgian-style stone manor on this site. Modified by later additions, notably the erection of the Neo-classical porch in the 1820's, the house served as Nelles' residence during his lengthy career as a justice of the peace, member of the legislative assembly, and commanding officer of the 4th Lincoln Militia.