In 1651 Nathaniel Sylvester settled on Shelter Island which was purchased to serve the corporate needs of four business partners - Nathaniel, his brother Constant and two Englishmen Thomas Middleton and Thomas Rouse - to provision their sugar plantations in Barbados It is described by Sylvester Manor as being built with “six or seven convenient rooms”. The house also provided a home for Shelter Island’s first European settlers, Nathaniel and Grizzell Sylvester, and their twelve children, as well as the enslaved and indentured servants who worked on the property. Beyond a brief reference in Grizzell’s 1687 will, there are no known descriptions of the 17th-century plantation-era residence.
The original house remained intact until c. 1735, when Nathaniel’s grandson Brinley Sylvester won a long legal battle to claim his inheritance. He built a new residence close to the original home site. Though the new construction repurposed several beams, doors and other architectural elements from the original dwelling, Brinley’s house was a new, fashionable Georgian-style residence that was inherited by his two daughters Mary and Margaret. Mary Sylvester Dering, her husband Thomas and their children lived in the Manor House from the mid-1700’s with their enslaved people working the farm. Following the American Revolution and through the abolition of slavery at the Manor, various family members strove to keep the house and lands within the family.
