Upon Jackson’s death, his adopted son Andrew Jackson Jr. inherited the property. The following year, he began selling off small outlying parcels of land. He made some improvements to the property such as a new carriage drive and gates and a new fence around the garden, but did little to improve the farm or its finances. He tried to diversify his moneymaking ventures with an iron works and lead mine in Kentucky, but those efforts were unsuccessful. By 1853, mounting debts forced him to mortgage The Hermitage.
In 1856, Andrew Jackson Jr. sold a 500-acre section of the 1,050-acre farm, including the mansion and outbuildings, for $48,000 to the State of Tennessee. The State bought the property with the intent that it would be put to a public use, such as a school, but funding was unavailable so the State allowed the Jackson family to remain at The Hermitage as tenants. Between 1856 and 1861, the U.S. House of Representatives and Senate contentiously debated whether to accept Tennessee’s offer of The Hermitage for a branch of West Point, but ultimately rejected the idea. In 1857, Andrew Johnson proposed converting The Hermitage into an “Executive Mansion” for the governor. That year, Andrew Jackson Jr. sold the remaining 550 acres of The Hermitage farm to private buyers. In 1858, the Jackson family vacated the property and relocated to a cotton plantation in Mississippi, taking all but five of their slaves whom they left behind to serve as caretakers.
From 1859 to 1861, Tennessee politicians proposed several new uses for The Hermitage, including a State Military School and a model farm for the Tennessee Agricultural Bureau, but none came to fruition. In 1860, Governor Isham Harris became the first political leader to advocate for outright preservation of The Hermitage, but the looming Civil War prevented any such action. In the fall of 1860, Andrew Jackson Jr. and family returned to The Hermitage as tenants after their Mississippi cotton plantation failed. Only a handful of slaves returned with them. Although several important battles took place near Nashville and in the surrounding region, no military action took place near The Hermitage. In fact for the bulk of the war, the Union controlled Nashville and The Hermitage. During the early years of the Civil War, some Hermitage slaves left the property for freedom, so that by its end few remained.
Andrew Jackson Jr. died in 1865 leaving his widow, Sarah, to oversee The Hermitage. After the Civil War, Sarah and her son, Andrew Jackson III conducted a very small farming operation with paid day labor and tenant farmers. The Hermitage farm fell into disrepair and the buildings began a slow deterioration. The state government was without funds for rebuilding vital infrastructure, much less maintaining Andrew Jackson’s home. In 1865, Governor William G. Brownlow instructed repairs be made to Jackson’s tomb, and a survey completed for the entire property. In 1866, Governor Brownlow made several unsuccessful proposals for its use, including a public institution for invalid soldiers. The following year, the Tennessee Legislature authorized a public auction of The Hermitage; however, it never followed through.
In the 1870s and 1880s, as Nashville grew into a southern commercial center, increasing numbers of people, from newspaper journalists to wealthy Nashvillians, began to make excursions to The Hermitage. Tennessee politicians continued to explore options regarding the proper use of this state- owned property. In 1883, the State approved $350 for repairing the Tomb and building an iron fence around it. The state undertook no other action until 1888, a year after the death of Andrew Jackson’s daughter-in-law, Sarah Yorke Jackson, when the legislature proposed converting the Hermitage mansion into a hospital for invalid Confederate soldiers. This led to a public outcry for preservation of the landmark and ultimately to the creation of an organization of Tennessee women who fought to save The Hermitage.