The first Ilam Hall was built by the Port family in the 16th century but this was demolished by Jesse Watts Russell to make way for his much grander hall of the 1820s. Most of the hall was demolished in the 1920s before Sir Robert McDougall bought the estate and donated it to the National Trust in 1934. Since then, the main remaining part of the hall has been used as a Youth Hostel and the grounds have been open to the public.

The manor of Ilam originally formed part of the estates of Burton Abbey but Henry VIII split it between three squires: the Ports at Ilam, the Meveralls at Throwley Hall and the Hurts at Castern Hall.

At the centre of the village there is a Gothic cross, erected, in 1840, in memory of Mrs. Watts Russell, whose family lived for generations at Ilam Hall. Her husband, Jesse, was a wealthy businessman. He bought the Ilam estate from the Ports, created a mansion with towers and turrets and rebuilt the village, between 1821 and 1826, to the spectacular design of John Shaw.
Some major parts of Ilam Hall were demolished in 1934. The remaining section is now a youth hostel, leased from the National Trust, who have owned the Hall since 1934. Together with a 50-acre estate, it was a gift to the nation by Sir Robert McDougall.

The Hall has associations with notable literary figures from the 18th Century. The family of the witty dramatist, William Congreve, (1670 -1729), lived at the Hall for some time. Congreve made a study out of a secluded grotto, in the woods, by the River Manifold and it is said that he there wrote much of “The Old Bachelor”, the comedy that first brought him fame, at the age of 18. A path along the riverbank leads to the grotto and Congreve’s stone desk and seat can still be seen.
Dr. Samuel Johnson was also a visitor to the Hall and is said to have derived the inspiration for the “happy valley”, in his novel “Rasselas(1759), from this same stretch of the Manifold. Johnson is said to have written the novel in the evenings of a single week to pay off his mother’s debts and funeral expenses.