In 1826 the 27 people excluded from the Zion Baptist chapel formed a separate society, meeting in the chapel on Wicker Hill, formerly used by the Zion congregation. They later bought land in the Lower Courts, now Castle Street, and built a chapel in 1828 which, with galleries, could hold 150 people, but by 1829 the congregation was said to be 250. In 1833 a Sunday school was established with 40 pupils. Despite the initial success the congregation did not flourish and the chapel was sold to Jonas Purnell. It re-opened in 1844 and, under the pastorate of Joseph Rudman, the congregation revived and the chapel had to be enlarged. Land was purchased and the chapel was almost completely rebuilt in 1850 by William Smith. There were galleries around 3 sides, a vestry and a Sunday schoolroom and the chapel could accommodate 300-400 people. Rudman seems to have over extended the resources of the congregation and they were left burdened with debts. Rudman left in 1852. The graveyard was closed in 1856, when the cemetery was built, and the chapel closed soon afterwards. In 1863 Jonas Purnell sold the buildings to the Rev. J.D. Hastings, vicar of St. James's, and they were converted to the Church of St. Stephen.