This was established as a chapel of ease to St. James's in 1863 at the southern end of Castle Street on a site opposite the entrance to the present St. Stephen's Place. The church bought the former Bethel (Baptist) chapel and added a tower with spire to the building. The church closed in 1924 and was demolished in 1926, when the stones were used to build the Co-operative Hall (now demolished) at the top of Castle Street. The registers for christenings (1863 - 1920) and marriages (1864-1922) are held in the Wiltshire and Swindon History Centre.