This was originally a resting place for pilgrims on their journey to the shrine of St. Joseph of Arimathera at Glastonbury and it was sited in the vanished Hazelbury parish. In the 13th century Hazelbury was granted to Bradenstoke Priory whose Augustinian canons probably built the original chapel. Tradition has it that it was founded c.1235 by Sir John du Plessis. In 1340 it was rebuilt by Richard Plaisted of Castle Combe and a preaching licence was granted in that year. The building was of one storey with separate accommodation for men and women for mass but designed so that both sexes could take part in the celebration. A hospice was built next to the chapel and called the Bell from the bell cote on top of the chapel. This later became an inn of the same name and remained so until the latter part of the 19th century.
In the 15th century the chapel was heightened and a porch, with a holy water stoup, was built. After the dissolution the chapel was converted into flats by the addition of an upper floor in the later 15th century. The building was later used for storage and the upper floor was lost in the restoration of 1893. This restoration had only been partial and it was completed in 1930 by George Kidston of Hazelbury Manor. From that time weekly services have been held here and the chapel is also open on some afternoons.