The original vicarage was a large, elegant, gabled, typically Victorian one, which cost almost as much as the church itself. It was built in 1864 next to the church on the same site as the present vicarage, but farther from St. James’s Road down which it faced. A small grant was made by the Church authorities but most of the cost was borne by Revd. and Mrs. Fitz Wygram. The vicarage garden extended all the way down St. James's Road, which was ideal for the Sunday School infants for their games and tea. It was also used for St. James's Day celebrations.

St. James's Road in the 1890s

There was much controversy when Revd. Harvey decided to sell a large part of the vicarage grounds bordering St. James’s Road for building. This move on his part gave rise to much bitterness in the vicinity and several old-established families "left the church". In 1939 the residents of the large houses opposite were so disgusted by the invasion of their privacy that many of them gathered up their goods and chattels, held last regretful tennis parties and moved away in search of "fresh fields and pastures new". Of those that remained there was one "die-hard" remnant of the village gentry who was so deeply wounded that he never again walked on the side of the road occupied by the new houses, but crossed the road immediately opposite his own house and walked past the offenders with averted face. Upon his death, however, the unfortunate gentleman was wheeled on the church bier along the offending side of the road, which must have done violence to his feelings, always supposing that he was, at that moment capable of having any!

With the money gained from the sale, Revd. Harvey had the old, rambling and uneconomical vicarage demolished and in 1937 built a new more suitable one which is still in use today.

The clearing of the large, very overgrown and neglected hedge at the front of the vicarage in 1990 resulted in the front of the house being seen clearly for the first time in years.

The vicarage and its grounds do not belong to the parish but to a body known as the Parsonages Board which is responsible for the maintenance and upkeep of vicarages throughout the Diocese.