To the right of the path into the churchyard is a large motte or mound of Norman origin. There are traces of an oval shell keep (a shell keep was a wall around the outside of the mound. This was usually done because of fears about the weight and stability of a tower being built on top of the mound) and a gatehouse with twin round towers. The sites of these towers can still clearly be seen from some surviving masonry foundations. Not much is known of its history until 1287 when it was in the hands of a Thomas Brampton who was likely to be related to Brian Brampton, who had been empowered by various Marcher Lords to defend the Teme valley against the Welsh after incursions around 1215. Thomas may have come into possession of Byton around 1262-4 following further Welsh attacks under Llywelyn ap Gruffydd that resulted in the destruction of nearby Presteigne castle and this may be the time of its construction.