The first castle on this fine defensive site is believed to have been erected by Harold Godwine in 1064 during his campaign to sort out one Gruffudd ap Llewelyn, a dissident Welsh Prince who had destroyed an even earlier fortification in 1052 at Old Radnor, which is how the two villages got their 'Old' and 'New' names. After the Norman invasion in 1093 the castle became part of the estate of the de Braose family, Marcher Lords who'd helped William during the invasion and had come originally from Bruis near Cherbourg. Things seem to have been fairly quiet here until 1195 when another dissident Welsh Prince called Rhys ap Gruffudd (aka Lord Rhys) started a revolt in which he captured the castle in 1196 from William de Braose, known as the 'Ogre of Abergavenny'. It was recaptured a year later and things returned to normal for the Ogre until 1209 when King John had him give it up to the Crown. This did not go down too well with the de Braose's and Reginald, son of the Ogre, with actual help from a Welsh leader called Llewelyn ap Ioworth (from 1216 known as Llewelyn the Great), recaptured it in 1214. He went on to marry Llewelyns daughter Gladys. The recapture did not go down too well with King John who attacked it in 1216 and ended up burning it to the ground. After the Magna Carta had put a handle on King John, Reginald rebuilt it and strangely changed his allegiances to support the new king Henry III against his former ally Llewelyn amongst others. In 1230 during the ensuing Welsh revolt Llewelyn took the castle and captured Reginald, his son in law. Reginald was executed. Llewelyn died in 1240 and the castle passed back to Reginalds son William, who died without a male heir. His daughter Maud got married to Roger Mortimer in 1233 so the castle passed into that families possessions. The town defenses are also of this time as by now a Norman Borough had grown to service the castle. These were paid for by means of 'murage grants' which were a tax on any goods bought into the town for selling. There were ditches and banks on three sides of the town and four gatehouses, the whole site being walled. Much of the earthworks can still be seen, particularly below the SW corner of the castle. Roger also had the castle rebuilt in stone in 1275. By this time most of this area supported the Crown and many men from these parts fought with Edward I on his campaigns around Wales which subdued the country and saw the building of the spectacular castles of north, west and south Wales.
The area seems to have been comparatively quiet again for the next 125 years. In the next and last Welsh revolt of 1401 - 1408, this time led by Owain Glyndwr much of Radnorshire was laid waste, including castles and towns at Bleddfa, Knighton and here. In 1845 whilst foundations were being dug out for the adjacent church a mass grave was discovered containing headless skeletons and a separate pile of skulls which are believed to be the remains of Edward Mortimers garrison of 60 men taken after the castle fell during Glyndwrs attack. This massacre was accompanied by the complete destruction of the town.
After this the castle was left to decay and in 1535 there is a mention of repairs to a gatehouse which was being used as the prison for the area. The castle was briefly repaired during the civil war and king Charles stayed here in 1642 during his tour to recruit men in this area. Parliamentary forces captured it soon afterwards and at this point it was dismantled.