The province Mylopotamos, one most mountainous provinces in Crete, hosted several fortresses. Two of the most important were the fort in the seaside village of Panormos and the fort Panokastri in Agios Mamas. The seaside village Panormos located 20km west of Rethymnon at the beginning of the 20th century was called Castelli Mylopotamos or Roumeli Castelli. It is believed that it is built on the site of the Roman town Panormos. Before the 20th century, when there were no roads connecting the Cretan cities, Roumeli Castelli was one of the most important commercial harbours of Crete, serving the entire Mylopotamos province.
Next to the pier of the village harbour and along the steep coastline, the Genoese pirate Enrico Pescatore built in the early 13th century the fortress Castel Milopotamo. When the Venetians conquered Crete, the fortress was restored and a settlement (bourg) started developing around the fort, later evolving into the current village Panormos.
The egg-shaped fort was oriented from north to south and was equipped with seven towers and two gates, one on the sea side and one on the south side. In 1341, the local family of Kapsokalives besieged the fort, in which Alexios Kallergis was hiding, but soon quited. Kallergis was a famous Cretan noble family, who gained special privileges from the Venetians.
In 1538 the pirate Hairendin Barbarossa destroyed Castel Milopotamo, but the Venetians restored it immediately, as it was located in a strategic position. Later the Turks, after having occupied the castle of Rethymnon, started moving towards Candia, the capital of Crete. Thus, when passing from Panormos, they besieged and occupied the fort in 1647. The Venetians tried to regain it by under the orders of the General Gildasi (Gil d 'Has), but they failed. Today there is almost nothing left from the fort except a small part of the wall forming a pile of stones next to the church of Agia Sophia.