Historians are unsure if ancient Sitia existed as a settlement on the site of the modern town, although there have been found Minoan graves, Geometric and Greek figurines, Roman buildings and a Christian basilica. It sure it existed during the First Byzantine, the Arab, the Second Byzantine and the Venetian Eras.
The Byzantine settlement of Sitia was surrounded by walls, thus was a castle. There are Venetian maps illustrating the Byzantine walls, parts of which have been excavated occasionally. These walls formed an isosceles triangle, which’s base was the beach side, starting next the toll office of Sitia and finishing at the Kazarma area. The top of the castle was protected by the fortress of Kazarma.
The walls and the fortress were repaired in 1204, after the conquest of the city by the Genovese pirate Henry Pescatore. Repairs were made also by the Venetians at various times. Such a repair was done in 1508, after a devastating earthquake.
During the invasion of Barbarossa in 1538, the walls of the castle suffered enormous damages and the Venetian rulers Paravitsini and Savorgan asked for their demolition. To not do so, the residents were forced to contribute financially (1500 ducats) so as to repair the entire defensive system of the city. But still, the seaside part of wall was later severely damaged and could not be repaired, thus demolition was proposed again. But the reaction of Sitia’s baili (Venetian diplomats) Bemb and Gongara prevented the demolition of the walls.
In 1626, a few years before the Cretan War, the city was declared "defenseless". In 1630 a report submitted by the military architect Fr. Basilicata, to the person responsible for the defense of the island, captain general Pietro Giustiniano, mentioned that the city's location was not suitable for fortification and security. However, since it was impossible to transfer the town, building new stronger walls should become a priority. Indeed, Venetian engineers were ordered to repair the walls, which were demolished, so as to rebuild stronger ones. However, it was the time when Crete was conquered by the Turks. The residents of Sitia were evacuated were transferred to the only free castle, Heraklion, and a small garrison remained in Sitia till 1651.
The walls of Sitia were never rebuilt and the city ceased to exist for two centuries. In 1869 Ayni Pasha founded the new town of Sitia, which was named Aynie by the Turks, after him (this name did never prevail). Today the walls are invisible and the fort of Kazarma is the only remaining proof that Sitia has been one of the castles of Crete.