Historians are unsure if ancient Sitia existed as a settlement on the site of the modern town of Sitia, although there have been found Minoan graves, Geometric and Greek figurines, Roman buildings, and a Christian basilica. It sure existed during the First Byzantine, the Arab, the Second Byzantine, and the Venetian Eras.
The Byzantine settlement of fortress of Kazarmawas surrounded by walls, therefore it was a castle. There are Venetian maps illustrating the Byzantine walls, parts of which have been excavated occasionally. These walls formed an isosceles triangle, which’s base was the beachside, starting next to the toll office of Sitia and finishing at the Kazarma area. The top of the castle was protected by the fortress of Kazarma.
The walls and the fortress were repaired in 1204, after the conquest of the city by the Genovese pirate Enrico Pescatore. Repairs were made also by the Venetians at various times. Such a repair was done in 1508, after a devastating earthquake.
During the invasion of Barbarossa in 1538, the walls of the castle suffered enormous damages and the Venetian rulers Paravizzni and Savorgan asked for their demolition. To not do so, the residents were forced to contribute financially (1500 ducats) so as to repair the entire defensive system of the city. But still, the seaside part of the wall was later severely damaged and could not be repaired, thus demolition was proposed again. But the reaction of Sitia’s baili (Venetian diplomats) Bemb and Gongara prevented the demolition of the walls.
In 1626, a few years before the Cretan War, the city was declared "defenseless". In 1630 a report submitted by the military architect Francisco Basilicata, to the person responsible for the defense of the island, captain-general Pietro Giustiniano, mentioned that the city's location was not suitable for fortification and security. However, since it was impossible to transfer the town, building new stronger walls should become a priority. Indeed, Venetian engineers were ordered to repair the walls, which were demolished, so as to rebuild stronger ones. However, it was the time when Crete was conquered by the Ottomans. The residents of Sitia were transferred to the only free castle, Heraklion, and a small garrison remained in Sitia till 1651.
The walls of Sitia were never rebuilt and the city ceased to exist for two centuries. In 1869 Ayni Pasha founded the new town of Sitia, which was named Aynie by the Turks, after him (this name did never prevail). Today the walls are invisible and the fort of Kazarma is the only remaining proof that Sitia has been one of the castles of Crete.