It is uncertain whether ancient Sitia occupied exactly the same site as the modern town of Sitia. However, Middle and Late Minoan tombs, Geometric and later figurines and pottery, Roman buildings and an Early Christian basilica have been discovered in the area. The existence of a settlement during the First Byzantine, Arab, Second Byzantine and Venetian periods is considered certain.

The Byzantine settlement was surrounded by defensive walls and formed an organised fortified town. Venetian maps depict these fortifications, small sections of which have occasionally been identified. The enclosure was roughly triangular, with its base facing the sea. It began near the Roman fish tanks, beyond the present customs building, and extended uphill towards the Kazarma, where the two sides of the fortification met.

The name Kazarma derives from the Venetian term Casa di Arma. The building served as the barracks of the garrison and the administrative headquarters of medieval Sitia. It was not the entire castle, but one of the most important structures within the fortified town. In the early 13th century, the Genoese adventurer Enrico Pescatore repaired or reinforced the fortifications, while the Venetians carried out further works on several occasions.

Following the destructive earthquake of 1508, the walls were repaired, but they suffered extensive damage during the raid of Hayreddin Barbarossa in 1538. The Venetian officials Paravizzini and Savorgnan proposed their demolition, but the inhabitants raised 1,500 ducats for the restoration of the town’s defensive system. Demolition was proposed again at a later date, but the bailiffs of Sitia, Bembo and Gongara, prevented it.

By 1626, Sitia was regarded as virtually undefended. In a report submitted in 1630, the military engineer Francesco Basilicata stated that the location of the town was difficult to fortify effectively. Nevertheless, its protection was considered necessary, as relocating the entire settlement was impossible. Engineers and funds were sent to Sitia, and sections of the old walls were demolished with the intention of replacing them with stronger fortifications.

The works were not completed before the Ottoman advance across Crete. The inhabitants were evacuated, and many were transferred to Candia, present-day Heraklion. A small garrison remained in Sitia until 1651, when the Venetians finally abandoned the town and destroyed much of its fortification.

The walls were never rebuilt, and Sitia remained largely abandoned for more than two centuries. The modern town was founded in 1869 according to the urban plan of Avni Pasha and was officially named Avnie, although this name was never widely adopted by its Greek inhabitants. Today, the walls are no longer visible, and the Kazarma Fortress is the most important surviving monument of medieval fortified Sitia.