Members of Madagascar's military stormed the presidential palace and overran it Monday evening. President Marc Ravalomanana was in another palace at the time, reports said. The military took over the palace in Antananarivo, the capital, about 6 p.m. local time (3 p.m. GMT), intensifying the turmoil over whether Ravalomanana stays in office, a government worker told CNN. Ravalomanana pledged Sunday that he would remain in office despite fierce opposition, and urged the country's military to not intervene. "I am aware of the feelings of anxiety that you feel in the face of intentions to seize power by force," Ravalomanana told a crowd gathered at the president's palace Sunday in Lavoloha, according to a statement issued by his office. He pledged to "remain faithful to the people and never leave them alone." An official with his office told CNN that the president said he would organize a referendum if that would help solve the crisis. The president's government has been at odds with opposition leader Andry Rajoelina, who recently was ousted as mayor of Antananarivo. Rajoelina has called on Ravalomanana to quit.