Hungary's populist Prime Minister Viktor Orban suffered a blow on Sunday in his revolt against the European Union after low voter turnout voided his referendum aimed at rejecting a contested migrant quota plan. Although a whopping 99.8 percent of voters backed his bid to reject the proposal, overall turnout fell well short of a 50-percent threshold. Only 3.3 million of the eight-million-strong electorate cast a valid vote, and the National Election Committee declared the referendum void after counting the ballots on Sunday evening. Opposition figures swiftly called on Orban to step down over the vote, after rights groups had accused him of whipping up anti-migrant fears despite there being only a few hundred asylum seekers in Hungary. But the firebrand leader downplayed the significance of the low turnout and vowed there would be "legal consequences" regardless. "Brussels or Budapest, that was the question, and the people said Budapest," he defiantly told supporters gathered in the capital on Sunday evening. "I will propose to change the constitution (which) shall reflect the will of the people. We will make Brussels understand that it cannot ignore the will of Hungarian voters." Orban did not reveal further details of the proposed amendment. "It looks like (Orban) wants to continue his fight with the EU on its migration policy, and the constitutional amendment is his way of doing that as it might trigger legal fights" with Brussels, analyst Bulcsu Hunyadi said. The fire-brand leader has emerged as the standard-bearer of those opposed to German Chancellor Angela Merkel's "open-door" policy, in the wake of the bloc's worst migration crisis since World War II. The EU migrant quota proposal — spearheaded by Merkel and approved by most governments in the bloc last year after antagonistic debates — seeks to ease pressure on front-line countries Italy and Greece, the first port of arrival for most migrants. But implementation has been slow. Eastern and central European nations vehemently oppose the plan aimed at relocating 160,000 people, many of who fled war in Syria. Neighboring Austria's Foreign Minister Sebastian Kurz said on Sunday the EU should stop clinging to its troubled plan. "The target is totally unrealistic," he told the German daily Welt am Sonntag, warning that disagreements over the plan could threaten "the cohesion of the entire European Union." Hungary has not accepted a single one of the 1,294 refugees allocated to it under the scheme and instead joined Slovakia in filing a legal challenge against it. Top EU officials had warned the referendum threatened to further split the quarreling bloc, already weakened by Britain's vote in June to leave the union — a decision Orban has blamed on the EU's handling of the migrant crisis. The referendum asked voters: "Do you want the EU to be able to mandate the obligatory resettlement of non-Hungarian citizens into Hungary even without the approval of the National Assembly?" In an editorial, Orban warned on Saturday that Hungarians had "a duty" to fight the failed "liberal methods" of the "Brussels elite." "It's true that the campaign was exaggerated but no-one can tell me if these migrants really are refugees of war," Zoltan, a 38-year-old lawyer and ‘No' voter, said earlier on Sunday. More than 400,000 refugees trekked through Hungary toward northern Europe in 2015 before Hungary sealed off its southern borders with razor wire in the autumn and brought in tough anti-migrant laws, reducing the flow to a trickle. Other countries on the so-called Balkan migrant trail followed suit, leaving some 60,000 migrants stranded in Greece. Many of those migrants live in grim conditions in camps dotted around the Aegean islands and the mainland, desperate to continue their onward journey. The EU said last week it hoped to relocate half of them by the end of 2017. A deal struck in March with Ankara to halt the influx looks shaky in the wake of a coup attempt in Turkey in July. German Interior Minister Thomas de Maiziere meanwhile said on Sunday that Berlin wants to reinstate an EU rule, suspended in 2011, to return asylum seekers who entered the bloc via Greece to be forced to return there. "We will take up discussions on this in a meeting with (EU) interior ministers" later in October, he told the Greek daily Kathimerini.