So, Lebanon is going to the polls again on Sunday. Excited? No, me neither. Elections in the Middle East tend to follow a pattern as predictable as “Star Wars.” You turn up at the polling station, queue patiently in the heat, cast your vote, have your fingers marked with ink, go home, rejoice at democracy in action and, when you wake up the next morning, absolutely nothing has changed.

Iraq currently provides a nonstop political cabaret illustrating the problem vividly. On the face of it, the main gainers from last October’s national elections were the Sadrists. They might not have increased their total vote count, but these votes were distributed where they made most impact.

Their main opponents, the grouping of Shiite parties now known as the Coordination Framework, were dismayed — but not discouraged. They simply prevented a new government from being formed with a variety of stalling tactics straight out of the playbook used to malign effect by Nouri Al-Maliki back in 2010, including the co-opting of the chief justice and the cynical exploitation of ambiguities in the 2005 Iraqi Constitution. They have been helped by divisions among the two main Kurdish parties, the Kurdistan Democratic Party and the Patriotic Union of Kurdistan, which each demand the presidency for themselves.

The Coordination Framework has the impudence to claim it is acting in the name of democracy and constitutional propriety — on the grounds that no government should be formed that does not guarantee political control to the Shiite bloc as a whole. They object to the Sadrists’ attempt to exclude them and include independents and Sunni and Kurdish parties instead. They are supported in all this by Iran, which is worried about losing influence if Iraqi politics were to become genuinely more open and responsive to the real, material needs of all Iraqis, not just a small number of ideologically motivated and power-hungry stooges. So much for the national interest…

But this is exactly what you get if you have a system of “muhasasah,” known in English as consociationalism — the distribution of political representation along communal lines, as defined by self-appointed gatekeepers. Some will say that this is a practical way to keep communal tensions in check, by guaranteeing proportionate shares in political benefits to mutually suspicious groups. In practice, it guarantees corruption, political opportunism and a freeze on any positive political development.

And of all the countries in the region, the one with the longest experience of this slow-motion car crash is poor Lebanon. I understand the reasons for the National Pact of 1943 — and why the 1989 Taif Agreement failed to do more than tweak the representational framework to take account of demographic changes. After all, civil wars are exhausting and stopping them is always a priority. But consociationalism is not a long-term answer. It promotes the representation or well-being not of individuals or the community as a whole but of predatory small groups and their leaders.

Electoral democracy is a process not an outcome. It is the product not the cause of a political ideology.

Sir John Jenkins

In both Lebanon and now Iraq, it has produced professionally communalist politicians who make decisions not on the basis of voter intentions as revealed through elections but in negotiations behind closed doors with other elite groups whose main aim is to preserve their power and the access to state resources that this power affords — and that in turn supports the patronage on which such a system depends. This has provided fertile ground for external actors such as Iran to sponsor the growth of militias like Hezbollah in Lebanon and Asa’ib Ahl Al-Haq and Kata’ib Hezbollah in Iraq. In both places, they have blocked change and hobbled good governance and, in the interests of their sponsor, now effectively hold these entire countries hostage.

Where does this end? In Iraq, in massive economic inequality, environmental catastrophe (with chronic water shortages, agricultural failure and, in recent days, some of the worst sandstorms in living memory), wholly inadequate national infrastructure, lawlessness and the looting of state coffers.

In Lebanon, the same again, as vividly illustrated by the complete failure of accountability for last year’s Beirut port explosion, the collapse of the central bank, a disastrous economic situation and rapidly rising poverty.

If you look at the evidence of elections, social surveys and other opinion polling across the Middle East and North Africa since 2011 (or in Iraq since 2003), it is clear that many if not most Arabs — and indeed Iranians, Kurds, Amazigh, Tuareg, Turkmen, Armenians, Assyrians and Yazidis — want a say in choosing clean, competent, effective, accountable and responsible governments. The absence of such governments was a major driver behind the events of the Arab Spring.

But if you then consider the actual outcomes of these elections, you see a graphic illustration of the observation of the Italian Marxist, Antonio Gramsci — made in the context of 1930s Europe — that the new cannot be born, the old will not die and the struggle between the two gives rise instead to a variety of more or less morbid symptoms. More particularly, you see the continued grip that systems of tribe, clan, ethnic, religious, sectarian and other group affiliation have on the politics and sociology of the region and its constituent parts.

Neither Lebanese nor Iraqi elections have produced a permeable and removable class of politicians that represent the interests of their constituents to the best of their ability and judgment. They instead confirm in office a set of elites whose power derives not from the ballot box but from the accumulation of social capital, patronage and the purposeful construction of ethnic, communal or sectarian boundaries.

This story is repeated with variations across the region. Some observers thought that the Arab Spring would produce better and more accountable governance. Instead, it produced insecurity, social turmoil, the instrumentalization of religion, the rise of often violent identity politics and, where elections were held, unresponsive and corrupt confessional elites that looked very much like the old ones. And, as a result, in all elections in the region since 2011, we now see the slow ebbing of popular confidence in the ballot box in response to endemic and persistent problems of misgovernance, corruption and state capture. If voting changes nothing, why bother voting?

Electoral democracy is a process not an outcome. It is the product not the cause of a political ideology. In Europe — whose political liberalism is an exception to be explained rather than a normative rule to be exported — the electoral systems expressed in diverse ways in different countries are the result of a highly contingent set of historical experiences and are underpinned by an articulated ideology of individual rights and freedoms whose origins can be traced back to Roman and common law.

And in the West, modernity was a cultural before it was an institutional project. Successful electoral democracy requires the development of sustained habits of mind and social practices and a shared sense of the past and the future. It needs an acceptance that power can be transferred peacefully, a living memory of efficient and non-predatory state behavior, and an unintimidated civil society. It needs a common sense of justice and acceptance of the rule of law. And it needs strong, independent and impartial state institutions to arbitrate.

So, the real question is this: How do we think the conditions can arise in which functioning electoral democracy can arise and be sustained in Lebanon, Iraq, Tunisia or anywhere else in the Middle East and North Africa? At the heart of this is a question not about democracy but about the state and about governance. Most people want strong and accountable states that deliver security, prosperity, services and jobs. Political systems like those in Lebanon and Iraq have failed catastrophically to meet this desire. Sunday’s elections in Lebanon will not fix the problem. They will simply illustrate it.

Everywhere these systems persist, there is probably a majority of people in favor of something different. But first the existing systems must be swept away or — at the very least — radically changed. And that brings its own huge risks, especially in places where murderous militias are embedded. Nevertheless, there is perhaps some comfort to be taken in the courage of those people — often the young — who have taken to the streets over the past few years in Beirut, Basra, Baghdad, Sidon, Tripoli and Tyre to demand fundamental change. In Iraq, some have even got themselves elected. If their counterparts in Lebanon could come together around a single agreed platform, they might just make some progress. It will be slow, it will be hard and it will be dangerous. And it will need first to construct a strong and effective state rather than simply a set of Potemkin ballot boxes. But something has to give, doesn’t it?

• Sir John Jenkins is a senior fellow at Policy Exchange. Until December 2017, he was corresponding director (Middle East) at the International Institute for Strategic Studies, based in Manama, Bahrain, and was a senior fellow at Yale University’s Jackson Institute for Global Affairs. He was the British ambassador to Saudi Arabia until January 2015.