* Australian govt in caretaker mode after close vote
* Major party leaders negotiating with independents
* PM Turnbull says still confident of forming govt (Updates with police potentially investigating Labor campaign texts, comment from independent candidate)
By Jonathan Barrett and Tom Westbrook
SYDNEY, July 3 (Reuters) - Australia's political parties began horsetrading on Sunday to break an anticipated parliamentary deadlock after a dramatic election failed to produce a clear winner, raising the prospect of prolonged political and economic instability.
The exceptionally close vote leaves Prime Minister Malcolm Turnbull's centre-right Liberal Party-led government in a precarious position, potentially needing the support of independent and minor parties.
It has also opened the door to the possibility, albeit less likely, that the main opposition Labor Party could win enough backing from the smaller parties to form government itself, although Turnbull said on Sunday he remained "quietly confident" of returning his coalition to power for another three-year term.
"I can promise all Australians that we will dedicate our efforts to ensuring that the state of new parliament is resolved without division or rancour," Turnbull, who accused Labor of waging a dirty tricks campaign, told reporters in Sydney.
Police said they were considering whether to investigate thousands of text messages sent to voters on Saturday by the Labor Party purporting to be from the state healthcare service Medicare, warning the service would be privatised by a coalition government.
Labor leader Bill Shorten said Australians had rejected Turnbull's mandate for reform like cuts to healthcare and a A$50 billion ($37 billion) corporate tax break over 10 years.
"What I'm very sure of is that while we don't know who the winner was, there is clearly one loser: Malcolm Turnbull's agenda for Australia and his efforts to cut Medicare," Shorten told reporters in Melbourne.
The election was meant to end a period of political turmoil in which Australia which has had four prime ministers in three years. Instead it has left a power vacuum in Canberra and fuelled talk of a challenge to Turnbull's leadership of the Liberal Party, less than a year after he ousted then prime minister Tony Abbott in a party-room coup.
If the coalition fails to form a government, it would be the first time in 85 years an Australian ruling party has lost power after its first term in office.
The uncertainty is likely to spook markets when they reopen on Monday, with analysts warning Australia's triple A credit rating could be at risk and predicting a fall in the Australian dollar and the share market. counting could take a week or more, and the coalition will rule under caretaker provisions in the interim.
Official electoral data for the House of Representatives showed a 3.4 percent swing away from the government, with about two-thirds of votes counted.
Electoral Commission projections give the coalition 67 seats in the 150-seat lower house, against Labor's 71 and five to independents and the Greens Party. A further seven seats were in the balance.
That leaves independents, whose election campaigns ranged from economic protectionism to anti-gambling and policies to improve the treatment of asylum seekers, as kingmakers.
Small parties are likely to hold the balance of power in the Senate, with Pauline Hanson's One Nation on track to win between two and four seats, marking the return of the anti-immigration activist to parliament after an almost 20-year absence.
LEADERSHIP DOUBTS
Confirming that talks were already underway, independent member of parliament Andrew Wilkie said Turnbull had called him just to "open lines of communication".
Stirling Griff, a Nick Xenophon Team candidate who is on track to win a senate seat, said the party was scrambling to pull together its wishlist for negotiations after being caught off-guard by the coalition's poor performance.
Speculation has also begun about the ability of Shorten and Turnbull to hold onto the leadership of their parties.
Turnbull was under the most pressure, having ousted Abbott on a promise of stability and then called the election in a risky bid to sweep out independents in the upper house who were blocking his economic agenda.
Turnbull had some of the highest poll ratings of an Australian leader on record shortly after he snatched the top job from Abbott in September.
But that popularity soured as he appeared to bend his centre-right values on issues like climate change and same sex marriage under pressure from the right of his party.