"A dozen or so" Conservative MPs could support a vote of no confidence to stop a no-deal Brexit, defence minister Tobias Ellwood has told the BBC.
that many backbenchers and ministers would rebel if the UK faced leaving the EU on the 31 October deadline without a legal agreement.
Boris Johnson, the favourite to succeed Theresa May as Tory leader, says the UK must leave by that date, come what may.
If the government lost a confidence vote, it could trigger an election.
A vote of no confidence lets MPs decide whether they want a government to continue.
A 14-day countdown is started if a majority of MPs vote for the motion - and a general election will be called if, during that period, the government or any other alternative government cannot win a new vote of confidence.
The current government has a working Commons majority - its effective numerical advantage over all the other parties - of just four and depends on the backing of the Democratic Unionists.
This makes it highly vulnerable to defeat if a small number of its MPs side with Labour and other opposition parties.
Theresa May survived a no confidence vote in January after MPs rejected her Withdrawal Agreement with the EU for the first time. At the time, no Tory MPs backed the move.
But MPs have suggested this could change if the next prime minister - whether it is Boris Johnson or his rival Jeremy Hunt - tries to take the UK out of the EU on 31 October without a legally-binding agreement.
A no-deal exit would see the UK leave the customs union and single market overnight and start trading with the EU on World Trade Organisation rules.
Opponents say it would cause huge disruption at the borders and be catastrophic to many firms reliant on trade with the continent.
But it remains the default legal outcome unless the EU grants the UK another extension to the Brexit process.
Asked if Conservatives opposed to a no-deal Brexit had the "numbers", Mr Ellwood told Panorama: "I believe that absolutely is the case.
"I think a dozen or so members of Parliament would be on our side, would be voting against supporting a no deal, and that would include ministers as well as backbenchers."
Tory former chancellor and pro-European Ken Clarke suggested he would be prepared to bring down a Conservative government to avoid a no-deal Brexit.
A vote of no confidence "might trigger an election, it might trigger a change of government without an election under the law we now have", he told BBC Radio 4's The World at One.
When asked what he would do if Labour put forward a confidence motion, he replied: "It depends on the circumstances at the time...
"But I am not going to vote in favour of a government that says it is going to pursue policies which are totally incompatible with everything the Conservative Party has stood for under all those prime ministers for the decades that I have been in Parliament."
But Conservative MP Marcus Fysh, a supporter of Mr Johnson's, said such "negative" comments from his colleagues were "unwise".
He said Mr Johnson was the only person able to reassure MPs and the public that the UK was properly prepared for the eventuality of a no-deal exit and could make the best of it.
"Boris Johnson has not even had a chance to put these plans into action," he told the BBC's Victoria Derbyshire show.
"Once people see them coming forward, I think they will have confidence we can do this."
Brexit was supposed to happen on 29 March but the UK was given an extra seven months to try and get Parliament's approval for the terms of withdrawal, rejected three times by MPs.
In his Daily Telegraph column on Monday, Mr Johnson said of the new 31 October deadline: "This time we are not going to bottle it. We are not going to fail."
But Mr Hunt challenged his rival to reveal whether he would call an election, something he himself has ruled out, if MPs refused to allow the UK to leave without a deal on that date.
He said he feared a government led by Mr Johnson would rapidly collapse, because he would be unable to hold together a coalition of supporters that range from MPs who back no deal to others who feel it would be totally unacceptable.
"If you are not clear about exactly what you are going to do, that coalition will collapse immediately and you will have Jeremy Corbyn in Number 10," the foreign secretary said.
The Race For No 10 will be broadcast on Panorama on BBC One at 20.30 BST on 24 June.