Libya's Supreme Court has ruled that parliament's election of Prime Minister Ahmed Maiteeq a month ago was unconstitutional, state media reported, a decision which means his predecessor will stay on for now.

OPEC producer Libya's government and parliament are struggling to impose authority on a country awash with arms and militias who ousted former leader Muammar Gaddafi in 2011 but now defy state authority.

Maiteeq's appointment as head of Libya's interim government had followed a chaotic vote in parliament, which is divided between Islamists and more moderate forces as well as along tribal and regional lines, and some lawmakers and judicial experts disputed the outcome.

Outgoing Prime Minister Abdullah al-Thinni, a career army officer who resigned in April, had refused to hand over power to Maiteeq after some lawmakers questioned the validity of the vote and said he would wait for a court decision.

"The ruling stated... the appointment of Mr Ahmed Maiteeq as premier of the interim government was unconstitutional," television reported.

There was no immediate reaction from Maiteeq, who was elected by independent and Muslim Brother Islamists, but parliament's second deputy speaker Salah Makhzoum told reporters lawmakers would respect the ruling.

"From this moment ... Abdullah Al-Thinni is the caretaker prime minister until Congress (parliament) learns the court's reasons for deciding that Maiteeq's election was unconstitutional," he said.

Parliament will discuss the matter further on Tuesday.