"Very seriously, it would be good if Rahul Gandhi had attended the Budget Session... That is something that we offered on a platter to the opposition," Tharoor said during a discussion on his latest book, 'India Shastra', with journalist Rajdeep Sardesai.
The former Union minister at the same time said he respected Rahul's move as he has "every right to make his decision".
"I am very confident that he would come back renewed and bring an agenda for change," Tharoor said.
The Congress MP, who had in the past angered his party by praising Narendra Modi and was even replaced as its official spokesman over the issue, meanwhile, flagged the contradictions between the rhetoric and the actions of the Prime Minister.
Tharoor said he found a lot of contradictions in Modi's speech and actions.
"Obviously, the speech to church leaders last week was too late to put in my book, but up until last week, he had used extremely negative language as well as specific actions which I have enumerated in the book of violence against minorities and communalism.
"The second contradiction is that he is advocating a very liberal agenda. If you listen to what he said, he seeks to fulfil that agenda on the back of a profoundly illiberal faith which is the Sangh culture," he said.
Tharoor said he felt that "this approach by Modi will flounder".
"You want foreign investment, you want to open up trade opportunities while the very followers you are dependent on for electoral victory are driving away foreigners and foreign investment and so on.
"Why will an Arab investor feel safe if the Muslims don't feel safe in the country. Why would the European want to come with large sums of money if the churches are being attacked. These are real questions. So his silence was actually undercutting his own image," he said as he stressed that one cannot give out different messages at the same time.
"You can't have one sort of message for RSS and the Sangh Parivar and try and give a different one for the rest of the country. In today's day and age, it does not work that way," he said.