Abe's government has made "womenomics" -- or boosting women's participation in the workplace -- a priority, as the country's workforce drops amid a rapidly ageing population.
But Koichi Hagiuda, a senior member of the ruling Liberal Democratic Party (LDP), touched off a firestorm on Sunday when he said men rearing children might be "unwelcome" for them.
"Children need an environment where they can stay with their mothers ... if you ask infants under three which parent they like more, the answer should be mama, even though there are no firm statistics to support it," said Hagiuda, 54, the LDP's executive acting secretary-general.
Those remarks came after another MP, Kanji Kato, doubled down on comments suggesting young couples should produce at least three children, saying he had received popular backing.
But the leader of the main opposition Constitutional Democratic Party lashed out on Tuesday, saying the comments were "intolerable."
"There are many people who cannot give birth to children despite wanting to and there are many single-father families," Yukio Edano said. "Don't they notice these facts?"
Sumire Hamada, from rights group Asia-Japan Women's Resource Center, told AFP that Hagiuda's comments were "out of the question."
"What happened to the government's pledge to build a society where men can participate in child-rearing?
"These comments overturn what the government has said, and I'm sure many fathers have been angered" by Hagiuda's "rude remarks," she said.
Another campaigner said the remarks could encourage men to persist in the long working-hours culture endemic in Japan.
Tetsuya Ando, founder of the organisation Fathering Japan, told AFP: "When he said children under three like mothers more than fathers, that's unacceptable."
"That kind of remark puts pressure on working mothers to stay at home while removing fathers' rights to rear children," said Ando, 55, himself a dad-of-three.