The rights group said that the killings took place on 29 November close to the village of Shayalat al-Imam, located some 70 kilometres (40 miles) south of Mosul, the last IS-held Iraqi city that is the target of a massive military operation launched two months ago.
Iraqi security forces were present for at least one execution but did not attempt to intervene, HRW quoted residents as saying.
"The Iraqi government should make clear that government-backed militias don't have a green light to abuse or execute captives regardless of what they think they're guilty of," Lama Fakih, HRW's deputy Middle East director, said in a statement.
According to HRW, residents of Shayalat al-Imam said that the militiamen ordered them to assemble in an open area south of the village.
They saw militiamen kill a man named Ahmed, whose brother said he had briefly joined IS but then left the militant group and returned to his family.
Residents also said they saw the bodies of three more men who had been in the custody of the paramilitary group, but did not witness those executions, according to HRW.
The rights watchdog quoted a community leader as saying that the militiamen were from a group known as Hashd al-Jubur, meaning they were members of Iraq's Jubur tribe.
The Iraqi government turned to paramilitary groups that are now part of an umbrella organisation known as the Hashd al-Shaabi, or Popular Mobilisation Forces, in 2014 to combat a major offensive by IS that overran around a third of the country.
These forces - the main units in which are Iranian-backed Shia militias, but which also include Sunni Arab and Christian units - played a major role in halting the militant drive and later in pushing them back.
But they have been repeatedly accused of carrying out abuses included summary executions, kidnappings and destruction of property in the course of the war against IS.