The group's abuses against Yazidi women have been well documented, but the watchdog said it had now recorded cases of arbitrary detentions, beatings, forced marriages and rape of women who have fled the IS-held town of Hawijah.
"Little is known about sexual abuse against Sunni Arab women living under [IS] rule," said Lama Fakih, deputy Middle East director at HRW.
"We hope that the international community and local authorities will do all they can to give this group of victims the support they need," she said.
Among those who have suffered under IS rule is 26-year-old Hanan, who was captured by the militants after attempting to escape the town with other women.
Hanan, whose husband had already fled Hawijah, was told his escape made her an apostate - and that she should marry the local militant leader.
But when she refused, she was blindfolded, beaten with plastic cables, suspended by her arms and then raped.
"The same guy raped me every day for the next month, without a blindfold, always in front of my children," Hanan told HRW.
The rights group said the issue was not given enough attention and that too little was being done to tackle the stigma that prevents many more victims from coming forward.
The Islamic State group proclaimed a "caliphate" across swathes of Iraq and Syria in June 2014. The group went on to commit mass rape and sexual enslavement of women from the Yazidi minority - who are neither Arab nor Muslim - claiming they were polytheists or devil-worshippers. Women from other minority sects were also made to suffer in the most brutal of ways.
The accounts collected by HRW often refer to IS fighters accusing the women of apostasy, or abandoning their faith, before their abuse.