Mattis, speaking during a press conference with Israeli counterpart Avigdor Lieberman, did not say how many chemical weapons Syria might have.
But he did hint that the US would strike the regime again if any more sarin gas attacks take place in Syria.
"The bottom line is there can be no doubt in the international community's mind that Syria has retained chemical weapons in violation of its agreement and its statement that it had removed them all," Mattis said.
"It's a violation of the United Nations Security Council resolutions, and it's going to have to be taken up diplomatically, and they'd be ill-advised to try to use any again. We've made that very clear with our strike."
Israel has backed US claims that the Assad regime still posseses a chemical arsenal.
Syria still possesses "a few tonnes" of chemical weapons, an Israeli military official has said, and almost certainly used them on its civilian population.
A chemical massacre on Khan Sheikhoun on 4 April killed over a hundred people and led to US air strikes on a regime airbase believed to be the launchpad for the bombing.
Syria claims to have handed over its chemical weapons stockpiles in 2013, under a deal brokered by Russia to avoid threatened US military action when another sarin attack took place.
Agencies contributed to this story.