The Al-Aqsa Foundation for the Islamic Waqf and Heritage said around 140 Israelis toured the mosque, sparking physical and verbal scuffles with Muslims praying at the compound.
Israeli police spokesman Mickey Rosenfeld said five Palestinian citizens of Israel were detained for attacking the group, who were on a security-escorted tour for the Jewish Sukkot festival.
A Jewish Israeli citizen was also detained for refusing police orders to leave the compound, he said.
After a similar tour on Tuesday, at least 12 people were detained by Israeli forces, witnesses said.
The Al-Aqsa compound, containing the mosque and the Dome of the Rock, is the third holiest site in Islam and abuts the site where Jews believe the ancient Second Temple stood.