Left-arm seamer Zaheer missed the second Test in Sydney that started on Wednesday due to a heel injury in his left foot.
Fast bowler Vikram Rajvir Singh will join the squad as a replacement.
"Zaheer will not be able to take part in the rest of the series. He will return immediately," a board statement said.
Vikram Rajvir, 23, has played five Tests and was last month called into the squad for the final Test against Pakistan after injuries ruled out Zaheer and Munaf Patel.
Zaheer's injury further weakens India's pace department already hit by fitness woes. S Sreesanth was ruled out of the tour with injury and the board had said Munaf needs match practice before playing international cricket again.
The visitors trail the four-Test series 1-0 after losing the opening game in Melbourne by 337 runs.