Blatter previously said football clubs should be docked points or relegated if their fans were found guilty of racist abuse and violence, however he is now suggesting such measures would be difficult to enforce.
Blatter had said last January that slapping financial penalties on clubs or ordering them to play matches behind closed doors did not go far enough.
But in comments Friday he appeared to recognise the practical difficulty of enforcing harder-hitting measures.
"How far should we go? Where should we stop?" the boss of world football's governing body said at a meeting of the company Early Warning System, which monitors matches on FIFA's behalf to fight match-fixing.
"Can we bring an end to violence or racism by docking points or relegating a team? Or would such measures lead people to come to games to get the match abandoned," he said.
"We should do all we can but there's a danger that if we have matches replayed or if we punish clubs on the sporting front, it will open the door to hooligan groups who will come to deliberately cause trouble."
FIFA later released a statement insisting that Blatter had not backtracked on his previous stance but was merely asking rhetorical questions about the feasibility of implementing tough sanctions.
"FIFA President Blatter today reiterated … the need for punishment that hits hard against any form of racism," said the statement.
"As mentioned previously the FIFA President underlined that fines achieve little, but points deductions and exclusion from competitions would be effective measures. He also stressed that prevention must go hand in hand with punishment.
"He acknowledged that the implementation of these sanctions might pose some challenges, such as a potential risk that the system could be abused by some spectators for the only purpose of getting a very unfair advantage for their team.
"Such issues will be carefully examined and will in no way alter the FIFA President's very strong commitment towards the eradication of the racism scourge from football and his zero tolerance towards any form of racism.
"At no stage did the FIFA President say any words that could be interpreted as him backing away from his firm position to sanction actions of any form of racism."
Back in January, Blatter's remarks came after AC Milan's Kevin-Prince Boateng and his team-mates walked off a pitch during a friendly with fourth-division Italian side Pro Patria in protest at a solid hail of abuse from rival fans.
After initially disagreeing with Boateng's decision to take the laws of the game into his own hands, Blatter swung behind the player and said tough action was needed and football should show the way because of its worldwide following.
In November 2011 Blatter caused controversy by denying there was racism on the field of play, indicating that if abuse did arise at a match the issue should be settled by a handshake at the end of the match.
He swiftly apologised for using "unfortunate words" and insisted he backed "zero tolerance" when it comes to racism anywhere in the game.
Boateng was recently named a member of FIFA's anti-racism taskforce.
On Friday, Blatter also said that a proposal for uniform sanctions across all of FIFA's 209 member associations would be put to a vote at its congress in May.
"The same standard must apply for each and every league and national association," he said.