On the first day of regional public hearings in Ballarat, Philip Nagle said he felt the church did not think the abuse and associated compensation claims were important and he was made to feel what happened to him was no big deal.
"I was intimidated and bullied into signing an agreement that was on their terms," Mr Nagel told the parliamentary inquiry into child sex abuse on Friday.
"They didn't care, they just wanted me to go away."
He said it was made clear to him that if he didn't agree to their terms any compensation claim would be tied up in the courts for years.
"It was even implied to me that what happened to me was not so bad," Mr Nagel said.
He said he signed the inadequate agreement under duress because the ongoing battle was "killing" his family.
Mr Nagle was sexually abused by his grade five teacher when he was a student at St Alipius Catholic Primary School in Ballarat in the 1970s.
The inquiry also was told that Victoria Police are investigating more than 30 suicides among former students of St Alipius.
Mr Nagel called on the "good people" in the Catholic Church to right past wrongs.
China: Cross removed from a third christian house of worship.