Hundreds of grassroots referees have told the BBC they fear for their safety when refereeing and are dissatisfied with current measures to tackle abuse.
Some described being punched, headbutted and spat at.
Almost all the respondents had experienced some form of verbal abuse.
The president of the Referees' Association in England, which distributed the questionnaire to its 7,000 members, says the abuse of match officials is having a significant impact on their mental health and they are only ever "one decision away from a smack in the mouth".
"One day in this country a referee will lose his or her life. It happened in Holland a few years ago and they really changed their culture in football," said Paul Field.
The Football Association (FA) says a small minority of people abuse referees but such behaviour is "completely unacceptable".
It has pledged to "continue to do everything we can" to stamp it out, including stronger sanctions and the launch of a three-year strategy to help tackle the issue.
A trial of referees wearing body cameras in adult grassroots football is planned to start this year.
Field said anyone banned at grassroots level for abusing referees should not be allowed to attend any matches and called for the FA to inform elite clubs, with a view to preventing them from buying tickets for games.
He said some parents' behaviour on the sidelines was "shocking" while players often imitate the behaviour of top-level professionals who question decisions.
There were 122 referees aged 17 and under who responded to the questionnaire - more than 100 said they had been verbally abused by coaches/managers (105), spectators (109) or players (102).
*The figures in the questionnaire are not statistically representative.
Ryan, 30, from Lancashire, once had to hide in a back street for an hour after being chased by players.
"Every week when you go out as a referee, you think what's going to happen this week. I sometimes don't want to turn up. It's freezing cold and you're going to stand there for 90 minutes to be abused, for £30, which is what you get paid," he said.
"Without a referee turning up at the weekend, you're not going to have football and grassroots will eventually end up going into the abyss."
He called for referees to be issued with body cameras, which the FA is hoping to pilot in a trial this year.
Megan, 18, from Oxfordshire, had a parent come on the pitch with raised fists after she sent a child off.
"He was yelling abuse at me and saying this is why girls shouldn't be in football," she said.
"I think he got a six-match ban and fined, but that isn't enough.
"It was probably the scariest experience I've had when I've been watching, playing or reffing football. It really had an impact on my mental health, I was just worried all the time. I took two weeks off and worked my way back.
"All referees get abuse and I also get abuse because of my gender - I'm female in a male-dominated area. I've had people comment on my chest in the middle of a game."
Adrian, 59, Portsmouth: "In nearly 24 years of refereeing, I have been threatened, verbally abused, been told they know where I live, also been assaulted five times. Why I carry on I don't know - suppose the love of football."
Bill, 74, Leamington Spa: "I have been stopped from driving my car by players lying in front of my car and jumping on the bonnet. Quite a few years ago I was punched from behind and kicked on the floor. I don't look my age and have vast experience with a good reputation so I can handle myself. But younger refs are walking away."
Joe, 18, Romford: "I've been assaulted by a grown adult player when I was 16, threatened multiple times by managers and parents. A player at under-14 level threatened to have me stabbed and 'get his gang on me'. Managers have threatened to see me in the car park. Players have also said this.
"We need legislation at government level which offers us protection. The police need to take assaults on referees as criminal matters not just 'a football matter', and the FA needs to invest more and increase sanctions."
Jacob, 15, Essex: "I've had players at under-13 level swearing at me and parents undermining my authority."
Former Premier League striker Chris Sutton - whose father was a referee - was asked by Radio 5 Live to referee his first match, a junior game, to find out what it's like on the other side.
"Any form of abuse is totally unacceptable," said the former Norwich, Blackburn, Celtic and Chelsea forward who presents the BBC's 606 football phone-in show.
"It's an issue that needs to be dealt with, otherwise grassroots football won't survive."
Sutton has admitted to previously confronting a young referee while watching his son play and now wants to address the issue.
"I walked on the field because I was concerned about how badly injured he was. I shouldn't have done that. I realise how difficult it is," he said.
Sutton said players at all levels should stop swearing at referees.
"We see Premier League and other professional players do that on a regular basis. I've done it. It doesn't make it right and all players have a responsibility to tone it down."
A FA spokesperson said: "We have over 28,000 referees in England, and they are the lifeblood of our game.
"We understand the challenges that some of them face, and we have been very clear that all forms of abuse, whether on or off the pitch, are completely unacceptable.
"While it is only a small minority of people who behave badly to referees, this is still too many, and we will continue to do everything we can to stamp out this behaviour from our game.
"Through stronger sanctions, leading innovations and a new three-year refereeing strategy coming soon, we are determined to tackle this issue and build a safer and more inclusive environment for our match officials to have happy and fulfilling long term experiences as referees."
*BBC Radio 5 Live sent the questionnaire out via the Referees' Association in England to its 7,000 members.
It was also sent out to the Referees' Association of Northern Ireland and the Referees' Association of Wales but insufficient responses were received. There is no independent referees' association for Scotland and the Scottish FA declined to send it out to its referees.