Hybrid sounds right. A human (or humans?) could also validate or invalidate algorithmic evaluations since the algorithms might miss sarcasm, context clues etc. If it's incentivized, it might reduce the hurdle of the high ask.

Still, I think natural language processing is far enough along, and there are open source projects that a dev could copy to automate something like this more easily than you suggest. Imperfect example, but: there have to be bunches of projects like this demo Tone Analyzer that uses Watson in open source repositories...

Something like this demo might not demonstrate completely accurate results, but wouldn't custom training input allow somebody to make a better model?