the only occasion I have met when the wearing of a seat belt was not good was in a boat yard, where the truck had a negative lift mast to lower boats to the water and lift them out. When asked the question, I was with a H & S Officer, and he agreed with me that, provided that a risk assessment showed that it was safer to be able to get out of a truck that had fallen into the dock, than to be trapped under water when wearing the belt, it was better not to wear a seat belt.
Otherwise, my advice as a trainer was for the operator to always wear the belt. If he didn't do so, he would be at least partly liable for any injury that occurred as a result of not wearing one.