I've been your parents, sort of, and that's made this hard to read. I'm the critical one, the one who thinks people are doing things wrong, and I know intellectually that I'm probably driving my adult kids away with it, but the thought of just... not saying anything, not offering my perspective... it feels like abandonment on my end. Like if I'm not arguing for how I think they should live, then I'm complicit in whatever goes wrong. That's insane, I know. My therapist has pointed this out. But that might be what your dad is experiencing - not that he's being mean on purpose, but that his way of loving is broken and he doesn't know how to fix it.
Here's what I wish my kids would do: come less frequently, but when they come, be a little more generous about the small stuff. Let me be wrong about my opinions without defending yours quite so hard. That's not fair of me to ask, I realize. You've got every right to defend your choices. I guess what I'm saying is that reduced visits might actually be perfect - less frequent means less ammunition, less time for the pattern to fully develop. You're not abandoning him. You're probably actually helping him avoid deeper shame and failure by not creating situations where that pattern gets reinforced. Four times a year is still a lot.