There are plenty of legitimate reasons Byron tends to fall short as a parent, including but not limited to having an affair with a college student and forcing his daughter to keep it a secret. This in turn makes him the world's biggest hypocrite when he finds out Aria is dating Ezra and subsequently tries to run him out of town. He wants to come off like a good guy but I'm not buying it.