Your boss is not your mom. If he wants to put guilt on you for riding I say "screw him" happy employees make better employees. someone should point that out to him. I let my guys get away with murder but they are there when I need them and theres no complaints. If he values you as an employee he will be cool if not there no loss in leaving a job where you are not appreciated.