boss was being annoying af last night

my boss is like super chill during the day but at night he turns into a totally different person. it's like his normal self goes to sleep and some grumpy crabby version comes out. been working nights for months now, everyone else leaves around 10pm or so, and i'm stuck here til midnight making sure everything is running smoothly cuz we get deliveries late at night. so last night he just kept hovering over my shoulder and breathing down my neck all night long. asking me to check on stuff that doesn't even need checking, nitpicking every little thing. i tried to be professional and just do my job but it was so hard not to snap back at him or like throw something. finally around 11:30pm he asked if i had finished the inventory list yet, which i had done an hour ago but told him i hadn't cuz it made me look bad in front of other managers. i knew what i was doing but couldn't help feeling defensive and insecure because he always does this to me. so when i said no he got super frustrated and started yelling about how incompetent i am and how i don't take my responsibilities seriously. i wanted to tell him off right there in front of everyone, like really let him have it. but instead i just nodded and went on with my work pretending nothing happened cuz i know if i confront him he'll come back at me harder later when no one is around to witness it. deep down i feel bad for not standing up for myself more but at the same time part of me enjoys getting under his skin. it's like a twisted game and i want to win so bad.