Oopsie

I accidentally told my boss he smelled like old books instead of complimenting his new cologne. Now every time we’re in the break room and a book is nearby, I panic that he thinks I’m insulting him again. How do I fix this? Did I ruin our working relationship with one ill-timed sniff?