Having physical access to technology, even if the devices themselves have been properly maintained and secured, increases risk dramatically, so physical security to your space still matters.
I suggest developing a strategy for verifying that personnel accessing your office are legitimate, as well as keeping a visitor log to know who came and went.
It’s also important to be able to track visitor movement through the office. Locking the server-room door becomes especially important when it’s across the hall from an office that people frequent. I have been to a number of offices and seen server closets just wide open, and I could have walked in and really done a lot of damage if I wanted to.
Think about everyone who has access to your office. No matter whether it is you, your staff, a vendor, or a visitor, ultimately you will be responsible for dealing with the consequences of potential security incidents. Make sure to consider ALL aspects of keeping your office secure, including physical security, in your disaster recovery plan and update it regularly as you change.