I agree with "walk confidently" but I find making eye contact, smiling, nodding, or even saying "hello" while you pass does wonders to subdue people's suspicions.
It's more the reason why you avoid eye contact. Looking nervous while shifting eyes to avoid eye contact looks suspicious. Avoiding contact because you are otherwise distracted thumbing through forms in a clipboard makes it look like you have a purpose for being there.
I remember when I was contractor I had to get some screens from another building to kick off the project. They didn't have badges, just somebody at reception. I walked in, took screens almost $30000 worth, carted them out and left. This was my first day, nobody knew me, but nobody challenged me. Only afterwards I realized how crazy this was.
Looking lost or "looking for something" while wandering in is another great tactic.