Where do you look when passing people on the street?

I usually look straight ahead, but I often sense the other person looking directly at me. Do some people expect you to acknowledge them while passing by on a public street?