There many factors to consider:
If you are building a garage, you might want the doors facing South if possible. The snow melts faster, and you will be protected from the cold north winds.
Building design is also a consideration; a good example of this is the Salt Box style, with the long rear portion of the roof facing North. Perhaps in younger years, you had to wait outside for the schoolbus. On those cold, windy days, I'll bet you stood with your back to the North and your collar up.
Be sure to choose the high ground; proper drainage away from the foundation is very important. Don't try to fight Mother Nature...you will never win.
If you are replacing an existing building, step back and "wipe the slate clean." Pick the best location regardless of what was done in the past; your predecessor may not have made the best choice.