Sounds like you have roofs turned on. While it makes the game much prettier, it can be annoying. Turn off roofs in your graphics settings to fix your issue. Or, there is an option to selectively remove roofs, which you could try to see if you like.
You may think the buildings are annoying you wait till you get to Prifddinas where there are trees everywhere and boy are they annoying, they are very pretty I have to admit, but annoying, very annoying, very much in the way.