This paper demonstrates the use of High Dynamic Range (HDR) fisheye images of the sky dome in lighting simulations. The objective is to improve the accuracy of simulations with site specific sky conditions. The luminance information stored at a pixel level in an HDR photograph is used to light the simulated environment through an Image Based Rendering (IBR) technique. The results show that image based sky models can provide a more accurate and efficient method for defining the sky luminance distributions and the impact of surrounding urban fabric and vegetation as compared to generic CIE sky models and explicit modeling of surrounding urban fabric and forestry.