Recent observations as well as previous research show a 2-3% increase in the natural frequencies of Caltech's Millikan Library, a nine-story reinforced concrete frame and shear wall building, during rainstorms. It has been a mystery about whether changes in the building structure or the soil underneath cause the phenomenon. The Southern California Seismic Network (SCSN) installed a 24-bit 3-axis rotational sensor in the basement of the Millikan Library in 2008. It acts as an excellent data source to examine the mode shape changes that accompany frequency changes of the building. During the rainstorm period in late 2010, the authors ran several shaking tests of the building at its natural frequencies, and the ratio of the roof translational response to the basement rotational response was used to reveal the mystery. Experimental results suggest that the building concrete swells in response to moisture and cracks on the building surface are forced to close. The corresponding increase in the Young's modulus of the building structure causes an increase in the measured natural frequencies. Furthermore, when the authors watered the ground adjacent to the Millikan Library, no changes in the building's natural frequencies were observed. The irrelevance of "soil saturation" supports the suggested hypothesis. Finally, a 1-dimensional Timoshenko Beam model with soil-structure interaction is used to verify the proposed explanation for the mystery.