Changes in temperature and relative humidity affect splitting tensile strength of concrete but the mechanisms are left unknown. This study aimed to clarify the mechanism through experiments using control mortar specimen and concrete specimens with two types of coarse aggregate of different shrinkage behavior mixed with the control mortar. Specimens were shaped in a thin plate for a faster equilibrium with the ambient drying conditions. Dying of different temperatures or relative humidities were maintained till the mass of specimens become constant. Then length changes, mass changes and splitting tensile strength were determined. It was shown that changes in strength of control mortar became more significant as drying conditions became severer while the change was not always monotonic. Compared with existing studies on shrinkage of cement pastes, the shrinkage of mortar specimens was supposed to be affected by the changes in strength of cement paste and by increase in strength as a result of multiple crack formation that occur due to fine aggregates. Changes in strength of concretes were affected by the change in mortar strength particularly at a relative humidity ranging from 0.80 to 0.43, where effects of fine cracks due to restraint of aggregate were imposed and strength reduction became more significant when coarse aggregate with a lower shrinkage was used. The restraining effect of coarse aggregate was confirmed also during heating higher than 40°C resulting in reduction of strength. It was concluded that the changes in strength of concrete during drying were mainly affected by both changes in mortar strength and restraint of coarse aggregate.