Fermi-LAT observation revealed that the solar-disk gamma-ray flux is about 7 times higher than predicted during the solar minimum and it has a time dependence associated with the solar activity levels. While it is expected that the time dependence of the solar gamma-ray flux has relation to the time variations in the galactic cosmic-ray (GCR) flux and/or the solar magnetic field, the details are still unknown. It was observed that the latitudinal and radial dependence of the GCR flux changes with a solar activity levels by the heliospheric network of spacecraft, Pioneer 10/11, Voyager 1/2, Ulysses, and IMP-8. Therefore, there is a possibility that the GCR flux near the sun also may have a time variation and it may affect the resultant solar gamma-ray flux. I will present our results of the time variation in the GCR flux in the solar system calculated by using the drift model of the solar modulation, and discuss the possible effect of the solar modulation of GCR on the time variation of the solar gamma-ray.