Low-voltage organic phototransistors (OPTs) are promising for optoelectronic applications such as photosensors and memory. In this paper, pentacene and Ta₂O₅ were used as the active layer and the gate dielectric, respectively. Commonly used polymer materials, such as poly(4-vinylphenol) and poly(methyl methacrylate), were spin-coated on the high transparency and high permittivity Ta₂O₅ layer as modification layers. The effects of the interface modification were investigated. It was found that the modification layers can block the electron injection process as the lowest unoccupied molecular orbital levels are higher than that of pentacene, and then reduce photosensitivity and memory effect. However, it was also observed that, for the modified device, the photoresponsivity, photocurrent/dark current ratio and retention time can be improved by modulating the optical writing process through varying a series of gate voltages accompanied by light illumination. Through analysis of energy levels, the mechanism of the tuning is proposed that the polymeric layer acts as a tunneling layer for high-energy electrons and a blocking layer for low-energy electrons. Our study verifies that interface modification benefits not only the OPT performance but also its applications in different fields.