For organic solar cells (OSCs), obtaining a high open circuit voltage (V_OC) is often accompanied by the sacrifice of the circuit current density (J_SC) and filling factor (FF), and it is difficult to strike a balance between V_OC and J_SC × FF. The trade-off of these parameters is often the critical factor limiting the improvement of the power conversion efficiency (PCE). Extended backbone conjugation and side chain engineering of non-fullerene acceptors (NFAs) are effective strategies to optimize the performance of OSCs. Herein, based on the quinoxaline central core and branched alkyl chains at the β position of the thiophene unit, we designed and synthesized three NFAs with different sized cores. Interestingly, Qx-BO-3 with a smaller central core showed better planarity and more appropriate crystallinity. As a result, PM6:Qx-BO-3-based devices obtained more suitable phase separation, more efficient exciton dissociation, and charge transport properties. Therefore, the OSCs based on PM6:Qx-BO-3 yielded an outstanding PCE of 17.03%, significantly higher than the devices based on PM6:Qx-BO-1 (10.57%) and PM6:Qx-BO-2 (11.34%) although the latter two devices have lower V_OC losses. These results indicated that fine-tuning the central core size can effectively optimize the molecular geometry of NFAs and the film morphology of OSCs. This work provides an effective method for designing high-performance NFA-OSCs with high V_OCs.