This cross-sectional study investigated the association between different types of physical activity (PA) and occupational psychological and physical stress responses among workers in Japan. Stress responses were assessed using the Brief Job Stress Questionnaire. Work-related PA (time spent sitting, sitting bouts, standing, walking, engaging in heavy labor, and moderate-to-vigorous PA [MVPA]) and exercise-based PA (frequencies [times/wk] of flexibility and muscle-strengthening activity, and walking) were measured using a questionnaire. Multiple linear regression was performed to examine the association between each type of PA and stress responses. Participants who engaged in >108 min/d of work-related MVPA exhibited a statistically significant association with higher psychological stress responses when compared to those who engaged in 0–42 min/d of work-related MVPA. For exercise-based PA, participants who engaged in flexibility activity or walking five or more times/wk, or muscle-strengthening activity one to three times/wk, demonstrated significantly lower psychological stress responses compared to those who did not exercise. Participants who engaged in flexibility activity five or more times/wok demonstrated significantly lower physical stress responses compared to those who did not exercise. This study suggests that work-related MVPA is associated with higher psychological stress responses, while exercise-based PA is associated with lower psychological or physical stress responses.