The Johnson and Johnson Ad26.COV2.S single dose vaccine, designed as an emergency response to the pandemic, represents an attractive option for the scale-up of COVID-19 vaccination in resource-limited countries. We examined the effect of prior infection with ancestral (D614G) or Beta variants on Ad26.COV2.S immunogenicity approximately 28 days post-vaccination. We compared healthcare workers who were SARS-CoV-2 naive (n=20), to those infected during the first wave prior to the emergence of Beta (n=20), and those infected in the second wave (n=20), when Beta was the dominant variant. We demonstrate that a priming exposure from infection significantly increased the magnitude of spike binding antibodies, neutralizing antibodies and antibody-dependent cellular cytotoxicity activity (ADCC) against D614G, Beta and Delta variants. The magnitude of antibody boosting was similar in both waves, despite the longer time interval between wave 1 infection and vaccination (7 months), compared to wave 2 (2 months). ADCC and binding cross-reactivity was similar in both waves. However, neutralization cross-reactivity varied by wave, showing that the antibody repertoire was shaped by the spike sequence of the infecting variant. Robust CD4 and CD8 T cell responses to spike of similar or higher magnitude as those elicited by infection were induced after vaccination. In contrast to antibody responses, prior infection was not required for the generation of high magnitude T cell responses, and T cell recognition of the Beta variant was fully preserved. Therefore, Ad26.COV2.S vaccination following prior infection, even >6 months previously, may result in substantially enhanced protection against COVID-19, of particular relevance in settings of high SARS-CoV-2 seroprevalence. Furthermore, the dominant impact of the infecting variant on neutralization breadth after vaccination has important implications for the design of second-generation vaccines based on variants of concern.