Steady illumination of a non-centrosymmetric semiconductor results in a bulk photovoltaic current, which is contributed by real-space displacements ('shifts') of charged quasiparticles as they transit between Bloch states. The shift induced by interband excitation via absorption of photons has received the prevailing attention. However, this excitation-induced shift can be far outweighed by the shifts induced by intraband relaxation and interband recombination, owing to: (i) an anomalous shift of quasiparticles as they scatter with phonons, as well as to (ii) topological singularities of the interband Berry phase. Both (i) and (ii) make the photocurrent extraordinarily sensitive to the frequency and polarization of the light source, and potentially lead to large non-linear conductivities of order mA/V^2 without experimental fine-tuning. A case study of BiTeI showcases these effects in a realistic material.