Astronomical data grow in quality and quantity, and yet many scientific questions cannot be answered by a single data set alone. While independent analyses are currently the norm, they miss out on substantial scientific synergies offered by the spatial overlap of current and future surveys. In my talk I will show how to build a coherent hyperspectral model from multiple data sets of the same celestial scene: imaging and spectroscopy observed from ground and space. I will introduce several cutting-edge methods from constrained optimization and machine learning. Combined, they form a framework for "data fusion". I will discuss its benefits for cosmological analyses such as gravitational lensing and studies of galaxy clusters, AGNs and the evolution of galaxies. I will conclude with an outlook on how the computing infrastructure of LSST and WFIRST can support data fusion for the entire astronomical community, enabling large-scale inference and discovery at an unprecedented scale.