The effects of mutations often depend on the genetic background in which they occur. For example, a mutation that is beneficial in a lab strain of yeast S.cerevisae may be deleterious in a vinyard strain. This dependence is often called “epistasis”. Some types of epistasis lead to qualitatively interesting evolutionary phenomena like speciation, but other types have only a quantitative effect on evolutionary trajectories. How prevalent are different types of epistasis? To approach this question, we start evolution from different strains of S.cerevisae that are either closely or distantly related. In both cases, we find that the initial genotype has a strong effect on the rate of adaptation, but surprisingly little effect on the pool of further adaptive mutations. So far, we discovered only one mutation that substantially changes the spectrum of further adaptive mutations. Other than this single exception, the rate of adaptation depends almost entirely on the fitness of the initial strain, but not on the specific mutations that it has, such that fitter strains adapt predictably slower than less fit strains. We also find that this dependence of the rate of adaptation on the initial fitness is caused by “diminishing returns” epistasis between adaptive mutations, which means that the same mutation confers smaller benefits to fitter strains than to less fit strains. As a consequence, fitness follows predictable trajectories even though adaptation at the genetic level is highly random.