We define the anisotropic Rabi model as the generalization of the spin-boson Rabi model: The Hamiltonian system breaks the parity symmetry; the rotating and counterrotating interactions are governed by two different coupling constants; a further parameter introduces a phase factor in the counterrotating terms. The exact energy spectrum and eigenstates of the generalized model are worked out. The solution is obtained as an elaboration of a recently proposed method for the isotropic limit of the model. In this way, we provide a long-sought solution of a cascade of models with immediate relevance in different physical fields, including (i) quantum optics, a two-level atom in single-mode cross-electric and magnetic fields; (ii) solid-state physics, electrons in semiconductors with Rashba and Dresselhaus spin-orbit coupling; and (iii) mesoscopic physics, Josephson-junction flux-qubit quantum circuits.