Heat pumps are a key technology for decarbonizing the residential heating sector. However, their efficiency depends heavily on the control strategy. In this work, we apply Constrained Reinforcement Learning (CRL) to the control of heat pumps. We formulate the problem as a safety-constrained Markov Decision Process (CMDP) and compare different CRL algorithms. Our results show that CRL can effectively learn policies that minimize energy consumption while satisfying comfort and safety constraints.