In this work the feasibility of Reinforcement Learning (RL) for Dynamic Spectrum Management (DSM) in the context of next generation multicell Orthogonal Frequency Division Multiple Access (OFDMA) networks is studied. An RL-based algorithm is proposed and it is shown that the proposed scheme is able to dynamically find spectrum assignments per cell depending on the spatial distribution of the users over the scenario. In addition the proposed scheme is compared with other fixed and dynamic spectrum strategies showing the best tradeoff between spectral efficiency and Quality-of-Service (QoS).