This article investigates the handoff optimization in an uplink non-orthogonal multiple access-based small cell network (SCN) with successive interference cancellation that takes both the base station (BS) association and power control into consideration. Due to user mobility and varying communication environment, it is important to associate a user with the proper BS and control its transmit power by considering the user mobility for the SCNs. To overcome overly frequent re-association and handoffs between BSs, a novel joint optimization of handoff, BS association, and total transmit power is presented that maximizes the system-wide and handoff utilities and minimizes power consumption simultaneously. We define the handoff utility function by considering the BS association in each time frame with the previous time frame. Then, we formulate two inter-related optimization problems under the minimum average-rate and maximum allowable transmit power of mobile users (MUs) constraints. The designed optimization problems are combined through the weighted sum method into a single-stage optimization problem. The transformed problem is solved by adopting the coalition formation game and primal decomposition theories. We design the coalition value function by collectively considering the handoff number of each BS in every time frame, transmit power of all MUs, and the number of associated MUs to every BS. The simulation results show the efficiency of the proposed handoff algorithm in comparison to another BS association algorithm in terms of the number of handoffs.