In this article, a distributed nonmodel based generalized Nash equilibrium (GNE) seeking algorithm is proposed for a class of constrained noncooperative games with unknown cost functions. In the game, the strategy of each agent is restricted by both the coupled equality constraint and local inequality constraints. By virtue of the exact penalty method, an auxiliary cost function is constructed with the cost function and the local constraints. The main feature of the proposed algorithm depends on the capability to estimate the gradient information of auxiliary cost functions with only the values of costs. This is obtained by the extremum seeking control (ESC). To deal with the coupled constraints, only the Lagrange multiplier is transmitted among agents with some prior information about the coupled constraints. Moreover, a diminishing dither signal is introduced in the seeking algorithm to remove undesirable steady-state oscillations occurred in the classical ESC. As a result, the nonlocal convergence of the designed seeking algorithm to the GNE of the game is obtained by the singular perturbation theory, averaging analysis and Lyapunov stability theory. Numerical examples are given to verify the effectiveness of our proposed method.