© 2024 American Physical Society. Excited-state quantum phase transitions (ESQPTs) strongly influence the spectral properties of collective many-body quantum systems, changing degeneracy patterns in different quantum phases. Level degeneracies in turn affect the system's dynamics. We analyze the degeneracy dependence on the size of two-level boson models with a u(n+1) dynamical algebra, where n is the number of collective degrees of freedom. Below the ESQPT critical energy of these models, the energy gap between neighboring levels that belong to different symmetry sectors gets close to zero as the system size increases. We report and explain why this gap goes to zero exponentially for systems with one collective degree of freedom but algebraically in models with more than one degree of freedom. As a consequence, we show that the infinite-time average of out-of-time-order correlators is an ESQPT order parameter in finite systems with n=1, but in systems with n>1, this average only works as an order parameter in the mean-field limit.