I'm sure the game was fine on the other consoles but it was just borderline unplayable at times on the Wii with the motion controls.

The books/movies beyond Goblet also I feel like were just not easily as adaptable into a video game? The movies from Phoenix and onwards felt a lot more coherent to the central story and development of characters, whereas Stone, Chamber, Prisoner, and Goblet were a lot more fun, creative, and whimsical (not that that makes them superior, even though I do think they're superior) and put a lot of emphasis into the otherworldliness of the wizarding world. Translating that more serious story/character centric style into an adventure game is trickier, I think.

Sorry this turned into a HP analysis.