Brain Burner

A game that is difficult to master, requires hard and sustained thinking, and leaves the players exhausted.

Adjective form: brain-burny (e.g.: a brain-burny dungeon crawl).
Brain melter is a common synonym.
Brain burner is sometimes used to designate puzzles as well as crossword puzzles. It was also a nickname for early mobile phones, around 2000.

Now and then, however… I want to come out of a game with my brain matter dribbling out of my nose.Chessiecat on Reddit

A brain burner would be the opposite of a Beer and Pretzels game, a filler game, or any kind of light game. Although it does not have to be hard to learn, nor very long to play, a brain burner will present the players with numerous and difficult decisions or calculations relying on many factors that require intense reflection.