Creating a Javascript game with ChatGPT

Once a year we at getDigital launch our infamous getDigital Easter puzzleWe pose over 50 nerdy questions, the level of difficulty of which is constantly increasing: For the last question, you can expect to spend several days trying to solve it. However, you can have a lot of fun even with the simpler questions and at the end, all participants will be entered into a draw to win great prizes.

We put a lot of effort into creating the questions for the puzzle a long time in advance: creating a question usually takes more effort than solving it, so we spend over 100 hours on it every year. We really wanted to use ChatGPT for one question this year. The idea was to create a simple Javascript jump-and-run that required nerd knowledge as well as skill. We first made a very general request to ChatGPT (version 4):

The result is not so convincing yet: This came out (clicking with the mouse makes the block jump into the air, but you can't do much else). So I was a bit confused:

Now it got really fascinating: then ChatGPT generated the code to make the blue block move and jump with the arrow keys! This is what the game looked like afterwards.

Let's continue:

It became this version.

In the next steps, I added a countdown timer and platforms that the player can jump onto. This is what it looked like afterwards.

As you can see, it doesn't work properly: the player block floats above the platforms and doesn't touch them. So I asked ChatGPT to fix the bug and now things got interesting. ChatGPT pretty much randomly tried to change something in the checkPlatformCollision() function to fix the bug, but only made it worse:

Every other change ChatGPT added only made it worse. In the end, I asked it to revert the code before the bug fixes and went looking for the bug myself: there was simply a logic error in the code of the checkPlatformCollision() function that determines whether the player has reached the top of the platform. To find this error, however, I had to take a close look at the logic and consider which checks would be correct: ChatGPT was apparently unable to do this. Instead, it changed things relatively helplessly, which seemed more like wild trial and error than real thought.

The process continued in the same way: It worked fascinatingly well to integrate further functionalities into the game, but whenever there were bugs that required a detailed examination of the logic, ChatGPT failed. Another example was a bug in the collision detection with the enemy blocks: ChatGPT's code only checked for collisions with the top edge of the block, while the player could safely touch the bottom edge. Again, I only figured this out by carefully checking the collision logic, ChatGPT only made it worse every time I tried to solve it. Even when I wrote exactly where the bug was.

I think the limitations I observed are significant: ChatGPT is not able to really understand what it is producing. It "simply" only outputs what matches well with what is in the request and the responses produced so far. This works surprisingly well, but one should not make the mistake of thinking that ChatGPT develops a real understanding of what it is writing.

We'll use the final game with all of my bug fixes in the 2023 Easter puzzle. It looks like this.