A quick test of chatGPT Solidity programming skills
The good part
When asked to recreate part of our own codebase, chatGPT, the latest demo of OpenAI based on
the large language model GPT3, managed to do incredibly well at first sight but quickly showed
some weaknesses.
After one single request, chatGPT was indeed able to generate from scratch a very simple ERC20
smart contract for the Khamseen Token and was even able to take into account dependencies and
code inheritance.
After multiple prompts, chatGPT even succeeded in generating an almost production-ready function for our reward mechanism:
The not so good part
However, while attempting to write our own
reward function, it failed to understand on its own the
logic and technical restrictions of smart contract development.
Multiple errors and unconventional choices were made during the first attempts:
- balances and total supply variables were wrongly updated with erroneous amounts
- an infinite while loop has been written
- float values were used instead of integers. When specifically asked about this error chatGPT
surprisingly "knew" that this was incorrect:
The main flaw of this specific version of chatGPT was that too many rounds of prompts were needed to correct or to simply remove lines of code. Any solidity programmer would have come up with a solution faster than chatGPT, at least in its current state. The model got stuck a multiple occasion on a specific version of the code even after being requested to update it according to our instructions. An other important point is that the model didn't manage to reproduce the same code from identical prompt across multiple sessions which raises reproductibility issues for software engineers. Those flaws are however quickly disappearing since the first release and the model keep being improved from the community feedback.
The verdict
chatGPT represents an impressive advancement for A.I applications. The fact that chatGPT wasn't designed
to generate code and still manage to do so well would suggest that a fine-tuned model would do even
better and go far beyond currently existing A.I programming assistants like Github's Copilot.
An end-to-end A.I model able to generate production-ready code would disrupt the entire Tech industry
and change its entire economical landscape. It is still early to pronounce the end of human
software engineers but A.I seems to be on its way to replace the last tricky and costly part of the Tech
industry: humans.
Some experts speculate that large workforce of engineers might not be needed anymore in the near
future and that most tech companies would only need a few engineers to supervise and review A.I
generated code.
On the other hand, this could also be a positive news for every tech worker with an entrepreunarial spirit. In a world
where code would be cheap, only the battle of ideas will remain, and it would not surprise anyone
that a young tech-savvy engineer with ambitious ideas and an appetite for risk would do better
than any MBA type manager.
On a lighter note, chatGPT might also indicate the need for future software engineers to improve their
"A.I prompt engineering skills" instead of their programming skills. Human-to-A.I communication
may be indeed the last piece of the puzzle until full-automation.