I’ve heard that ChatGPT is capable of generating complex coding projects. I’ve been curious to give it a try for myself. So I decided to use one of the most reliable measurements of technical skills known to software hiring managers the world over: a take-home coding project. But instead of giving ChatGPT a one pager describing what I was looking for and hearing back with a git repo link in a few days, it took the infamous large language model quite a bit of hand holding to get the job done.

I’ve posted the resulting project with all the prompts and responses and code in this Github repo: davekonopka/password-checker The commit history shows each step along the way.

Define the needful

Take-home coding projects have become a mainstay of the software hiring process. They’re slightly less tormentuous for job candidates than on-the-spot brain teasers or psuedo-coding bubble sort exercises. You get a little more breathing space and time to showcase your talents with a small, typically throw-away, project.

I wanted to push ChatGPT a bit. So I picked a “hard” level LeetCode exercise as the project base: 420. Strong Password Checker

Given a string password, return the minimum number of steps required to make password strong. if password is already strong, return 0.

The solution needed to be built into an HTTP web service with all the accompanying trappings of a production bound web application: decent test coverage, some metrics, a Dockerfile, and maybe some load tests.

I was impressed that given the basic LeetCode text and a bit more explanation, ChatGPT produced a password-checker function right away that generally worked. Things got a little more complicated quickly after that though.

Can you slow down and speak in painfully short, specific requests?

I listen to podcasts at near 2x speed. It’s gotten to where normal speech sounds like slow motion to me. ChatGPT on the other hand likes to communicate at 1/2x speed. If you give it too much information too quickly you’re likely to get less useful results.

In my limited experience generating this project it helped to keep my prompts limited to a single task with as much specific detail as possible. If you have specific libraries you want to use, mention that. Be clear about what you want external behavior to look like. Layer on each aspect that you want in a new prompt. This will improve the result and make it easier to tweak project layers as you go.

Of course this methodology requires you to have significant understanding of what you’re trying to accomplish and the tools used to do it. ChatGPT coding is not at a point where it can deliver complex, working solutions with largely uninformed prompts. Responses can seem magical at times but it’s not a replacement for human understanding or skill. Yet.

Expect to have a ton of back and forth with your new AI friend. Be kind and polite with your prompts. Who knows, maybe someday it will save your life when the robots take over.

Getting stuck in some corners

I ended up getting stuck with broken code a few times, mostly with unit tests. A first attempt at adding unit tests ended up with failures. A second attempt also ended in failures. It’s clear the ChatGPT is not trying out its code before sharing it.

Later after adding a CLI and web server to the project, attempting to add unit tests resulted in failure and messy restructuring of the underlying code. I gave up after two attempts at fixing things, backed out the tests for these features, and reverted the code changes. I figured it’d reflect better on ChatGPT to have a gap in tests than broken code.

Teach me, ChatGPT

An interesting side effect of this project was seeing how ChatGPT could teach me about new tech. It often gave detailed summaries describing how a solution could work. This amounted to an interactive tutorial that I could ask for more information whenever I needed to.

At the very beginning it provided an overview of the pieces required to accomplish my prompt. When I introduced testing for the CLI and daemon interfaces, it offered ways to refactor the underlying code and an explanation of how integration testing might differ from unit testing.

I can see myself using ChatGPT projects like this to teach myself new tools and methodologies. One thing I’m concerned about is being lied to about things I don’t fully understand yet. I’ve heard that ChatGPT can hallucinate and confidently assert faulty facts. This would be a more problematic issue while learning new concepts.

Hiring decision: Thumbs up or thumbs down?

In the end I was able to produce a working project with:

I’d give ChatGPT a thumbs up to proceed to the next round of hiring interviews. It would almost certainly score well on a technical deep dive interview. I’m not sure how well it would fare on a behavioral interview but I’d give it good odds. And I will certainly keep exploring with it for future projects of my own.

Perhaps another important question after seeing this is: Will I be able to tell if future human candidates’ take-home projects were done solo or with ChatGPT’s help?