Craft CMS and Chat-oriented Programming, CHOP

The narration of this post was created with Bespoken plugin for Craft CMS.

What is CHOP? A post on Sourcegraph titled The death of the stubborn developer defines it as "Chat-Oriented Programming," meaning the developer is constantly conversing with an AI model, most likely ChatGPT, while programming. In short, LLMs are decent at executing small, well-defined tasks, leaving "the more difficult planning and coordination" to the human developer. According to the post, the chop methodology "gives enterprise engineers a minimum boost of 30% in productivity across the board." That's enough of a summary of the post. I encourage you to read the whole thing.

After I read the piece, I want to test the CHOP methodology when applied to Craft CMS development to understand its implications for the Craft developer.

My test case involved using ChatGPT to create a small plugin that will add functionality to my site, which I've wanted for a while.

Plugin goals

Automatically check my Craft site to see if an update is available.
If an update is available, notify my devices using the Pushover API.

By the way, I am not affiliated with Pushover other than using the service. It's an inexpensive way to customize notifications sent to my devices.

Approaches

Option 1: specific small task questions on a familiar topic, Javascript

To demonstrate what I mean by a specific question, I asked ChatGPT the following:

I'm using Typescript in my project. I have a data request that returns a response object I need to parse in React. The object does not contain unique IDs. Tell me the most efficient way to add a unique ID to each item as I render it.

You can read the complete response by visiting this link.

ChatGPT did a good job by providing four options. For what it's worth, my go-to option is the 4th one on this list, using crypto.randomUUID().

I assume that ChatGPT has been trained on many pieces of Javascript and React content. The internet is flooded with this type of stuff, both good and bad.

I'm pretty satisfied with ChatGPT's response in this case, but that's also an easy question.

Option 2: specific task questions on a less familiar topic, Craft CMS

Let's face it: Craft CMS content is less prevalent online than questions about React and Typescript. There's less content for the AIs to have vacuumed up.

Since I'm familiar with Craft and have created plugins, I could use the Craft CMS Generator to scaffold a plugin and then ask ChatGPT for assistance with specific tasks related to PHP, Yii, or Craft CMS. As demonstrated above, being precise with questions helps direct ChatGPT to the right place to find an answer. If you're a Craft CMS developer using ChatGPT, you're probably already asking for this type of assistance from chatbots.

Option 3: broad task questions on a less familiar topic

For this test, I’m curious to explore how much help ChatGPT can be in creating the plugin. I prefer not to assume that the person asking the question is already well-versed in the topic. Instead, I’d like to encourage a more thoughtful exchange that prompts ChatGPT to engage in deeper reasoning. After all, this is a test, so we want to challenge ChatGPT and see what it can do.

On with the test

I started by opening up ChatGPT and giving it the following prompt.

I want to create a plugin that will send a Pushover API message when Craft CMS has updates available.

ChatGPT replied with a lengthy answer that effectively outlined the task at a broad level. (By the way, I've added a link to the full chat transcript at the end of this post so you can read the whole conversation and draw your own conclusions.)

Let's look at the beginning of ChatGPT's answer.

Creating a Craft CMS plugin to send a Pushover notification when updates are available involves the following steps. Here’s a high-level overview of what you’ll need to do:
Set Up Your Plugin:
Create a plugin using Craft’s scaffolding tools.
Define your plugin's details in the composer.json and Plugin.php files.
Check for Updates:
Use Craft’s updates service to check if updates are available. The service provides details about plugin, module, or Craft core updates.
Send the Pushover Notification:
Use Pushover’s API to send messages. You’ll need a Pushover account, an application token, and a user key.
Set Up a Scheduled Task or Listener:
Run the update check periodically via a cron job or as part of Craft’s queue system.
Handle Configuration:
Allow the admin to configure the Pushover API token and user key through your plugin settings.

If you know nothing about the process, this answer will give you a good series of general steps to follow in building your plugin.

After this portion of the response, the reply drifted into questionable territory.

The first hallucination

How does the AI suggest we scaffold out the plugin? It sounds as simple as the following snippet of code that it provided.

php craft plugin/create

I tried this on my local development site to see the result.

The error this line produced said: Unable to resolve the request: plugin create.

Caused by: Exception 'yii\base\InvalidRouteException' with message 'Unable to resolve the request: plugin/create'

in /var/www/html/vendor/yiisoft/yii2/base/Controller.php:149

The problem is that there is no plugin/create function, although the general idea is not far off. As mentioned in the Option 2 section above, the Craft CMS Generator allows you to scaffold a plugin, but ChatGPT doesn't appear to know about this, so it made up some pseudo code. The correct response would be to install the generator and run the php craft make plugin command.

ChatGPT was on the right general path, but it got the specifics wrong.

Another hallucination

Let's dive a little deeper into the response. One of the key things the plugin will need to do is query for possible updates to Craft CMS. How does the ChatGPT response say to do this? "Use Craft's updates service to detect updates." This sounds generally correct. But how do we do that specifically? Let's look at the code it provided.

The code block can be seen in the original article, but it suggests querying a property that does not exist on the Update model.

$updates = Craft::$app->updates->getUpdates();
if ($updates->isUpdateAvailable) {
    $message = "Updates are available for Craft CMS.";
    // Call your function to send a Pushover message.
    $this->sendPushoverNotification($message);
}

If you try this code, you'll find that isUpdateAvailable isn't a valid property, although, like the previous example, this is on the right track. I threw the first line into PHPStorm, and the autocomplete in the IDE showed me that I could do the following.

See the original article for a corrected version of the code.

$updates = Craft::$app->getUpdates()->getUpdates( true );

Now we're getting somewhere. Let's examine what's inside of our $updates variable by dumping its contents.

The code block is excluded from this narration. In short, the code block shows a nested object containing an array of available Craft CMS releases.

craft\models\Updates { ▼
  -_events: []
  -_eventWildcards: []
  -_behaviors: []
  -_errors: null
  -_validators: null
  -_scenario: "default"
  +cms: craft\models\Update { ▼
    -_events: []
    -_eventWildcards: []
    -_behaviors: []
    -_errors: null
    -_validators: null
    -_scenario: "default"
    +status: "eligible"
    +renewalPrice: null
    +renewalCurrency: null
    +renewalUrl: null
    +releases: array:3 [▼
      0 => craft\models\UpdateRelease { ▼
        -_events: []
        -_eventWildcards: []
        -_behaviors: []
        -_errors: null
        -_validators: null
        -_scenario: "default"
        +version: "5.5.6.1"
        +date: DateTime @1733875200 { ▶}
        +critical: false
        +notes: """
          <ul>
          <li>Fixed a bug where Tags fields had “Replace” actions. (<a href="https://github.com/craftcms/cms/issues/16310">#16310</a>)</li>\n
          <li>Fixed styling issues. (<a href="https://github.com/craftcms/cms/issues/16298">#16298</a>, <a href="https://github.com/craftcms/cms/issues/16312">#16312</a>) ▶
          </ul>
          """
      }
      1 => craft\models\UpdateRelease { ▶}
      2 => craft\models\UpdateRelease { ▶}
    ]
    +phpConstraint: "^8.2"
    +packageName: "craftcms/cms"
    +abandoned: false
    +replacementName: null
    +replacementHandle: null
    +replacementUrl: null
  }
  +plugins: array:1 [▶]
}

Although ChatGPT was wrong about the specific code, it again pointed in the general right direction, which I think is helpful. The "releases" array buried in the response is a key piece of data needed for the plugin.

Closing thoughts

If you read the full transcript, you'll see I asked ChatGPT several times about the errors in its responses. ChatGPT apologized and confidently provided another code block that produced other errors. The transcript goes on with a back-and-forth like this.

Based on the limited experiment I've outlined, chatbots in late 2024 are doing a decent job pointing a Craft developer in the general right direction, but they don't write error-free code. However, pointing you in the general right direction still might save you some time.

When I circle back to the "boost of 30% in productivity" claim that started this small exploration, it's impossible to say if it truly is a 30% increase, but I think it is worth adding to your toolkit as a Craft developer. What do you think? My socials are on my Contact page. Share your thoughts.

Read the full ChatGPT transcript of this experiment.