Craft CMS and Chat-oriented Programming, CHOP

What is CHOP? A post on Source­graph titled The death of the stub­born devel­op­er defines it as ​“Chat-Ori­ent­ed Pro­gram­ming,” mean­ing the devel­op­er is con­stant­ly con­vers­ing with an AI mod­el, most like­ly Chat­G­PT, while pro­gram­ming. In short, LLMs are decent at exe­cut­ing small, well-defined tasks, leav­ing ​“the more dif­fi­cult plan­ning and coor­di­na­tion” to the human devel­op­er. Accord­ing to the post, the chop method­ol­o­gy ​“gives enter­prise engi­neers a min­i­mum boost of 30% in pro­duc­tiv­i­ty across the board.” That’s enough of a sum­ma­ry of the post. I encour­age you to read the whole thing.

After I read the piece, I want to test the CHOP method­ol­o­gy when applied to Craft CMS devel­op­ment to under­stand its impli­ca­tions for the Craft devel­op­er. 

My test case involved using Chat­G­PT to cre­ate a small plu­g­in that will add func­tion­al­i­ty to my site, which I’ve want­ed for a while.

Plugin goals

• Auto­mat­i­cal­ly check my Craft site to see if an update is avail­able. 
• If an update is avail­able, noti­fy my devices using the Pushover API.

By the way, I am not affil­i­at­ed with Pushover oth­er than using the ser­vice. It’s an inex­pen­sive way to cus­tomize noti­fi­ca­tions sent to my devices.

Approaches

Option 1: specific small task questions on a familiar topic, Javascript

To demon­strate what I mean by a spe­cif­ic ques­tion, I asked Chat­G­PT the fol­low­ing:

I’m using Type­script in my project. I have a data request that returns a response object I need to parse in React. The object does not con­tain unique IDs. Tell me the most effi­cient way to add a unique ID to each item as I ren­der it.

You can read the com­plete response by vis­it­ing this link.

Chat­G­PT did a good job by pro­vid­ing four options. For what it’s worth, my go-to option is the 4^th one on this list, using crypto.randomUUID().

I assume that Chat­G­PT has been trained on many pieces of Javascript and React con­tent. The inter­net is flood­ed with this type of stuff, both good and bad. 

I’m pret­ty sat­is­fied with Chat­G­P­T’s response in this case, but that’s also an easy ques­tion.

Option 2: spe­cif­ic task ques­tions on a less famil­iar top­ic, Craft CMS

Let’s face it: Craft CMS con­tent is less preva­lent online than ques­tions about React and Type­script. There’s less con­tent for the AIs to have vac­u­umed up. 

Since I’m famil­iar with Craft and have cre­at­ed plu­g­ins, I could use the Craft CMS Gen­er­a­tor to scaf­fold a plu­g­in and then ask Chat­G­PT for assis­tance with spe­cif­ic tasks relat­ed to PHP, Yii, or Craft CMS. As demon­strat­ed above, being pre­cise with ques­tions helps direct Chat­G­PT to the right place to find an answer. If you’re a Craft CMS devel­op­er using Chat­G­PT, you’re prob­a­bly already ask­ing for this type of assis­tance from chat­bots.

Option 3: broad task questions on a less familiar topic

For this test, I’m curi­ous to explore how much help Chat­G­PT can be in cre­at­ing the plu­g­in. I pre­fer not to assume that the per­son ask­ing the ques­tion is already well-versed in the top­ic. Instead, I’d like to encour­age a more thought­ful exchange that prompts Chat­G­PT to engage in deep­er rea­son­ing. After all, this is a test, so we want to chal­lenge Chat­G­PT and see what it can do.

On with the test

I start­ed by open­ing up Chat­G­PT and giv­ing it the fol­low­ing prompt.

I want to cre­ate a plu­g­in that will send a Pushover API mes­sage when Craft CMS has updates avail­able.

Chat­G­PT replied with a lengthy answer that effec­tive­ly out­lined the task at a broad lev­el. (By the way, I’ve added a link to the full chat tran­script at the end of this post so you can read the whole con­ver­sa­tion and draw your own con­clu­sions.) 

Let’s look at the begin­ning of Chat­G­P­T’s answer.

Cre­at­ing a Craft CMS plu­g­in to send a Pushover noti­fi­ca­tion when updates are avail­able involves the fol­low­ing steps. Here’s a high-lev­el overview of what you’ll need to do:

1. Set Up Your Plu­g­in:
   • Cre­ate a plu­g­in using Craft’s scaf­fold­ing tools.
   • Define your plug­in’s details in the composer.json and Plugin.php files.
2. Check for Updates:
   • Use Craft’s updates ser­vice to check if updates are avail­able. The ser­vice pro­vides details about plu­g­in, mod­ule, or Craft core updates.
3. Send the Pushover Noti­fi­ca­tion:
   • Use Pushover’s API to send mes­sages. You’ll need a Pushover account, an appli­ca­tion token, and a user key.
4. Set Up a Sched­uled Task or Lis­ten­er:
   • Run the update check peri­od­i­cal­ly via a cron job or as part of Craft’s queue sys­tem.
5. Han­dle Con­fig­u­ra­tion:
   • Allow the admin to con­fig­ure the Pushover API token and user key through your plu­g­in set­tings.

If you know noth­ing about the process, this answer will give you a good series of gen­er­al steps to fol­low in build­ing your plu­g­in. 

After this por­tion of the response, the reply drift­ed into ques­tion­able ter­ri­to­ry.

The first hallucination

How does the AI sug­gest we scaf­fold out the plu­g­in? It sounds as sim­ple as the fol­low­ing snip­pet of code that it pro­vid­ed.

php craft plugin/create

I tried this on my local devel­op­ment site to see the result.

The error this line pro­duced said: Unable to resolve the request: plu­g­in cre­ate.

Caused by: Exception 'yii\base\InvalidRouteException' with message 'Unable to resolve the request: plugin/create'

in /var/www/html/vendor/yiisoft/yii2/base/Controller.php:149

The prob­lem is that there is no plugin/create func­tion, although the gen­er­al idea is not far off. As men­tioned in the Option 2 sec­tion above, the Craft CMS Gen­er­a­tor allows you to scaf­fold a plu­g­in, but Chat­G­PT does­n’t appear to know about this, so it made up some pseu­do code. The cor­rect response would be to install the gen­er­a­tor and run the php craft make plugin com­mand.

Chat­G­PT was on the right gen­er­al path, but it got the specifics wrong. 

Another hallucination

Let’s dive a lit­tle deep­er into the response. One of the key things the plu­g­in will need to do is query for pos­si­ble updates to Craft CMS. How does the Chat­G­PT response say to do this? ​“Use Craft’s updates ser­vice to detect updates.” This sounds gen­er­al­ly cor­rect. But how do we do that specif­i­cal­ly? Let’s look at the code it pro­vid­ed.

The code block can be seen in the orig­i­nal arti­cle, but it sug­gests query­ing a prop­er­ty that does not exist on the Update mod­el.

$updates = Craft::$app->updates->getUpdates();
if ($updates->isUpdateAvailable) {
    $message = "Updates are available for Craft CMS.";
    // Call your function to send a Pushover message.
    $this->sendPushoverNotification($message);
}

If you try this code, you’ll find that isUpdateAvailable isn’t a valid prop­er­ty, although, like the pre­vi­ous exam­ple, this is on the right track. I threw the first line into PHP­Storm, and the auto­com­plete in the IDE showed me that I could do the fol­low­ing.

See the orig­i­nal arti­cle for a cor­rect­ed ver­sion of the code.

$updates = Craft::$app->getUpdates()->getUpdates( true );

Now we’re get­ting some­where. Let’s exam­ine what’s inside of our $updates vari­able by dump­ing its con­tents.

The code block is exclud­ed from this nar­ra­tion. In short, the code block shows a nest­ed object con­tain­ing an array of avail­able Craft CMS releas­es.

craft\models\Updates { ▼
  -_events: []
  -_eventWildcards: []
  -_behaviors: []
  -_errors: null
  -_validators: null
  -_scenario: "default"
  +cms: craft\models\Update { ▼
    -_events: []
    -_eventWildcards: []
    -_behaviors: []
    -_errors: null
    -_validators: null
    -_scenario: "default"
    +status: "eligible"
    +renewalPrice: null
    +renewalCurrency: null
    +renewalUrl: null
    +releases: array:3 [▼
      0 => craft\models\UpdateRelease { ▼
        -_events: []
        -_eventWildcards: []
        -_behaviors: []
        -_errors: null
        -_validators: null
        -_scenario: "default"
        +version: "5.5.6.1"
        +date: DateTime @1733875200 { ▶}
        +critical: false
        +notes: """
          <ul>
          <li>Fixed a bug where Tags fields had “Replace” actions. (<a href="https://github.com/craftcms/cms/issues/16310">#16310</a>)</li>\n
          <li>Fixed styling issues. (<a href="https://github.com/craftcms/cms/issues/16298">#16298</a>, <a href="https://github.com/craftcms/cms/issues/16312">#16312</a>) ▶
          </ul>
          """
      }
      1 => craft\models\UpdateRelease { ▶}
      2 => craft\models\UpdateRelease { ▶}
    ]
    +phpConstraint: "^8.2"
    +packageName: "craftcms/cms"
    +abandoned: false
    +replacementName: null
    +replacementHandle: null
    +replacementUrl: null
  }
  +plugins: array:1 [▶]
}

Although Chat­G­PT was wrong about the spe­cif­ic code, it again point­ed in the gen­er­al right direc­tion, which I think is help­ful. The ​“releas­es” array buried in the response is a key piece of data need­ed for the plu­g­in. 

Closing thoughts

If you read the full tran­script, you’ll see I asked Chat­G­PT sev­er­al times about the errors in its respons­es. Chat­G­PT apol­o­gized and con­fi­dent­ly pro­vid­ed anoth­er code block that pro­duced oth­er errors. The tran­script goes on with a back-and-forth like this. 

Based on the lim­it­ed exper­i­ment I’ve out­lined, chat­bots in late 2024 are doing a decent job point­ing a Craft devel­op­er in the gen­er­al right direc­tion, but they don’t write error-free code. How­ev­er, point­ing you in the gen­er­al right direc­tion still might save you some time. 

When I cir­cle back to the ​“boost of 30% in pro­duc­tiv­i­ty” claim that start­ed this small explo­ration, it’s impos­si­ble to say if it tru­ly is a 30% increase, but I think it is worth adding to your toolk­it as a Craft devel­op­er. What do you think? My socials are on my Con­tact page. Share your thoughts.

Read the full Chat­G­PT tran­script of this exper­i­ment.