Thursday, December 9, 2021

Rightpoint Commerce’s Magento Extension Evaluation Framework

Max Chadwick, Technical Lead, Commerce
Technology / Platforms

Extension quality is an important topic in the Magento ecosystem, and is an area I haven’t been shy on sharing my thoughts about in the past (see my talk “What Agencies Want from Extension Providers”). A poorly built extension can quite literally be the root cause of a critical downtime event during peak Black Friday holiday traffic (something we’ve seen before). As such, Rightpoint Commerce puts an emphasis on proper vetting of Magento extensions/extension providers before recommending our clients install them.

Over time, we’ve identified signals that indicate a good quality extension, and signs that tell us to veer away. Earlier this year, we developed our own in-house framework for evaluating the quality of Magento extensions.

Now, we’ve decided to share this framework with the general public. The goal here is to promote extension quality and share our framework so it can be leveraged by other organizations operating in the Magento space. We’re excited to hear what feedback the Magento community has and any suggestions for improvement.

Evaluation Criteria

Before sharing the exact system, I’d like to provide details on what aspects of the extension are assessed. These criteria are each assessed individually, and then used to calculate an overall grade rating, which will be outlined later. The criteria are as follows:

  • Code Review Outcome – Rightpoint will conduct a line-by-line code review of the extension. During this code review we will identify a number of issues. Those issues will then be assigned a severity (Critical, High, Medium, Low) based on the reviewer’s judgement. A full overview of our code review process is not in scope for this blog post but, for example, a pre-authentication remote code execution security vulnerability would be classified as “critical” whereas a best practice violation such as an indentation issue would be classified as “low”.
  • Supports Composer Installs – This should be fairly self-explanatory. The module’s official installation instructions should support (and prefer) a composer-based install approach, rather than ZIP file/app/code-based installs.
  • Adobe Commerce (aka Magento Enterprise Edition) Compatibility – This may not be relevant to everyone in the Magento ecosystem but for us the vast majority of Magento work we’re doing is with Adobe Commerce. As such, we make sure that the extension is officially listed as compatible.
  • Listed in Magento Marketplace – This should also be fairly self-explanatory; the module should be available via Magento marketplace. This is important as code available from marketplace is known to have passed Magento’s EQP checks.
  • Compatible with latest major Magento version – This should again be self-explanatory. At the time of writing this the latest major Magento version is 2.4, so, if the extension only listed compatibly with 2.3 it would fail this evaluation.
  • Demo site available – We check whether or not there is a demo site available to preview the extension functionality. This is useful to review the module’s functionality pre-purchase or even for checking how the module is intended to function post-purchase to triage issues during testing and determine whether they were caused by the integration with the Magento store, or are in fact intended behavior for the module.
  • Documentation – As part of the review process we read through the documentation to assess how detailed/thorough it is. For the purposes of the rubric, we just confirm that there even is documentation at all.
  • Changelog – The importance of changelogs is well documented and known within the software development space. We validate that the provider maintains one for the extension in question.
  • Code Obfuscation – Obfuscated code (e.g. ionCube encoded) cannot be read and therefore is challenging to support and debug. As such, obfuscated code is strongly discouraged.
  • Automated Test Coverage – Automated tests are another indicator of good code quality. As such we review the code base to check for automated code coverage as part of the code review process. We evaluate this as either “Thorough”, “Minimal” or “None”.

The Rubric

Without further ado, here is the rubric we use for evaluation Magento extensions, again, I hope you find this useful and would be interested to hear any thoughts on it. Questions? Get connected with one of our experts today.