The new struggle for software selection because AI
One of the job of IT people is to select software, in terms of both applications or software libraries, this process may have different requirements depending on your needs or your organization needs, but in general there are some basic requirements for everybody, like:
- Good Quality software
- Consistent APIs
- Well tested
- Easy to maintain
- Right level of documentation
- Good background organization
- An invested company or set of individual interested to push the project
- Enough economic resources to keep the project alive.
- Good amount of users/community
- Enough people out there that know how to use the software
- Enough online and offline information for solve problems
- Easy to recruit already skilled people
With the advent of LLM coding, the evaluation of these criteria shifted, as today is really easy to create something that at first look seems good quality software with extensive tests and documentation, and an apparent good organization behind it, but in reality is just some quick LLM generated project done by an individual in its free time, and when you start to dig through the project in details you find that it may lack of precision and coherency, with a lot of LLM boilerplate and not clearly defined scope, this is not just due to the use of LLMs but because to make a project good is needed a lot of time to clearly define limits and details, does not matter if you produce the software with humans, or trough tutoring of LLMs.
Also with the time passing, the number of this quick generated projects is increasing a lot, so when trying to select a software for someone needs the apparent options seems way more than some time ago, but more often than not when going through the details all these options are not real options.
The result is that now the effort to select something is higher than it was before, really often the projects to go through are an order of magnitude more than before the AI wave, and is harder to understand which of these project is good quality and will be here in 6 months and which will be not, one good criteria is often how much people invested on making a project, if people invested more time, is more likely that they will invest time in future to maintain and push the project forward, even though the effort invested cannot anymore be measured trough project size, text generation now is cheap. A lot of AI projects are just a few weeks/months of free time code generation which sometimes seems better to re-generate them with LLMs for your specific needs instead of use existing one.
This is increasing the cost for selecting software, decision maker now need to spend more time evaluating projects, the superficial technical criteria used in the past, like checking tests results, code coverage, documentation presence and size are not that useful anymore, all of these things now can be generated quickly and do not determine any more quality, more often than not the only criteria left from before AI, is community size, making the software selection looks more like a popularity context than a technical evaluation.
All of this even for my specific cases is adding stress on my IT work, I do come by a lot of interesting projects in recent times, but a lot of them are LLM generated project with various level of tutoring, and this stress me because I would love to use them, but most of these projects will be over as quickly as they have been generated, as soon as the developer lose interest on them, or the tokens to maintain them became too expensive, so recently I started to check for signs of LLM coding and try to avoid using the projects that use LLMs if alternatives are available, just because is more efficient for me to ignore LLM generated project that actually evaluate them.
There are a bunch of tools can be used to quickly check LLM use in a project, a few I used recently:
I would love to see on software repositories badges or clear stated hint for software automatically generated, or have somewhere o general index that score the "AIness", maybe together with all the other software criteria selection.
Looking for to see more people sharing their approach e feeling on this new challenges, I want to hear more on what tools and data you are using to filter out the noise over the content, please do not say "I just used XYZ AI " :).
IT world is changing fast, maybe faster than what people(I) can handle, and I really need to vent the stress that is adding on our work!
Have a good day!
Ko-fi