AI-assisted development is set to revolutionize the IT industry, bringing about significant changes. With this in mind, we wanted to conduct a test project to allow us to compare the process and tools of conventional software development with that powered by generative AI. This initiative is driven by the recognition that AI is not just a passing trend, but a transformative force of the future.
In order to do this, we asked two similar groups of designers and developers to carry out the same task:
Create a Tool Registry, where you can add tools and also search for tools. Group 1 was allowed to use traditional development methods and tools, while group 2 utilized AI assisted tools. The groups were given a two-week timeframe to carry out the task.
The principal idea of the project was to test the new AI assisted tools and how they performed in design and development work. Challenges, pain points and areas of success were all documented along the way.
Are there differences in the efficiency and quality of the work? Which represent the best ways of working? Are some parts of the process easier and some harder when making a comparison? Does one way of working offer time savings over another?
Choose your stack
The two groups were made up of 4-5 members, including one designer and three or four developers.
Group 2, let’s call them the AI team, kicked off the project by discussing the tech stack. They then sought advice from ChatGPT regarding the most suitable recommendations for this particular project.
The group also decided to build a web application and chose React to build it with. They also wanted to test Github Copilot and subsequently got a GPT-4 license to go with it.
“Would it be more beneficial to choose a tech stack with less experience? Or will it end up in chaos :D“
– Developer, AI team
Based on the teams’ thoughts and ChatGPT suggestions for this type of project, the technology stack that was chosen consisted of:
Front-end: React with Next.js
Back-end: Node.js
Framework: Express.js
Database: PostgreSQL
Authentication: Passport.js
Group 2, let’s refer to them as the traditional team, started the project with the limited time frame in mind and made a quick decision to rely on a stack they knew would work. However, this was without have prior experience in these technologies.
Instead of relying solely on AI tools for development, an alternative approach was opted for which embraced minimalism and leveraged pre-existing, reliable components. Typically, the selection of software libraries is influenced by discussions in forums, blogs, and recommendations from peers. The decision to adopt new software becomes easier when there is comprehensive documentation and helpful examples available. Positive feedback and ratings from users also play a significant role in making informed choices. The traditional team decided to go with a pretty similar kind of stack:
Front-end: React
Back-end: PostgREST, Express.js
Database: PostgreSQL
Getting on with it (week 1)
None of our AI team members had much experience in AI tools previously, so it took some time to familiarize themselves with these. However, when our team began issuing commands to the AI tools, they quickly observed its exceptional functionality.
Setting up the database turned out to be quite easy task for the AI team (with the help of AI, naturally).
“We got almost everything done by copy-pasting from ChatGPT. 😊”
– Developer, AI team
AI team’s developer successfully created a project template for an application and set up the application environment by copying commands from ChatGPT, which took significantly less time than a manual setup. He managed to generate simple application logic with the help of AI. However, he found an error in the application logic through the unit tests generated by the AI and this required some manual work to put right.
“I asked AI to create new tables in the database, based on my schema suggestion. Actually, I asked AI to create migration scripts. Very, very simple, and AI had no problems in creating the scripts.”
– Developer, AI team
One of the developers in the team decided to experiment with creating an app using Next.js. Despite encountering a steep learning curve, he successfully set up both the frontend and backend. Seeking further assistance, he turned to GPT-4 and requested it to generate a new form for adding a tool to the registry. However, the initial output was somewhat rudimentary in appearance. To improve its aesthetics, the developer employed AI to stylize the form, resulting in a more modern and visually appealing CSS form.
Since he hadn't tried Next.js before, he was pretty happy with the development, even though GPT-4 gives lots of outdated and/or incorrect answers. Also, you need to be able to ask the right kind of questions in order to get the optimal response.
“It would have been much more efficient if had read that Next.js documentation (which was shared in our team’s discussion) beforehand 😀 then I would have been able to create better questions for GPT-4.”
– Developer, AI team
Kicking off with the design
The designer from the AI team set about exploring AI-powered visual tools. Approaching the project with a growth mindset, she aimed to expand her knowledge and gain confidence in utilizing AI-assisted tools. To facilitate UI designs, she utilized ChatGPT to generate Tailwind CSS components. Additionally, she employed Midjourney for generating visually captivating images to enhance the project's visuals.
She was amazed by the capabilities of ChatGPT in creating user interfaces. Upon submitting the request “Create a banner for my Tool registry web page”, it generated a visual look and text that fit perfectly with the context. With a step-by-step approach she could swiftly create the required user interface for the registration page. By leveraging the power of AI, she achieved remarkable speed in the UI development process. She was also able to get the CSS / HTML code of the UI generated and share it with the developers.
Evolution of the navigation bar using ChatGPT. We started with a very simple prompt of creating a navigation bar for a tools library and then perfected it to the desired result that we were happy with.
How the different modules evolved to the final layout. ChatGPT also took the liberty in this case to explain how each of these changes are done and taught me more about the different CSS classes and how the code is actually created.
“When I started to explore AI tools, I was a bit scared but soon became impressed. It gives you instructions and teaches you, and we had a proper discussion. It doesn’t understand the UX, but it knows the UI inside out because it has in use all the knowledge in the world.”
– Designer, AI team
One of the notable advantages for our designer, who specializes in UX rather than UI design, was the relief from having to meticulously consider the design's intricate components, such as button specifications. This time-saving benefit became apparent when she simply conveyed her requirements to the AI, allowing it to handle the details. She was also astounded by the text generation capabilities of ChatGPT.
What AI cannot do is tasks demanding creativity, as it creates very generic content. A human is always required when creating appealing visual designs.
How the traditional team got on
With well-defined roles in place, the team swiftly embarked on their respective tasks. The designer wasted no time and promptly began crafting mock-ups using Figma, while the developers efficiently achieved rapid progress by leveraging an abundance of pre-existing code from previous repositories. The culmination of individual efforts enabled the team to make significant headway in their project.
“The benefit using my previous code is that I know it works and it suits my ways of working.”
– Developer, the traditional team
Where the team struggled was handling weird environment related issues. They got somewhat frustrated with repetitive tasks and looking up things that have been resolved multiple times before. AI tools would have made their job much faster. The team’s developer stated that ChatGPT would give him a shortcut to the necessary data without all the sponsored content and inaccurate data that Google provides.
The traditional team often spent considerable time searching for forgotten commands and configuration options across multiple data sources for various tools. They relied on conversations with colleagues and lively discussions on Slack, yet they pondered whether AI tools could offer a more direct means of accessing the same information. Searching through numerous websites and reading extensive texts consumed a significant portion of their day, particularly when dealing with unfamiliar topics.
Fortunately, the team made a wise decision in opting for Postgres, as it swiftly facilitated the integration of the interface. Surprisingly, they encountered minimal hurdles in getting the entire tech stack to seamlessly work together. As a result, the team's tool registry was rapidly taking shape, progressing at an impressive pace.
The final touches (week 2)
Towards the final stages of the project, the AI team significantly sped up their progress by leveraging AI to import the user interface (UI). The team's designer provided the developer with HTML files containing the designs she had created. They thoroughly examined the user flow within the application and even had some time to explore the visual imagery tools offered by Midjourney.
Meanwhile, the traditional team engaged in discussions regarding their initial choices and decided to make some adjustments to the selected technologies. Due to time constraints, they opted not to utilize tool images and simplified certain implementations. Eventually, the team completed their registry and deliberated on the best approach to share it for evaluation purposes in the future.
The outcome
The teams agreed in the beginning of the project that registries have these features: - 1. A login page
- 2. A home page, which allows users to search and get basic information on how to rent and add new tools to the registry
- 3. Search tools function
- 4. Add a new tool to the library –function by filling in a form
This is how the tool registries looked. The traditional team’s outcome looked like this:
Our AI team created three applications in total:
How much time was used for each phase?
The teams summarized their use of time:
The AI team:
UX design 15%
Frontend design 15%
Frontend implementation 30%
Backend design 10%
Backend implementation 30%
The traditional team:
UX design 10%
Frontend design 10%
Frontend implementation 30%
Backend design 10%
Backend implementation 40%
Evaluating the design and technical execution
We asked external experts to view the solutions’ design and technical execution, keeping in mind that the time used for the project was really limited and lot of new tools were tested.
The design
From the design point-of-view, the traditional team’s UI was evaluated as fairly clean, with working typography hierarchy and color contrasts with no distracting elements. The look and feel is consistent and the basics are in order, however it looks a bit wireframe-y instead of a finished product. The outcome was seen as a solid base for good UX/UI, but it needs some finetuning and accessibility fixes together with greater visual appeal.
When it comes to the AI team’s design, there were lots of issues, in relation to both look and feel and user experience. The UX was seen as inconsistent and misleading, and there were fundamental accessibility issues. The app’s basic functionalities do work, but beyond that there’s a serious risk of the user getting lost, confused, or irritated. The app looks and feels broken and unfinished.
Technical execution
The technical execution of the traditional team’s app was seen as better quality than the AI team’s. In addition to the very limited time constraint, the traditional team focused on one app and AI team wanted to test new tools and created several apps, which affected the functionality of the outcome.
The traditional team’s app was a bit tricky to install, and the instructions were unclear and not up-to-date. There were also some clear bugs in the code and newer JS features were not used at all. The code also included lots of unnecessary syntax. Module architecture was on right track, but component architecture had a bit too many blocks. No unit tests or quality analysis were executed. The team got a better evaluation on the front-end side.
The AI team estimated that 90% of their code was generated with AI of their app which was evaluated (the other two apps were not evaluated). There is a mix of different styles in the code. Some of the code is not production ready. When viewing the component architecture, this AI team’s is simpler with two components (backend and database), but there’s not real component thinking behind it.
What did we learn?
There are both pros and cons in using AI tools compared to non-AI-assisted development.
Firstly, we cannot rely purely on generative AI. It's essential to know what you are doing in the first place, or things might go fundamentally wrong. Especially for junior developers, it’s very important to read the tutorials and be in control, or you might get overwhelmed with the speed of things. It’s like driving a self-driving car – use it to assist you, but do not get lazy and trust it blindly, or you’ll surely end up in an accident. Data security issues also provide plenty of pitfalls ahead.
For seasoned developers, it's common to possess a database of their tried-and-tested generated code. This collection often comprises scoped components that are devoid of business logic, have undergone testing, and align with their preferred development practices.
In addition, when working with AI, acquiring the skill of asking precise and targeted questions becomes crucial. By providing accurate prompts, developers can obtain more accurate data and insights from AI models. Furthermore, proficiency in data validation is essential, as AI models like ChatGPT have been trained on data up until 2021 and may lack awareness of more recent developments.
“ChatGPT lacks the knowledge of the very latest changes in technology, but Copilot is pretty up-to-date. Combining all these different tools might be the key to success here. However, we should not forget the old way of working! Existing customized project templates, toolkits, scripts etc. are all still useful.”
– Developer, AI team
Generative AI can be a huge accelerator for routine tasks and the checking of information. Thus, our small project showed that a lot of learning, testing, and quality control is needed if we use it in code generation as extensively as our AI team did.
Thirdly, carry out a small project like this emphasized the importance of putting processes and documentation in place. Following a structured way of working was essential in order to be effective and to avoid a mess.
Fourthly, AI assisted tools can bring design and development work closer to each other. When the tool gives you the code of your design straight away, you get a lot of insight as to how the design is executed.
Fifthly, this particular project was viewed as an excellent opportunity to not only apply previously learned techniques but also to explore and familiarize themselves with new tools. The traditional team became intrigued by the prospect of testing AI tools on a broader scale, recognizing the potential for expanding their skills and knowledge in the realm of artificial intelligence.
“It would be interesting to continue with a customer pilot where we could test the AI methods and tools on bigger scale. To find the ways to build something that adds value for a real client faster and with improved quality.”
– Developer, traditional team
Finally, our AI team learned that utilizing generative AI tools requires a good internet connection, especially when playing around with visual content. Getting answers takes a long time if the connection is poor, but hopefully, these issues become reduced when the tools evolve.
Key takeaways from this project
- AI-assisted tools can mean design and development is faster, but we must have a good understanding of the development and what we are developing.
- The quality of the code can be lower with AI tools compared to your own, so prepare to use more time for testing and checking quality.
- You may want to use multiple AI tools for different kinds of tasks.
- Generative AI is a great assistant in UI design, but does not understand e.g. accessibility or what constitutes a good user experience.
- Generative AI can help to add understanding and teaches you as you go.
- The importance of asking the right questions cannot be minimized with generative AI.
- Human creativity is always needed, no matter what.
Written by:
Nea Kolmonen
Client Marketing Lead
Interim Head of Marketing