AI-Enhanced API Design: A New Paradigm in Usability and Efficiency

3.1 API Production Experience

A survey was sent to 4,000 software engineers at a Google API interest group. The group is a mailing list consisting of members interested in any aspect of APIs (e.g., API Usability, API Security, Designing APIs and Frameworks). In the email, we requested responses from only those with at least 1 year of experience designing API specs. In total, we received 234 responses to the survey. Among the 234, 100 agreed to be part of this study, where they were tasked to design APIs based on sample product specifications that we provided. The goal of the product was to build a Social Network App for kids to share ideas and information with their friends. Requirements included things such as “Parent creates an account and child account” or “Feed displays relevant posts based on child’s followers and interactions (likes and comments). Participants were asked to spend 2 hours recording their screen while designing the API. After completing the design, they were also required to submit a survey to reflect on the experience. For their time, participants were rewarded $80. We balanced the 100 participants into 3 groups using the Gower distance [5]:

The Gower Distance To create balanced experimental groups, we leveraged advanced statistical techniques including Gower Distance [5] for measuring dissimilarities and Partitioning Around Medoids (PAM) [20] for clustering. This allowed us to split the 100 participants into 3 groups with minimal differences in key attributes like experience level. We further employed propensity score matching, assessing distribution of categorical variables and numerical means, to mitigate selection bias. This rigorous approach enabled comparable groups, enhancing experimental validity. Out of the 100 participants, we received responses from 34, which we divided into three groups: 11 in Group I, 11 in Group II, and 12 in Group III.

3.2 API Consumption Experience

We recruited 33 professionals with an average of 12 years of software engineering experience through LinkedIn and API discussion channels. Approximately 71% were familiar with AIPs, and 56% had prior experience using AIPs for API design. The API specifications evaluated in the interviews were randomly selected from those developed by engineers in the API production phase of our study. Each 30-minute video interview consisted of both open-ended questions and a structured survey. After gauging participants’ AIP familiarity and design criteria, we administered surveys with 5 side-by-side API specification comparisons randomly drawn from the 3 pipeline groups.

To enable comprehensive comparisons, we manually matched elements across the groups’ API specs. For example, when comparing "create account" flows, we systematically paired the Group 1 design against relevant sections in Group 2 and Group 3. This matching process ensured an unbiased evaluation of all specifications. The process included 4 steps. Survey Creation: Three distinct surveys were developed, each with five questions to enable random pairwise comparisons between API specifications across the groups. API Documentation Generation: Using buf.build, we generated detailed API documentation for each specification, forming the basis of our analysis and comparison. For further details, if needed, see the screenshot/figure 1 included in the Appendix. Code Snippet Creation: Alongside the documentation, we produced code snippets illustrating the APIs’ functionality, providing a practical context for survey comparisons. For further details, if needed, see the screenshot/figure 2 included in the Appendix. Pairwise Analysis: During our surveys, participants undertook informed pairwise comparisons of API specifications across three distinct buckets. This approach ensured a thorough comparison, with each API element from one bucket being systematically paired against corresponding elements from the other two buckets. For instance, in scenarios like "create an account", "send a post", and "retrieve a list of posts with paging", participants selected their preferred API version from two choices presented in each scenario. This structured approach allowed us to draw meaningful conclusions regarding the strengths and weaknesses of the APIs, ultimately contributing to a more comprehensive understanding of their respective documentation and usability.

The API Architect evaluated the API designs from the three pipeline groups using several criteria. These included a Usability score, a quantitative assessment of the API specification’s usability on a scale from 0 to 100; an Evolvability score, measuring the ease of modifying or expanding the API specification in the future, also on a 0 to 100 scale; and an AIP adherence score, quantifying the extent to which the specification follows AIP conventions, again rated between 0 and 100.

4.1 API Production Experience

Comparing the perception of engineers with the actual design assessment, 94% of the engineers believed they completed more than 32% of the requirements specification in the 2-hour period. However, the analysis shows just about 58% of the engineers actually did so. 44% believed we could have allotted more than 2 hours period for the API design.

A majority of engineers (65%) reported that the API design process felt easy. Of this group, 29% used both AIPs and linter, 38% used just AIPs, and 33% used neither protocols. For the 35% of engineers who found API design difficult, first-time AIP users in particular noted challenges toggling between documentation and coding environments.

Despite these learning curves, for those leveraging AIPs, 91% agreed that AIPs were helpful for design. Furthermore, among those also using the linter tool, 75% found it beneficial when combined with AIPs. While adapting to new protocols poses initial difficulties, especially for novice users, a significant majority of engineers found value in AIPs for enhancing API design. Integrating linter alongside AIPs provided additional gains in perceived productivity.

To evaluate completion rate and errors, each API design response was manually checked against the 29 product requirements. Each requirement was scored as either 1 (satisfied) or 0 (unsatisfied). The percentage of requirements satisfied determined the completion rate for each response. Additionally, the number of errors flagged by the API Linter was recorded where applicable. The analysis revealed the following completion rates and error counts across groups:

To investigate the practical significance of the observed differences in completion rates among the three groups, we computed Cohen’s d, a widely used measure of effect size. In addition to calculating effect sizes, we examined the precision and certainty of our estimates using confidence intervals and standard errors. The effect size analysis revealed moderate effect sizes of 0.4 and 0.6 respectively when Group I is compared with Groups II and III, while the comparison between Groups II and III revealed a small effect size. Standard errors are relatively small. However, the wide confidence intervals indicate that our estimates are associated with considerable uncertainty due to the small sample size.

This results of the API production experience provided insights into the impacts of governance protocols. Despite initial learning curves, especially for new AIP users, a significant majority found AIPs valuable for enhancing design. The controlled experiment also quantified productivity gains, with the group leveraging both AIPs and linter achieving a 10 percentage point higher completion rate compared to the other groups.

4.2 API Consumption Experience

Our interviews revealed that most participants (56%) had less than 7 years of experience in the industry, and a significant portion (44%) had more than 7 years of experience, suggesting a diverse range of experience levels among API consumers. There were two main groups of API consumers, based on their evaluation criteria. The first group (33%) prioritized technical proficiency and dexterity, and they valued APIs that were readable and easy to use. The second group (30%) prioritized broader business goals and adaptability, and valued APIs that were agile and end-user-acceptable. Only 4% of API consumers prioritized security-related aspects. The survey found that 73% of the total participants preferred API designs that incorporated AIPs, and 76% of that percentage preferred designs that included the use of both AIPs and API Linter which contributed to the improved structure, standardization, and documentation of the APIs. However, 38% expressed a preference for alternative designs without AIPs or linter, citing completeness as a concern.

When comparing Group I and Group III API specifications, out of 120 pairwise matchups, Group I was preferred 73 times compared to 47 times for Group III. A two-proportion z-test showed this distribution has a z-score of 3.156 and a p-value of 0.008, indicating a statistically significant preference for the Group I designs at the p < 0.05 level. However, the comparison between Group II and Group III did not reveal a significant difference in preferences. Out of 150 matchups between these groups, Group II was preferred 79 times versus 71 times for Group III. The z-test resulted in a z-score of 1.05 and a p-value of 0.147, meaning no substantial preference difference between Group II and III. Users exhibited a definitive preference for Group I APIs over Group III, but Group II vs. Group III did not show a statistically significant distinction. This suggests integration of both AIPs and linter notably enhances API usability over no protocols, while AIPs alone do not confer a similarly sizable advantage.

On average, the API Architect scored the Group I specifications highest across all criteria (usability = 74.4, evolvability = 69.6, AIP adherence = 73.6). This aligns with the 73% rate of human experts preferring Group I designs, demonstrating strong agreement between human evaluator preferences and the Architect assessments. To test if Architect-measured differences between pipelines were statistically significant, two-proportion z-tests were conducted. Comparing Group I and Group III AIP adherence scores (where p₁ = mean Group 1 score = 73.6, p₂ = mean Group 3 score = 72.8) yielded p = 0.04818 < 0.05.

The respondents with less than 7 years preferred APIs that were "readable" and "easy to use." These considerations underscored their emphasis on technical proficiency and dexterity, as they expressed their preference for APIs that are not only functional but also comprehensible. Specifically, phrases such as "Readability: I want to understand what it’s doing if I read the code in order" encapsulated their criteria for assessing API quality. Conversely, respondents with more than 7 years of experience leaned toward facets such as "end-user acceptability" and "agility" when scrutinizing APIs. These preferences reflected their wealth of leadership and product-oriented experience, demonstrating a nuanced perspective that prioritizes broader business goals and adaptability. Phrases like "Does the API deliver business value?" and "Does it achieve a goal? How easy does it solve the problem?" captured the essence of their evaluative framework.

4.3 API Architect Evaluation

To benchmark the human-generated API designs, the AI-based API Architect developed specifications for the same product requirements within 3 minutes. Six industry experts with an average of 20 years of API design experience evaluated the Architect’s APIs based on overall quality, evolvability, completeness, and AIP adherence. The average overall quality score was 5 out of 10, highlighting some deficiencies compared to human-level design proficiency. The evolvability scored highest at 7 out of 10, as the Architect’s systematic approach enables future extensibility. However, completeness was rated lower at 5.8 out of 10, indicating key functionality gaps in the generated designs. Finally, AIP adherence received a 4.5 out of 10 score, reflecting deviations from best practice conventions.

Qualitative feedback emphasized issues like the API Architect’s divergence from resource-oriented principles and lack of HTTP bindings. As one expert noted, "It doesn’t seem resource-oriented at all...no definition of User or Post, just duplicate fields in different requests/responses." Another stated the designs don’t follow "the most common AIPs (Pagination, field names on resources, etc)." Several responses highlighted missing functionality, with one commenting "It doesn’t look like the parental controls are there at all." These example quotes align with the quantitative analysis pointing to limited completeness and AIP compliance.

5.1 API Production Experience

The analysis of API production experience provides valuable insights into RQ1 on how AIPs and tools like the API Linter influence development velocity and requirement fulfillment. The completion rate data reveals that Group I, leveraging both AIPs and linter, achieved the highest rate of 71.11%, surpassing Groups II and III by nearly 10%. This aligns with RQ1 by quantitatively showing how comprehensive integration of structured protocols boosts productivity.

Furthermore, the effect size analysis indicates a moderate effect when contrasting Group I to Groups II and III, demonstrating the notable influence of the AIPs and linter tandem on productivity. Given the substantial Cohen’s d values, these tools tangibly improve engineers’ capacity to build functional, production-ready APIs within constrained timeframes. While the confidence intervals are wide owing to small sample sizes, the consistently higher completion rates and API quality (lower error counts) underscore the benefits of AIP adoption. However, the perceived difficulty variance across groups highlights an avenue for refinement. The additional rigor imposed by strictly adhering to AIPs and resolving linter errors was seen as challenging, especially by first-time users. This indicates that more work is required towards tightening the integration within existing developer workflows. As API design practices evolve across organizations, maintaining an emphasis on minimizing disruption and flattening learning curves will be key. Nonetheless, our findings strongly validate that leveraging AIPs and linter collectively enhances the API development experience along critical factors of velocity and requirement fulfillment. In this context, the importance of effective API governance systems, as discussed comprehensively by Krintz et al. [7], becomes evident, emphasizing the critical role of APIs in digital IT infrastructure and the need for structured management and governance systems.

5.2 API Architect

Participants typically invested 2-3 hours to design APIs, while the API Architect leveraged GPT-4 to generate specifications in just minutes. However, expert assessment revealed the Architect’s outputs were rated average 5 out of 10 overall, highlighting deficiencies compared to manual approaches. While the API Architect demonstrated promising velocity, its technical proficiency and best practice compliance requires substantial improvement to match human designers. This tempers notions of AI achieving comparable API design quality and prompts further evaluation of how to best integrate AI assistance without compromising end results. The time savings warrant exploration of AI acceleration of rote tasks, provided the overall workflow incorporates sufficient human oversight.

The expert assessment of the API Architect provides valuable insights into the current capabilities of AI for API design. While the Architect demonstrated promising velocity by drafting specifications in minutes, the modest scores it received highlight meaningful gaps compared to human expertise. With average ratings between 4.5 and 7 out of 10 on overall quality, completeness, and AIP adherence, the results reveal clear room for improvement. The greatest strength lies in the Architect’s systematic approach, reflected in the higher evolvability score. However, reviewers cited critical weaknesses in resource modeling, protocol compliance, and missing functionality. This tempers RQ1 by indicating substantial room for improving technical proficiency of generative AI tools. However, the Architect shows potential for accelerating rote specification generation to augment designers through expert fine-tuning.

5.3 API Consumption Experience

The API consumption research analyses provide compelling insights into RQ2 regarding the impact of AIPs and linter on the user experience. Across all participants, including those possessing prior AIP proficiency, the study revealed a resounding 73% preference towards Group I designs harnessing AIPs and linter in tandem. This directly addresses the core research question regarding measurable improvements in efficiency and user satisfaction.

The statistically significant edge of Group I over Group III validates that Purposeful API design standards substantively enhance comprehension and utility for consumers. Moreover, the alignment between human evaluator preferences and API Architect scoring demonstrates the objectively higher quality of Group I specifications. Notably, while familiarity with AIPs contributes to an informed perspective, it does not universally translate to favorability, as 38% of participants opted for alternatives lacking AIPs or linter. Additionally, differentiation emerged across experience levels, with less seasoned engineers prioritizing readable and usable designs, while more seasoned ones focused on business value and problem-solving efficacy. This bifurcation spotlights the need for versatility in API design thinking, catering to multifaceted consumer viewpoints.

The consumption analysis provides unambiguous quantitative and qualitative proof that integrating AIPs and linter into API design workflows yields superior outcomes for end users. The areas warranting further attention involve enhancing intuitive integration for producers and accommodating the diversity of consumer preferences. Overall, the research strongly confirms that adherence to API development best practices translates to substantially improved experience for API consumers.