Improving API Documentation Usability with Knowledge Pushing
Uri Dekel and James D. Herbsleb
Institute for Software Research, School of Computer Science
Carnegie Mellon University
5000 Forbes Avenue, Pittsburgh, PA 15213 USA
The documentation of API functions typically conveys detailed specifications for the benefit of interested readers. In some cases, however, it also contains usage directives, such as rules or caveats, of which authors of invoking code must be made aware to prevent errors and inefficiencies. There is a risk that these directives may be “lost” within the verbose text, or that the text would not be read because there are so many invoked functions. To address these concerns for Java, an Eclipse plug-in named eMoose decorates method invocations whose targets have associated directives. Our goal is to lead readers to investigate further, which we aid by highlighting the tagged directives in the JavaDoc hover. We present a lab study that demonstrates the directive awareness problem in traditional documentation use and the potential benefits of our approach.
Modern software systems combine code written by many individuals and make heavy use of external libraries and Application Programming Interfaces (APIS). Stakeholders in these settings are not likely to be fully acquainted with all current knowledge about artifacts and services in the project and third-party code. When focused on a particular code fragment, however, it may be critical for them to be wellversed in all the services that it uses. A lack of awareness of usage guidelines and caveats can result in runtime failures and maintenance difficulties.
Since many API functions are meant for widespread use, their authors are motivated to invest significant effort in creating elaborate documentation that fully specifies everything that a client may need to know about a function. Such specifications are crucial for assuring correctness during inspections and the development of testing plans [5, 12]. Unfortunately, the potential consumers of this documentation spend much of their time browsing code  that includes numerous method invocations. They are therefore limited in the time and effort they can spend on any particular call and may therefore miss important information.
Consider, for example, the documentation of method setClientId from the Java Messaging Service (JMS) API, which is depicted in Fig. 1 as it is displayed in the Eclipse IDE. The detailed narrative covers many details, including purpose, configuration, and exceptions. Stakeholders skimming the text may miss the highlighted sentence deep within the third paragraph, which defines a protocol that explicitly forbids prior method invocations on this object.
This problem is compounded by the significant fan-out (number of outgoing edges in the call graph) of many functions. Sifting through the documentation of one invoked function is challenging enough, so searching all targets for important knowledge is even less practical. For instance, consider the code excerpt of Fig. 2, which creates a message queue in JMS. When writing or examining this relatively straightforward code we must decide which, if any, of the four invoked methods should have their documentation examined for additional requirements. The IDE support does not offer any cues drawing us to (or away from) any particular call, though we might be inclined to examine the complex-looking calls that take one or more arguments.
It turns out, however, that the documentation of the seemingly straightforward call to createQueueConnection mentions that connections are created in a “stopped mode” and no messages will be delivered until their start method is invoked. Since this detail is not mentioned in the queue’s receive method, a lack of awareness of this directive here may result in the program hanging when messages are eventually retrieved.
Casual observations confirm that developers only investigate the documentation of a small portion of invoked methods. We suspect that this may also have an indirect effect on the willingness of authors of project artifacts to document less “visible” functions. Such functions are often written with specific assumptions, expectations, and limitations in mind, but developers presumably weigh the potential future benefits to their peers against the immediate costs of capturing this knowledge. Increasing the prospects that the documentation would actually be read may create better incentives for preserving it.
We also note that project artifacts are likely to have associated action items or bug reports . Stakeholders need to become aware of these caveats in invoked functions to avoid depending on a faulty implementation.
About this work
The goal of our work is to make developers examining a code fragment more aware of important directives that are associated with the invoked functions. We use this term to distinguish knowledge that has immediate implications for the clients from specifications that can be actively consulted to improve one’s understanding. We believe that such awareness not only will help stakeholders avoid or fix certain invocation errors, but also will assist those learning to use the API from code samples.
This paper presents a solution based on the premise that if we: 1) explicitly identified important directives in the function’s documentation, 2) could unobtrusively signal which call targets have associated directives, and 3) offered lightweight means to explore the utility of the information without changing context, then: developers will be more likely to become aware of directives.
We implemented this approach as part of our eMoose memory aid for software practicioners, which currently supports JAVA developers in the Eclipse IDE. Our Eclipse plugin manages a knowledge space that maps atomic knowledge items (KIs) to specific functions. All KIs in this paper are directives or to-do items. The space is populated by manually tagged text in source code comments, and by downloadable collections of KIs. As part of this work, we systematically surveyed core parts of several major APIS, including the JAVA standard library, Eclipse, JMS, and apachecommons. We tagged several thousands of directives and packaged them for users.
Our plug-in continuously tracks the contents of the JAVA editor window and identifies method calls whose static (or possible dynamic) targets have associated directives. As can be seen in Fig. 3, it then highlights these calls by surrounding them with a box and placing a small icon on their line. These cues should alert users to the availability of potentially relevant directives in certain calls while offering some assurance of their absence on the other targets.
When the user hovers over the decorated call, the usual tooltip showing the documentation of the target method (Fig. 1) is augmented with a lower pane listing the directives (Fig. 4) to facilitate their consumption.
Two natural concerns about this approach are whether these interventions have the desired beneficial effects and whether these effects would be offset by distraction and information overload. In addition, there is at present only limited evidence that developers actually miss important directives with standard techniques. We address these and other concerns with results from a comparative lab study in which developers were tasked with fixing errors within small code fragments. eMoose users were significantly more successful than non-users, and without being significantly distracted.
Contributions and importance
The first major contribution of this paper is in demonstrating, via a controlled lab study, that developers indeed fail to become aware of important directives in the functions they invoke. This carries significant implications for function authors, and should raise questions about documentation practices and API usability. Since developers frequently learn new APIS from code examples, our findings also have implications for current learning practices.
Our second contribution is in demonstrating (within the limitations of the lab) that decorating method invocations is an effective way to alert readers to potentially important information associated with these targets, and without significant overload. These results are also important because such cues may be effective for other types of information and perhaps for other mediums and link semantics.
Before we proceed, it is important to clarify the difference between our approach and the very active research field of automated conformance checking. Techniques that enforce design-by-contract (e.g., [1, 6]) allow function authors to formally specify a usage contract and then use static or dynamic analysis to ensure the conformance of invoking code. While these techniques can be extremely useful in automatically detecting certain violations, they require significant investments and skills from those authors. In addition, we note that some function documentations convey contracts that are too abstract to be formally specified. Others convey important information whose violation is not necessarily an error, such as performance caveats.
There is therefore a need for a complementary approach for the vast majority of directives that have not yet been formalized and may never be. Our approach focuses on increasing the clients’ awareness of a directive rather than offering automated assurances. It leverages natural text in existing documentation under a premise that manually tagging directives is significantly more practical and general than writing formal specifications.
The novelty of our approach lies in both the distinction made between directives and the rest of the narrative, and in the idea of “pushing” them to the awareness of clients. This may reduce the risk for errors and potentially improve the effectiveness of existing documentation practices.
Outline: The rest of this paper is organized as follows: Sec. 2 discusses method documentation in JAVA, and Sec. 3 describes the nature and types of directives. Our tool is described in Sec. 4. We present the design of our lab study in Sec. 5 and its results in Sec. 6. We discuss these results in Sec. 7, and the study’s limitations in Sec. 8. We conclude and discuss current research directions in Sec. 9
Method documentation in JAVA
Methods in JAVA are documented via a JavaDoc comment block placed just before their declaration in the source code. Though any text is allowed, official guidelines [5, 12] recommend a specific structure: 1) A “summary sentence containing a concise but complete description of the API item”. 2) An “implementation-independent description and specification that must include boundary conditions, parameter ranges and corner cases.”. 3) A series of tagged lines that list parameters, return values, exceptions, and other metadata, even if they are obvious or redundant with the documentation text.
While the documentation of every class in the API is typically provided as an HTML file generated by the JavaDoc tool, most IDEs provide means to read the JavaDocs of specific methods from within the editor. Selecting a method from a class outline or auto-complete list presents its JavaDocs, but more importantly, hovering over a call presents the JavaDocs of its target in a tooltip window, which we call the JavaDoc hover.
In our survey of APIS we found many materials that bloat JavaDocs and might make directives even more difficult to spot. These include general descriptions that would fit better at the class level, and implementation details that are only relevant to maintainers. A prevalent problem with the JavaDocs of toolkits that rely on subclassing, such as SWING, is that they mix the information relevant to clients of a class instance with information that is relevant only to subclass developers who override the method.
The subclassing mechanism also presents significant challenges for the consumption of documentation. In object-oriented languages like JAVA, polymorphism  allows a variable declared with a certain type (termed static type) to contain at runtime instances of subtypes (called dynamic types). When a method is invoked on the variable and there is an overriding version in a dynamic type, the latter is invoked. However, since the dynamic type of a variable cannot be predicted and as it may change during runtime, the actual target of such calls cannot be determined.
For this reason, most IDEs merely present the JavaDocs for the static target, potentially leaving users unaware of new or conflicting directives in an overriding version. This can have severe consequences if conformance  is violated and the documented behavior of the overriding version conflicts with that of the overridden. Though deprecated, we encountered such violations even in quality library code.
Though directives play a central role in our approach, there are no strict criteria to distinguish knowledge that should be “pushed” into the awareness of clients from materials that can “wait” for the user to actively seek them. These decisions ultimately fall to whoever creates the eMoose knowledge-items for that API. Our intention, however, is for candidate directives to meet two requirements. First, they must demand or imply concrete steps which the client can follow and for which it is straightforward to come up with a violating example. Second, they should capture nontrivial, infrequent, and possibly unexpected information. For example, since many methods specify policies for null argument values, these policies should not be considered directives.
We note that in our survey of standard APIS, many directives were relatively straightforward to identify and distinguish from the rest of the narrative. Authors frequently used the imperative to address clients or talked about them in the passive. They also frequently emphasized these clauses with phrases such as be aware that or note that, and words such as must, should, warning, etc. Most difficulties in determining whether a clause constituted a directive occurred when such constructs were not used.
To illustrate the breadth of important issues covered by directives, we now present prominent types which we frequently encountered in our survey of API documentation.
We begin with imperative directives, which represent contract elements whose violations can have immediate consequences, delayed and unpredictable runtime effects, or future compatibility and extension issues.
Restrictions: Many methods explicitly restrict the set of clients and contexts from which they may safely be invoked, such as “do not call from UI thread” or “for use only by debugging code”. Interestingly, many of these restrictions are defined in abstract and human-readable terms that would complicate formal specification and automated conformance checking. For instance, some refer to conceptual sets of artifacts that may be difficult to enumerate, such as “toolkit code” or “debugging infrastructure”, and others to program states that are difficult to evaluate, such as: “only from actions that are installed on a button”.
Protocols: Many methods are designed to be used as part of a sequence of actions that set or transform the object’s state. Their documentation includes protocol constraints that specify what should occur before and after the call. Common examples state that the method must only be called once, or that it must be called prior to or only after a call to some other method. As with restrictions, many protocols are described in general natural text descriptions that would be difficult to formally specify and validate. For example: “All data in returned stream must be read prior to getting value of other column”, “To reuse a closed internal frame, add it to a container”, or “Cursor must be on the insert row before this call”.
Locking: Some methods present clear synchronization requirements for their use .
Parameters and return values: While all methods are expected to specify the nature and ranges for all parameters and return values, some convey unexpected restrictions or requirements which can be considered directives. For instance, the replaceAll method in String warns users from including $ and \ characters in the replacement string, since it is implemented via regular expressions. Many methods that receive or return complex objects or platform resources indicate whether internal copies are used, and what the disposal responsibilities are.
We now turn to directives that are more informative in nature. These are meant to be taken into consideration but not necessarily acted upon, and may therefore be outside the domain of tools for automatic conformance checking. In some cases, however, ignoring these directives can lead to actual errors.
Alternatives: The documentation of some methods suggests that a different method be used. Though often related to encapsulation or deprecation, some alternatives offer fundamentally different functionality, so a lack of awareness may result in breakdowns. For example, the JavaDocs for putLayer in SWING’s JLayeredPane suggest using setLayer to get “desired side-effects like repainting”.
Limitations: Some methods are less robust and comprehensive than their name may suggest; they may specifically delineate the inputs or situations they are capable of handling or describe limitations on their outputs. Other methods explicitly announce that certain events or side effects will not occur. For instance, adding or removing items from SWING containers does not cause visual change until validate is called.
Side effects: Some methods have additional effects to
those conveyed in their signature and summary sentence.
For example, many getters perform lazy creation. Setting
certain properties in SWING automatically sets additional
properties. Changing the row sorter in a
JTable will also
clear selections and reset row heights. Disposing of the last
displayable window in the JVM may terminate it.
Performance: Some methods use algorithms or services that have performance implications of which clients must be aware. For example, many methods return copies of complex objects and therefore suggest caching or avoiding excessive calls. Other examples include: “should only be called on highly available system unless a progress monitor is set up”, or “names should be provided to avoid querying the entire registry”.
Threading: In addition to defining locking requirements or restricting calls to specific threads, some JavaDocs present additional details related to multithreading behavior. For example, some mention lock changes and blocking, while others suggest that clients manually synchronize groups of calls.
Security: Some methods cause security vulnerabilities that may be relevant in certain contexts.