During my internship, I conducted a comprehensive product evaluation of Connected Papers, a software tool designed for applied scientists and researchers. Using expert analyses, in-depth interviews (IDIs), and consumer reviews, I developed data-driven recommendations to enhance the tool’s effectiveness and user experience.
To enhance user satisfaction and overall usability, multiple research phases and methods were conducted and an evaluation report was submitted to project stakeholders.
View Evaluation Report
Consumer Reviews
Performed IDI’s with fiveusers to understand reasoning behind poor usability ratings.
Expert Analysis
Performed IDI’s with fiveusers to understand reasoning behind poor usability ratings.
Task Analysis
Performed IDI’s with fiveusers to understand reasoning behind poor usability ratings.
Interviews
Performed IDI’s with fiveusers to understand reasoning behind poor usability ratings.
A look into what the project entailed across the many changes, iterations, and explorations! So much to delve into.
Research & Discovery
Understanding what current solutions are serving the market and what issues our target segment may have using these.
.svg)
Brainstorming & Ideating
We went through many rounds of brainstorming to uncover user insights that guided our design process.
.svg)
Wireframing
Both individually & collaboratively, we sought to gray box solutions and decide on a path forward for Masterclass.
.svg)
High Fidelity & Final Designs
With branding, illustrations, badging, logos, packaging and marketing, our final design embodies not only our target market, but the originality of our team.
Before implementing the evaluation methods, an evaluation proposal was submitted to justify the approach and persuade stakeholders of the potential for a good return on investment (ROI).
View Proposal
Criterion 1: Accuracy - Bibliographic Citations
Criterion 2: Usability, Efficiency, Satisfaction - Task Analysis & IDI’s
To better understand the existing usability issues, I scrapped the web for consumer reviews. While majority of consumers gave positive ratings, many mentioned that the software took a significant amount of time to learn along with other small things like bibliographic citations.
Hypothesis
Based on the consumer reviews, we hypothesized that certain tasks involved in creating a graph were more difficult and time-consuming. This hypothesis guided the task analysis.
To evaluate the accuracy of Connect ed Papers, I tested the bibliographic citation generator and compared its outputs to academic standards.
Error 1: Author
The authors' names are inverted (last name first, followed by initials with no full names).
Error 2: Publication Date
The publication year is in parentheses followed by a period.
Error 3: Article Title
The title of the article is in sentence case and not in quotes
Error 4: Volume
The volume number is in italics.
Three tasks were crafted to measure the usability and efficiency of the software. During task completion participants performed a cognitive speak aloud. Additionally screens were recorded to capture task time and clicks.
Task 1: Select an Article & Generate a Graph
Theme a: Browsing results
Participants spent the most time on step 3, browsing the search results. Investigation revealed that students primarily focused on reviewing article titles and descriptions to determine which article they would select as their origin article. This insight was mentioned by all participants in some capacity.
.png)
Theme b: Multiple search queries
Participants viewed Connected Papers similar to academic databases and applied strategic search techniques, including boolean operators. However, they encountered extended search durations because the software lacks comparable search capabilities found in other databases. This not only lengthens the process of finding an origin article but also detracts from the tool's overall usability and efficiency.
Theme c: Search strategies
Several participants needed multiple search queries to meet two objectives: finding a relevant article and locating one with sufficient citations for graph creation. Specifically, Participants 2 (P2) and 5 (P5) each entered two search queries to find their origin paper, while Participants 3 (P3) and 5 (P5) struggled to select articles due to insufficient citations for graph creation.
Task 2: Explore the graph until you feel you have a good understanding of its functions.
Theme a
Most participants expressed a strong need for initial training on using the generated graph, as they often resorted to "trial and error" to navigate its functions. This confusion extended Task 2 completion time to 4 minutes and 1 second, this is reflected in the softwares usability rating of 6.8/10.
Theme b
Participants viewed Connected Papers similar to academic databases and applied strategic search techniques, including boolean operators. However, they encountered extended search durations because the software lacks comparable search capabilities found in other databases. This not only lengthens the process of finding an origin article but also detracts from the tool's overall usability and efficiency.
Task 3: Navigate to your article's seminal and derivative works
Theme a
Several participants were unfamiliar with the terms "seminal" and "derivative" works, asking evaluators for explanations before using the tool. This underscores the need for clear definitions and explanations within the platform to aid user comprehension and effective use.
After participants completed each task and became familiar with the software, they were asked three questions to gauge its perceived usability, efficiency, and satisfaction.
Despite discrepancies between task times and interview feedback, as well as varying correlations between usability, efficiency, and satisfaction ratings, participants attributed these differences to the novelty of the software. All participants described Connected Papers as niche, new, and cool.
Based on the data collected from expert analysis, task analysis, and interviews, recommendations were created and prioritized according to their severity and ease of implementation.
Connect with People
When conducting research, my attention to detail often focuses on the logistical aspects of the study. In this project, I made sure to connect with participants and truly listen while maintaining an unbiased relationship.
Importance of Probing
Interviewing participants about their ratings on usability, efficiency, and satisfaction, I found that using probing questions elicited insightful and important details that otherwise might have been overlooked.
Planning = Success
I spent many hours rigorously reading evaluation literature, writing the evaluation proposal, and writing the evaluation plan. I found this to be extremely helpful when I needed clarity during the interview process and final write-up.
Laugh along the way
Research, due to its scholarly nature, is often time-consuming and cognitively demanding. Finding opportunities to take a deep breath or share a laugh can make the process more manageable and enjoyable.
.png)
.png)