User Review Analysis for Public Food Aid Mobile Apps
User Experiences and Missing Design Opportunities in
Mobile Applications for Government Food Assistance
My Role
UX Researcher,
Principal Investigator
Deliverables
Timeline
February — July 2023
Research Objectives
Evaluating mobile apps for the Supplemental Nutrition Assistance Program (SNAP) from the end-user (beneficiaries) standpoint
Identifying what end-user values are supported or violated on SNAP apps
Ideating design recommendations that support end-user values for SNAP apps
How are public values supported or violated in existing SNAP mobile apps?
How might SNAP mobile apps be designed to support public values for beneficiaries?
How might SNAP mobile apps be designed to support public values for beneficiaries?
Method
Design Evaluation Framework
I evaluated the SNAP mobile apps using the Value-sensitive Design framework [1], consisting of:
Conceptual Investigation
Identifying stakeholders and their values
Empirical Investigation
Exploring how stakeholders perceive and experience technologies and embedded values
Performed secondary research to identify public values ideal to embed in government mobile apps
Conducted content analysis of mobile app reviews and r/foodstamps Reddit posts to understand which public values are supported or violated
Technical Investigation
Analyzing existing features or designing new ones to support values
Mapped each feature into each public value elements and translated insights into design opportunities
[1] Batya Friedman, Peter Kahn, and Alan Borning. 2002. Value sensitive design: Theory and methods. University of Washington technical report 2 (2002), 12.
Conceptual Investigation
Secondary Research Findings
Secondary research revealed four types of public values that government systems should support:
Quality of Information & Service
Accuracy, timeliness, relevancy, clarity, and detail of information. Transactional functionalities.
e.g., virtually completing, and submitting application forms or accessing archival data
Organizational Efficiency
Cutting processing time and costs
e.g., shared public information, IT enabled contact points, and well-trained public staff
Openness
Transparency and accountability of government systems
e.g., the disclosure of budget and expenses
Responsiveness
The extent to which a public organization interacts with citizens
e.g., online case tracking, ability to make and timely responses to inquiries
Empirical Investigation
Content Analysis Process
1. Collecting app reviews
Searched for the keywords
SNAP, food stamps, and EBT (Electronic Balance Transfer) in the Android and iOS app storesCriteria for selecting app stores:
(1) app store listing is in English;
(2) app allows SNAP benefit management (discerned by reading their app store descriptions);
(3) app is available in both Android and iOS app storesExtracted top-50 "most helpful reviews'' from each app store
For apps with fewer than 50 reviews, all reviews were sampled.
573 Android app reviews with average 57 words per review (SD=32)
523 iOS app reviews with average 79 words per review (SD=52)
2. Collecting r/foodstamps posts
Web crawling to collect the top 3,000 posts ranked by the number of votes
Filtered out deleted posts → 2966 posts
3. Thematic Data Analysis
Individually developed open codes with another research member
App Reviews: Identified the features in the current apps and their strengths and pain points
Reddit Posts: Identified features that could potentially address an issue raised in each postCollectively created a codebook
Created a codebook, comprising code names, definitions, and exemplary quotes, with each code symbolizing an app feature that appeared consistently across multiple app reviews and posts.Mapped the codes with our pre-established conceptual framework of four public values in ideal government systems
Performed another round of data coding
4. Mapping Features to Values
Quality of Information & Service: Balance update and management, SNAP eligible items/stores
Organizational Efficiency: Reduced Need for In-person Visits, Online Document Submission
Openness: Explanation for SNAP eligibility/allocation, Filing an Appeal or Complaints
Responsiveness: Interview/renewal notification, App Support
Empirical & Technical Investigation
Key Findings from Content Analysis
Value 1: Quality of Information and Service
Feature 1
Balance update and management
Automatic refresh of EBT balances in mobile apps improved efficiency and convenience for SNAP beneficiaries
“I hate when I’m in a rush and needing to find out what my ebt food balance is and I’m unable to find my last receipt...[the app] has saved me a lot of time and frustration”
- Prov A21
Users expressed relief and reduced anxiety, especially during purchases, by quickly verifying their balance
“when I’m shopping I could just open the app and be reassured of my balance so I don’t get that ‘holding up of the line, or awkward stare for it not going through’”
-EE A7
Some apps offered additional budget management tools like next benefit deposit date, transaction histories, and suggested budget plans
However, lack of such information could lead to suboptimal budgetary decisions
a Reddit poster, unaware of the next EBT deposit date, felt uncomfortable spending benefits and restricted food purchases to cheap, low-quality, or limited items.
Had the poster been informed of the next deposit date, the poster could have averted poor diet diversity and financial stress
Transaction histories were desired for transparency and accuracy, allowing users to cross-verify balances and monitor discrepancies
“the app [could] include SNAP purchase
transactions, in order to verify accuracy of balance shown”
- NAH A22
Feature 2
Showing SNAP Eligible Items/Stores
Users often seek clarification on where SNAP EBT cards can be used, facing potential embarrassment if denied at certain retailers
Only three apps included this feature, and even their data accuracy was questionable
“Today I went into subway and asked if I could use my snap and they said no. I just need clarification on whether or not I could use my
card on certain restaurants.”
- R144
Users often seek clarification on where SNAP EBT cards can be used, facing potential embarrassment if denied at certain retailers
Only three apps included this feature, and even their data accuracy was questionable
“Today I went into subway and asked if I could use my snap and they said no. I just need clarification on whether or not I could use my card on certain restaurants.”
- R144
Apps listing SNAP-eligible items are valued for saving time
Beneficiaries expressed embarrassment when having to abandon purchases due to items not being SNAP-eligible
“I’ve been freaking myself out about going to the store...I worry it’ll look obvious that I’m worried about affording stuff...and utterly horrified of having to put groceries back
at the register because I can’t afford them. Is there some kind of list I can look at so I can make a plan before I go?”
- R204
Features in apps that help users prepare shopping lists based on SNAP eligibility contribute to maintaining dignity in public settings
“I could make my [shopping] list based on ad prices or prices I knew well enough to estimate. I knew before I
got to the store if I was within my budget”
- Prov G40
SNAP beneficiaries report feeling embarrassed by cashier attitudes when using EBT cards, highlighting the need for discreet transaction designs
“The cashiers always make it a point for everyone to know
I’m using pebt, and when I pull my card out they go from friendly to annoyed and judgemental. I get really embarrassed because everyone in line just looks at me”
- R742
Value 2: Organizational Efficiency
Feature 1
Reduced Need for In-person Visits
SNAP app users appreciate the cost and time savings due to fewer in-person government office visits
Reviews indicate challenges in accessing physical offices, especially for those without personal transport or loss of income from taking time off work, emphasizing the socioeconomic benefits of digital services
“This app...saves me...money...i don’t have to drive myself to the Building...therefore i won’t need to spend gas...that can be used on other necessities”
- MCB G29
“[take] time off work, which caused loss of income”
- CP G12
SNAP apps help alleviate feelings of embarrassment and preserve dignity by providing assistance without face-to-face interactions, reducing the psychological strain associated with in-person visits to government offices, bypassing long lines and negative attitudes at government offices
“Using this app is a plus...I don’t...have...to mentally prepare myself...if the worker assigned to [me] has had a bad day”
- NAH A3
“sincerely "Thanks" you for...helping us maintain our dignity by providing accessible information online”
- YTB A14
Feature 2
Online Document Submission
Traditional communication via mail between government offices and SNAP beneficiaries led to delays, missed notifications, and even termination of benefits
“The current address I’m at is not feasible for me to
get mail from DHS...I’m looking into a UPS PO box but I don’t know if that will work?”
- R73
Online submission occasionally led to inadvertent benefit cancellations or denials, emphasizing the importance of ensuring correct document uploads in the right format to prevent errors
“The pictures may be clear on your phone but the caseworkers will not get a clear picture on their end, so they will deny your case, saying you didn’t turn in your documents or that they were too blurry. It happened to me 7 times already”
- YTB A43
Value 3: Openness
Feature 1
Explanation for EBT Denial or Amount
Many Reddit posters expressed confusion and frustration over not receiving clear explanations for their SNAP application denials, highlighting a communication gap and the need for increased transparency throughout the application process
“Someone went over my application with me and said everything looks good and told me that my household is
qualified for food stamps. I logged on last night and saw my application was actually denied and no reason stating why.”
-R1584
Some applicants received explanations for their ineligibility, but they were too technical or lacked details, making it hard to comprehend the decision-making process
Specific information on denial reasons, such as residency requirements or income calculations, and actionable steps for correction could increase applicants' chances of approval upon reapplication
“They have me listed as ineligible and the reason listed is TECHNICAL_ELIGIBILITY_IN_SANCTION_PERIOD. Does anyone know [what] this means?”
- R295
“I moved to IL a little over a month ago, and just received notice that my application was denied because I don’t meet residency requirements. Do I have to live in the state for
a certain amount of time to receive benefits, or is there something weird going on here?”
- R1998
Beneficiaries also expressed confusion over sudden changes in benefit amounts, seeking detailed explanations for discrepancies to address concerns about potential complications and ensure the correct utilization of benefits
“there was a deposit of $199...I’m supposed to get
$160...I’m not exactly sure what to do or who I should call about this. Does anyone here have experience with this and what should I do?”
- R95
Feature 2
Filing an Appeal or Complaints
Current SNAP apps lack a feature to request appeals, forcing applicants to call government offices, leading to long waiting periods and suggesting the need for app improvements to support the appeal process
“we got denied for "Too much income"...I’ve tried...calling multiple times...just to get the auto reply...to call back
later”
- R2719
Some beneficiaries reported not receiving necessary assistance from caseworkers when attempting to appeal, with instances of caseworkers discouraging appeals and making negative statements, with reports of hostility, discrimination, and false accusations hindering beneficiaries' ability to navigate the benefit decision-making process and access entitled benefits
Fear of potential retaliation led to reluctance in filing complaints against caseworkers, highlighting the need for measures to improve interactions with caseworkers and protect the rights and dignity of beneficiaries during the appeal or complaint filing process
“yelled at me and told me that what I was asking for would be fraud. . . She said I could ask for a hearing but the ruling wouldn’t be in my favor”
- R71
“the interviewer was very harsh and hostile towards me...asked very personal questions...said I alarmed the system(?) because i was "too young for food stamps" and that he was worried I could be a victim of human trafficking...I was denied again”
- R97
“if I request a new caseworker, would they make a note of it somewhere so that people could potentially discriminate against me in that way?”
- R140
Value 4: Responsiveness
Feature 1
Interview/Renewal Notification
Many app users expressed frustration over untimely app notifications, specifically related to document renewal submissions or upcoming interviews, leading to benefit denial or cancellation
“the app doesn’t notify you...until a few days later and by that time your case will get rejected. They need to update this app and actually put some care into this. [This] is people's food”
- NAH A7
Concerns were raised in 15 Reddit posts and 14 app reviews regarding missing calls from caseworkers, with reports of failed communication attempts leading to benefit denial
“I was sent a letter to...have my phone interview...I called and called and called and left several messages...and no one got back to me...Now, they’ve denied me, saying I didn’t complete my phone interview and that it was my responsibility”
- R2
Feature 2
App Support
40 app reviews expressed dissatisfaction with the app support service or lack thereof, highlighting the potential for technical glitches or lack of support to lead to benefit delays, denial, or cancellation
Dissatisfaction with the lack of real-time or immediate technical support was evident, with instances of users experiencing account lockouts and delays in resolving technical issues, leading to missed deadlines and loss of benefits
“I don’t like feeling helpless...I’ve been trying to do what I gotta do to not delay the process but the app wants to keep crashing”
- NAH A17
Outsourcing the development of SNAP mobile apps to third-party developers has resulted in insufficient app support, creating a situation where neither developers nor government staff adequately address user technical issues
Developers dismisses issues as not their problem and government employees lacked the necessary expertise to assist effectively
User demands for improved government staff training to better understand app functionality and provide support were noted
“it has no option to speak to a representative about this
app..my county assistance...are just as lost as me”
- CE A1
“I was told it’s not our problem”
(CE G6)
Inefficiency in technical app support has led some beneficiaries to believe it was a deliberate strategy by the government to hinder benefit access, fostering distrust
“Guessing this is the new way of not letting people recertify so they can deny your benefits”
- NAH A15
Lack of app support contributed to users feeling neglected and marginalized, while a well-maintained app conveyed respect towards beneficiaries, highlighting the ethical implications of effective technical support
“Too bad the developers...have apparently abandoned the app and those that depended on it”
- EE A25
“This app shows that you guys actually do care about people in need...and you do it with dignity and respect”
- Prov A15
Empirical & Technical Investigation
Implications from Findings
We identified major features that could support each public values.
However, the analysis revealed that dignity, encompassing self-image in relation to treatment from others and individual agency, is inadequately represented in current SNAP mobile apps. Current designs of SNAP apps and programs were found to compromise beneficiaries' self-esteem and agency, prompting the need for design implications to uphold their dignity.
Implication 1
Enhancing Self-Esteem of SNAP Beneficiaries
[1] Lynn Dombrowski, Ellie Harmon, and Sarah Fox. 2016. Social justice-oriented interaction design: Outlining key design strategies and commitments. In Proceedings of the 2016 ACM Conference on Designing Interactive Systems. 656–671.
[2] Jill P Dimond, Michaelanne Dye, Daphne LaRose, and Amy S Bruckman. 2013. Hollaback! The role of storytelling online in a social movement organization. In Proceedings of the 2013 conference on Computer supported cooperative work. 477–490.
Implication 2
Enhancing Agency of SNAP Beneficiaries with Contestability
[1] Daniel N Kluttz and Deirdre K Mulligan. 2019. Automated decision support technologies and the legal profession. Berkeley Technology Law Journal 34, 3 (2019), 853–890.
[2] Henrietta Lyons, Senuri Wijenayake, Tim Miller, and Eduardo Velloso. 2022. What’s the appeal? Perceptions of review processes for algorithmic decisions. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems. 1–15.
[3] Gerald S Leventhal. 1976. What should be done with equity theory? New approaches to the study of fairness in social relationships. (1976).
[4] Jason A Colquitt, Donald E Conlon, Michael J Wesson, Christopher OLH Porter, and K Yee Ng. 2001. Justice at the millennium: a meta-analytic review of 25 years of organizational justice research. Journal of applied psychology 86, 3 (2001), 425.