PSA: To learn more about how to spot, avoid, and report scams—and how to recover money if you’ve paid a scammer—visit ftc.gov/scams. If you spot a scam, report it to the FTC at ReportFraud.ftc.gov
Operations Asset Risk Assessment and Management Product Design
Reports of Amazon scams increased by 500% between June 2020 and 2021, with victims losing more than $27 million to scammers.
Overview
As the Lead Product, UX, UI Designer, I was responsible for the vision, the ideation, the discovery, the end-to-end design, and the interactive prototypes that spans and accommodates across all tenants of the global Amazon business umbrella for content risk management solutions.
The new Content Risk Assessment Operational Workflow design began in 2021 and was first released and operational in June 2023 to address modern, current Risk Assessment Operations workflow needs and to support new and emerging media types.
An MVP in November 2022 launched a new workflow for Amazon Worldwide Global teams Books Content Risk Assessment Operations.
Project Role
- Product Design
- Visual Design
- Icon Design
- Illustration and Branding
- CX User Research
- CX User Studies
- CX/UX/UI Wireframing
- User Journey + Onboarding
- CX/UX/UI Interaction with React
- CX/UX/UI Prototyping with React
- CX/UX/UI Workflow + Prototyping with React
- CX/UX/UI Usability Testing + Observation
Challenge
Amazon has many different types of fraud to thwart. Scammers are adopting new modes of fraud at scale daily and Amazon must be proactive to keep the businesses it operates and the customers it serves safe from fraud and abuse.
Operations Experience Challenge Breakdown
- Create a UX framework and a reusable UI for various Amazon businesses to spin up their Content Risk Management Workflow that relates to their specific needs based on their business requirements and regulations.
- Design a platform to quickly assess fraudulent User Generated Content (UGC) for all business divisions and operational workflows within the Amazon Worldwide Corporation and all its holdings.
- Design a suite of tools to identify and reduce harm, risk, and fraud before it's made available to the public for consumption and purchase.
- Create a UX and UI layout that works within legacy systems and is flexible for new systems, third-party apps, and Salesforce.
- Design for Operational Excellence and End-to-End workflows for Operations, Specialists, Managers, Accounts, and more to determine and take action on Fraud.
- Design for the complexity of Global Worldwide Regulations and Compliance on assets and IP. For example, Germany's regulations and Costa Rica's regulations on content from World War II are different. Amazon must comply with national, international, along with the regional laws on ecommerce, copyright, and more.
- View general log and diagnostics information on the issues resolved, the patterns of fraud, and the rate of decision-making to inform the AI and ML to support a new robust decision engine and algorithm to reduce the need to human-centered reviews.
Project Goals
Create a Fraud and Risk Assessment Platform for any and all Amazon businesses that comply with international regulations. Ensure it can work with any and all technologies within the Amazon Worldwide Umbrella and adaptable to support new and evolving content types.
Create a method, workflow, UX and UI for image fingerprinting using Amazon Rekognition to be integrated into the new Fraud and Risk Assessment Platform. For the first time, by integrating and using Amazon Rekognition, it will add advanced computer vision capabilities to the Operations and Risk Management tools. The service is powered by proven deep learning technology and it requires no machine learning expertise to use. Features can be added to detect images within images, objects, text, unsafe content, it can analyze images and videos, and compare image content including embedded text within images to quickly scan for fraud, plagiarism, image content violations, and any risk to IP and copyright laws.
Create standardized new UX workflows for Image Analysis based on tenant needs:
- Object, Scene, and Concept Detection - Detects and classify objects, scenes, concepts, and celebrities in images.
- Text Detection - Detect and recognize printed and handwritten text in images in a variety of languages.
- Unsafe Content - Detect and filter explicit, inappropriate, and violent content and images. Detects granular unsafe content labels.
- Custom Labels - Build custom classifiers to detect objects specific to your use case, such as logos, products, characters.
- Image Properties - Analyze image properties like quality, color, sharpness, contrast.
Create standardized new UX workflows for Video Analysis based on tenant needs:
- Object, Scene, and Concept Detection - Detects and classify objects, scenes, concepts, and celebrities in videos.
- Text Detection - Detect and recognize printed and handwritten text in videos in a variety of languages.
- People pathing - Track identified people as they move across video frames.
- Facial Analysis - Detect, analyze, and compare faces in streaming or stored videos.
- Unsafe Content Detection - Detect explicit, inappropriate, and violent content in videos.
- Video segmentation - Automatically identify useful segments of video, such as black frames and end credits.
Amazon Fraud Detector is particularly useful when we enter new markets or geographies, since the enrichment of inputs like IP Address and Email Address provides an immediate boost even when we do not have robust historical data. By using Amazon Fraud Detector’s automated Online Fraud Insights model, we started capturing 35% more fraudulent account signups in new geographies. Moreover, Amazon Fraud Detector allows us to train models for new parts of our business in half a day – the process previously took at least a week.”
Tim Wallach, Head of Fraud Prevention - AWS
Key Deliverables
- Create prototype and research plan for new product design and workflow
- Identify usability issues on resolving Fraud reviews with contextual content to efficiently determine the outcome of the review
- Determine the critical features needed to resolve Operational tickets in need of human review.
- Design a new product to visualize and review image-based content with a new image-based fingerprinting framework
- Create functional prototypes in Figma, in React with a new, common design system, and research plans for new product designs and workflows
- Design UI, UX, icons, a design system with tokens, and symbols for tools and product integration into the new Content Risk Management Platform
- Product design and prototype for integration into global customer service intake services
Operations Asset Risk Assessment and Management
Catching, Identifying, and Addressing Issues of Fraud before Publishing to Amazon Worldwide and Amazon Books
From The New York Times
Fake books, false accusations
User Research
The means and ways of fraud are frustrating for both the seller, the customer, and for Amazon as the platform provider. Amazon Discussions board allows customers to discuss and identify scams and Amazon's policies on fraud, risk, trust, and safety.
What matters most to customers?
Core UX Customer: Internal Accounts, Fraud, Risk, Safety, Trust Amazon Worldwide Operations Teams
Stakeholders: Amazon Worldwide Operations, Amazon Books, Amazon Legal, Amazon Engineering
- Fast and Efficient workflow is key. Speed, accuracy, and a clean, clear design matters most to customers. A clear path from starting a task to completing a task in the quickest, most successful way.
-
Tenants want a tool that enables them to rapidly develop business-specific CRM workflows and complete their specific tasks, search, find context, escalate, and more at a rapid rate and maintain their business SLA. They want a CX that only shows their required needs. For example, if a T2 is reviewing a PDF text-only document for
Content Appropriateness and Disappointing Content, they have no need to see any UX components for video or images. - C4A will allow tenants to customize an E2E workflow to suit their needs with the use of “widgets” or customizable components CX implementations such as UI components for images, text, like reusing carousel structures, image comparison widgets, and accessing the Rekognition API.
Hybrid Push Queue Research Plan
This research's goal is to understand the needs of a hybrid queue for C4A. For C4A, the 'hybrid' workflow is a push/pull hybrid queue with the ability to stop auto-loading tasks from the push queue model.Research Goals
- Determine the best user experience for a hybrid push / pull queue workflow
- Identify user workfl ow needs based on tasks and workfl ow of a baseline user for C4A
- Determine user expectations for hybrid use of push / pull queue workflow
- Determine if the autoload task switch in UI is intuitive
- Determine habits and needs (selecting tasks, searching for tasks, auto-loading tasks, etc)
Protocol - Survey Questions
These questions center around the T3 Specialists. A Specialist reviews tasks, reviews, accounts, appeals, calibrations, and escalations. The role requires a user to pick-up tasks (pull), complete auto-loaded tasks (push), and use search tools to find tasks (pull).Start Work
Q: You have just logged into C4A as a T3 Specialist at the beginning of your shift. How will you start work?- Press the Start Work button.
- Select a task in the queue and press the Start Work button.
- Select a task in the queue and double-click to open the task directly
- expect the next task in your queue to auto-load by default.
- expect to return to the dashboard.
End Work
Q: You have been working in C4A all day and it's the end of your shift. How will you end your shift?
- Turn off the auto-load task button. Complete last task, end day. [hybrid workflow]
- Exit review (lose completed task work), end day. [hybrid workflow]
- Complete last task, end day. [pull-only workflow]
Q: Should auto-loading of tasks be the default T3 Specialist workfl ow experience for C4A?
- yes, if the Start Work button is used.
- yes! yes, the next task should auto-load by default. User must turn off auto-load to stop.
- no, always return to the queue after completing a task, even if the Start Work button is used.
- yes
- no, only a pull queue
- it depends on a tenant by tenant basis
T1 MANAGER: HYBRID QUEUE NEEDS. FOR T1 MANAGERS WHO ALSO HAVE T2 AND/OR T3 ROLES:
Q: As a T1 Manager, what is the expected default queue experience?
- Pull queue: assigning tasks, maintaining queues of tasks, completing tasks
- Push queue: auto-loading assigned tasks to next available user
- Hybrid queue: both push/pull capabilities
- a task will load for review. After completing the review, you will return to dashboard.
- a task will auto-load for review. After review, another task will load for review.
- you expect a way to stop auto-loading reviews as needed.
Other questions on queues:
QUEUE DETAILS:
Q: As a T3 Specialist, what information do you need in your queue?
- Customer ID
- Task Name
- Title Information
- Account ID
- ASIN
- Age of task
- Policy flags and violations
- Author / owner
- Year published / created
- Status of title
- State of review
- Publisher ID / other ID
- Queue name (what queue this task belongs to)
- Customer ID
- Task Name
- Title Information
- Account ID
- ASIN
- Age of task
- Policy flags and violations
- Author / owner
- Year published / created
- Status of title
- State of review
- Publisher ID / other ID
- Queue name (what queue this task belongs to)
We will conduct a survey in the Field Sense questionnaire system on 10/21/2022. Participant survey will be distributed after reviewing the T2 Push Queue workfl ow. Screens of a T3 hybrid queue dashboard can be viewed within the meeting of the C4A Product Council during the survey.
The Field Sense questionnaire system does not support media files or urls.
Participant Profile
Participants will be individually selected for this study using PM guidance. All participants are part of the C4A Product Council.
Expected Outputs
- Aligning on default experience for hybrid queue workflow
- Understanding user needs for T3 push/pull hybrid model
- Aligning on default expectations for auto-loading and stopping autoloading
Between July 2020 and June 2021, over 96,000 people reported being targeted on the platform. Half of the Amazon fraud victims lost more than $1,000, while their total losses exceeded $27 million. One of the main businesses within Amazon - Amazon Direct Publishing within Amazon Books has many types of fraud that can be caught and stopped before it is published on the Amazon platform. A new suite of technology tools and a new fraud prevention and evaluation platform was needed. I designed and lead the Content Risk Management Product Design to better support the volume, complexity, and compliance needs of newer and evolving technologies.
In July 2021, I was tasked to create a new suite of products to address the greater needs of fraud prevention, risk assessment, fraud & abuse, rights management, accounts management, along with trust, policy, and compliance.
Amazon Books' KDP has unique needs for assessing customer's digital content for fraud, abuse, policy, and compliance standards. Along with Books, many other divisions will use the toolboxes and components provided as necessary to complete the needed Content Risk Management Operations Review workflow.
Amazon Scams to Watch Out for
Amazon has many different types of fraud to thwart daily. Unfortunately, scammers are adopting new modes of fraud at-scale daily.
The means and ways of fraud are frustrating for both the seller and Amazon as the platform. Amazon Discussions board allows customers to discuss and identify scams and Amazon's policies on fraud, risk, trust, and safety.
There are some pretty big scams hitting self-publishing authors on Amazon these days. It is worth Authors knowing what to watch for because the faster they can detect a scammer at work with their book on Amazon, the less revenue they'll lose and the less damage they will take long term.
I also designed a way for Authors to tag their work and their published manuscripts on KDP. This design would allow for mapping image and text to file for review to help prevent fraud and remove bad actors. The results will help us map patterns of deceit and fraud, better address nuance of privacy, copyright, accessibility, legal, and general regulations from various agencies.
Timeline
Within 2 months of being assigned the problem, a working prototype to visualize and resolve a new portal for Content Risk Assessment of KDP Digital Books, KDP Comix, KDP Vella, KDP Manga, Amazon Live, Amazon Shopping, and Amazon Video. By December 2022, a working MVP with 91% success rate on first run was observed.
Weeks later, in the early days of 2023, the MVP was proven in reducing the workload of human reviews to 3% via ML and AI assessments with accurate results and reporting. Adoption of the Books org for the production-ready Content Risk Assessment Operations Portal was integrated into Operations teams' workflows globally with 24/7 operational uptime and a 57% reduction in workload of human reviews.
The Decision Support Engine. was designed to bring self-service customization to the Content Risk Assessment Platform based on the individual business needs for content risk management.
Workflow
The typical operator experience can be frustrating to discover and resolve tickets in a timely manner because many decisions need supporting evidence to determine if the content should be blocked and should the account holder be banned or blocked for fraudulent activities on Amazon. The typical, legacy workflow has many opportunities for improvement, efficiencies, and modernization of the operational technology running in the backend.
Wireframes
Designing the User Experience for Operations Content Risk Reviews specializing in Books and Advertising content review policy and regulatory needs.
Unassigned and Expanded Queue Design
Assigned and Expanded Queue Design
Ideation
As discussions, teams, and scope changed - many revisions and ideas were designed to fit the current requirements. The final Operations portal was created within this agile approach to product development. Close alignment with dev and UX helped facilitate a close partnership with successful and reliable results.
User Journey
Design
A UX/UI Workflow experience for Investigative Reviews: primary focus on the Manager's user experience of assigning and reviewing team member's progress on timely review of User Generated Content.
A UX/UI Workflow experience for Investigative Review: primary focus on the Operator's user experience of reviewing User Generated Content.
Bulk upload of CSV into the Accounts Fraud and Abuse Management Team Operator's Enforcement portal. This action helps keep guidelines and policies up to date on a global scale.
Search allows for Managers and Specialists to access policies, lists, and data needed for Operational Excellence and for The Operator's Risk Assessment and Enforcement portal.
Preview of the Design Experience
I opted to go for a clean, optimized UI because much of the work to review can be filled with ugly, cluttered images and text. My goal was to make a user experience that offers some calm to the overwhelming task of keeping customers safe from harmful content.
Design: Version 1.0
Prototype
About the Prototype Color Scheme
Early discussions and design concepts were shared via internal Amazon products and systems for rapid communication and development.
Design Workshop
Organizing Stakeholders and Subject Matter Experts with the needs and nuance of the UX/UI within the Product, Customer, and User Design.
To BE updated.
Contextualizing the Workflow.
Workflow needs impacts the layout.
Objective: Design an intuitive UI to ....
To BE updated.
Workflow needs impacts the layout...
Objective: Design an intuitive UI to ....
To BE updated.
User Validation
Recap
