Decision Support Engine: Page Under Construction
Designing the Decision Support Engine for Amazon's Operations Content Moderation Platform
I created a new, robust suite of fraud and abuse moderation tools for content on Amazon's platform from 2021 - 2023 using Machine Learning and AI tools.
Product Design, UX Strategy, and the User Experience
Using ML, AI to automate content
moderation and compliance of all user-generated content (UGC) for all of Amazon's businesses and platforms.
Catching all fraud types at scale, with speed and accuracy to reduce human moderator's workload:
- Modern, versatile UI modular layout and customizable UX per business and content needs.
- Flexible, adaptable products to meet new and future content types for any business need.
- Harmful Actors Profiling for faster response and informing AI to catch fraud faster.
- Quick, accurate image fingerprinting, content scanning, and profiling to catch and prevent harm before it is published to Amazon.
- Adaptable decision support matrix to automate decisions for speed and accuracy.
- Brand safety protocols to meet internal and external standards, practices, and regulations.
- Easy logging, diagnostics, and repairs for any system issues for real-time moderation.
Project Role
Working with Software Engineering, User Research, Amazon Worldwide Operations, and Product Management to design a suite of Content Moderation.
- Product Design, Production Design, Project Management
- Visual Design + Wireframing
- Design Systems
- Information Architecture
- UX Design
- UX User Research + User Studies: Internal and External Users
- DX/UX/UI Prototyping in Figma and React
- UX Design Hand-off
General Role Responsibilities
- Facilitate discussions and resolve highly-debated topics using presentations and proposals with various C-level technical stakeholders and other high-level internal stakeholders
- Lead teams through a human-centered design process; inspiring and directing all aspects of project work and deliverables
- Planning and scoping projects for teams according to business needs and design opportunities, setting teams up for success with an opinionated view on best practices
- Balancing multiple teams autonomy and guidance through harmonious collaboration and teams engagement
- Rapidly iterate and prototype design concepts, lead and guide best practices for designers and developers
- Work as a collaborative team member with design and engineering colleagues, product managers, and operations.
- Structure and facilitate ideation sessions with various technical and non-technical stakeholders
- Help product managers assess and prioritize opportunities and constraints, with clear, candid communication
Challenge
Create a successful suite of tools for the Developer Experience unifying the tools with a fresh look, feel and user experience. The design must explore new concepts that will expand and unite various tools and departments.
Developer Experience Challenge Breakdown
- Visualize Decision Support Engine.
- Inspect the UI Decision Support Engine .
- Diagnose Decision Support Engine
- Decision Support Engine patterns paradigm for Content Moderation.
- Debug Decision Support Engine needs.
- View general log and diagnostics information about Decision Support Engine for any troubleshooting needs.
Project Goals
Design a successful product .... Design high-fidelity prototypes for integration of this product.
Key Deliverables
- Product design for Decision Support Engine, Content Moderation Suite of products and prototyping tools for integration into business' workflows
- Decision Support Engine Design System, Design icons, symbols for tools and product integration into the new Content Moderation Suite
- Create prototypes and a research plan for new product design and workflow: Discovery, Onboarding, User Flow, User Testing, User Journey
- Design a new product to visualize the rules-based implementation in the new Content Moderation framework focusing on the Decision Support Engine
- Determine the critical features needed to resolve runtime UI issues caused within Decision Support Engine.
- Anticipate and identify usability issues using the Decision Support Engine
- Design a cohesive suite of tools for troubleshooting and observing the Decision Support Engine, focusing on resolving and improving upon legacy workflow issues
Types of Content Moderation
- Pre-moderation: content requires a review and approval before being published. This is the best safeguard against legal or fraudulent risk but can be a time-consuming and laborious process that can delay the appearance of content, product displays, reviews or comments and, in turn, prove frustrating for users and reduce the time to market.
- Post moderation: content immediately goes live but is queued for review and any items that do not meet guidelines are removed. It is difficult moderating large volumes of material and increases the risk of inappropriate content being missed.
- Reactive moderation: users are asked to ‘flag’ any offensive or questionable material for moderator review. Because of Amazon's volume, relying only on this isn't cost-effective and can impact the brand reputation. It relies on a proactive audience and can heighten the risk of inappropriate material going undetected. Many tickets are filed through Customer Service representatives and other forums for human review.
- Distributed moderation: Amazon's online community is invited to rate or ‘star’ on published content, with material that does not meet a certain standard removed. Amazon uses this method for Kindle Vella, giving the Authors the ability interact directly with their audience and to moderate their Readers' posts.
- Automated moderation: utilizing ML, AI, and digital tools like Amazon's Rekognition to detect and block inappropriate content before it goes live, along with blocking known account IDs, customer IDs, and IP addresses of users already determined to be abusive. It serves as a first pass and is often used in post moderation. The absence of human analysis and interpretation can lead to worthy content being rejected and vice versa. The new Content Moderation framework I've designed combines the best of automated moderation with human review, with 98-99% of the content passing an error-free automated review with nuanced accuracy without need for additional human review.
User Research
In the August 2021 survey, 100% of the users surveyed had experience with the legacy platform
and its specific issues. The common issues were with tooling (100%), visual differences - no image fingerprinting (100%), and
dependency on antiquated tools from many different platforms that require manual input without tracking nor cohesion (100%).
Operations staff and content moderation reviewers have issues with the UI layout of more than 10 different legacy
systems and need a way to visualize and resolve tasks quickly with factual data and comparing content to ensure they are or
aren't compliant.
Research, analyzing data, reviewing the top issues on all popular platforms where users seek help or file issues, and asking why the issue is common and what is the root cause of the relationship of the code, framework paradigms,and UI helped me understand the major frictions points and how to design for them. Usability testing, interviewing, observing and engaging in social outreach honed the solution direction.
Lorem Ipsum.
Timeline
Within six weeks, launch an MVP of a text-review and test the ML automation's accuracy.
Within twelve weeks, launch the new design system for a Content Moderation UX/UI for a modular,
component-based platform.
Within twelve weeks, launch an MLP of a text-review and test the ML automation.
Within six weeks, launch the new design for a Decision Support Engine for Content Moderation for any
content type and compliance need as a modular, component-based platform.
Content Moderation UX/UI Design
The first reveal of the Content Moderation UX/UI prototype was created within a three week timetable.
Workflow
The typical Content Moderator Operator experience can be frustrating to resolve tasks
efficiently . The typical workflow for resolving these has many opportunities for helping the user understand the paradigms
that are creating this frustrating issue.
My role was to identify the blockers and needs of the Developer/User within
the workflow and within the paradigms of the new Content Moderation Decision Support Engine. I designed the Operator's portal
to visualize, contextualize the content to quickly to make decisions on content, users, and account holders.
As illustrated, high-level workflow does not have an easy path to discovering the root cause and solution to resolve the render overflow issue, as seen here.
Using the Decision Support Engine
Wireframes
Decision Support Engine Rule Maker
Designing the Decision Support Engine for visualizing how rules can be combined to create fact-based decisions.
Decision Support Engine Fact-Based Rules
Lorem Ipsum
Golden Path User Solution
Ideation
Resolving technical issues and generating facts-based outcomes that rely on a complex matrix of regulations.
Creating the Decision Support Engine UI for visualizing what it takes to generate a decision based on policy and legal regulations.
One of the key issues Operators' experience with the legacy system had was content was without useful context to understand issues to resolve the tasks.
Decision Support Engine Layout Limitations + Considerations
- Lorem Ipsum
- Lorem Ipsum
- Lorem Ipsum
- Lorem Ipsum
I explored the variations of content, top user needs and how they could be visualized for the user to reduce the time it takes
to create a decision and to inform ML and AI to make decisions.
Initial ideas and issues to reproduce the problem are
explored and studied. This application was used in User Research studies and iterations of testing as the design was
developed.
Decision Support Engine Fact-Based Rules Resolution Examples
Design
Decision Support Engine Fact-Based RulesUI Design Suite
Decision Support Engine Fact-Based Rules
+ Lorem Ipsum
Discovery: Decision Support Engine Fact-Based Rules
Launching Decision Support Engine Fact-Based Rules.
Preview of Decision Support Engine
Decision Support Engine Fact-Based Rules
Enable this
feature with the new setting.
LOrem Ipsum
Decision Support Engine Fact-Based Rules Navigation Design
Decision Support Engine Fact-Based Rules Performance
The Decision Support Engine is a powerful tool for visualizing and exploring the rules-based decision matrix. More to come.
Decision Support Engine Timeline
Decision Support Engine Rules
Decision Support Engine Workflow
Improve Rules Discoverability
Add Indication of Rules workflow and basic rule and debugging support allowing user to step through how the DSE generates code.
Lorem Ipsum
Lorem Ipsum
Content Moderation Decision Support Engine Debug State
User Lorem IPsum
Decision Support Engine Network
Decision Support Engine Logging
Decision Support Engine Info
Decision Support Engine App Size
Image: Snapshot
Image: Zoomed in on code rendering from DSE
Decision Support Engine Inspector
Decision Support Engine Layout
The Decision Support Engine is a powerful tool for visualizing and exploring
decision trees. The Decision Support Engine framework uses components as the core building block for generating rules-based
code to inform the platform, ML, AI, and manual reviews.
The Decision Support Engine helps users visualize and
explore the code trees, and can be used for understanding existing rules and diagnosing rule issues.
Prototypes: Decision Support Engine
Prototype demonstrating expected behavior for the Decision Support Engine to animate code changes of the Decision Support Engine and how it works with the Operator's workflow, the ML workflow, and informs the AI.
About the Prototype Color Scheme
Early discussions and design concepts were shared via Slack, Chime, Quip, Figma and other internal Amazon products for rapid communication and development.
Decision Support Engine Prototype Design: Version 0.x
Version 0.6 UX/UI Design.
Updating the UX/UI to better illustrate the flex options of the Decision Support Engine.
Objective:
Easy to understand workflow representing the layout alignment expectations and linear approach to
generating a rule.
Version 0.5 Contextualizing the Rule Workflow.
Visualize the way a Rule is generated.
Objective: Design an intuitive UI to show the Rule's content and how it will be utilized to generate an outcome.
Version 0.2 Illustrating the Code.
Visualize how code will render based on the Rule.
Objective: Design an intuitive UI and UX to work with the Rule policy constraints.
of Note: Iterating on visualizing the Decision Support Engine parameters, and application.
Version 0.1 Context for Content.
UX needs to indicate the actual outcome and how the rules will generate code.
Objective: Design an intuitive UI to show the application's rule content is continuing to inform the ML, AI, and backend.
Recap
Research shows the most common issue for Operators and Account Managers is understanding the requirements and policies for decision making and quickly finding fact-based evidence to render the decisions.
The alpha version of the Decision Support Engine was released in 2023 and demoed in 2022.
The Decision Support Engine is a powerful tool for visualizing rule trees. The Decision Support Engine framework uses ruless as the core building block and helps users visualize how Rules create outcomes on policy violations.
