Automate the Obvious, Review the Rest: Designing AI-Powered Content Moderation

Описание и применение

Система модерации контента, использующая AI, автоматически обрабатывает ясные случаи, а неоднозначные передаются модераторам, обеспечивая масштабируемость и безопасность.

Задача

Moderating user‑generated content at scale without delaying uploads or over‑relying on human review.

Результат

The system achieves scalable moderation by automating clear cases, escalating ambiguous ones, and maintaining an audit trail, reducing human workload while protecting the feed.

Рабочий процесс

• 1. User creates content (text or media).
• 2. Text: row saved, Hasura event triggers NestJS, creates content record, enqueues BullMQ job.
• 3. Media: uploaded to S3, ObjectCreated triggers SNS, fan‑out to ingest Lambda.
• 4. Text worker calls OpenAI Moderation API, converts max category score to risk score, applies thresholds (40/75).
• 5. Media scorer Lambda calls Amazon Rekognition, applies thresholds (75/90).
• 6. Results stored in content_moderation_status and moderation_results tables.
• 7. Content with status approved is shown in feeds; under_review or rejected stay hidden until moderator action or owner notification.

Доказательства

We needed a moderation system that could inspect captions, comments, ads, stories, images, and videos as soon as they were created without delaying uploads or posts for human review or automating every borderline case.

Источник

Text goes through NestJS, Hasura event triggers, BullMQ, and the OpenAI Moderation API. Images and videos go through S3, SNS, SQS, AWS Lambda, and Amazon Rekognition.

Источник

Risk below 40 → approved. The content can enter the feed. Risk from 40 up to (but not including) 75 → under_review. Risk 75 or above → rejected.

Источник

Images/video (Rekognition): start stricter on auto-approve than text (review at 75%, reject at 90%).

Источник