Mistral AI, an open-source developer of foundation models, has released a new content moderation API that powers its Le Chat service. The API is designed to help users implement content moderation in their applications according to their safety standards.

The API includes two endpoints: one for raw text and another for conversational content. The LLM classifier can categorize text into nine categories and supports multiple languages, including Arabic, Chinese, English, French, German, Italian, Japanese, Korean, Portuguese, Russian, and Spanish.

The company claims its content moderation system provides effective guardrails by addressing model-generated issues such as unqualified advice and Personal Identifiable Information (PII).

Analyst QuickTake : Mistral's moderation tool aims to provide organizations with the ability to filter and manage content generated by AI systems, ensuring that it adheres to community standards and regulatory requirements. This proactive approach is crucial, as GenAI is adopted in various sectors, raising concerns about misinformation, harmful content, and potential misuse. By implementing moderation capabilities, Mistral positions itself as a responsible player in the AI landscape, echoing OpenAI's earlier efforts to integrate safety measures into its offerings.