AI model evaluation platform LatticeFlow has launched Compl-AI, the first evaluation framework for determining compliance with the EU AI Act. The free, open-source framework evaluates large language models (LLMs) across 27 technical areas based on the Act's six ethical principles.

Compl-AI assesses LLM responses in areas such as prejudiced answers, general knowledge, biased completions, following harmful instructions, truthfulness, copyrighted material memorization, common sense reasoning, goal hijacking, and prompt leakage. The framework rates models on a scale from 0 (no compliance) to one (full compliance), with N/A scores for insufficient data.

The platform reveals shortcomings in existing models and benchmarks, particularly in areas like robustness, safety, diversity, and fairness. Additionally, the company claims the methodology can be extended to evaluate AI models against future regulatory acts, making it a valuable tool for organizations working across different jurisdictions.