LLama and ChatGPT soon in the sights of European regulators? A tool reveals major deficiencies in certain AI models

Certain artificial intelligence models do not comply with regulations provided for by the European Union, particularly in the areas of cybersecurity. A new tool deployed by the Swiss startup LaticeFlow makes it possible to test generative artificial intelligence models developed by large companies like Meta or OpenAI, offering them a score ranging from 0 to 1 depending on their compliance with European regulations. LatticeFlow has published a report showing that the companies Alibaba, Anthropic, OpenAI, Meta or Mistral all received a score of 0.75 or higher.

However, some models show significant deficiencies depending on Reuters. One of these recurring faults lies in discrimination (artificial intelligence sometimes tends to reproduce human biases linked to race or gender, for example). According to LatticeFlow, in the field, ChatGPT-3.5 Turbo benefits from a low score of 0.46 on LatticeFlow's LLM Checker tool. Qwen1.5 72B Alibaba Chat has a score of 0.37. In other words, artificial intelligence still has work to do to try to reduce these human biases linked to discrimination.

LatticeFlow's LLM Checker also allows you to test cybersecurity, including “prompt hijacking“, this type of cyberattack allows hackers to disguise a malicious request as a legitimate request and thus recover sensitive information. In this area, LLama 2 13B Chat de Meta obtains a low score of 0.42. “The European Union is still working on all the compliance criteria, but we can already see gaps in the models” according to Peter Tsankov, CEO of LatticeFlow. He then continues: “With a greater focus on optimizing compliance, we believe model providers can be well prepared to meet regulatory requirements“. Overall, it's Claude 3 Opusfrom Anthropic, which fares best with an average score of 0.89, making it the most secure and least discriminating artificial intelligence on the market today.

If the companies identified by such reports fail to resolve these problems, the deficiencies could cost up to 35 million euros in fines in the event of non-alignment with the European Union's AI Act.