Groundbreaking Fairness Evaluation Dataset From Sony AI Now Available as Global Benchmark for Ethical Data Collection and Responsible AI Development

FHIBE: Fair Human-Centric Image Benchmark confirms known biases and identifies undiscovered bias tendencies in AI models

First publicly available, globally diverse, consent-based human image dataset for evaluating bias across a wide variety of computer vision tasks

Research published today in Nature

Tokyo, Japan – November 5, 2025 – Sony AI today unveiled the Fair Human-Centric Image Benchmark (FHIBE, pronounced “Fee-bee”), a groundbreaking dataset created to establish a new global benchmark for fairness evaluation in computer vision models. While computer vision technologies are central to modern artificial intelligence (AI) applications – from smartphones to autonomous vehicles – FHIBE addresses the industry’s persistent challenge of biased and ethically compromised training data. It aims to catalyze industry-wide improvements for responsible and ethical protocols throughout the entire life span of data – from sourcing and management to utilization – including fair compensation for participants and clear consent mechanisms. The FHIBE dataset is publicly available beginning today and was published in Nature.

Groundbreaking dataset addresses persistent issues in AI

FHIBE was created to address issues with current publicly available datasets that lack diversity and are collected without consent, which can perpetuate bias and present a persistent challenge to AI developers and users. Additionally, the lack of adequate and available evaluation datasets can result in biased or harmful models being deployed, making it difficult to assess potential harms and the ability of a model to function equitably on a global scale. The team at Sony AI recognized the need to address these barriers and invested significant resources to provide this benchmark for public use by AI developers and researchers globally.

FHIBE is the first publicly available, consensually-collected, and globally diverse fairness evaluation dataset for a wide variety of human-centric computer vision tasks. It enables researchers and developers to rigorously evaluate bias and accuracy across a variety of computer vision tasks, including face detection and verification, pose estimation, and visual question answering.

“FHIBE is proof that fair and responsible practices can be achieved when ethical AI is a priority. AI is evolving rapidly, and it is imperative for us to investigate the ethical implications of how we collect and use data. For too long, the industry has relied on datasets that lack diversity, reinforce bias, and are collected without proper consent,” said Alice Xiang, Global Head of AI Governance at Sony Group Corporation and Lead Research Scientist for AI Ethics at Sony AI. “This project comes at a critical moment, demonstrating that responsible data collection – incorporating best practices for informed consent, privacy, fair compensation, safety, diversity, and utility – is possible. While this is an important step forward, it is just the beginning. We are setting a new precedent for future progress in AI that is fair, transparent, and accountable.”

Enabling nuanced assessments and more granular diagnoses sets the new industry benchmark

The dataset comprises 10,318 consensually-sourced images of 1,981 unique subjects, each with extensive and precise annotations. These annotations capture demographic and physical attributes, environmental factors, and camera settings – enabling nuanced assessments of fairness and bias across a wide range of demographic attributes and their intersections. The images were collected from subjects in over 81 countries/regions, making it one of the most globally diverse and most comprehensively annotated datasets in existence.

The research published today in Nature in an article titled, “Fair human-centric image dataset for ethical AI benchmarking,” examines FHIBE’s performance across both narrow computer vision and large-scale multimodal generative models. It showcases how FHIBE assesses biases across demographic attributes and their intersections while comparing FHIBE against existing human-centric fairness evaluation datasets.

Using FHIBE, the Sony AI research team affirmed previously documented biases and showed that FHIBE can support granular diagnoses on the factors leading to such biases. For example, FHIBE validated that some models had lower accuracy for individuals using “She/Her/Hers” pronouns while also discovering that this disparity can be traced to greater hairstyle variability – a factor previously overlooked in fairness research. Another example highlighted that a model, when asked neutral questions (e.g., “What is this person’s occupation?”), sometimes reinforced stereotypes, associating specific demographic groups with criminal activities.

The dataset is designed to evolve responsibly over time. Data subjects retain control over their personal information and they can withdraw their consent at any time – with no impact on compensation they received for the project. To maintain dataset integrity and diversity, Sony AI will remove and, to the extent possible, replace withdrawn images, ensuring that FHIBE remains a continually updated benchmark.

“Sony AI is leading the way in AI ethics research, focused on building fair, transparent, and accountable technologies that protect the interests of AI users, creators, and everyone in the global data community,” said Michael Spranger, President of Sony AI. “We have created a publicly available resource that addresses fairness and accuracy in computer vision models – ensuring they do not disenfranchise stakeholders – while inspiring broader change in AI ethics and data collection. FHIBE sets a new industry benchmark for AI datasets, proves that the process can be achievable, and shows that collecting data responsibly is possible, helping to build trustworthy AI from the ground up.”

The scale and complexity of this research reflect the significant challenges in creating a dataset that is comprehensive, globally diverse, and consensually-sourced. Over the course of three years, a global team of Sony AI researchers, engineers, and project managers worked to develop rigorous procedures for data collection, annotation, and validation. Their work was further supported by legal, privacy, Information Technology (IT), and Quality Assurance (QA) specialists.

Details on the FHIBE project and access to the publicly available benchmark dataset can be found at https://fairnessbenchmark.ai.sony/

A short film on the development of FHIBE can be watched here.

The benchmark can be downloaded at https://fairnessbenchmark.ai.sony/

About Sony AI

Sony AI, a division of Sony Research, was founded as a subsidiary of Sony Group Corporation on April 1, 2020, with the mission to “unleash human imagination and creativity with AI.” Sony AI aims to combine cutting-edge research and development of artificial intelligence with Sony Group’s imaging and sensing technology, robotics technology, and entertainment assets such as movies, music, and games to accelerate Sony’s transformation into an AI-powered company and to create new business opportunities. To achieve this, Sony AI is working across six Flagship Projects that are aimed at the evolution and application of AI technology in the areas of AI for Creators, Gaming and Interactive Agents, Ethics, Scientific Discovery, Imaging and Sensing, and Robotics. For more information, visit https://ai.sony/.

Source: zedreviews.com