Mind Supernova

AI + Humans: Data to Power LLMs & VLMs

We deliver high-quality, curated data by combining the latest AI & ML technologies with expert human feedback.

EXPERTS

How we blend AI and human expertise

Mind Supernova has a diverse network of experts to perform the LLM evaluation and red teaming to identify risks.

Taxonomy creation

We design tailored taxonomies to match the model's use cases and capabilities. By starting with unique taxonomies for each domain of knowledge, we end up with well-structured and representative datasets.

Performed by:

Domain superexpert

Data architect

Outcome: Taxonomy for each unique use case

Data generation

We augment state-of-the-art AI & ML technologies with expert human feedback in sophisticated data pipelines.

Our team has the expertise and experience to:

Input raw data:

Your proprietary data

Open-source dataset

Relevant raw data from the internet

Crowdsourced data

Performed by:

Technologies / LLM Pipeline

Human Experts

Outcome: Raw generated dataset

Data verification

Our experts perform comprehensive validations on generated data to curate an accurate and reliable dataset for your model's needs.

Input:

Synthetic data

Hybrid data

Performed by:

Human Experts

Outcome: High quality dataset

Auto-verifiable tasks for Deep Research Agent

Our team has built a dataset to enhance the Deep Research Agent. Each task includes a complex domain-specific prompt and a set of rubrics for automatic answer verification. The agent’s performance on extensive online research tasks was significantly improved through end-to-end RL using this data.

Client type:

Leading AI Company

Experts:

MS & PhD in Finance

Accounting

Economics

Medicine

Volume:

600 datapoints per domain

Application: Enhancing Deep Research Agent using end-to-end RL

Synthetic data verification and/or editing

Research conducted by Scale’s Safety, Evaluations, and Analysis Lab (SEAL) will enable model-assisted approaches.

Client type:

Coding AI agents startup

Experts:

Software architects

DevOps engineers

Backend engineers

Volume:

5,000 trajectories

500 per week

Application: Coding agent for repository maintenance and bug-fixing tasks

Why Scale

Reliable and Robust Performance Management

Mind Supernova Evaluation is designed to enable frontier model developers to understand, analyze, and iterate on their models by providing detailed breakdowns of LLMs across multiple facets of performance and safety.

Proprietary Evaluation Sets

High-quality evaluation sets across domains and capabilities ensure accurate model assessments without overfitting.

Rater Quality

Expert human raters provide reliable evaluations, backed by transparent metrics and quality assurance mechanisms.

Product Experience

User-friendly interface for analyzing and reporting on model performance across domains, capabilities, and versioning.

Targeted Evaluations

Custom evaluation sets focus on specific model concerns, enabling precise improvements via new training data.

Reporting Consistency

Enables standardized model evaluations for true apples-to-apples comparisons across models.

Companies are incentivized to game leaderboards in order to one-up one another. This makes it hard to tell whether AI systems are actually improving. That’s why it’s important that organizations such as Mind Supernova assess these AI systems with their private evaluations.

Dan Hendrycks

Director, Center for AI Safety

The work Mind Supernova is doing to evaluate the performance, reliability, and safety of AI models is crucial. Government agencies and the general public alike need an independent, third party like Mind Supernova to have confidence that AI systems are trustworthy and to accelerate responsible AI development.