Set up a benchmark challenge in your browser. Teams prepare their method on GitHub, train it on your data and submit it. Every method is scored on the same holdout data, and a leaderboard shows what works best.
You pose a research question that can be scored, such as "who will have a child in the next three years?". Teams from anywhere build a method to answer it. Because every method is tested the same way, on data nobody has seen, you learn which approaches really work.
A prediction task with a clear way to score the answers.
Teams download it to build and tune their method.
Ranks the methods, with a tab for every metric.
Create a project, add a benchmark challenge item, and you're ready to set it up.
Your colleagues can join as admins of the project, so you can organize the challenge together.
Add your logo, a header image, an introduction and the challenge rules that participants agree to.
Optionally add a privacy statement and a helpdesk page with your contact details. The default language is English, because challenges are usually international.
Prepare method links to your template repository. Download data gives teams your training data. Submit method collects their submissions until the deadline.
Submissions close automatically at the deadline, and participants see it in their own time zone. Running several phases? Add a submit task for each, each with its own leaderboard.
Teams build their method in a copy of your template repository on GitHub. The template fixes how a method reads the data and writes its predictions, so every submission can run on the holdout data in exactly the same way.
Each team forks your template repository.
They train and tune it on your training data, for example in Python or R.
They submit a link to one commit on Next.
After the deadline, every method runs in the same Docker setup.
Hidden from participantsSetting up a challenge for the first time? You don't have to do it alone.
Eyra helps you develop the template repository and the Docker setup that runs the submissions on your holdout data.
The PreFer template repository is open source. Adapt it to your own data and metrics.
Not sure how to set up a fair challenge? Eyra advises you on how to set up your challenge.
Participants see your challenge, follow the steps and submit their method. Until the deadline, they can update their submission.
After the deadline, export the submissions and run them on your holdout data. Upload the scores as a CSV file, and Next checks them against the submissions.
Then publish the leaderboard: open, with every team's name and code, or blinded, where teams only see their own name next to the scores.
In the Predicting Fertility data challenge (PreFer), teams predicted who would have a child within three years, using Dutch survey and register data. It ran on Next.
Create a free account and set up your challenge. Pricing depends on what you need, so get in touch to discuss your plans.