All lessonsLesson 2 · Intro to contest AI▾
Before you learn any model, it's worth understanding what you're dealing with. An AI contest problem almost always has the same pieces: some data, a target to predict, a scoring metric and a submission file. Whoever reads the statement properly starts with a big head start.
Anatomy of a problem
You get two datasets. A training one, which also has the correct answers (called labels, or the target), and a test one, which has the same columns but no answer. Your job is to predict the answer for the test set, based on what you learned from the training one.
- Training data: the rows you see in full, answer included. This is what the model learns from.
- Test data: the same columns, but the target column is missing. This is what you have to fill in.
- The target: what you predict. It can be a label (spam or not) or a number (a price).
- The submission file: a table with your predictions, in exactly the format the platform asks for.
The metric
Every problem has a metric, the formula your score is computed with. It's written in the statement and it matters a lot. If the metric is F1 and you optimize accuracy, you can climb on a score that earns you no points. A few examples:
- Accuracy: the percentage of correct answers. Simple, but misleading when the classes are imbalanced.
- F1: balances precision against recall, good when the classes are uneven.
- RMSE or MAE: for numbers, how far you are on average from the real answer.
A good rule: read the metric before anything else, and train with it in mind. If F1 is what's scored, validate locally on F1 as well.
Submission and leaderboard
Once you have the predictions, you put them in the required file and upload it. The platform compares it against the correct answers, which you don't see, and gives you a score on a leaderboard. You usually have a limited number of submissions, so don't waste them on random tries.
The leaderboard has two halves. The public one is computed on part of the test data and you see it throughout the contest. The private one is computed on the rest and is revealed only at the end. The ranking that counts is the private one. That split exists for a reason.
If you pick your model by the public score, you end up fitting that small slice of data rather than the real problem. It's called overfitting the leaderboard: you climb nicely on the public one, then drop on the private one, which is where the points are actually handed out.
The ONIA rules
From the olympiad rules, a few things worth knowing in advance:
- The contest lasts four hours.
- A problem's solution has two parts: the file with your answer and the Python code that produced it. You submit both on the platform.
- The number of submissions is limited, and a minimum amount of time has to pass between two of them. You find out both together with the problems.
- The score you see during the contest is provisional, computed on part of the test data. The final one is computed on the whole set.
- For each problem you pick two submissions. Only those count in the final ranking.
- Your account on platform.olimpiada-ai.ro has to be created and activated before the announced deadline. After it, you can't sign up anymore.
Where you train
Three platforms where you'll find problems:
- Kaggle (kaggle.com): the most widely used international ML platform.
- MLCompete (platform.olimpiada-ai.ro): the platform the olympiad runs on and where you practise between stages, with archive problems and training competitions.
- Nitro AI Judge (judge.nitro-ai.org): the platform RoAI and other AI contests run on.
Your first submission
MLCompete (platform.olimpiada-ai.ro) and Nitro AI Judge (judge.nitro-ai.org) are the platforms you'll practise on. Make an account on both.
- 01Enter a training competition and read which metric is scored.
- 02Download the data and open the sample submission file, to see exactly which columns and which format it asks for.
- 03Produce a file in the same format, even with random answers. The simplest is to upload the sample file itself, unchanged.
- 04Upload it and look at the score.
- Any problem with a sample submission file
- Titanic on Kaggle