I’m writing about an attention-to-detail error immediately after realizing it. It probably won’t matter, but if it ends up costing me a thousands-of-dollars prize, I’ll feel salty. I thought I’d grouse in advance just in case.
The last few years I’ve entered Kaggle’s March Madness data science prediction contests. I had a good handle on the women’s tournament last year, finishing in the top 10%. But my prior data source – which I felt set me apart, as I scraped it myself – wasn’t available this year. So, living my open-source values, I made a quick submission by forking a repo that a past winner shared on Kaggle and adding some noise.
Now, to win these contests – with a $25k prize purse – you need to make some bets, coding individual games as 1 or 0 to indicate 100% confidence that a team will win. If you get it right, your prediction is perfect, generating no penalty (“log-loss”). Get it completely wrong and the scoring rule generates a near-infinite penalty for the magnitude of your mistake – your entry is toast.
You can make two submissions, so I entered one with plain predictions – “vanilla” – and one where I spiced it up with a few hard-coded bets. In my augmented Women’s tournament entry, I wagered that Michigan, Michigan State, and Buffalo would each win their first round games. The odds of all three winning was was only about 10%, but if it happened, I thought that might be enough for me to finish in the money.
Michigan and Buffalo both won today! And yet I found myself in the middle of the leaderboard. I had a sinking feeling. And indeed, Kaggle showed the same log-loss score for both entries, and I was horrified when I confirmed:
In case Michigan State wins tomorrow and this error ends up costing me a thousand bucks in early April, the commit in question will be my proof that I had a winning ticket and blew it.
Comment if you see the simple mistake that did me in:
As of this writing – 9 games in – I’m in 294th place out of 505 with a log-loss of 0.35880. With the predictions above, I’d be in 15th place with a log-loss of 0.1953801, and ready to benefit further from my MSU prediction tomorrow.
The lesson is obvious: check my work! I consider myself to be strong in that regard which makes this especially painful. I could have looked closely at my code, sure, but the fundamental check would have been to plot the two prediction sets against each other.
That lesson stands, even if the Michigan State women fall tomorrow and render my daring entry, and this post, irrelevant. I’m not sure I’ll make time for entering these competitions next year; this would be a sour note to end on.