author: niplav, created: 2022-07-15, modified: 2023-04-14, language: english, status: notes, importance: 6, confidence: certain
Iqisa is a library for handling and comparing forecasting datasets from different platforms.
The eventual success of my archives reinforced my view that public permission-less datasets are often a bottleneck to research: you cannot guarantee that people will use your dataset, but you can guarantee that they won’t use it.
—Gwern Branwen, “2019 News”, 2019
Iqisa is a collection of forecasting datasets and a simple library for handling those datasets. Code and data available here.
So far it contains data from:
for a total of ~4.2m forecasts, as well as code for handling private Metaculus data (available to researchers on request to Metaculus), but I plan to also add data from various other sources.
The documentation can be found here, but a simple example for using the library is seeing whether traders with more than 100 trades have a better Brier score than traders in general:
import gjp import iqisa as iqs market_fcasts=gjp.load_markets() def brier_score(probabilities, outcomes): return np.mean((probabilities-outcomes)**2) def brier_score_user(user_forecasts): user_right=(user_forecasts['outcome']==user_forecasts['answer_option']) probabilities=user_forecasts['probability'] return np.mean((probabilities-user_right)**2) trader_scores=iqs.score(market_fcasts, brier_score, on=['user_id']) filtered_trader_scores=iqs.score(market_fcasts.groupby(['user_id']).filter(lambda x: len(x)>100), brier_score, on=['user_id'])
And we can see:
>>> np.mean(trader_scores) score 0.159194 dtype: float64 >>> np.mean(filtered_trader_scores) score 0.159018 dtype: float64
Concluding that more experienced traders are only very slightly better at trading.
Iqisa offers some advantages over existing datasets:
Since this is a project I'm now doing in my free time, it might not be as polished as it should be. Sorry :-/
If you decide to work with this library, feel free to contact me.
NAmore often than they are right now.
Credits go to Arb Research for funding the first 85% of this work, and Misha Yagudin in particular for guidance and mentorship.
Thanks also to hrosspet for making the library more usable.