home

author: niplav, created: 2022-07-15, modified: 2026-06-27, language: english, status: finished, importance: 6, confidence: certain

Iqisa is a library for handling and comparing forecasting datasets from different platforms.

Iqisa: A Library For Handling Forecasting Datasets

The eventual success of my archives reinforced my view that public permission-less datasets are often a bottleneck to research: you cannot guarantee that people will use your dataset, but you can guarantee that they won’t use it.

Gwern Branwen, “2019 News”, 2019

Iqisa is a collection of forecasting datasets and a simple library for handling those datasets. Code and data available here. Documentation is here.

So far it contains data from:

for a total of ~4.38m forecasts, as well as code for handling private Metaculus data (available to researchers on request to Metaculus), but I plan to also add data from various other sources.

The documentation can be found here, but a simple example for using the library is seeing whether traders with more than 100 trades have a better Brier score than traders in general:

import gjp
import iqisa as iqs
market_fcasts=gjp.load_markets()

def brier_score(probabilities, outcomes):
    return np.mean((probabilities-outcomes)**2)

def brier_score_user(user_forecasts):
    user_right=(user_forecasts['outcome']==user_forecasts['answer_option'])
    probabilities=user_forecasts['probability']
    return np.mean((probabilities-user_right)**2)

trader_scores=iqs.score(market_fcasts, brier_score, on=['user_id'])
filtered_trader_scores=iqs.score(market_fcasts.groupby(['user_id']).filter(lambda x: len(x)>100), brier_score, on=['user_id'])

And we can see:

>>> np.mean(trader_scores)
score    0.159194
dtype: float64
>>> np.mean(filtered_trader_scores)
score    0.159018
dtype: float64

Concluding that more experienced traders are only very slightly better at trading.

Advantages

Iqisa offers some advantages over existing datasets:

Requirements

Uses

Possible Projects

Known Bugs

Since this is a project I'm now doing in my free time, it might not be as polished as it should be. Sorry :-/

If you decide to work with this library, feel free to contact me.

Feature Wishlist

Potential Additional Sources for Forecasting Data

Acknowledgements

Credits go to Arb Research for funding the first 80% of this work, and Misha Yagudin in particular for guidance and mentorship.

Thanks also to hrosspet for making the library more usable.

Discussions