How a finding is established

Data

The research uses public data on Belgian companies only: official publications, the business register, annual accounts, planning applications, public contracts and awarded support. Every event is dated and linked to the company number. Data from Hunch clients is never used.

1,141,228 companies1,064,469 dated events2018 start of the longest series363 links tested

Study design

Each question takes the form of a retrospective cohort study. The cohort is every company that had event A within a fixed entry window. We measure what share of them has event B within a fixed follow-up period.

The follow-up period only starts after the entry window, so two changes from the same deed can never pass as cause and effect. When a series is still young, as with bankruptcies, both cohorts share the same follow-up window and their entry windows come before it.

Similar companies

Large, old and active companies experience more of everything. To remove that effect, all companies are divided into strata by sector (NACE section), size class (0, 1-4, 5-9, 10-49, 50-249 and 250 or more full-time equivalents from the latest annual accounts; without accounts, employer or not) and age class (0-2, 3-5, 6-10, 11-20 and more than 20 years). A stratum with fewer than 300 companies is merged, always in the same order: age first, then size. Within each stratum, the rate of B among companies without A is calculated, and from it the number of cases expected in the cohort.

factor = observed cases / expected cases = O / Σ_stratum (n_stratum × probability_{stratum without A})

A factor of 2 means B follows twice as often as among similar companies. The 95% confidence interval is the exact Poisson interval on the number of observed cases.

The comparison group decides what a factor means. Each test is therefore run three times. The main factor compares with companies without A that had another publication or event in the same entry period: simply being active cannot then create a link. The test is also run against all economically active companies (staff, annual accounts in the two years before entry, or a publication in that period) and against the full register. A finding only receives a status if the first two groups point in the same direction; all three factors appear in each technical note. A source covering one region is tested within that region only.

The absolute difference always accompanies the factor. A factor of 10 on an outcome that affects 0.1% of companies is still 1%. A finding is only featured if at least 2% of companies with A also have B and the difference with the comparison group is at least 1 percentage point.

Replication

A link only counts if it shows up again in a second, independent cohort: different companies, a later period. For planning applications the test is repeated for every year since 2018. Findings based on annual accounts currently rest on one financial year and get their second measurement with the next.

Multiple testing

Test hundreds of links and a few will turn up by chance. In the first measurement the expected share of false discoveries is therefore kept below 5% (Benjamini-Hochberg procedure). The second measurement has to be significant on its own (p < 0.05) and point in the same direction.

Results with no link or an inverse link are published in the same way as the others. An archive of only striking outcomes would give a distorted picture.

Review

Every new or changed finding is reviewed separately before publication, in two rounds. In the first round the review sees only the definitions and the design, without figures, and formulates at least three alternative explanations, each with the test that would rule it out: a shifted follow-up period, excluding cases shortly after the event or the largest stratum, a placebo event, a negative control with the outcome before the event, overlap of definitions, stability across measurements and the E-value for unmeasured confounding.

Those tests are run in code. In the second round the review assesses the figures and the test results and decides: publish, publish with a caveat, recalculate or reject. Rejected findings do not appear. Each published finding states which tests it passed and which caveat applies. The review is automated and does not replace peer review.

Results

Confirmed

Both measurements show a factor of at least 1.4, after correction for multiple testing.

Less often

Both measurements show that B follows clearly less often (a factor of 0.71 or lower).

No link

Enough cases, and both measurements rule out a link of 1.4 times or more in either direction.

Provisional

One measurement qualifies; the second is still to come.

Refuted

A hypothesis that the data contradict.

Running

Too few cases for a conclusion yet. Running tests only appear in the archive once completed.

Limitations

• A link is not proof of cause. The comparison accounts for sector, size, age and activity, not for every difference between companies.
• The size class relies on the annual accounts available; for most employers it is only known that they have staff, not how many.
• Bankruptcies and new establishments are followed for a shorter period than other events. Those results are updated as the series grows.
• Planning applications and Flemish support are tested within Flanders only. Annual accounts are not yet available for every company.

From finding to model

Confirmed links become candidate signals. A separate test on each client’s target audience decides whether and how a signal enters that client’s model. Which signals a model uses and how they combine are not published.