Threats, vulnerabilities, and new forms of attack within the cyber domain develop rapidly. To keep up with and respond to these trends, cyber security professionals must demonstrate reaction velocity, accuracy and a high tolerance for complexity. Publicly available information (PAI) can serve as an important aid to personnel engaged in cyber security analysis. However, evaluation of cyber analytical capacity – a pre-requisite for any measurement of quality or improvement – is still inchoate. This article covers the concept and design of an initial phase of research begun in October 2018 in The Netherlands to measure and improve cyber analysis techniques. The research program features a forecasting tournament to record participants’ probabilistic estimates on future cyber outcomes based exclusively on PAI knowledge acquisition. This phase of research seeks to address whether analysts’ predictions are more accurate in certain subjects within the cyber domain than in others and to assess how predictive accuracy in the cyber domain compares to accuracy in other domains in which forecasting tournaments have been organized.