2) Design provenance
a. Has the questionnaire been designed by relevant field experts/academics (i.e. with financial sector and statistical experience)? It is a concern if the questionnaire does not have relevant expert origins.
b. Can those academics be referenced and verified? Check their existence and relevance of their academic achievements.
3) Academic research and independent testing and scrutiny
Having multiple academics reviewing the questionnaire provides further evidence of the questionnaires’ good provenance.
a. Have academic researchers used the questionnaire or data set for any documented research, preferably published in internationally recognised peer-reviewed journals?
b. Does this research support the integrity of the questionnaire?
c. Are their academic papers available for scrutiny?
d. Has the questionnaire been checked or tested by an independent academic? It is important that there is independence in testing outside of the provider and that the results of testing were positive.
e. Ongoing development and analysis: How and by whom is the performance of the questionnaire monitored, tested and updated?
f. How regularly has it been tested? This should happen at (a minimum of) 5 yearly intervals.
g. Are any changes made to the questionnaire documented and communicated to the users. Have there been a number of different versions, if yes why have there been changes. If there have been no changes, why not?
4) Assumptions underpinning the questionnaire
a. What are the origins of the questionnaire? Is it empirically constructed or based on theory? Look for reference to academic papers or references. It is a good indicator if the questionnaire has a sound theoretical or empirical basis as opposed to been constructed or invented purely in-house.
b. What are the assumptions underpinning the questionnaire? Do these assumptions seem intuitively reasonable? For example, are there any assumptions about human nature or behaviour that seem unreasonable?
c. Is the questionnaire heavily influenced by a particular academic research paradigm (e.g. psychology and psychometrics)? Do your own research online to see if this paradigm is criticised or has any areas of weakness.
5) Relevance: Norms and test sample
a. Has the questionnaire been tested against a sample population that is real and relevant. For example, is the age group that it has been tested with similar to the age group that you may use the test.
b. Do the norms tables listed contain the scores of individuals from comparable groups to those to which the test will actually be given? Is there thorough information about the norms and standardisation process to develop an acceptable norm group?
