Linguistics has experienced an increasing focus on experimental evidence in recent times. This does not only concern external linguistic interfaces dealt with in psycho- or sociolinguistics. Proper linguistic sub-disciplines show a rapidly growing interest in experimental data as a base for theoretical linguistics, leading to new research strands such as experimental phonology, experimental syntax, experimental semantics or experimental pragmatics.
Despite an ongoing fundamental discussion on the general relevance of experimental data in theoretical linguistics, each linguistic sub-discipline has at commands a set of formal methods for data collection, which are frequently applied or newly developed. Here, linguists can refer to various sets of guidelines. Furthermore, we can orient ourselves by means of a large variety of study documentations. But as it turns out, many practical aspects remain unclear. Both methods on their own or together seem to not yet be exhaustive enough to answer all questions on proper procedure or to function as a base for a replication of individual studies.

As an example,
take the challenges in the field of experimental syntax, which are not restricted to acceptability judgments, but are rather very general in nature. Until today, the need for judgment data in syntactic theorizing is controversially discussed. Proponents of experimental approaches call into question the empirical base of theories based on informal methods (Wasow & Arnold 2005; Featherston 2007; Gibson & Fedorenko 2010; Häussler & Juzek 2017). Other researches point to the practicability and various achievements of informal theorizing, thus defending introspective approaches (Phillips & Lasnik 2003; Phillips & Wagers 2007; Phillips 2009).

Proponents of experimental approaches can make use of a set of established formal methods (Likert Scales, Forced Choice, Magnitude Estimation) and newly introduced ones (Thermometer Task (Featherston 2008), Sentence Fragment Arrangement Task (e.g. Störzer 2017)). They can draw on concrete guidelines for experimental practice (Cowart 1997; Schütze 1996/²2016; Schütze & Sprouse 2013) as well as on numerous study documentations. But various issues regarding conception or implementation remain unresolved or unaddressed in both sources, ranging from questions related to study design or sentence material to debates about an up-to-date procedure or about adequate method-specific statistical analyses. Moreover, guidelines are often too vague to ensure a uniform application.

Let us take a closer look at some of the questions left unanswered …

Regarding study design,
we first focus on factorial designs. We can observe that a great majority of studies constrain themselves to a small number of conditions, presumably simply motivated by practicability. But how can we approach complex phenomena or typological variation in such a limited design? Extending the numbers of conditions is not that simple, for it would confront us with questions such as how to keep the questionnaires at a processible length or how to analyze such data properly. Secondly, the adequate sample size is another open question. Häussler & Juzek (2017) present the issue as follows: “While there is no research into the question of how many participants are too many in a syntactic acceptability judgement task, we suggest testing 6-8 participants per condition as a rule of thumb.”

With respect to sentence material,
several aspects have to be worked out. While some properties of test items – e.g. corpus frequencies – are usually controlled for, others have often been neglected. For example, one of these seldom explicitly addressed but very important properties is the influence of semantics on syntactic ratings. In addition, especially in the case of scale-based studies, methodological reflections on filler and control items are necessary (Juzek & Häussler 2019).

Concerning the procedure,
we want to point to two exemplary issues. First, there is a need to reflect on instructions, for example on the adequate degree of formality or complexity (Juzek 2016; Juzek & Häussler 2019). Secondly, we observe a growing orientation towards online studies. While there are promising advantages, this type of experiment comes with its own set of challenges. For example, we have to reflect on how to approach the limited controllability. One possibility is a combined analysis of control items and of response times (Häussler & Juzek 2016), but what could be further solutions to this problem?

With regard to data analysis,
we are confronted with diverging applications of different models on judgment data, leaving open the question of an ideal method. Here, current critical examinations can point to advances and future directions (Schütze et al. 2013; Barr 2013; Song et al. 2017). But we are still far from definite standards.

In light of the various open questions, the purpose of this conference is to bring together experimental researchers working in different strands and different linguistic sub-disciplines or related areas in order to gather and critically discuss open issues. Thus, each individual contribution presents certain intricate practical aspects or corresponding pragmatic solutions, or methodological reflections and impulses on linguistic experimentation. With this, we want to contribute to a critical evaluation of current standards as well as to the development of new standards and guidelines in the field.