Category: corpus measurements

Corpus similarity measures remain robust across diverse languages

June 9, 2022

Li, H. & Dunn, J. (2022). “Corpus similarity measures remain robust across diverse languages.” Lingua. Corpus similarity remains a robust measurable property across 39 diverse languages. Measures are highly accurate on a register prediction task using comparable corpora. Accuracy is not dependent on specific registers, shown by out-of-domain experiments. Corpus similarity measures also work for … More Corpus similarity measures remain robust across diverse languages

Predicting Embedding Reliability in Low-Resource Settings

April 21, 2022

Dunn, J.; Li, H.; & Sastre, D. (2022). “Predicting Embedding Reliability in Low-Resource Settings Using Corpus Similarity Measures.” In Proceedings of the 13th International Conference on Language Resources and Evaluation. European Language Resources Association. 6461-6470. Abstract This paper simulates a low-resource setting across 17 languages in order to evaluate embedding similarity, stability, and reliability under … More Predicting Embedding Reliability in Low-Resource Settings

Representations of Language Varieties Are Reliable

March 2, 2021

Dunn, J. (2021). “Representations of Language Varieties Are Reliable Given Corpus Similarity Measures.” In Proceedings of the Eighth Workshop on NLP for Similar Languages, Varieties, and Dialects. Association for Computational Linguistics. 28-38. Abstract. This paper measures similarity both within and between 84 language varieties across nine languages. These corpora are drawn from digital sources (the … More Representations of Language Varieties Are Reliable

Measuring Linguistic Diversity During COVID-19

October 14, 2020

Dunn, J.; Coupe, T.; & Adams, B. (2020). “Measuring Linguistic Diversity During COVID-19.” Proceedings of the 4th Workshop on NLP and Computational Social Science. Association for Computational Linguistics. 1-10. Abstract. Computational measures of linguistic diversity help us understand the linguistic landscape using digital language data. The contribution of this paper is to calibrate measures of … More Measuring Linguistic Diversity During COVID-19

Mapping Languages: The Corpus of Global Language Use

March 8, 2020

Dunn, J. (2020). “Mapping Languages: The Corpus of Global Language Use.” Language Resources and Evaluation. 54: 999-1018. Abstract. This paper describes a web-based corpus of global language use with a focus on how this corpus can be used for data-driven language mapping. First, the corpus provides a representation of where national varieties of major languages … More Mapping Languages: The Corpus of Global Language Use

Geographically-Balanced Gigaword Corpora for 50 Language Varieties

March 8, 2020

Dunn, J. & Adams, B. (2020). “Geographically-Balanced Gigaword Corpora for 50 Language Varieties.” In Proceedings of the Language Resources and Evaluation Conference. European Language Resources Association. 2528-2536. Abstract. While text corpora have been steadily increasing in overall size, even very large corpora are not designed to represent global population demographics. For example, recent work has … More Geographically-Balanced Gigaword Corpora for 50 Language Varieties

Mapping Languages and Demographics

July 22, 2019

Dunn, J. and Adams, B. (2019). “Mapping Languages and Demographics with Georeferenced Corpora.” In Proceedings of Geocomputation 2019. Abstract. This paper evaluates large georeferenced corpora, taken from both web-crawled and social media sources, against ground-truth population and language-census datasets. The goal is to determine (i) which dataset best represents population demographics; (ii) in what parts … More Mapping Languages and Demographics

Computational Learning of Construction Grammars

October 4, 2018

Dunn, J. (2017). “Computational Learning of Construction Grammars.” Language and Cognition, 9(2): 254-292. Abstract. This paper presents an algorithm for learning the construction grammar of a language from a large corpus. This grammar induction algorithm has two goals: first, to show that construction grammars are learnable without highly specified innate structure; second, to develop a … More Computational Learning of Construction Grammars

Multi-Unit Directional Measures of Association

October 4, 2018

Dunn, J. (2018). “Multi-Unit Directional Measures of Association: Moving Beyond Pairs of Words.” International Journal of Corpus Linguistics, 23(2): 183-215. Abstract. This paper formulates and evaluates a series of multi-unit measures of directional association, building on the pairwise ΔP measure, that are able to quantify association in sequences of varying length and type of representation. … More Multi-Unit Directional Measures of Association