RuBLiMP includes 45k minimal pairs of sentences that differ in grammaticality and isolate morphological, syntactic, or semantic phenomena. In contrast to existing benchmarks of linguistic minimal pairs, RuBLiMP is created by applying linguistic perturbations to automatically annotated sentences from open text corpora and decontaminating test data, making the minimal pairs more diverse and natural.

Language: Russian
Repository: github.com/RussianNLP/RuBLiMP
Paper: aclanthology.org/2024.emnlp-main.522
License: Apache-2.0

Dataset Creation

Example: Vpervye kosmonavt spal v nevesomosti "For the first time an astronaut slept in zero gravity".

The RuBLiMP's creation approach comprises four stages:

(a) Extract sentences from publicly available corpora of Wikipedia texts, news articles, and books.
(b) Annotate each extracted sentence in the Universal Dependencies scheme with a multidomain morphosyntactic parser for Russian.
(c) Search the dependency trees for specific lexical units and linguistic structures and apply expert-written perturbation rules to create a pool of minimal pairs for a target paradigm.
(d) Compute Min-K% Prob for each grammatical sentence in the pool using a set of LMs. Select t (the threshold for the maximum Min-K% Prob value), which allows to find an intersection of 1k minimal pairs between the LMs. The minimal pairs in the intersection contain grammatical sentences that are not detected as the LMs' pretraining examples.

Dataset Structure

Data Instances

Each instance in the dataset is pair of sentences, where a certain linguistic phenomenon is isolated:

{
    'id': 240907, 
    'source_sentence': 'Это еще вчера все поняли.', 
    'target_sentence': 'Это еще завтра все поняли.', 
    'source_word': 'вчера', 
    'target_word': 'завтра', 
    'level': 'Semantics', 
    'phenomenon': 'Tense', 
    'PID': 'tense_marker',
    'subtype': 'tense_marker_simple',
    'domain': 'librusec',
    'tree_depth': 2
}

An example in English for illustration purposes:

{
    'id': 240907, 
    'source_sentence': 'Everyone understood this yesterday already.', 
    'target_sentence': 'Everyone understood this tomorrow already.', 
    'source_word': 'yesterday', 
    'target_word': 'tomorrow', 
    'level': 'Semantics', 
    'phenomenon': 'Tense', 
    'PID': 'tense_marker',
    'subtype': 'tense_marker_simple',
    'domain': 'librusec',
    'tree_depth': 2
}

Data Fields

id: minimal pair identificator from the corpus
source_sentence: original grammatical sentence
target_sentence: perturbed ungrammatical sentence
source_word: token in the original, undergoing the perturbation
target_word: token in the ungrammatical sentence after the perturbation
level: language level, targeted by the minimal pair (Syntax, Semantics or Morphology)
phenomenon: targeted phenomenon
PID: name of the paradigm
subtype: type of the paradigm, if applicable
domain: sentence source (librusec, wikipedia or wikinews)
tree_depth: depth, of the sentence's syntax tree

Phenomena

Morphology

Word Formation

Addition of Extra Morphemes: Uninterpretable Suffix Combinations (add_new_suffix)
Adding a new suffix to the noun or adjective to create a non-existing word
Addition of Extra Morphemes: Verb Prefixes (add_verb_prefix)
Adding a prefix to a verb to create a violation of prefix stacking rules.
Morpheme Permutation: Verb Prefixes (change_verb_prefixes_order)
Changing the order of the verb's prefixes to create a violation of prefix stacking rules.

Word Inflection

Replacement of Inflectional Affixes: Noun Declensions (Simple) (change_declension_ending)
Changing the inflectional suffixes of a noun to the suffixes of another declension
Replacement of Inflectional Affixes: Declensions of Nouns With Agreeing Dependents (change_declension_ending_has_dep)
Changing the inflectional suffixes of a noun to the suffixes of another declension in the presence of an agreeing noun modifier
Inflectional Affixes: Verbal Conjugation Swap (change_verb_conjugation)
Replacing the verb’s inflection with inflection of the opposite conjugation

Syntax

Government

Prepositional Government (adp_government_case)
Changing the case of a noun, governed by a preposition
Verbal Government: Direct Object (verb_acc_object)
Changing the case of a direct verb object
Verbal Government: Genitive Object (verb_gen_object)
Changing the case of an indirect verb object in Genitive case
Verbal Government: Object in Instrumental Case (verb_ins_object)
Changing the case of an indirect verb object in Instrumental case
Verbal Government: Nominalizations (nominalization_cas)
Changing the case of a dependent of a nominalization

Subject-Predicate Agreement

Subject-Predicate Agreement (Number) (noun_subj_predicate_agreement_number)
Changing the number of the predicate to be distinct from its subject's (or, sometimes, changing number of the subject to be distinct from its predicate's)
Genitive Subject-Predicate Agreement (Number) (genitive_subj_predicate_agreement_number)
Changing the number of the predicate to plural, when subject is genitive and the agreement must be the default singular neuter
Clausal Subject-Predicate Agreement (Number) (clause_subj_predicate_agreement_number)
Changing the number of the predicate to plural, when subject is a clause and the agreement must be the default singular neuter
Subject-Predicate Agreement in Presence of an Attractor (Number) (subj_predicate_agreement_number_attractor)
Changing the number of the verb to that, which is different from the subject, but the same as subject's dependent, or the attractor
Subject-Predicate Agreement (Gender) (noun_subj_predicate_agreement_gender)
Changing the gender of the predicate to be distinct from its subject's (or, sometimes, changing number of the subject to be distinct from its predicate's)
Genitive Subject-Predicate Agreement (Gender) (genitive_subj_predicate_agreement_gender) Changing the gender of the predicate to feminine or masculine, when subject is genitive and the agreement must be the default singular neuter
Clausal Subject-Predicate Agreement (Gender) (clause_subj_predicate_agreement_gender)
Changing the gender of the predicate to feminine or masculine, when subject is a clause and the agreement must be the default singular neuter
Subject-Predicate Agreement in Presence of an Attractor (Gender) (subj_predicate_agreement_gender_attractor)
Changing the gender of the verb to that, which is different from the subject, but the same as subject's dependent, or the attractor
Subject-Predicate Agreement (Person) (noun_subj_predicate_agreement_person)
Changing the person of the predicate to be distinct from its subject's
Genitive Subject-Predicate Agreement (Person) (genitive_subj_predicate_agreement_person) \ Changing the person of the predicate to first or second person, when subject is genitive and the agreement must be the default third person singular
Clausal Subject-Predicate Agreement (Person) (clause_subj_predicate_agreement_person) \ Changing the person of the predicate to first or second person, when subject is a clause and the agreement must be the default third person singular

Anaphor Agreement

Anaphor Agreement (Number) (anaphor_agreement_number)
Changing the number of the relative pronoun or of its head noun
Anaphor Agreement (Gender) (anaphor_agreement_gender)
Changing the gender of the relative pronoun

Noun Phrase Agreement

Noun Phrase Agreement (Number) (np_agreement_number)
Changing the number of an agreeing adjective
Noun Phrase Agreement (Gender) (np_agreement_gender)
Changing the gender of an agreeing adjective
Noun Phrase Agreement (Case) (np_agreement_case)
Changing the case of an agreeing adjective

Floating Quantifier Agreement

Floating Quantifier Agreement (Number) (floating_quantifier_agreement_number)
Changing the number of the quantifier or of the controller
Floating Quantifier Agreement (Gender) (floating_quantifier_agreement_gender)
Changing the gender of the quantifier or of the controller
Floating Quantifier Agreement (Case) (floating_quantifier_agreement_case)
Changing the case of the quantifier or of the controller

Reflexives

External Possessor (external_possessor)
Change a noun phrase or a pronoun to a reflexive pronoun sebya ‘self’ in a u-phrase inside the existential be-possessive construction.

Negation

Negative Concord (negative_concord)
Shifting the negative particle ne from a negated verb to another word in the sentence to violate negative concord rules.
Replacement of a Negative Pronoun with an Indefinite One (negative_pronoun_to_indefinite)
Replacing an negative pronoun in the construction without a negated verb to an indefinite pronoun
Replacement of an Indefinite Pronoun with a Negative One (indefinite_pronoun_to_negative)
Replacing an indefinite pronoun in the construction with a negated verb to a negative pronoun

Semantics

Argument Structure

Transitivity (transitive_verb)
Replacing a transitive verb with an intransitive one
Animate Subject of a Transitive Verb (transitive_verb_subject)
Swapping the subject and the direct object of a transitive verb or replacing the subject with a random inanimate word
Animate Subject of a Passive Verb (transitive_verb_passive)
Swapping the subject and the direct object of a transitive verb in a passive construction or replacing the subject with a random inanimate word
Animate Direct Object of a Transitive Verb (transitive_verb_object) \ Replacing the direct object with a random inanimate word
Animate Indirect Object of a Transitive Verb (transitive_verb_iobject) \ Swapping the subject and the indirect object of a transitive verb or replacing the indirect subject of a transitive verb with a random inanimate word

Aspect

Incompatibility of the Perfective with the Semantics of Duration (change_duration_aspect)
Replacing an imperfective verb with a perfective one in contexts with semantics of duration
Impossibility of the Perfective in Repetitive Situations (change_repetition_aspect)
Replacing an imperfective verb with a perfective one in contexts with semantics of repetition
Impossibility of the Perfective Under Negated Strong Deontic Verbs (deontic_imperative_aspect)
Replacing an imperfective verb with a perfective one in contexts with a negated deontic verb

Tense

Tense (single_verb_tense)
Changing verb tense in the presence of a temporal adverbial
Tense (coordination) (conj_verb_tense)
Changing the tense of a conjoined verb in the presence of a temporal adverbial
Tense Markers (tense_marker)
Changing a temporal adverbial in a sentence with a tense-marked verb

Additional Information

Dataset Curators

Ekaterina Taktasheva (@evtaktasheva), Maxim Bazhukov (@bamaxi), Kirill Koncha (@midwestcyr), Ekaterina Artemova (@KatyaA), Vladislav Mikhailov (@vmkhlv)

Licensing Information

Apache 2.0

Citation Information

@misc{taktasheva2024rublimprussianbenchmarklinguistic,
      title={RuBLiMP: Russian Benchmark of Linguistic Minimal Pairs}, 
      author={Ekaterina Taktasheva and Maxim Bazhukov and Kirill Koncha and Alena Fenogenova and Ekaterina Artemova and Vladislav Mikhailov},
      year={2024},
      eprint={2406.19232},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2406.19232}, 
}

Downloads last month: 1,691

Total file size:

7.51 MB

Paper for RussianNLP/rublimp

RuBLiMP: Russian Benchmark of Linguistic Minimal Pairs

Paper • 2406.19232 • Published Jun 27, 2024 • 1