Datasets:
The dataset viewer is not available for this dataset.
Need help to make the dataset viewer work? Make sure to review how to configure the dataset viewer, and open a discussion for direct support.
CIL-Dataset: Colombian Indigenous Languages
Parallel and monolingual corpora for machine translation between Spanish and four Colombian Indigenous languages: Wayuunaiki, Inga, Kamëntsá and Nasa Yuwe. Compiled for the thesis Data-Centric Strategies for Low-Resource Neural Machine Translation in Colombian Indigenous Languages (Universidad de los Andes, FLAG Lab).
Languages
| Language | Column | ISO 639-3 |
|---|---|---|
| Spanish | esp |
spa |
| Wayuunaiki | way |
guc |
| Inga | ing |
inb |
| Kamëntsá | kam |
kbh |
| Nasa Yuwe | nas |
pbb |
Contents
Parallel and monolingual corpora
Full corpora, one aligned segment per row.
| Config | File | Rows | Columns |
|---|---|---|---|
inga |
inga.csv |
4,552 | esp, ing, origen |
kamentsa |
kamentsa.csv |
281 | esp, kam, origen |
nasa_yuwe |
nasa_yuwe.csv |
4,031 | esp, nas, origen |
wayuunaiki |
wayuunaiki.csv |
83,579 | esp, way, origen |
wayuunaiki_monolingual |
wayuunaiki_monolingual.csv |
2,676 | way, origen |
The origen column is a provenance identifier formatted as
<language>__<type>_<source>__<year> (for example
wayuu__dictionary_negrete_amaya__2021). Sources include Bible translations,
dictionaries, grammars, constitutional texts, short stories and books.
Rephrasing subset (Wayuunaiki)
Train/validation/test splits used to study rogue memorization. In the training split, the Spanish side of biblical rows is replaced by an alternate human Spanish Bible translation, which reduces repeated Spanish targets. The Wayuunaiki side is never modified, and validation/test keep the original Spanish.
| Config | Split | Rows |
|---|---|---|
wayuunaiki_rephrasing_completo |
train / validation / test | 67,135 / 8,222 / 8,222 |
wayuunaiki_rephrasing_sin_dict |
train / validation / test | 7,469 / 763 / 764 |
Extra columns in the rephrasing files: source_row_id, esp_original,
esp_target_rephrased, verse_id, repeat_group_size, assigned_version,
rephrase_applied, rephrase_strategy.
Usage
from datasets import load_dataset
# a parallel corpus
inga = load_dataset("Flaglab/CIL-Dataset", "inga")
# the rephrasing splits
reph = load_dataset("Flaglab/CIL-Dataset", "wayuunaiki_rephrasing_completo")
print(reph["train"][0])
Trained models
NLLB-200 fine-tunes built on this data are available in the collection Machine Translation for Colombian Indigenous Languages.
Sources and licensing
The corpora were assembled from heterogeneous materials (Bible translations, dictionaries, grammars, constitutional texts, community stories and books). These sources carry their own copyrights and licenses. The dataset is released for non-commercial research use; downstream users are responsible for respecting the licensing of each underlying source. The rephrasing subset draws Spanish renderings from public Spanish Bible versions in the BibleNLP/eBible corpus.
- Downloads last month
- 30