BramVanroy/CommonCrawl-CreativeCommons-fine

Common Crawl Creative Commons Corpus Fine (C5f)

A filtered version of the Common Crawl Creative Commons Corpus (C5), only retaining samples that are also present in the FineWeb or FineWeb-2 datasets. As such, this dataset contains a high-quality subset of C5.

Created with this script.

For more information, see C5.

Progress

In the v1 release, the following crawls are included

• CC-MAIN-2019-30
• CC-MAIN-2020-05
• CC-MAIN-2023-06
• CC-MAIN-2024-51
• CC-MAIN-2024-46
• CC-MAIN-2025-05 (included in the original but not here because FineWeb and FineWeb-2 do not include it)
• CC-MAIN-2022-05

Languages

The following languages are included. This is a limited set due to computational and storage limitations.

• Afrikaans: afr
• German: deu
• English: eng
• French: fra
• Frysian: fry
• Italian: ita
• Dutch: nld
• Spanish: spa

Quantity

Counts for the fine release. Detailed number of tokens (Llama 3.3 tokenizer) and number of documents are given in the counts.json file.

Note: when comparing the original with this version, understand that: i. this version does not include crawl 2025-05 because FineWeb(-2) did not include it; ii. both 2024-46 and 2024-51 only contain English because FineWeb-2 did not include those crawls. So calculating quantity reduction after filtering should take this information into account.

Language	No. Docs (original)	No. Docs (C5f)	No. Tokens (original)	No. Tokens (C5f)
afr	312,262	5,753	358,873,448	8,214,345
deu	9,530,746	224,789	11,362,859,534	258,945,770
eng	92,635,372	17,528,954	87,537,859,958	16,629,260,476
fra	9,234,900	136,349	12,366,480,025	176,835,571
fry	230,910	3,240	197,430,774	3,879,970
ita	10,734,597	301,315	11,913,669,333	334,812,841
nld	2,827,636	60,572	2,757,074,705	60,488,015
spa	22,226,944	502,896	22,515,709,432	496,644,788
Total	147,733,367	18,763,868	149,009,957,209	17,969,081,776