corpuscrawler

Corpus Crawler

Corpus Crawler is a tool for Corpus Linguistics.

Modern linguistic research works on language corpora, which are large samples of “real world” text. This crawler helps to build such corpora: it follows links to publicly accessible web pages known to be written in a certain language; it removes boilerplate and HTML markup; finally, it writes its output into plaintext files. The crawler implements the Robots Exclusion Standard, and it is intentionally slow so it does not cause much load on the crawled web sites.

This is not an official Google product. But if you’re a linguistic researcher, or if you’re writing a spell checker (or similar language-processing software) for an “exotic” language, you might find Corpus Crawler useful.

To build corpora for not-yet-supported languages, please read the contribution guidelines and send us GitHub pull requests.

The crawled corpora have been used to compute word frequencies in Unicode’s Unilex project.

Supported Languages

IETF BCP47 Code	Language	Tokens¹
aai	Arifama-Miniafia	181K 💾
aak	Ankave	194K 💾
aau	Abau	313K 💾
aaz	Amarasi	308K 💾
abt	Ambulas	297K 💾
aby	Aneme Wake	233K 💾
acd	Gikyode	323K 💾
ace	Aceh/Acehnese	817K 💾
acf	Saint Lucian Creole French	236K 💾
ach	Acoli	178K 💾
acn	Achang	232K 💾
acr	Achi	239K 💾
acu	Achuar-Shiwiar	174K 💾
ade	Adele	267K 💾
adh	Adhola	166K 💾
adj	Adioukrou	233K 💾
ae	Avestan	129K 💾
ae-Latn	Avestan (Latin)	141K 💾
aey	Amele	218K 💾
agd	Agarabi	256K 💾
agg	Angor	214K 💾
agm	Angaataha	238K 💾
agn	Agutaynen	234K 💾
agr	Aguaruna	149K 💾
ahk	Akha	367K 💾
aia	Arosi	223K 💾
akb	Batak Angkola	220K 💾
ake	Akawaio	190K 💾
akh	Akha	408K 💾
akp	Siwu	191K 💾
alj	Alangan	185K 💾
alp	Alune	225K 💾
alt	Southern Altai	121K 💾
alz	Alur	160K 💾
am	Amharic	2,170K 💾
ame	Yanesha’	221K 💾
amf	Hamer-Banna	152K 💾
amk	Ambai	229K 💾
amm	Ama (Papua New Guinea)	246K 💾
amn	Amanab	207K 💾
amp	Alamblak	241K 💾
amr	Amarakaeri	151K 💾
amu	Guerrero Amuzgo	202K 💾
ann	Obolo	236K 💾
anv	Denya	214K 💾
aoj	Mufian	217K 💾
aom	Ömie	231K 💾
aon	Bumbita Arapesh	294K 💾
aoz	Uab Meto	197K 💾
ape	Bukiyip	294K 💾
apr	Arop-Lokep	373K 💾
apz	Safeyoka	235K 💾
ar	Arabic	19,593K 💾
arl	Arabela	206K 💾
asg	Cishingini	270K 💾
aso	Dano	290K 💾
ata	Pele-Ata	248K 💾
atb	Zaiwa	291K 💾
atg	Ivbie North-Okpela-Arhe	229K 💾
atq	Aralle-Tabulahan	202K 💾
auy	Awiyaana	164K 💾
av	Avaric	111K 💾
avn	Avatime	229K 💾
avt	Au	263K 💾
avu	Avokaya	391K 💾
awa	Awadhi	211K 💾
awb	Awa (Papua New Guinea)	179K 💾
ay	Aymara	482K 💾
ayo	Ayoreo	264K 💾
az	Azerbaijani	3,413K 💾
azg	San Pedro Amuzgos Amuzgo	271K 💾
azz	Highland Puebla Nahuatl	265K 💾
ba	Bashkir	666K 💾
ban	Balinese	211K 💾
bao	Waimaha	232K 💾
bav	Vengo	250K 💾
bba	Baatonum	792K 💾
bbb	Barai	289K 💾
bbo	Northern Bobo Madaré	211K 💾
bbr	Girawa	245K 💾
bch	Bariai	248K 💾
bcw	Bana	304K 💾
bdd	Bunama	171K 💾
be	Belarusian	1,441K 💾
be-tarask	Belarusian (Taraškievica)	108,431K 💾
bef	Benabena	239K 💾
bep	Besoa	204K 💾
bex	Jur Modo	254K 💾
bfd	Bafut	276K 💾
bfo	Malba Birifor	260K 💾
bg	Bulgarian	10,597K 💾
bgr	Bawm Chin	213K 💾
bgz	Banggai	186K 💾
bhl	Bimin	324K 💾
bhw	Biak	164K 💾
bi	Bislama	315K 💾
bib	Bissa	243K 💾
big	Biangai	229K 💾
bik	Central Bikol	183K 💾
bim	Bimoba	215K 💾
biv	Southern Birifor	221K 💾
bjr	Binumarien	226K 💾
bjv	Bedjond	268K 💾
bkl	Berik	306K 💾
bku	Buhid	204K 💾
bkv	Bekwarra	244K 💾
blh	Kuwaa	259K 💾
blt-Latn	Tai Dam (Latin)	262K 💾
blz	Balantak	199K 💾
bm	Bambara	30K 💾
bmh	Kein	253K 💾
bmq	Bomu	207K 💾
bmr	Muinane	122K 💾
bmu	Somba-Siawari	234K 💾
bmv	Bum	258K 💾
bn	Bangla	7,258K 💾
bnj	Eastern Tawbuid	239K 💾
bnp	Bola	263K 💾
bo	Tibetan	5,642K 💾
boa	Bora	133K 💾
boj	Anjam	255K 💾
bon	Bine	244K 💾
bov	Tuwuli	203K 💾
box	Buamu	274K 💾
bpr	Koronadal Blaan	204K 💾
bps	Sarangani Blaan	214K 💾
bqc	Boko	567K 💾
bqj	Bandial	175K 💾
bqp	Busa	162K 💾
bru	Eastern Bru	261K 💾
bs	Bosnian	8,993K 💾
bsn	Barasana-Eduria	225K 💾
bss	Akoose	199K 💾
btd	Batak Dairi	192K 💾
bts	Batak Simalungun	175K 💾
btt	Bete-Bendi	266K 💾
btx	Batak Karo	189K 💾
bua	Buriat	143K 💾
bud	Ntcham	207K 💾
buk	Bugawac	264K 💾
bus	Bokobaru	159K 💾
bvc	Baelelea	308K 💾
bvz	Bauzi	509K 💾
bwq	Southern Bobo Madaré	214K 💾
bwu	Buli	285K 💾
byr	Baruya	182K 💾
byx	Qaqet	387K 💾
bzh	Mapos Buang	251K 💾
bzi	Bisu	381K 💾
bzj	Belize Kriol English	240K 💾
ca-valencia	Valencian	24,295K 💾
caa	Chortí	307K 💾
cab	Garifuna	154K 💾
cac	Chuj	244K 💾
cak	Kaqchikel	259K 💾
cap	Chipaya	154K 💾
car	Galibi Carib	160K 💾
cax	Chiquitano	149K 💾
cbc	Carapana	256K 💾
cbi	Chachi	187K 💾
cbl	Bualkhaw Chin	210K 💾
cbr	Cashibo-Cacataibo	236K 💾
cbs	Cashinahua	198K 💾
cbt	Chayahuita	150K 💾
cbv	Cacua	265K 💾
cce	Chopi	204K 💾
ccp	Chakma	79K 💾
cdf	Chiru	193K 💾
ce	Chechen	669K 💾
ceb	Cebuano	1,067K 💾
ceg	Chamacoco	232K 💾
cfm	Falam Chin	438K 💾
cgc	Kagayanen	299K 💾
chj	Ojitlán Chinantec	305K 💾
chm	Mari	132K 💾
chr	Cherokee	119K 💾
chz	Ozumacín Chinantec	205K 💾
cjo	Ashéninka Pajonal	141K 💾
cjp	Cabécar	199K 💾
cjv	Chuave	286K 💾
cko	Anufo	272K 💾
cle	Lealao Chinantec	313K 💾
cme	Cerma	230K 💾
cmr	Mro-Khimi Chin	275K 💾
cnh	Hakha Chin	934K 💾
cni	Asháninka	122K 💾
cnk	Khumi Chin	237K 💾
cnl	Lalana Chinantec	308K 💾
cnt	Tepetotutla Chinantec	261K 💾
coe	Koreguaje	181K 💾
cof	Colorado	183K 💾
cok	Santa Teresa Cora	230K 💾
con	Cofán	151K 💾
cot	Caquinte	128K 💾
crh	Crimean Tatar	505K 💾
cs	Czech	3,141K 💾
csk	Jola-Kasa	177K 💾
cso	Sochiapam Chinantec	328K 💾
ctd-Latn	Tedim Chin (Latin)	852K 💾
ctu	Chol	203K 💾
cub	Cubeo	220K 💾
cuc	Usila Chinantec	278K 💾
cui	Cuiba	292K 💾
cuk	San Blas Kuna	187K 💾
cul	Culina	221K 💾
cv	Chuvash	111K 💾
cwe	Kwere	144K 💾
cwt	Kuwaataay	168K 💾
cy	Welsh	11,519K 💾
cya	Nopala Chatino	245K 💾
czt	Zotung Chin	227K 💾
da	Danish	655K 💾
daa	Dangaléat	208K 💾
dad	Marik	197K 💾
dah	Gwahatike	274K 💾
ddn	Dendi	210K 💾
de	German	46,431K 💾
ded	Dedua	146K 💾
des	Desano	210K 💾
dga	Southern Dagaare	458K 💾
dgi	Northern Dagara	257K 💾
dgz	Daga	219K 💾
din	Southwestern Dinka	196K 💾
dip	Northeastern Dinka	193K 💾
djk	Eastern Maroon Creole	307K 💾
dln	Darlong	776K 💾
dnw	Western Dani	254K 💾
dob	Dobu	179K 💾
dop	Lukpa	226K 💾
dsh	Daasanach	211K 💾
dtb	Labuk-Kinabatangan Kadazan	248K 💾
dtp	Kadazan Dusun	1,038K 💾
dts	Toro So Dogon	202K 💾
due	Umiray Dumaget Agta	247K 💾
dug	Duruma	172K 💾
duo	Dupaninan Agta	266K 💾
dwr	Dawro	254K 💾
dww	Dawawa	208K 💾
dyi	Djimini Senoufo	268K 💾
dyo	Jola-Fonyi	158K 💾
dyu	Dyula	1,156K 💾
dz	Dzongkha	61K 💾
ee	Ewe	421K 💾
eka	Ekajuk	213K 💾
el	Greek	5,470K 💾
emi	Mussau-Emira	176K 💾
emp	Northern Emberá	158K 💾
enb	Markweeta	147K 💾
enq	Enga	217K 💾
enx	Enxet	772K 💾
eri	Ogea	269K 💾
es	Spanish	32,670K 💾
ese	Ese Ejja	226K 💾
et	Estonian	3,658K 💾
eu	Basque	130K 💾
ewo	Ewondo	158K 💾
eza	Ezaa	963K 💾
fa	Persian	9,114K 💾
fa-AF	Dari	7,363K 💾
faa	Fasu	238K 💾
fai	Faiwol	256K 💾
fal	South Fali	198K 💾
far	Fataleka	286K 💾
fi	Finnish	4,837K 💾
fil	Tagalog	184K 💾
fip	Fipa	134K 💾
fit	Tornedalen Finnish	292K 💾
fj	Fijian	257K 💾
fo	Faroese	851K 💾
fon	Fon	266K 💾
for	Fore	169K 💾
fr	French	5,488K 💾
fue	Borgu Fulfulde	148K 💾
fuf	Pular	174K 💾
fuq	Central-Eastern Niger Fulfulde	156K 💾
fuv	Nigerian Fulfulde	13K 💾
ga	Irish	7,587K 💾
gag	Gagauz	245K 💾
gah	Alekano	210K 💾
gam	Kandawo	250K 💾
gaw	Nobonob	246K 💾
gbi	Galela	288K 💾
gd	Scottish Gaelic	17,105K 💾
gde	Gude	217K 💾
gdn	Umanakaina	306K 💾
gdr	Wipi	271K 💾
gej	Gen	236K 💾
gfk	Patpatar	294K 💾
ghs	Guhu-Samane	186K 💾
gil	Gilbertese	228K 💾
gkn	Gokana	267K 💾
gmv-Latn	Gamo (Latin)	127K 💾
gn	Guarani	142K 💾
gnd	Zulgo-Gemzek	364K 💾
gng	Ngangam	219K 💾
gnw	Western Bolivian Guaraní	263K 💾
gof	Gofa	124K 💾
gog	Gogo	173K 💾
gor	Gorontalo	211K 💾
gqr	Gor	218K 💾
grb	Northern Grebo	270K 💾
grt	Garo	141K 💾
gso	Southwest Gbaya	228K 💾
gsw-u-sd-chag	Swiss German (Aargau)	99K 💾
gsw-u-sd-chbe	Swiss German (Bern)	73K 💾
gsw-u-sd-chfr	Swiss German (Fribourg)	42K 💾
gu	Gujarati	702K 💾
gub	Guajajára	997K 💾
guc	Wayuu	211K 💾
gud	Yocoboué Dida	216K 💾
guh	Guahibo	204K 💾
gui	Eastern Bolivian Guaraní	197K 💾
gum	Guambiano	186K 💾
gun	Mbyá Guaraní	176K 💾
guo	Guayabero	203K 💾
guq	Aché	184K 💾
gur	Farefare	240K 💾
gux	Gourmanchéma	215K 💾
gv	Manx Gaelic	152K 💾
gvc	Guanano	241K 💾
gvf	Golin	276K 💾
gvl	Gulay	270K 💾
gwr	Gwere	157K 💾
gym	Ngäbere	294K 💾
gyr	Guarayu	176K 💾
ha	Hausa	1,775K 💾
hae	Eastern Oromo	163K 💾
hag	Hanga	202K 💾
haw	Hawaiian	2,221K 💾
hay	Haya	112K 💾
heh	Hehe	136K 💾
hi	Hindi	10,004K 💾
hif	Fiji Hindi	204K 💾
hig	Kamwe	261K 💾
hil	Hiligaynon	208K 💾
hla	Halia	273K 💾
hne	Chhattisgarhi	207K 💾
hnn	Hanunoo	212K 💾
hns	Caribbean Hindustani	312K 💾
ho	Hiri Motu	240K 💾
hot	Hote	222K 💾
hr	Croatian	8,188K 💾
ht	Haitian	1,101K 💾
hto	Minica Huitoto	182K 💾
hu	Hungarian	600K 💾
hub	Huambisa	160K 💾
hui	Huli	232K 💾
hus	Huastec	236K 💾
huu	Murui Huitoto	165K 💾
huv	San Mateo Del Mar Huave	197K 💾
hvn	Sabu	312K 💾
hy	Armenian	25,972K 💾
ian	Iatmul	224K 💾
iba	Iban	179K 💾
icr	Islander Creole English	248K 💾
id	Indonesian	6,634K 💾
ifa	Amganad Ifugao	810K 💾
ifb	Batad Ifugao	835K 💾
ife	Ifè	300K 💾
ifk	Tuwali Ifugao	214K 💾
ifu	Mayoyao Ifugao	258K 💾
ify	Keley-I Kallahan	863K 💾
ig	Igbo	13K 💾
ign	Ignaciano	161K 💾
ik	Inupiaq	96K 💾
ilo	Iloko	169K 💾
imo	Imbongu	280K 💾
inb	Inga	151K 💾
ino	Inoke-Yate	236K 💾
iou	Tuma-Irumu	225K 💾
ipi	Ipili	312K 💾
iri	Irigwe	243K 💾
irk	Iraqw	184K 💾
iry	Iraya	205K 💾
it	Italian	13,569K 💾
itv	Itawit	242K 💾
iu	Inuktitut	98K 💾
iws	Sepik Iwam	307K 💾
izr	Izere	216K 💾
izz	Izii	908K 💾
ja	Japanese	2,116K 💾
jac	Popti’	221K 💾
jae	Yabem	186K 💾
jam	Jamaican Creole English	254K 💾
jbu	Jukun Takum	264K 💾
jic	Tol	285K 💾
jiv	Shuar	134K 💾
jmc	Machame	150K 💾
jun	Juang	178K 💾
jv	Javanese	177K 💾
jvn	Caribbean Javanese	211K 💾
ka	Georgian	4,978K 💾
kaa	Kara-Kalpak	135K 💾
kab-Arab	Kabyle (Arabic)	715K 💾
kab-Tfng	Kabyle (Tifinagh)	1,338K 💾
kab	Kabyle	66K 💾
kac	Kachin	1,057K 💾
kao	Xaasongaxango	205K 💾
kaq	Capanahua	164K 💾
kbh	Camsá	193K 💾
kbm	Iwal	298K 💾
kbp	Kabiyè	571K 💾
kbq	Kamano	156K 💾
kbr	Kafa	147K 💾
kcg	Tyap	279K 💾
kdc	Kutu	140K 💾
kdi	Kumam	195K 💾
kdj	Karamojong	163K 💾
kdn	Kunda	144K 💾
kek	Kekchí	406K 💾
ken	Kenyang	200K 💾
keo	Kakwa	215K 💾
ker	Kera	267K 💾
kew	West Kewa	247K 💾
kez	Kukele	173K 💾
kgf	Kube	175K 💾
kgr	Abun	356K 💾
khz	Keapara	196K 💾
kia	Kim	525K 💾
kij	Kilivila	155K 💾
kj	Kuanyama	1,474K 💾
kjb	Q’anjob’al	263K 💾
kje	Kisar	235K 💾
kjh	Khakas	128K 💾
kjs	East Kewa	251K 💾
kk	Kazakh	642K 💾
kki	Kagulu	125K 💾
kkj	Kako	263K 💾
kln	Kalenjin	149K 💾
km	Khmer	29,110K 💾
kma	Konni	230K 💾
kmg	Kâte	127K 💾
kmo	Kwoma	213K 💾
kms	Kamasau	293K 💾
kmu	Kanite	214K 💾
kn	Kannada	126K 💾
kne	Kankanaey	230K 💾
knf	Mankanya	164K 💾
knj	Western Kanjobal	1,350K 💾
knk	Kuranko	228K 💾
kno	Kono	360K 💾
knv	Tabo	243K 💾
kog	Cogui	189K 💾
kpf	Komba	174K 💾
kpg	Kapingamarangi	967K 💾
kpr	Korafe-Yegha	262K 💾
kpw	Kobon	288K 💾
kpx	Mou

README 内容过长，已截取前 64 KB。完整内容请前往 GitHub 仓库查看。