Tea@programming.dev to Technology@lemmy.worldEnglish · edit-22 days agoAI crawlers cause Wikimedia(The umbrella organization of Wikipedia and a dozen or so other crowdsourced knowledge projects) Commons bandwidth demands to surge 50%.diff.wikimedia.orgexternal-linkmessage-square77fedilinkarrow-up1709arrow-down16
arrow-up1703arrow-down1external-linkAI crawlers cause Wikimedia(The umbrella organization of Wikipedia and a dozen or so other crowdsourced knowledge projects) Commons bandwidth demands to surge 50%.diff.wikimedia.orgTea@programming.dev to Technology@lemmy.worldEnglish · edit-22 days agomessage-square77fedilink
minus-squarepewter@lemmy.worldlinkfedilinkEnglisharrow-up1·1 day agoThere’s a chance this isn’t being done by someone who only wants Wikipedia’s data. As the amount of websites you scrape increases, your desire to use the easy tools loses out to creating the most general tool that can look at most webpages.
There’s a chance this isn’t being done by someone who only wants Wikipedia’s data. As the amount of websites you scrape increases, your desire to use the easy tools loses out to creating the most general tool that can look at most webpages.