FeatherCast

The voice of The Apache Software Foundation

Apache Big Data Seville 2016 – Crawling the Web for Common Crawl – Sebastian Nagel

January 2, 2017
asfinfra

Crawling the Web for Common Crawl – Sebastian Nagel

Common Crawl is non-profit organization which regularily crawls a significant sample of the web and makes the data accessible free charge to everyone interested in running machine-scale analysis on web data. The presentation will demonstrate how to use the Common Crawl data covering data formats and tools as well as examples and derived datasets. The monthly crawls are run by Apache Nutch on Apache Hadoop. Sebastian will also share his experience from running a web-scale crawl on a small budget.

More information about this talk

Leave a Reply

Powered by WordPress.com.
%d bloggers like this: