Jiang, Jingtian, Nenghai Yu, and Chin-Yew Lin. "Focus: learning to crawl web forums." Proceedings of the 21st International Conference on World Wide Web. 2012.

This paper introduces FoCUS (Forum Crawler Under Supervision), a supervised web-scale forum crawler designed to efficiently gather relevant forum content with minimal overhead. Forums, despite their varied layouts and software packages, share similar implicit navigation paths via specific URL types that connect entry pages to thread pages. FoCUS reduces the forum crawling problem to recognizing these URL types and learns accurate regular expression patterns of navigation paths from an automatically created training set using results from weak page type classifiers. These robust classifiers, trained with as few as five annotated forums, can be applied to a large set of unseen forums. Test results demonstrate FoCUS's effectiveness, achieving over 98% effectiveness and 97% coverage across forums using over 150 different software packages.