Python Scrapy web crawler for vBulletin

This project is to build a web scraper for vBulletin forums sites using Scrapy ([login to view URL]). I have not used Scrapy yet, so I would like to understand the general capabilities it has, this is a proof of concept to learn more about it in a real world example. As part of this project I would also like an overview of what is possible with Scrapy, both from a content extraction as well as setting up one or multiple crawl processes to distribute the load across multiple servers and IP addresses. Also, I would like to understand the possibilities of making the scraper intelligent enough to not have to re-crawl the entire site to get the latest posts. For example, if we have already scraped a thread, then start on the last page working back until we reach a post that we have already scraped. I understand that this would require some type of state tracking, but I am not sure what all is possible with Scrapy. The output of the scrape should be the following: • Forum ID • Forum Name • Forum URL permalink • Forum Thread Count • Forum Posts Count • Thread ID • Thread Name • Thread Link • Thread Rating • Thread Comment Count • Thread View Count • Post ID • Post URL permalink • Post Title • Post Content • Post Date • Post Author • Post Author ID • Post Author URL • Post Author Join Date • Post Author Location • Post Author Post Count