Anu Pidaparti commented on Nov 7, 2014

Re: Crawling data for the first time

Does the rel=next link always return the same response? In other words, if we have million content items that need to be retrieved, and if there is some failure after we process few pages, what can we use to resume crawling from where it stopped rather than starting all over?

Dimitrios Arethas commented on Oct 1, 2014

Re: Crawling data for the first time

What about sub-communities?