At a set frequency the crawler should rescan the target website for new content to add to its knowledgebase.