Better website crawling | Voters

Better website crawling

in progress

Laurent Breillat

When working on a large website with hundreds of pages, some improvements could help a lot.
I can imagine uploading a sitemap with all pages we wanna use (for example exporting all posts from Wordpress but not pages).
Or straight up Worpdress plugin ?

April 5, 2023

Gary

Basic crawling of a Zendesk knowledge base does not work at all. It only retrieves the page url that is submitted. Please fix this as well.

Karim Wael

marked this post as

in progress

Yasser Elsaid

marked this post as

complete

Pawel Kazakow

Thank you for implementing this feature. It would be great to see the status of the crawling and view the text extracted from each URL.

Yasser Elsaid

Just implemented the sitemap feature! Also will be increasing the time limit much higher for crawling

Chris

Yasser Elsaid: i would like suggest to add a function that stores the url to the "sitemap.xml" and rescans the "sitemap.xml" regulary (as an advanced option to crawling it only once to add the urls to the data source > website)

RCO

Agreed