Better website crawling | Voters

Better website crawling

in progress

Laurent Breillat

When working on a large website with hundreds of pages, some improvements could help a lot.
I can imagine uploading a sitemap with all pages we wanna use (for example exporting all posts from Wordpress but not pages).
Or straight up Worpdress plugin ?

April 5, 2023

Support Login

The ability to crawl Zoho Desk Knowledge Base would be great. It loads dynamically, so the current crawler does not find the pages.

Kim Alex

Try out the crawling at Chat Data: https://www.chat-data.com/

It can crawl even if your sitemap has 3000-10000 website links . There is no limit to the number of URLs as long as your chatbot is under the capacity.

Gary

Basic crawling of a Zendesk knowledge base does not work at all. It only retrieves the page url that is submitted. Please fix this as well.

Karim Wael

marked this post as

in progress

Yasser Elsaid

marked this post as

complete

Pawel Kazakow

Thank you for implementing this feature. It would be great to see the status of the crawling and view the text extracted from each URL.

Yasser Elsaid

Just implemented the sitemap feature! Also will be increasing the time limit much higher for crawling

Chris

Yasser Elsaid: i would like suggest to add a function that stores the url to the "sitemap.xml" and rescans the "sitemap.xml" regulary (as an advanced option to crawling it only once to add the urls to the data source > website)

RCO

Agreed