There are lots of online sitemap creators on the web.
can anyone describe how to work them?
can we create a crawler like them?
how to work online sitemap creators?
What are Sitemaps?
Sitemaps are an easy way for webmasters to inform search engines about pages on their sites that are available for crawling. In its simplest form, a Sitemap is an XML file that lists URLs for a site along with additional metadata about each URL (when it was last updated, how often it usually changes, and how important it is, relative to other URLs in the site) so that search engines can more intelligently crawl the site.
Web crawlers usually discover pages from links within the site and from other sites. Sitemaps supplement this data to allow crawlers that support Sitemaps to pick up all URLs in the Sitemap and learn about those URLs using the associated metadata. Using the Sitemap protocol does not guarantee that web pages are included in search engines, but provides hints for web crawlers to do a better job of crawling your site.
You need to read the full description including specification about Sitemap standard in http://www.sitemaps.org/.
You can easily write a crawler for this. See article: How to make a sitemap of any website using a crawler.
Web crawlers usually discover pages from links within the site and from other sites. Sitemaps supplement this data to allow crawlers that support Sitemaps to pick up all URLs in the Sitemap and learn about those URLs using the associated metadata. Using the Sitemap protocol does not guarantee that web pages are included in search engines, but provides hints for web crawlers to do a better job of crawling your site.
You need to read the full description including specification about Sitemap standard in http://www.sitemaps.org/.
You can easily write a crawler for this. See article: How to make a sitemap of any website using a crawler.