linux command for website scan

36 views
asked Mar 8, 2013 in General by Jake (580 points)
Hi all
I am looking for a linux script or command that scans a website and displays the number of  webpages in a website. Does any one know any such scripts? Your help is highly appreciated
Thanks

2 Answers

0 like 0 dislike
answered Mar 8, 2013 by SmartHost (9,060 points)
Try to pull a robots.txt file or sitemap.xml file then  use a sitemap generator to generate a link list.
0 like 0 dislike
answered Mar 8, 2013 by FixHost (12,380 points)
There are lots of web based tools that generate  xml sitemaps.
...