Web Hosting Geeks | Web Hosting Experts
Need help? Call us 1(866)TOP-HOST

14318 reviews 1707 hosting providers

Questions & Answers

linux command for website scan

Mar 8, 2013 by Jake
Hi all
I am looking for a linux script or command that scans a website and displays the number of  webpages in a website. Does any one know any such scripts? Your help is highly appreciated

2 Answers

0 votes
Mar 8, 2013 by SmartHost
Try to pull a robots.txt file or sitemap.xml file then  use a sitemap generator to generate a link list.
0 votes
Mar 8, 2013 by FixHost
There are lots of web based tools that generate  xml sitemaps.