|
Home > Archive > IIS ASP > May 2004 > trying to collect links on my site, how to crawl it?
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
| Author |
trying to collect links on my site, how to crawl it?
|
|
| Danny 2004-05-30, 11:53 am |
| I am trying to crawl my site to get a list of links. I am using the regular
expressions to get the href tags from the pages and reading the links using
xmlhttp module.
is there an efficient way to loop through the links? If you start with the
home page, how do you do it? How do yo ukeep track of the pages you have
been to and which to do next.
I am not sure how to go about tihs.
I tried some loop but it seemed to take too long to crawl my site.
Thanks again
| |
| Aaron Bertrand [MVP] 2004-05-30, 11:53 am |
| For all .asp and .html files etc., you could build the list of files to hit
using FileSystemObject... store the list in a database or text file, and
scroll through it...
--
Aaron Bertrand
SQL Server MVP
http://www.aspfaq.com/
"Danny" <dannywork5@hotmail.com> wrote in message
news:Wlctc.55736$cz5.22135337@news4.srv.hcvlny.cv.net...
>I am trying to crawl my site to get a list of links. I am using the
>regular
> expressions to get the href tags from the pages and reading the links
> using
> xmlhttp module.
>
> is there an efficient way to loop through the links? If you start with
> the
> home page, how do you do it? How do yo ukeep track of the pages you have
> been to and which to do next.
>
> I am not sure how to go about tihs.
>
> I tried some loop but it seemed to take too long to crawl my site.
>
> Thanks again
>
>
|
|
|
|
|