SRL Archive - Computer Help and Tutorials

Wget is win, we all know that. One of the most simple looking, but actually most advanced terminal client.

So, you want to download an entire site's content, eh? Using wget it is easy!

Now you must install wget (it is installed on 99% of Linuxes by default)

Now you have `wget http://some.cool.web.site/

But oh noes, they have a robots.txt file to disallow you to index some parts of their website. No problem, ignore that stupid text file. [Note: some sites have this file to prevent abuse, or waste of their bandwidth; use it wisely]
`wget -e robots=off http://some.cool.web.site/

You trying to ignore the directories above your current one? Append -np :)
`wget -np -e robots=off http://some.cool.web.site/dicrecory/I/want/

Your downloads always timing out, and leaving incomplete files? Append -c.
`wget -np -c -e robots=off http://some.cool.web.site/file.that.likes.to.timeout.sh

You dislike how wget dumps all the files into one folder? Want them in recursive folders, such as how the server has it? -r is your friend.
`wget -r -np -c -e robots=off http://some.cool.web.site/oh/noes/so/many/folders/on/this/website/

Tired of websites blocking the wget useragent? (websites usually do this to prevent abuse and kiddies from spidering their site with wget that abuses their bandwidth) -U is fun to mess around with.
`wget -r -np -c -U "Mozilla/5.0 (X11; U; Linux i686; nl; rv:1.7.3) Gecko/20040916" -e robots=off http://some.cool.web.site/admin/who/dislikes/wget/
Be careful when using this command; spoofed Useragents can be dangerous if you use a blacklisted one, the admin may IP ban you. I recommend using the Useragent from your Firefox.

Final result!
`wget -r -np -c -U "Mozilla/5.0 (X11; U; Linux i686; nl; rv:1.7.3) Gecko/20040916" -e robots=off http://some.cool.web.site/


Some more features, not used enough by me to care.
`wget -help



Have fun leeching, and messing around with the power of wget.
~ Harry

So, theoretically, you could make your useragent the googlebot and download hidden sections?

Member and junior member scripts aren't indexed though o_O.

From: R0b0t1
So, theoretically, you could make your useragent the googlebot and download hidden sections?

Sush, brah. Don't expose my secrets.

@Da Owner: How do you think Google ads work in the Members section?

Well, the pages or content in there isn't contained within the Google index as far as I've seen.

Point me to a Google search where it shows a member page.

From: Da 0wner
Well, the pages or content in there isn't contained within the Google index as far as I've seen.

Point me to a Google search where it shows a member page.

They do not index it, as it is not allowed in the robots.txt, but they are still able to view it, but not cache it, so they can show ads.

Oh, well how can this be used to acquire member scripts? If you'd like you can MSN me it :p.

The goal of this thread is not how to obtain higher level scripts :3
Also, vBulletin patched it I think.. it only allows Google's IP.

Well I don't want scripts. I don't even macro. I just want to know how for educational purpose :D.

Na, Google can't view the members sections. Noone can :)
The robots.txt also can NOT disallow google to view members sections.♠

I told you Harry :).

I'm going to be in the car for like 10+ hours, is it okay if I download this forum to read scripting tutorials? There are so many and I'm not sure which ones I will want to read… Just double checking that I'm not going to get IP banned or something. Can anyone shed light on this?

From: Versace
I'm going to be in the car for like 10+ hours, is it okay if I download this forum to read scripting tutorials? There are so many and I'm not sure which ones I will want to read… Just double checking that I'm not going to get IP banned or something. Can anyone shed light on this?

i dont know if it will work, but going file -> work offline might do the trick