lardmaster
2011-09-18 06:38:02
Hello All,
If any of you happen to remember me, I am back (again)!
This time, I bring you, my all new, hastily-hacked-together Runescape StockExtractor!
You might ask. Well, to put it simply, it extracts the price history of items from the Runescape Grand Exchange website, and saves them in a handy comma delimited file on your hard drive.
You might ask again. If you are asking this question, the answer is, probably not much, at least for now. However, in due time, this could allow a system which *automatically* analyzes, tracks, and predicts the value of Runescape commodities.
The project is a QT project, and if you don't have the necessary libraries/ compilers installed, you probably should wait until this is more than just a proof of concept before trying to run it.
If you do, however, and you compile the program here's a basic tutorial on how it works:
There are two basic functionalities bundled into it.
One is a test extractor: If you click on all three Load… buttons, and select the appropriate image files on your hard drive (downloaded from a page such as this, It will load in the images. Then, if you click the "Output" button, it will ask you where to save the output, and it will save the extracted price history to a given output. This is generally useless except for debugging.
The other is the page scraper: If you click the Crawl button, it will begin scraping the runescape website for *all* items price history for the past 120 days. By default it will output to the directory "dump" in your working directory. You can choose a different output directory with the "Set Dump Dir" button.
The output format of the scraper is as follows:
For each item, a directory is created with its name.
Inside each directory a file is placed with the current timestamp. This is the pricing history. The format of the file is
timestamp,price
timestamp,price
timestamp,price
the timestamps are in 64 bit millisecond format.
Also inside each directory is placed a file "id", which contains the numerical id number the runescape website uses to identify the item.
Yes, I know, the GUI is hideous. Oh well, I'll work on it later.
There are still sometimes problems with extracting the prices. If there is an error, the problematic item will just be skipped. You can read the command line output to see when an error occurs.
Here is the source, I used qtcreator to make it, so it might be easy to use that to open it up. I also included an example dump of the letter A and partway through the letter B, just to demonstrate it does work.
http://www.mediafire.com/?0cvskwb2dquvxfg
Because runescape is continually losing history older than 180 days, it might be a smart idea to start dumping now, and figure out what to do with the dump later.
Obviously this is just the first step. Other programs should be able to interface with this *very* easily, as the data format is really simple. Market analyzers are the eventual goal. Oh yeah, and I release this under the GPL license, so feel free to contribute. This program was just hacked together in a single night, so I still plan on doing some sort of more coherent release later.
Thank you for your time!
Lardmaster
If any of you happen to remember me, I am back (again)!
This time, I bring you, my all new, hastily-hacked-together Runescape StockExtractor!
What does it do?
You might ask. Well, to put it simply, it extracts the price history of items from the Runescape Grand Exchange website, and saves them in a handy comma delimited file on your hard drive.
Yeah, but what does it really do for *me*?
You might ask again. If you are asking this question, the answer is, probably not much, at least for now. However, in due time, this could allow a system which *automatically* analyzes, tracks, and predicts the value of Runescape commodities.
The project is a QT project, and if you don't have the necessary libraries/ compilers installed, you probably should wait until this is more than just a proof of concept before trying to run it.
If you do, however, and you compile the program here's a basic tutorial on how it works:
There are two basic functionalities bundled into it.
One is a test extractor: If you click on all three Load… buttons, and select the appropriate image files on your hard drive (downloaded from a page such as this, It will load in the images. Then, if you click the "Output" button, it will ask you where to save the output, and it will save the extracted price history to a given output. This is generally useless except for debugging.
The other is the page scraper: If you click the Crawl button, it will begin scraping the runescape website for *all* items price history for the past 120 days. By default it will output to the directory "dump" in your working directory. You can choose a different output directory with the "Set Dump Dir" button.
The output format of the scraper is as follows:
For each item, a directory is created with its name.
Inside each directory a file is placed with the current timestamp. This is the pricing history. The format of the file is
timestamp,price
timestamp,price
timestamp,price
the timestamps are in 64 bit millisecond format.
Also inside each directory is placed a file "id", which contains the numerical id number the runescape website uses to identify the item.
Yes, I know, the GUI is hideous. Oh well, I'll work on it later.
There are still sometimes problems with extracting the prices. If there is an error, the problematic item will just be skipped. You can read the command line output to see when an error occurs.
Here is the source, I used qtcreator to make it, so it might be easy to use that to open it up. I also included an example dump of the letter A and partway through the letter B, just to demonstrate it does work.
http://www.mediafire.com/?0cvskwb2dquvxfg
Because runescape is continually losing history older than 180 days, it might be a smart idea to start dumping now, and figure out what to do with the dump later.
Obviously this is just the first step. Other programs should be able to interface with this *very* easily, as the data format is really simple. Market analyzers are the eventual goal. Oh yeah, and I release this under the GPL license, so feel free to contribute. This program was just hacked together in a single night, so I still plan on doing some sort of more coherent release later.
Thank you for your time!
Lardmaster