ba3
2010-06-05 21:31:25
I have recently been trying to merchant on the GE and grew frustrated at the lack of ability to actually get past data on items or at the lack of ability to compare items readily to each other on site.
The only way to improve this would be to somehow create a "dump" of the GE database everyday, so you would have the data yourself in entirety.
Of course, currently, there were no programs to do that. Even the huge fansites relied on people to manually update prices for them, and some of their prices were over a week old. Thats not good enough if you need info now to decide to buy or sell a product when your whole RS net worth is invested in it.
So, Because I was so suprised that a ge data dumper didn't exist, I wrote one myself. The program is not going to be released however. I will release daily CSV dumps of the entire GE database on my site, which I won't mention in this post at this time.
Now, on to the program. It basically, the first time you run it, or when you rebuild your item id database, bruteforces through item ids 1-20000 looking for a an item on the ge. If saves that list and can use it over and over before needing that part updated again.
Then, it just scrapes data from the GE about the item numbers that you have found earlier to be good, reformat them (fixes 122.3k into real numbers for graphind such as 122300).
Here are two screenshots and a sample beginning of a database dump.
I want to know though, do any of you think people would actually want this dump data? Would you use it?
2,Cannonball,Ammo for the Dwarf Cannon.,313,330,347,-7.0%,-14.9%,-0.9%
6,Cannon base,The cannon is built on this.,120800,127100,133500,+7.6%,+6.4%,-1.4%
8,Cannon stand,The mounting for the multicannon.,122800,129300,135700,+8.5%,+12.5%,-1.4%
10,Cannon barrels,The barrels of the multicannon.,122300,128700,135200,+8.6%,+9.3%,-5.4%
12,Cannon furnace,This powers the multicannon.,122500,129000,135400,+6.2%,+11.5%,-1.6%
36,Candle,A candle.,172,181,190,+5.8%,+12.4%,+0.5%
39,Bronze arrowtips,I can make some arrows with these.,9,10,10,+0%,+0%,+0%
40,Iron arrowtips,I can make some arrows with these.,17,18,19,-5.2%,-5.2%,+5.8%
41,Steel arrowtips,I can make some arrows with these.,34,36,36,+0%,+0%,+2.8%
42,Mithril arrowtips,I can make some arrows with these.,64,67,68,+4.6%,+1.5%,+17.5%
43,Adamant arrowtips,I can make some arrows with these.,152,160,160,+0%,+0%,+1.2%
44,Rune arrowtips,I can make some arrows with these.,437,460,460,+0%,+0%,+0%
45,Opal bolt tips,Opal Bolts,1,1,2,+0%,+0%,+0%
46,Pearl bolt tips,Pearl Bolt Tips,23,24,25,+41.1%,+14.2%,+242.8%
47,Barb bolttips,I can make bolts with these.,38,40,42,+8.1%,+14.2%,+0%
48,Longbow (u),I need to find a string for this.,12,12,13,+0%,+0%,+0%
50,Shortbow (u),I need to find a string for this.,4,4,5,+0%,+0%,-20.0%
52,Arrow shaft,A wooden arrow shaft.,4,5,6,+0%,+66.6%,+25.0%
53,Headless arrow,A wooden arrow shaft with flights attached.,20,21,22,-27.5%,+10.5%,-19.2%
54,Oak shortbow (u),An unstrung oak shortbow; I need a bowstring for this.,10,10,11,+0%,+0%,+0%
56,Oak longbow (u),An unstrung oak longbow; I need a bowstring for this.,16,16,17,+0%,+0%,+0%
58,Willow longbow (u),An unstrung willow longbow; I need a bowstring for this.,32,32,34,+0%,+0%,+0%
60,Willow shortbow (u),An unstrung willow shortbow; I need a bowstring for this.,20,20,21,+0%,+0%,+0%
62,Maple longbow (u),An unstrung maple longbow; I need a bowstring for this.,88,93,98,+12.0%,-7.0%,+27.3%
64,Maple shortbow (u),An unstrung maple shortbow; I need a bowstring for this.,47,50,53,+25.0%,+25.0%,+25.0%
66,Yew longbow (u),An unstrung yew longbow; I need a bowstring for this.,408,430,452,+0.2%,+0.2%,-2.0%
68,Yew shortbow (u),An unstrung yew shortbow; I need a bowstring for this.,164,173,182,-4.9%,-14.3%,+16.1%
70,Magic longbow (u),An unstrung magic longbow; I need a bowstring for this.,1175,1237,1299,-1.4%,-3.8%,-1.9%
72,Magic shortbow (u),An unstrung magic shortbow; I need a bowstring for this.,708,745,782,-2.8%,-11.3%,-12.0%
91,Guam potion (unf),I need another ingredient to finish this guam potion.,706,743,780,-16.9%,-7.2%,-0.9%
93,Marrentill potion (unf),I need another ingredient to finish this marrentill potion.,246,259,272,+96.2%,+87.6%,+58.8%
95,Tarromin potion (unf),I need another ingredient to finish this tarromin potion.,1012,1065,1118,+4.6%,-11.1%,+22.1%
97,Harralander potion (unf),I need another ingredient to finish this harralander potion.,997,1050,1103,-3.1%,-33.5%,-24.6%
99,Ranarr potion (unf),I need another ingredient to finish this ranarr potion.,8275,8711,9147,-1.3%,-28.5%,-23.8%
The only way to improve this would be to somehow create a "dump" of the GE database everyday, so you would have the data yourself in entirety.
Of course, currently, there were no programs to do that. Even the huge fansites relied on people to manually update prices for them, and some of their prices were over a week old. Thats not good enough if you need info now to decide to buy or sell a product when your whole RS net worth is invested in it.
So, Because I was so suprised that a ge data dumper didn't exist, I wrote one myself. The program is not going to be released however. I will release daily CSV dumps of the entire GE database on my site, which I won't mention in this post at this time.
Now, on to the program. It basically, the first time you run it, or when you rebuild your item id database, bruteforces through item ids 1-20000 looking for a an item on the ge. If saves that list and can use it over and over before needing that part updated again.
Then, it just scrapes data from the GE about the item numbers that you have found earlier to be good, reformat them (fixes 122.3k into real numbers for graphind such as 122300).
Here are two screenshots and a sample beginning of a database dump.
I want to know though, do any of you think people would actually want this dump data? Would you use it?
2,Cannonball,Ammo for the Dwarf Cannon.,313,330,347,-7.0%,-14.9%,-0.9%
6,Cannon base,The cannon is built on this.,120800,127100,133500,+7.6%,+6.4%,-1.4%
8,Cannon stand,The mounting for the multicannon.,122800,129300,135700,+8.5%,+12.5%,-1.4%
10,Cannon barrels,The barrels of the multicannon.,122300,128700,135200,+8.6%,+9.3%,-5.4%
12,Cannon furnace,This powers the multicannon.,122500,129000,135400,+6.2%,+11.5%,-1.6%
36,Candle,A candle.,172,181,190,+5.8%,+12.4%,+0.5%
39,Bronze arrowtips,I can make some arrows with these.,9,10,10,+0%,+0%,+0%
40,Iron arrowtips,I can make some arrows with these.,17,18,19,-5.2%,-5.2%,+5.8%
41,Steel arrowtips,I can make some arrows with these.,34,36,36,+0%,+0%,+2.8%
42,Mithril arrowtips,I can make some arrows with these.,64,67,68,+4.6%,+1.5%,+17.5%
43,Adamant arrowtips,I can make some arrows with these.,152,160,160,+0%,+0%,+1.2%
44,Rune arrowtips,I can make some arrows with these.,437,460,460,+0%,+0%,+0%
45,Opal bolt tips,Opal Bolts,1,1,2,+0%,+0%,+0%
46,Pearl bolt tips,Pearl Bolt Tips,23,24,25,+41.1%,+14.2%,+242.8%
47,Barb bolttips,I can make bolts with these.,38,40,42,+8.1%,+14.2%,+0%
48,Longbow (u),I need to find a string for this.,12,12,13,+0%,+0%,+0%
50,Shortbow (u),I need to find a string for this.,4,4,5,+0%,+0%,-20.0%
52,Arrow shaft,A wooden arrow shaft.,4,5,6,+0%,+66.6%,+25.0%
53,Headless arrow,A wooden arrow shaft with flights attached.,20,21,22,-27.5%,+10.5%,-19.2%
54,Oak shortbow (u),An unstrung oak shortbow; I need a bowstring for this.,10,10,11,+0%,+0%,+0%
56,Oak longbow (u),An unstrung oak longbow; I need a bowstring for this.,16,16,17,+0%,+0%,+0%
58,Willow longbow (u),An unstrung willow longbow; I need a bowstring for this.,32,32,34,+0%,+0%,+0%
60,Willow shortbow (u),An unstrung willow shortbow; I need a bowstring for this.,20,20,21,+0%,+0%,+0%
62,Maple longbow (u),An unstrung maple longbow; I need a bowstring for this.,88,93,98,+12.0%,-7.0%,+27.3%
64,Maple shortbow (u),An unstrung maple shortbow; I need a bowstring for this.,47,50,53,+25.0%,+25.0%,+25.0%
66,Yew longbow (u),An unstrung yew longbow; I need a bowstring for this.,408,430,452,+0.2%,+0.2%,-2.0%
68,Yew shortbow (u),An unstrung yew shortbow; I need a bowstring for this.,164,173,182,-4.9%,-14.3%,+16.1%
70,Magic longbow (u),An unstrung magic longbow; I need a bowstring for this.,1175,1237,1299,-1.4%,-3.8%,-1.9%
72,Magic shortbow (u),An unstrung magic shortbow; I need a bowstring for this.,708,745,782,-2.8%,-11.3%,-12.0%
91,Guam potion (unf),I need another ingredient to finish this guam potion.,706,743,780,-16.9%,-7.2%,-0.9%
93,Marrentill potion (unf),I need another ingredient to finish this marrentill potion.,246,259,272,+96.2%,+87.6%,+58.8%
95,Tarromin potion (unf),I need another ingredient to finish this tarromin potion.,1012,1065,1118,+4.6%,-11.1%,+22.1%
97,Harralander potion (unf),I need another ingredient to finish this harralander potion.,997,1050,1103,-3.1%,-33.5%,-24.6%
99,Ranarr potion (unf),I need another ingredient to finish this ranarr potion.,8275,8711,9147,-1.3%,-28.5%,-23.8%