SRL Archive - OSR Help

I'm making a database of all tradable items on the Grand Exchange.
Does anyone have a CSV of all the tradable items, with or without their Jagex inventory number? The inventory number is the last thing in the URL when you access the Grand Exchange page for that item.

There should be about 3153 items in your list.

If anyone can provide me with such a list, I'll make sure you are well compensated.

JIM

make a list of stuff you CAN'T trade on the GE…

From: brain
make a list of stuff you can't trade on the ge…

huh?

Make a list of the stuff that you can't sell or buy on the Grand exchange. That way you exclude their ID numbers. That way it's faster and shorter.

I think he means gather a list of items you cant trade in the GE and subtract them from the total item list?

I don't know what you're making the list for….I was just saying it would be a shorter list if you made a list of stuff that you can't trade on the GE.

Just made this little thing:

program GEItemGatherByTad;

const
  FromID = 0;
  ToID = 25000;

procedure TrashHTML(var StuffToTrash: String);
begin
  while((pos('<',StuffToTrash) <> 0) and (pos('>',StuffToTrash) <> 0)) do
    delete(StuffToTrash,pos('<',StuffToTrash),pos('>',StuffToTrash));
end;

procedure grabprices;
var s:string;
    i : integer;
begin
  for i:=FromID to ToID do
  begin
    s := between('<div class="brown_box main_ge_page vertically_spaced">','</div>',
                 getpage('http://itemdb-rs.runescape.com/viewitem.ws?obj='+inttostr(i)));
    TrashHTML(s);
    s := copy(s,2,length(s));
    if(s <> '') then
      addtoreport(inttostr(i)+': '+s);
  end;
end;

begin
  ClearReport;
  grabprices;
end.

Currently running it (Does about 1 ID per second, so if there's like 20,000 items in RS, it'll be about 5 hours). Anyone know how many items there actually are?

EDIT: This is what the output looks like:

2: Cannonball

6: Cannon base

8: Cannon stand

10: Cannon barrels

12: Cannon furnace

36: Candle

39: Bronze arrowtips

40: Iron arrowtips

41: Steel arrowtips

42: Mithril arrowtips

Let me know if that's OK before leaving it go for 5 hours :p

http://pastebin.com/d15e959a3

Have fun.

From: Timer
http://pastebin.com/d15e959a3

Have fun.

That's missing 0-50?

In any case, here is 0-50, I'll let the script keep running though (It's only at 1600, about 3.5 hours to go)

2: Cannonball

6: Cannon base

8: Cannon stand

10: Cannon barrels

12: Cannon furnace

36: Candle

39: Bronze arrowtips

40: Iron arrowtips

41: Steel arrowtips

42: Mithril arrowtips

43: Adamant arrowtips

44: Rune arrowtips

45: Opal bolt tips

46: Pearl bolt tips

47: Barb bolttips

48: Longbow (u)

EDIT: Also missing 9980

9980: Roast bird meat 

Rofl. I got my output from a script I made and ran for 10 minutes.
I staarted at 50, oops. :p

From: Timer
Rofl. I got my output from a script I made and ran for 10 minutes.
I staarted at 50, oops. :p

How'd you get it to go through the GE so fast? Or are you doing it some other way?

Anyone know how many items there actually are?

By my count, there are 3153 tradable items on the Grand Exchange.

I'm trying to pull the prices with a PHP script but it's only grabbing one price every 30 seconds. I think Jagex has put in some sort of limit on how many times per minute you can access that site, maybe they are afraid of D.O.S. attacks from hackers?

Thanks anyway.

JIM

From: jimthesoundman
By my count, there are 3153 tradable items on the Grand Exchange.

I'm trying to pull the prices with a PHP script but it's only grabbing one price every 30 seconds. I think Jagex has put in some sort of limit on how many times per minute you can access that site, maybe they are afraid of D.O.S. attacks from hackers?

Thanks anyway.

JIM

Timer's list has 2891, I think. My Scar script should finish in a couple hours, though I'm going to be going to sleep so I'll see what it comes up with in the morning.

By how many items I mean how many are there in RS? I want to know how many IDs I should let it run for. As for Jagex limiting it, I'm not positive, but that doesn't seem to be the case, at least with my Scar script, since it is getting around 1 ID per second done (it opens each ID's page and checks whether it's valid or not).

From: Tad
Timer's list has 2891, I think. My Scar script should finish in a couple hours, though I'm going to be going to sleep so I'll see what it comes up with in the morning.

By how many items I mean how many are there in RS? I want to know how many IDs I should let it run for. As for Jagex limiting it, I'm not positive, but that doesn't seem to be the case, at least with my Scar script, since it is getting around 1 ID per second done (it opens each ID's page and checks whether it's valid or not).

The staff of light is the newest item (I think), and it's ID is 15486 in the GE DB, so maybe stop it at 16000? Just in case..

Well I stayed up :p

Here it is, 3133 items

http://www.pastebin.com/m41907f07

Are you positive 20 are missing? :\ I did get 35 "connection closed gracefully" errors, so I guess that caused it to miss some. I'll be running it again overnight and have it debug which ones it misses.

When I made a script like that I got up to 15k items.. :/

From: Zyt3x
When I made a script like that I got up to 15k items.. :/

Then you did something wrong.. :p

From: i luffs yeww
Then you did something wrong.. :p

Lol definitely :X

From: Zyt3x
When I made a script like that I got up to 15k items.. :/

There's only 15.5k items in Runescape :p

I did it again, this time manually rechecking the ones with failed connections, and I only got 3148…

http://www.pastebin.com/m2b975034

http://pastebin.com/me1ed2fd
This is list from 2008 though.

But isn't it easier just to parse all of these pages? That way you can be sure you got all items.

That's what I used to do, but it seems I took my script down when it became outdated which is most peculiar of I. Maybe a rewrite is in order!

Just check if you get a con error > retry for max. 5 times

Here I present you with a CSV of what I believe is all GE items and nearly all prices (a few items failed due to my silly connection). It's zip'd as the silly forum doesn't allow large txt files :(

Or you can view a pastebin of the data here though it has extra lines for awesomeness.

From: mixster
Here I present you with a CSV of what I believe is all GE items and nearly all prices (a few items failed due to my silly connection). It's zip'd as the silly forum doesn't allow large txt files :(

Or you can view a pastebin of the data here though it has extra lines for awesomeness.

Judging from Jim's number, appears to be missing 205 items :\

If he's basing his number on all RS items, then it would be a bit high due to non-GE'able items and the like.

From: mixster
If he's basing his number on all RS items, then it would be a bit high due to non-GE'able items and the like.

Well he says there are 3153 items on the GE (@Jim: Where'd you get that number?). My extremely slow gatherer managed to grab 3148 items. Yours and Timer's seem to have about 2900 (unless I counted incorrectly). In all, there are nearly 16,000 items in RS (quite surprising; only 20% of items are tradeable?).

EDIT: I recall there being some items that are tradeable yet can't be put on the GE (some trees/plants from the Fally Garden shop, for instance). Does 3153 include those, Jim?

Where'd you get the 16,000 items..? >_> That wouldn't make any sense.

From: i luffs yeww
Where'd you get the 16,000 items..? >_> That wouldn't make any sense.

Wouldn't each ID represent an item? Or can NPCs, objects, etc not have the same ID, so 16,000 would be the aggregate of that? (I got ~16,000 from the most recent GE item, which has an ID of 15490)

Either way, doesn't really matter

EDIT: According to mormonman's post below, there are in fact 16,000 items, 80% of which are untradeable.

List of all items(I do believe). stolen from method's pastebin
http://uppit.com/v/V53ZSFJP

Yeah, noted. :p The reason the GE DB doesn't go in order (and they usually go in twos) is because items that are noted are a different ID than the item on the ground, or the item in the inventory. But I don't think that just because it's noted it's a different item..

(Technically it is, but I think that's stupid. :p)

And mormon, that list hasn't been updated for a few months.. :p

From: i luffs yeww
Yeah, noted. :p The reason the GE DB doesn't go in order (and they usually go in twos) is because items that are noted are a different ID than the item on the ground, or the item in the inventory. But I don't think that just because it's noted it's a different item..

(Technically it is, but I think that's stupid. :p)

And mormon, that list hasn't been updated for a few months.. :p

Only difference is noted -> unnoted (ground items, etc. are all the same as their corresponding inventory item). But yes, you're right, from a player's perspective there's only about 10,000.

As for the list, It's only missing about 50. Mormonman: Any idea how Method created that? Would be handy to just have a database if ref ever comes back.

Anyone know if my old script on Jim's other thread still works?

From: Tad
Only difference is noted -> unnoted (ground items, etc. are all the same as their corresponding inventory item). But yes, you're right, from a player's perspective there's only about 10,000.

As for the list, It's only missing about 50. Mormonman: Any idea how Method created that? Would be handy to just have a database if ref ever comes back.

I printed out the information when the client loaded all of the item definitions during the loading process.

I'm looking into the differences between Methods list and my own to see what is missing and the like - it may be that my silly connection that led to some items having no price was also responsible for some pages being missed or the like. If something comes up, I'll edit with details.

New item list for you guys: http://uppit.com/v/YIQPBXNW

From: mixster
I'm looking into the differences between Methods list and my own to see what is missing and the like - it may be that my silly connection that led to some items having no price was also responsible for some pages being missed or the like. If something comes up, I'll edit with details.

Method's list includes items that aren't on the G.E.

Indeed, but by eliminating general things like noted versions then looking for things that look like they would be on the GE and checking that out should narrow it down a lot.

http://schumaker40.site90.com/Auth.scar

That page contains a list of items from the GE a few months ago.

Got them using this:

program New;
var
  i : Integer;
  C : String;
begin
  For I := 0 To 25000 Do
  Begin
    C := between('<div class="subsectionHeader">' + chr(10), chr(10) + '</div>' + chr(10) + '<div id="item_additional" class="inner_brown_box">', GetPage('http://itemdb-rs.runescape.com/Feather/viewitem.ws?obj=' + IntToStr(I)));
    If(C <> '') then
    begin
      WriteLn(C)
      Writeini('item', Inttostr(I),  IntToStr(I) + ' - ' + C, ScriptPath + 'ItemIDS.ini');
    End;
  end;
end.

It seems to be pretty slow now, as it took about 30 seconds to get me the first three or four items, but it got about one every second when I last used it. Possibly just my internet connection, though.

~Sand

Looks like I was right about the slow connection causing a jump - it missed out a few pages under 's' - everything from Spined boots upto Talon beast pouch, so I'm rerunning now with less stuff going on to hopefully make it not jump and I'll attach the new list afterwards.

Edit: As you only asked for id's and names, I've removed the prices since it made it a lot easier to do reliably. 3149 items now, yay.

what are you going to use this for?