SRL Archive - Web Development

I'm trying to scrape Runescape's Grand Exchange database for values, etc and setup a webpage where it presents costs, etc for leveling skills with realtime prices.

So far, I figure I need to use curl.

This is what I have thus far:

<?php
    $ch = curl_init("http://services.runescape.com/m=itemdb_rs/results.ws?query=tarromin+potion&price=all&members=");
    curl_setopt($ch, CURLOPT_FILE, $fp);
    curl_setopt($ch, CURLOPT_HEADER, 0);
    $output = curl_exec($ch);
    curl_close($ch);
    echo $output;
?>

There echo was just for debugging purposes.

I have a php and mysql book I'm currently reading. I figure I'll find in the string, the location in table search_results_table, row_b, column 3 the guide price and whether it fell, pushed, or rose.

That's basically all I need for now.

Then I need to add or modify a row in my mysql dbase for that item. That part I can find in my book.

Any tips or input how to find the guide price in the string or different ways to do it?

Or you could just do:
[PHP]<?php

$url = "http://serv...rest of the url/";
$content = file_get_contents($url);
echo $content;

?>[/PHP]

EDIT: Heres an example output of something I made quickly (Googled a function to parse a table, cbf to make one myself):

Array
(
    [0] => Array
        (
            [] => 
            [Item] =>  Tarromin potion (unf)
            [Guide Price] => 450
            [Change Today (gp)] => -2
        )

    [1] => Array
        (
            [] => 
            [Item] =>  Tarromin seed
            [Guide Price] => 2
            [Change Today (gp)] => 0
        )

    [2] => Array
        (
            [] => 
            [Item] =>  Agility potion (1)
            [Guide Price] => 197
            [Change Today (gp)] => 0
        )

    [3] => Array
        (
            [] => 
            [Item] =>  Agility potion (2)
            [Guide Price] => 116
            [Change Today (gp)] => 0
        )

    [4] => Array
        (
            [] => 
            [Item] =>  Agility potion (3)
            [Guide Price] => 146
            [Change Today (gp)] => 0
        )

    [5] => Array
        (
            [] => 
            [Item] =>  Agility potion (4)
            [Guide Price] => 220
            [Change Today (gp)] => -11
        )

    [6] => Array
        (
            [] => 
            [Item] =>  Ash potion (unf)
            [Guide Price] => 113
            [Change Today (gp)] => 0
        )

    [7] => Array
        (
            [] => 
            [Item] =>  Attack potion (1)
            [Guide Price] => 132
            [Change Today (gp)] => 0
        )

    [8] => Array
        (
            [] => 
            [Item] =>  Attack potion (2)
            [Guide Price] => 212
            [Change Today (gp)] => +2
        )

    [9] => Array
        (
            [] => 
            [Item] =>  Attack potion (3)
            [Guide Price] => 217
            [Change Today (gp)] => +3
        )

    [10] => Array
        (
            [] => 
            [Item] =>  Attack potion (4)
            [Guide Price] => 294
            [Change Today (gp)] => -1
        )

    [11] => Array
        (
            [] => 
            [Item] =>  Avantoe potion (unf)
            [Guide Price] => 3,538
            [Change Today (gp)] => +133
        )

    [12] => Array
        (
            [] => 
            [Item] =>  Cadantine potion (unf)
            [Guide Price] => 1,155
            [Change Today (gp)] => 0
        )

    [13] => Array
        (
            [] => 
            [Item] =>  Combat potion (1)
            [Guide Price] => 185
            [Change Today (gp)] => 0
        )

    [14] => Array
        (
            [] => 
            [Item] =>  Combat potion (2)
            [Guide Price] => 32
            [Change Today (gp)] => 0
        )

    [15] => Array
        (
            [] => 
            [Item] =>  Combat potion (3)
            [Guide Price] => 14
            [Change Today (gp)] => 0
        )

    [16] => Array
        (
            [] => 
            [Item] =>  Combat potion (4)
            [Guide Price] => 27
            [Change Today (gp)] => -1
        )

    [17] => Array
        (
            [] => 
            [Item] =>  Crafting potion (1)
            [Guide Price] => 938
            [Change Today (gp)] => 0
        )

    [18] => Array
        (
            [] => 
            [Item] =>  Crafting potion (2)
            [Guide Price] => 1,154
            [Change Today (gp)] => 0
        )

    [19] => Array
        (
            [] => 
            [Item] =>  Crafting potion (3)
            [Guide Price] => 2,026
            [Change Today (gp)] => 0
        )

)

Source:
[PHP]<?php

function parse_table($html) {
preg_match("/<table.*?>.*?<\/[\s]*table>/s", $html, $table_html);
preg_match_all("/<th.*?>(.*?)<\/[\s]*th>/", $table_html[0], $matches);
$row_headers = $matches[1];
preg_match_all("/<tr.*?>(.*?)<\/[\s]*tr>/s", $table_html[0], $matches);
$table = array();
foreach($matches[1] as $row_html) {
preg_match_all("/<td.*?>(.*?)<\/[\s]*td>/", $row_html, $td_matches);
$row = array();
for($i=0; $i<count($td_matches[1]); $i++) {
$td = strip_tags(html_entity_decode($td_matches[1][$i]));
$row[$row_headers[$i]] = $td;
}
if(count($row) > 0)
$table[] = $row;
}
return $table;
}

$content = file_get_contents("http://www.frement.net/srl/t.html");

print_r(parse_table($content));

?>[/PHP]

EDIT2: Modified source to get the table from the site:
[PHP]<?php

function str_between($start, $end, $string){
$string = " ".$string;
$ini = strpos($string, $start);
if ($ini == 0) {
return "";
}
$ini += strlen($start);
$len = strpos($string, $end, $ini) - $ini;
return substr($string, $ini, $len);
}

function parse_table($html) {
preg_match("/<table.*?>.*?<\/[\s]*table>/s", $html, $table_html);
preg_match_all("/<th.*?>(.*?)<\/[\s]*th>/", $table_html[0], $matches);
$row_headers = $matches[1];
preg_match_all("/<tr.*?>(.*?)<\/[\s]*tr>/s", $table_html[0], $matches);
$table = array();
foreach($matches[1] as $row_html) {
preg_match_all("/<td.*?>(.*?)<\/[\s]*td>/", $row_html, $td_matches);
$row = array();
for($i=0; $i<count($td_matches[1]); $i++) {
$td = strip_tags(html_entity_decode($td_matches[1][$i]));
$row[$row_headers[$i]] = $td;
}
if(count($row) > 0)
$table[] = $row;
}
return $table;
}

$content = file_get_contents("http://services.runescape.com/m=itemdb_rs/results.ws?query=tarromin+potion&price=all&members=");
$content = str_between('<td class="navright">', '<div class="results_nav">', $content);
print_r(parse_table($content));

?>[/PHP]

Ooooh thanks.

I have to admit, i was staring at that parse table code for a while wondering wtf was going on.

I'll have to weed out some of the stuff in that function (and only return the price + up/down movement and then exit) that was for debugging purpose and add some failsafes in case it doesn't find the text, but i appreciate the help!

Also, curl was like 3x faster; try it :).

[php]
<?php
function str_between($start, $end, $string){
$string = " ".$string;
$ini = strpos($string, $start);
if ($ini == 0) {
return "";
}
$ini += strlen($start);
$len = strpos($string, $end, $ini) - $ini;
return substr($string, $ini, $len);
}

function contains($string, $substring) {
$pos = strpos($string, $substring);
if($pos === false) {
return false;
}
else {
return true;
}
}

function parse_table($html) {
preg_match("/<table.*?>.*?<\/[\s]*table>/s", $html, $table_html);
preg_match_all("/<th.*?>(.*?)<\/[\s]*th>/", $table_html[0], $matches);
$row_headers = $matches[1];
preg_match_all("/<tr.*?>(.*?)<\/[\s]*tr>/s", $table_html[0], $matches);
$table = array();
foreach($matches[1] as $row_html) {
preg_match_all("/<td.*?>(.*?)<\/[\s]*td>/", $row_html, $td_matches);
$row = array();
for($i=0; $i<count($td_matches[1]); $i++) {
$td = strip_tags(html_entity_decode($td_matches[1][$i]));
if(contains($td, "Tarromin pot")) {
$price = strip_tags(html_entity_decode($td_matches[1][$i + 1]));
$movement = strip_tags(html_entity_decode($td_matches[1][$i + 2]));
return array($price, $movement);
}
$row[$row_headers[$i]] = $td;
}
if(count($row) > 0)
$table[] = $row;
}
return false;
}

$ch = curl_init("http://services.runescape.com/m=itemdb_rs/results.ws?query=tarromin+potion&price=all&members=");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$contents = curl_exec ($ch);
curl_close ($ch);
$contents = str_between('<td class="navright">', '<div class="results_nav">', $contents);
print_r(parse_table($contents));
?>
[/php]

I didn't notice much difference. Anyway, glad you got it working.

I'd suggest you use some sort of timeout on the page request, so it won't hang and 503 after ~30 seconds if RS goes down or the transfer attempt fails etc.

Don't remember how to do on php-curl (or if you even are using it and I just don't realize it :p ), but if you're using file_get_contents:

$context = stream_context_create(array('http' => array('timeout'=>1)));
@$page = file_get_contents("http://website.etc/page", 0, $context);

This would timeout after 1 second ^.