SRL Archive - Web Development

Is This Possible?


scuz 10

2011-04-01 04:32:38

I'm working on a project where I need to get data from different parts of a website. Since the website isn't made with html, i can't edit the url's to get to different parts of the website to get everything i need.

Is there an automated way to get to different parts of a website where your browser url does not change when you click on the links? Is it possible to make a php script or something like that that automatically visits the different parts of the website and parses them?

Thanks..

nielsie95

2011-04-01 04:37:26

What's the website? Is it made using flash? If not, it probably use a frame to wrap the website. If that's the case then yes, it is possible. I'm sure there are a few website crawlers for PHP out there. If you want to create one yourself, you might want to look into a solution to get all the links for you.

scuz 10

2011-04-01 04:42:22

I do not know much about web languages, but from the little I do know, i think it's made using javascript.

nielsie95

2011-04-01 05:07:31

That one's indeed pretty annoying. It does use forms, so you could try to submit to that url with some values, but it would probably take some time to see how it works. I'm not sure what other solutions there are, but I'm sure there are. (It would be easier to parse http://courses.ucsd.edu/, but I'm not sure if it contains the same info?).

scuz 10

2011-04-01 05:13:59

From: nielsie95
That one's indeed pretty annoying. It does use forms, so you could try to submit to that url with some values, but it would probably take some time to see how it works. I'm not sure what other solutions there are, but I'm sure there are. (It would be easier to parse http://courses.ucsd.edu/, but I'm not sure if it contains the same info?).



Doesn't have all the information I need, and I'm looking for a robust solution that I can apply to similar websites. I appreciate you finding that though, please let me know if you find solution.

Also, is there any specific syntax to submitting the url with values, or do I just submit it with another word after it?

nielsie95

2011-04-01 05:19:32

If you look at the <form> tags, you'll see that certain childs have "name=" attributes. If the form uses "method=get", then it would just be url.pl?name=value&name2=value2. But I believe the forms in your site use post, which complicate things a bit. You need something else to add those values to the request then (Simba/Scar or some other solution).

scuz 10

2011-04-01 05:23:24

Does this help?

<?xml version="1.0" encoding="UTF8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www-act.ucsd.edu/DTD/xhtml1-strict.dtd">
<html xml:lang="en" lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>
<title>UCSD :: My TritonLink</title>
<link type="text/css" href="/jlinkservices/decorators/templates/tritonlink/bnb/styles/screen.css" rel="stylesheet" media="screen, projection"/>
<link rel="stylesheet" href="/jlinkservices/decorators/templates/tritonlink/bnb/styles/print.css" type="text/css" media="print"/>
<script type="text/javascript" src="/jlinkservices/decorators/templates/tritonlink/bnb/js/master.js"></script>
<link type="text/css" href="/jlinkservices/decorators/templates/tritonlink/bnb/components/sifr/sifr.css" rel="stylesheet" media="screen, projection"/>
<script type="text/javascript" src="/jlinkservices/decorators/templates/tritonlink/bnb/components/sifr/sifr.js"></script>
<link type="text/css" href="/jlinkservices/decorators/templates/tritonlink/bnb/components/lea/lea.css" rel="stylesheet" media="screen, projection"/>
<link type="text/css" href="/jlinkservices/decorators/templates/tritonlink/bnb/components/homepage/homepage.css" rel="stylesheet" media="screen, projection"/>
<link type="text/css" href="/jlinkservices/decorators/templates/tritonlink/bnb/components/cwp/cwp.css" rel="stylesheet" media="screen, projection"/>
<script type="text/javascript" src="/jlinkservices/decorators/templates/tritonlink/bnb/components/cwp/cwp.js"></script>
<link type="text/css" href="/jlinkservices/decorators/templates/tritonlink/bnb/components/slideshow/slideshow.css" rel="stylesheet" media="screen, projection"/>
<script type="text/javascript" src="/jlinkservices/decorators/templates/tritonlink/bnb/components/slideshow/slideshow.js"></script>
<link type="text/css" href="/jlinkservices/decorators/templates/tritonlink/bnb/components/tree/tree.css" rel="stylesheet" media="screen, projection"/>
<script type="text/javascript" src="/jlinkservices/decorators/templates/tritonlink/bnb/components/tree/lib/tree.js"></script>
<link type="text/css" href="/jlinkservices/decorators/templates/tritonlink/bnb/components/tablesorter/tablesorter.css" rel="stylesheet" media="screen, projection"/>
<script type="text/javascript" src="/jlinkservices/decorators/templates/tritonlink/bnb/components/tablesorter/tablesorter.js"></script>
<link type="text/css" href="/jlinkservices/decorators/templates/tritonlink/bnb/components/collapse/collapse.css" rel="stylesheet" media="screen, projection"/>
<script type="text/javascript" src="/jlinkservices/decorators/templates/tritonlink/bnb/components/collapse/collapse.js"></script>
<link type="text/css" href="/jlinkservices/decorators/templates/tritonlink/bnb/components/toggler/toggler.css" rel="stylesheet" media="screen, projection"/>
<script type="text/javascript" src="/jlinkservices/decorators/templates/tritonlink/bnb/components/toggler/toggler.js"></script>
<link type="text/css" href="/jlinkservices/decorators/templates/tritonlink/bnb/components/autocomplete/autocomplete.css" rel="stylesheet" media="screen, projection"/>
<script type="text/javascript" src="/jlinkservices/decorators/templates/tritonlink/bnb/components/autocomplete/autocomplete.js"></script>
<link type="text/css" href="/jlinkservices/decorators/templates/tritonlink/bnb/components/applications/applications.css" rel="stylesheet" media="screen, projection"/>
<link type="text/css" href="/jlinkservices/decorators/templates/tritonlink/bnb/components/search/search.css" rel="stylesheet" media="screen, projection"/>
<!–[if lt IE 7]><script type="text/javascript" src="/jlinkservices/decorators/templates/tritonlink/bnb/js/suckerfish-ie6-fix.js"></script><![endif]–>
<link type="text/css" href="/jlinkservices/decorators/templates/tritonlink/override.css" rel="stylesheet" media="screen, projection"/>
</head>
<body>
<div id="cwp-doc">
<div id="cwp-hd">
<a href="http://www.ucsd.edu/portal/site/ucsd"><img src="/jlinkservices/decorators/templates/tritonlink/bnb/images/global/ucsd_logo_screen.gif" alt="UCSD Logo" id="cwp-logo-screen"></img></a>
<img src="/jlinkservices/decorators/templates/tritonlink/bnb/images/global/ucsd_logo_print.gif" alt="UCSD Logo For Printing" id="cwp-logo-print"></img>
<a href="#" class="access-aid">Skip navigation links</a>
<h2 class="access-aid">Navigation</h2>
<div id="cwp-sso">
<h3 class="access-aid">Account</h3>
<a href="https://www-act.ucsd.edu/security/logout?url=http%3A%2F%2Fwww.ucsd.edu%2Fportal%2Fsite%2Fucsd%2Fmenuitem.a49a5057074b8fa15245209d364b01ca%2F%3Fvgnextoid%3Dd7075f6e9cf63110VgnVCM10000045b410acRCRD" class="cwp-sso-login">Sign off</a>
<!– <span class="cwp-sso-name">Hello ScheduleOfClasses User</span> –>
</div>
<h3 class="access-aid">Search</h3>
<div id="cwp-search">
<form action="http://www.ucsd.edu/portal/site/ucsd/tools/search" id="site-search" class="first" title="Search The UCSD Website">
<div class="cwp-search-tr">
<fieldset>
<legend>Search</legend>
<div>
<label for="q2" class="access-aid">Query</label>
<input name="q" id="q2" class="query text-field"/>
<button class="go-button" type="submit"><span>Go</span></button>
</div>
<div>
<a href="#" id="advanced-search-link">Advanced</a>
</div>
</fieldset>
</div>
</form>
<form action="http://www.ucsd.edu/directory/faculty_staff" id="people-search" class="last" title="Search For People At UCSD" onSubmit="dirVerify(this)">
<div class="cwp-search-tr">
<div class="cwp-search-tl">
<fieldset>
<legend>Find People</legend>
<div>
<label for="q3" class="access-aid">Query</label>
<input id="q3" name="entry" class="query text-field"/>
<button class="find-button" type="submit"><span>Find</span></button>
</div>
<div>
<input type="radio" id="q2" name="dir" value="faculty_staff"/>
<label for="qf">Faculty/Staff</label>
<input type="radio" id="q2" name="dir" value="student" checked="checked"/>
<label for="qs">Students</label>
</div>
</fieldset>
</div>
</div>
</form>
</div><!– cwp-search –>
<p id="cwp-section-title"><a href="http://www.ucsd.edu/portal/site/ucsd/menuitem.a49a5057074b8fa15245209d364b01ca/?vgnextoid=d7075f6e9cf63110VgnVCM10000045b410acRCRD">Current Students</a></p>
<!– $Id: Tabs_CurrStudents.html 895 2008-02-21 15:55:32Z devdss $ –>
<div id="cwp-nav">
<h3 class="access-aid">Global Navigation</h3>
<ul id="cwp-primary-nav">
<li id="cwp-first-primary-nav">
<div><a href="/mytritonlink/view">My TritonLink</a></div>
<ul> <li><a href="http://www.ucsd.edu/current-students/my-tritonlink/forms/index.html">Forms</a></li> <li><a href="http://www.ucsd.edu/current-students/my-tritonlink/tools/index.html">Tools</a></li> </ul>
</li>
<li>
<div><a href="http://www.ucsd.edu/current-students/academics/index.html">Academics</a></div>
</li>
<li>
<div><a href="http://www.ucsd.edu/current-students/finances/index.html">Finances</a></div>
</li>
<li>
<div><a href="http://www.ucsd.edu/current-students/student-life/index.html">Student Life</a></div>
</li>
<li>
<div><a href="http://www.ucsd.edu/current-students/beyond/index.html">Beyond UCSD</a></div>
</li>
</ul>
</div>
</div><!– cwp-hd –>
<div id="cwp-bd">
<div id="cwp-breadcrumb-nav">
<h3 class="access-aid">You are here</h3>
<ul>
<li>
<a href="http://www.ucsd.edu/portal/site/ucsd/">UC San Diego</a>
</li>
<li>
<a href="http://www.ucsd.edu/portal/site/ucsd/menuitem.a49a5057074b8fa15245209d364b01ca/?vgnextoid=d7075f6e9cf63110VgnVCM10000045b410acRCRD">Current Students</a>
</li>
<li>
<a href="http://www.ucsd.edu/portal/site/ucsd/menuitem.4277fbddcd85b310b9936d10d34b01ca/?vgnextoid=465bdc247ef63110VgnVCM10000045b410acRCRD">My TritonLink</a>
</li>
</ul>
</div><!–/cwp-breadcrumb-nav–>
<div class="cwp-ja">
<div class="cwp-u first">
<div class="cwp-page-nav" style="background:#fffdf4 url(/jlinkservices/decorators/templates/tritonlink/bnb/images/page_nav/triton_link.jpg) center top no-repeat;padding-top:110px;">
<h3 class="access-aid">Section Navigation</h3>
<h4>Tools</h4>
<ul class="cwp-tree oneatatime arrows">
<!– $Id: SectionNav_Tools_CurrStudents.html 895 2008-02-21 15:55:32Z devdss $ –>
<li class=""><a href="/cgi-bin/tritonlink.pl/9/students/transcripts/academic_history.pl">Academic History</a></li>
<li class=""><a href="/studentAddresses/addresses">Addresses</a></li>
<li class=""><a href="/studentEBill/StudentBilling">Billing and Payments</a></li>
<li class=""><a href="/studentCareerPortfolio">Career Portfolio</a></li>
<li class=""><a href="/classPlanner/planner">Class Planner</a></li>
<li class=""><a href="/studentEnrolledClasses/enrolledclasses">Classes/Wait Lists</a></li>
<li class=""><a href="/studentDars/select">Degree Audit</a></li>
<li class=""><a href="/cgi-bin/tritonlink.pl/1/students/finances/financial_aid.pl">Financial Aid</a></li>
<li class=""><a href="/studentHealthWaiver/waiver">Health Fee Waiver</a></li>
<li class=""><a href="/studentHolds/holds">Holds</a></li>
<li class=""><a href="/studentMajorMinorChange/majorminorchange">Major and Minor</a></li>
<li class=""><a href="/cgi-bin/tritonlink.pl/9/students/academic/classes/schedule_of_classes.pl">Schedule of Classes</a></li>
<li class=""><a href="/studentTranscriptOrder/transcriptorder">Transcripts</a></li>
<li class=""><a href="/studentVerification/verification">Verifications</a></li>
<li class=""><a href="https://aventeur.ucsd.edu/students/vac/">Virtual Advising Center</a></li>
<li class=""><a href="https://acs-webmail.ucsd.edu/">WebMail</a></li>
<li class=""><a href="/cgi-bin/tritonlink.pl/9/students/academic/classes/acd/webreg.pl">WebReg</a></li>
<li class=""><a href="http://www.ucsd.edu/portal/site/ucsd/menuitem.135225ab0c7ce3c0c0020010d34b01ca/?vgnextoid=750846c543c2311046c543c231107eb5ef84RCRD">More Tools …</a></li>
</ul>
<h4>Forms</h4>
<ul class="cwp-tree oneatatime arrows">
<li class=""><a href="http://www.ucsd.edu/portal/site/ucsd/menuitem.135225ab0c7ce3c0c0020010d34b01ca/?vgnextoid=f869a78f2c741110a78f2c741110cdca5105RCRD#academic">Academic</a></li>
<li class=""><a href="http://www.ucsd.edu/portal/site/ucsd/menuitem.135225ab0c7ce3c0c0020010d34b01ca/?vgnextoid=f869a78f2c741110a78f2c741110cdca5105RCRD#finances">Finances</a></li>
<li class=""><a href="http://www.ucsd.edu/portal/site/ucsd/menuitem.135225ab0c7ce3c0c0020010d34b01ca/?vgnextoid=f869a78f2c741110a78f2c741110cdca5105RCRD#studentlife">Health and Well Being</a></li>
<li class=""><a href="http://www.ucsd.edu/portal/site/ucsd/menuitem.135225ab0c7ce3c0c0020010d34b01ca/?vgnextoid=f869a78f2c741110a78f2c741110cdca5105RCRD#career">Career Planning</a></li>
</ul>
</div>
</div>
<div class="cwp-u cwp-default-styles">
<!– BEGIN WEBAPP –>
<!– /opt2/actj2/secured/cgi-bin/sis/schedule_of_classes.pl –>
<!– START FORM –><LINK href="/jlinkservices/css/schedule.css" type=text/css rel=stylesheet>
<LINK href="/jlinkservices/css/studentLink.css" type=text/css rel=stylesheet>
<script src="/jlinkservices/js/domtab2.js" type="text/javascript"></script>
<SCRIPT language="JavaScript" type="text/javascript">
<!–
function submitForm()
{
if (document.options.nw.checked) {
window.open("","results","scrollbars=1,directories=0,statusbar=0,resizable=1,menubar=0,location=0,toolbar=0");
document.forms[currentTab].target="results";
} else {
document.forms[currentTab].target=this.name;
}
document.forms[currentTab].soc_term.value = document.term.soc_term.value;
document.forms[currentTab].hide_full_sections.value = document.options.hide_full_sections.checked;
document.forms[currentTab].submit();
}
function redisplayForm()
{
//only reload if on subject or department tabs
document.redisplay.soc_term.value = document.term.soc_term.value;
document.redisplay.redisplayed.value = "y";
current_tab = document.redisplay.tab_name.value;
//if (current_tab == "" || current_tab == "subject" || current_tab == "department")
document.redisplay.submit();
}
// –>
</SCRIPT>
<CENTER>
<noscript>
<div style="border:solid 2px red;">
<h1>&iexcl;This application requires JavaScript!</h1>
<CENTER>We're terribly sorry, but the Schedule of Classes application requires JavaScript to work.
<BR>
To use this application you must go into your browser settings and enable JavaScript.
<P>If you need help, please contact the TritonLink Customer<BR>Service Representative by clicking
the link in the lower right corner.
</CENTER>
<p>
</p>
</div>
</noscript>
<FORM name="redisplay" method="POST" action="schedule_of_classes.pl">
<INPUT type="hidden" name="tab_name" value="">
<INPUT type="hidden" name="soc_term" value="">
<INPUT type="hidden" name="redisplayed" value="">
<INPUT type="hidden" name="have_schedinfo" value="1">
<INPUT type="hidden" name="adv_schedule" value="1">
<INPUT type="hidden" name="dept_subj" VALUE="subj">
<INPUT type="hidden" name="hide_full_sections" value="">
<INPUT type="hidden" name="jlinkevent" value="subject">
</FORM>
<DIV CLASS="application-content">
<TABLE BORDER=0 CELLPADDING=0 CELLSPACING=0>
<TR>
<TD>
<SPAN STYLE="font-size: 13px; font-weight: bold;">
Select search term:
</SPAN>
</TD>
<TD>
<FORM name="term">
<SELECT name="soc_term" onchange="redisplayForm();" STYLE="font-size: 13px;">
<OPTION STYLE="font-size: 13px;" selected="SELEcted" value=S211>Summer Session II 2011<OPTION STYLE="font-size: 13px;" value=S111>Summer Session I 2011 <OPTION STYLE="font-size: 13px;" value=S311>Special Summer Session 2011<OPTION STYLE="font-size: 13px;" value=SA11>Summer Sessions (All) 2011<OPTION STYLE="font-size: 13px;" value=SP11>Spring Quarter 2011<OPTION STYLE="font-size: 13px;" value=WI11>Winter Quarter 2011<OPTION STYLE="font-size: 13px;" value=FA10>Fall Quarter 2010 <OPTION STYLE="font-size: 13px;" value=S210>Summer Session II 2010<OPTION STYLE="font-size: 13px;" value=SU10>Summer Med School 2010<OPTION STYLE="font-size: 13px;" value=S110>Summer Session I 2010 <OPTION STYLE="font-size: 13px;" value=S310>Special Summer Session 2010<OPTION STYLE="font-size: 13px;" value=SA10>Summer Sessions (All) 2010<OPTION STYLE="font-size: 13px;" value=SP10>Spring Quarter 2010<OPTION STYLE="font-size: 13px;" value=WI10>Winter Quarter 2010<OPTION STYLE="font-size: 13px;" value=FA09>Fall Quarter 2009 <OPTION STYLE="font-size: 13px;" value=S209>Summer Session II 2009<OPTION STYLE="font-size: 13px;" value=SU09>Summer Med School 2009<OPTION STYLE="font-size: 13px;" value=S109>Summer Session I 2009 <OPTION STYLE="font-size: 13px;" value=SA09>Summer Sessions (All) 2009 </SELECT>
</FORM>
</TD>
</TR>
<TR>
<TD WIDTH="20%">
<SPAN STYLE="font-size: 13px; font-weight: bold;">
Find courses:
</SPAN>
</TD>
<TD STYLE="width: 560px;">
<DIV id='mainnav'>
<LI><A HREF="#subject">
by&nbsp;subject</A></LI>
<LI><A HREF="#department">
by&nbsp;department</A></LI>
<LI><A HREF="#specific">
by&nbsp;code(s)</A></LI>
<LI><A HREF="#sectionid">
by&nbsp;section&nbsp;id</A></LI>
<LI><A HREF="#profandtitle">
by&nbsp;professor&nbsp;or&nbsp;title</A></LI>
</DIV>
</TD>
</TR>
<TR STYLE=mainrow>
<TD COLSPAN=2 STYLE="border: 1px solid #adadd6; background-color: #ffffff; height: 300px; width: 400px;">
<DIV class="layer" id="sectionid">
<FORM name="sectionid" method="post" action="class_schedule.pl">
<INPUT type="hidden" name="have_schedinfo" value=1>
<INPUT type="hidden" name="adv_schedule" value=1>
<INPUT type="hidden" name="soc_term" value="">
<INPUT type="hidden" name="hide_full_sections" value="">
<INPUT type="hidden" name="jlinkevent" value="sectionid">
<INPUT type="hidden" name="soc_day" value="MTWRFS">
<TABLE BORDER=0 WIDTH="635px">
<TR>
<TD>
multiple<BR>section ids
</TD>
<TD>
<TEXTAREA style="width: 200px;" ROWS=13 name="multiple_sec_id"></TEXTAREA>
</TD>
<TD ROWSPAN=2>
<TABLE BORDER=0 WIDTH="100%">
<TR>
<TD>
<SPAN STYLE="font-weight: bold;">Section ID Search Tips</SPAN>
</TD>
</TR>
<TR>
<TD>
<LI>Search by six-digit section IDs separated by anything!
</TD>
</TR>
<TR>
<TD>
<LI>commas, spaces, returns, periods, smileys, your name
</TD>
</TR>
<TR>
<TD>
<LI>Example:<P><P>123456 543210.987654,<BR>323475*565473Joe456321
</TD>
</TR>
</TABLE>
</TD>
</TR>
</TABLE>
</FORM>
</DIV>
<DIV class="layer" id=department>
<FORM name="department" method="post" action="class_schedule.pl">
<INPUT type="hidden" name="soc_term" value="">
<INPUT type="hidden" name="have_schedinfo" value="1">
<INPUT type="hidden" name="adv_schedule" value="1">
<INPUT type="hidden" name="dept_subj" VALUE="dept">
<INPUT type="hidden" name="hide_full_sections" value="">
<INPUT type="hidden" name="jlinkevent" value="dept">
<TABLE BORDER=0 CELLPADDING=3 CELLSPACING=0>
<TR>
<TD class="rightborder">
<SPAN style="font-style: italic;">if you select the list you may use your keyboard to jump to a department name</SPAN><BR>
<SPAN style="font-weight: bold;">select one or more departments</SPAN>
<SELECT size=18 name="adv_soc_dept" MULTIPLE style="width: 285px; font-family: 'lucida console', monospace;">
<OPTION SELECTED style="font-family: 'lucida console', monospace;" value="ANTH">Anthropology
<OPTION style="font-family: 'lucida console', monospace;" value="BENG">Bioengineering
<OPTION style="font-family: 'lucida console', monospace;" value="BIOL">Biology
<OPTION style="font-family: 'lucida console', monospace;" value="CHEM">Chemistry and Biochemistry
<OPTION style="font-family: 'lucida console', monospace;" value="COGS">Cognitive Science
<OPTION style="font-family: 'lucida console', monospace;" value="COMM">Communication
<OPTION style="font-family: 'lucida console', monospace;" value="CSE">Computer Science & Engineering
<OPTION style="font-family: 'lucida console', monospace;" value="CONT">Contemporary Issues Program
<OPTION style="font-family: 'lucida console', monospace;" value="CGS">Critical Gender Studies Program
<OPTION style="font-family: 'lucida console', monospace;" value="CAT">Culture, Art & Technology Program
<OPTION style="font-family: 'lucida console', monospace;" value="ECON">Economics
<OPTION style="font-family: 'lucida console', monospace;" value="EDS">Education Studies
<OPTION style="font-family: 'lucida console', monospace;" value="ECE">Electrical & Computer Engineering
<OPTION style="font-family: 'lucida console', monospace;" value="ETHN">Ethnic Studies
<OPTION style="font-family: 'lucida console', monospace;" value="HIST">History
<OPTION style="font-family: 'lucida console', monospace;" value="HDP">Human Development Program
<OPTION style="font-family: 'lucida console', monospace;" value="HUM">Humanities Program
<OPTION style="font-family: 'lucida console', monospace;" value="INTD">Interdisciplinary
<OPTION style="font-family: 'lucida console', monospace;" value="LAWS">Law and Society Program
<OPTION style="font-family: 'lucida console', monospace;" value="LIT">Literature
<OPTION style="font-family: 'lucida console', monospace;" value="MATH">Mathematics
<OPTION style="font-family: 'lucida console', monospace;" value="MAE">Mechanical & Aerospace Engineering
<OPTION style="font-family: 'lucida console', monospace;" value="MUS">Music
<OPTION style="font-family: 'lucida console', monospace;" value="PHIL">Philosophy
<OPTION style="font-family: 'lucida console', monospace;" value="PHYS">Physics
<OPTION style="font-family: 'lucida console', monospace;" value="POLI">Political Science
<OPTION style="font-family: 'lucida console', monospace;" value="PSYC">Psychology
<OPTION style="font-family: 'lucida console', monospace;" value="RSM">Rady School of Management
<OPTION style="font-family: 'lucida console', monospace;" value="SIO">Scripps Institution of Oceanography
<OPTION style="font-family: 'lucida console', monospace;" value="SOC">Sociology
<OPTION style="font-family: 'lucida console', monospace;" value="SE">Structural Engineering
<OPTION style="font-family: 'lucida console', monospace;" value="THEA">Theatre and Dance
<OPTION style="font-family: 'lucida console', monospace;" value="USP">Urban Studies & Planning Prog
<OPTION style="font-family: 'lucida console', monospace;" value="VIS">Visual Arts
<OPTION style="font-family: 'lucida console', monospace;" value="WCWP">Warren College Writing Program
</OPTION>
</SELECT>
</TD>
<TD>
<TABLE BORDER=0 CELLSPACING=3 CELLPADDING=0>
<TR>
<TD>
<TABLE BORDER=0>
<TR>
<TD COLSPAN=3 style="font-weight: bold;">
only show me courses which match any of the following criteria
</TD>
<TR>
<TD COLSPAN=3>
<TABLE BORDER=0 CELLSPACING=0 CELLPADDING=0>
<TR>
<TD NOWRAP>lower division:</TD>
<TD NOWRAP>
<INPUT name=schedoption1 CHECKED type=checkbox>1-99
</TD>
<TD NOWRAP>
<INPUT name=schedoption11 type=checkbox>87&90s
</TD>
<TD NOWRAP>
<INPUT name=schedoption12 type=checkbox>99s
</TD>
</TR>
<TR>
<TD NOWRAP>upper division:</TD>
<TD NOWRAP>
<INPUT name=schedoption2 CHECKED type=checkbox>100-198
</TD>
<TD NOWRAP>
<INPUT name=schedoption4 type=checkbox>195s
</TD>
<TD NOWRAP>
<INPUT name=schedoption5 type=checkbox>199s
</TD>
</TR>
<TR>
<TD>graduate:</TD>
<TD NOWRAP>
<INPUT name=schedoption3 type=checkbox>200-297
</TD>
<TD NOWRAP>
<INPUT name=schedoption7 type=checkbox>298s
</TD>
<TD NOWRAP>
<INPUT name=schedoption8 type=checkbox>299s
</TD>
<TD NOWRAP>
&nbsp;
</TD>
<TD NOWRAP>
&nbsp;
</TD>
</TR>
<TD>&nbsp;</TD>
<TD NOWRAP>
<INPUT name=schedoption13 type=checkbox>300+
</TD>
<TD NOWRAP>
<INPUT name=schedoption10 type=checkbox>400+
</TD>
<TD NOWRAP>
<INPUT name=schedoption9 type=checkbox>500+
</TD>
<TR>
</TR>
</TABLE>
</TD>
</TR>
</TABLE>
</TD>
</TR>
<TR>
<TD COLSPAN=3>
&nbsp;
</TD>
</TR>
<TR>
<TD STYLE="border: 1px solid #cccccc; background-color: #fafafa;">
<TABLE BORDER=0>
<!– <TR>
<TD COLSPAN=3>
<SPAN style="font-style: italic;">(<B>optional</B>) the course title or professor name is</SPAN>
</TD>
</TR>
<TR>
<TD>
course title
</TD>
<TD>
<SELECT name="adv_soc_title_type">
<OPTION value="begin" SELECTED>begins with
<OPTION value="contain">contains
<OPTION value="exact">is exactly
</SELECT>
</TD>
<TD>
<INPUT name="adv_soc_title" size=25>
</TD>
</TR>
<TR>
<TD>
professor&nbsp;name
</TD>
<TD>
<SELECT name="adv_soc_instructor_type">
<OPTION value="begin" SELECTED>begins with
<OPTION value="contain">contains
<OPTION value="exact">is exactly
</SELECT>
</TD>
<TD>
<INPUT name="adv_soc_instructor" size=25>
</TD>
</TR>
–>
<TR>
<TD COLSPAN=3>
<SPAN style="font-style: italic;">(<B>optional</B>) the course meets primarily on</SPAN>
</TD>
</TR>
<TR>
<TD ALIGN=CENTER COLSPAN=3>
<TABLE BORDER=0>
<TR>
<TD NOWRAP>
<INPUT CHECKED name=soc_day VALUE=M type=checkbox>Mon
</TD>
<TD NOWRAP>
<INPUT CHECKED name=soc_day VALUE=T type=checkbox>Tues
</TD>
<TD NOWRAP>
<INPUT CHECKED name=soc_day VALUE=W type=checkbox>Wed
</TD>
<TD NOWRAP>
<INPUT CHECKED name=soc_day VALUE=R type=checkbox>Thurs
</TD>
<TD NOWRAP>
<INPUT CHECKED name=soc_day VALUE=F type=checkbox>Fri
</TD>
<TD NOWRAP>
<INPUT CHECKED name=soc_day VALUE=S type=checkbox>Sat

nielsie95

2011-04-01 05:27:09


<FORM name="department" method="post" action="class_schedule.pl">
<INPUT type="hidden" name="soc_term" value="">
Those are the things you should be looking for. But you might want to search around for a bit on the interwebs for a crawler as I'm sure there are which support things like this.

scuz 10

2011-04-01 05:37:40

From: nielsie95

<FORM name="department" method="post" action="class_schedule.pl">
<INPUT type="hidden" name="soc_term" value="">
Those are the things you should be looking for. But you might want to search around for a bit on the interwebs for a crawler as I'm sure there are which support things like this.


I've been searching for a crawler and I can't really find one, I think my problem is that I don't really know what to search.

Back to what you said before, using the "department" thing, could I do something like:

https://www-act.ucsd.edu/cgi-bin/tritonlink.pl/2/students/academic/classes/schedule_of_classes.pl?department="CHEM"


Also, can you elaborate a little more on how I'd use scar/simba with "post" ?

nielsie95

2011-04-01 05:40:42

From the source above, it looks like you're looking for:

class_schedule.pl?adv_soc_dept=CHEM

The option CHEM is part of the select with name adv_soc_dept, which is part of the form which submits to class_schedule.pl. But the form uses POST, so this might not work.

scuz 10

2011-04-01 05:44:43

From: nielsie95
From the source above, it looks like you're looking for:

class_schedule.pl?adv_soc_dept=CHEM

The option CHEM is part of the select with name adv_soc_dept, which is part of the form which submits to class_schedule.pl. But the form uses POST, so this might not work.


It seems to just ignore any requests I put at the end, is this because of post?

nielsie95

2011-04-01 05:48:07

Could be, or maybe you need to add more.

lordsaturn

2011-04-02 23:25:18

@scuz I think you need to make your own website. Learn javascript. Then you will much better understand what it is you need.

rogeruk

2011-04-05 23:41:22

From: scuz 10
I'm working on a project where I need to get data from different parts of a website. Since the website isn't made with html, i can't edit the url's to get to different parts of the website to get everything i need.

Is there an automated way to get to different parts of a website where your browser url does not change when you click on the links? Is it possible to make a php script or something like that that automatically visits the different parts of the website and parses them?

Thanks..


You just want to crawl the pages and extract data yea?.
http://simplehtmldom.sourceforge.net/

I used this to grab data from grand exchange.

[php]$porh2 = $html->find('h2');[/php]

This grabs the data inbetween <h2> tags in the HTML. It can grab links, images, everything.

I believe its got a "level" option too, how far to follow links

footballjds

2011-04-06 00:13:32

Click me for Get to post conversion

;) solves your get/post problems

rogeruk

2011-04-06 08:32:30

From: iTry
Click me for Get to post conversion

;) solves your get/post problems


I think he would need the source, since its a project

nielsie95

2011-04-06 08:47:42

From: rogeruk
You just want to crawl the pages and extract data yea?.
http://simplehtmldom.sourceforge.net/


Check the link in my first post. It's a similar project ;)

The GET to POST site can help with experimenting, but when you're going to write your own crawler you probably don't need it. I tried looking for a crawler which would support these kinds of sites, but didn't find one that easily. I think it is possible, but it would probably require some time (and even more time if you're not very familiar with things like HTML/JS).

footballjds

2011-04-06 09:43:59

From: nielsie95
Check the link in my first post. It's a similar project ;)

The GET to POST site can help with experimenting, but when you're going to write your own crawler you probably don't need it. I tried looking for a crawler which would support these kinds of sites, but didn't find one that easily. I think it is possible, but it would probably require some time (and even more time if you're not very familiar with things like HTML/JS).


Yes, he will definitely need to learn a lot more. The Get to Post is solely for Static posts and/or experimenting.