SRL Archive - Discussions and Debates

Hey everyone,
I recently decided to take over updating SRL Stats into a usable condition again - by rewriting it from scratch. I'm making this thread for 3 reasons.
1) People will hopefully recognize it's being done and maybe noone else will try to work on it while I am.
2) To keep everyone updated on the status of how I'm coming with it.
3) To make sure if for some reason I start getting lazy, any of you can bitch at me because I made a public thread like this. :)

On that note, I now have a database setup that I'm capable of programmatically commiting or getting information from. What does that mean is left? Actually a fair amount. I need to create a web based access point for it so Simba can call it, I need to write SRL code to use it, and I will need to write something to graph everything into pretty formats for anyone who wants to actually look up the stats on a per user/script basis online and any other feature that may come up.

Speaking of which, make some random functionality requests and they may make it in. Or maybe they won't. At least I'm leaving it as possible.

3) To make sure if for some reason I start getting lazy, any of you can bitch at me because I made a public thread like this. :)

What?!?! Kevin… lazy… is that even possible? O.o
Good job though :) Keep it up or I'll use my rune battleaxe on you.

Good job on taking some action on this. Apparently all that needs to be done for the official SRL Stats to work again is starting the records from scratch, but it's been filled up for so long so it can't be that simple to fix.

Anyway, good work on this and maybe you can talk with the srl devs about how/if you could work together on it so that we don't end up with two separate SRL Stats systems :p

Keep this update and I'll monitor it often. Also, you're looking at the future "Master of 'Monkfish Caught' variable".

From: The Killer
3) To make sure if for some reason I start getting lazy, any of you can bitch at me because I made a public thread like this. :)

What?!?! Kevin… lazy… is that even possible? O.o
Good job though :) Keep it up or I'll use my rune battleaxe on you.

Yeah… Of course… All my up to date scripts scream dedicated :p

From: BMWxi
Good job on taking some action on this. Apparently all that needs to be done for the official SRL Stats to work again is starting the records from scratch, but it's been filled up for so long so it can't be that simple to fix.

Anyway, good work on this and maybe you can talk with the srl devs about how/if you could work together on it so that we don't end up with two separate SRL Stats systems :p

There appeared to be a little more than that, but it's perfectly fine by me to instead create a new one for a few reasons. It makes it better as far as I'm concerned because it's easier to update and it's easier to add new functionality that I'm looking forward to.

Looking forward to seeing the progress, Kevin.

Just a thought, but if this gets put on the villavu domain, then I could use villavu's cookies to verify authenticity for user's to create scripts and variables in the system and would not need to make any of my own authentication to verify any of us are who we say we are.

From: Kevin
Just a thought, but if this gets put on the villavu domain, then I could use villavu's cookies to verify authenticity for user's to create scripts and variables in the system and would not need to make any of my own authentication to verify any of us are who we say we are.

but how would users feel about having to enter their villavu info into the script? 'omg i gun get haxd'

I have a few questions.

First of all, if we are going to start with a new database - then why not use
the system I wrote? There is really very little wrong with it, as I stated
before, it is not very hard to fix, but very annoying considering the huge
database in it. We're talking 10-20GBs of database. So why are you rewriting the
system? What does the database look like? Is it anything like mine – or will it
be back to the very basics – like Dankness' system?

Secondly, does it contain a full history of all the commits?

And finally. what will it be written in?

It's nice to see people wanting to work on this, but I'm saddened that the old
(pretty darn good) work is not being picked up. I'd like to see more details,
because if you're reinventing the wheel, make sure you do it properly.

Wizzup beat me to it, Why not just reset?

From: Wizzup?
I have a few questions.

First of all, if we are going to start with a new database - then why not use
the system I wrote? There is really very little wrong with it, as I stated
before, it is not very hard to fix, but very annoying considering the huge
database in it. We're talking 10-20GBs of database. So why are you rewriting the
system? What does the database look like? Is it anything like mine – or will it
be back to the very basics – like Dankness' system?

Secondly, does it contain a full history of all the commits?

And finally. what will it be written in?

It's nice to see people wanting to work on this, but I'm saddened that the old
(pretty darn good) work is not being picked up. I'd like to see more details,
because if you're reinventing the wheel, make sure you do it properly.

From what I've read his will have different features like graphs for scripts/users and the ability for, whomever declared the script in Stats, to add and remove both scripts and variables. I believe this was a desired feature from long ago.

http://villavu.com/forum/showthread.php?t=107181&p=1277020#post1277020
http://villavu.com/forum/showthread.php?t=107181&page=2&p=1277083#post1277083

From: Flight
From what I've read his will have different features like graphs for scripts/users and the ability for, whomever declared the script in Stats, to add and remove both scripts and variables. I believe this was a desired feature from long ago.

http://villavu.com/forum/showthread.php?t=107181&p=1277020#post1277020
http://villavu.com/forum/showthread.php?t=107181&page=2&p=1277083#post1277083

Current one has support for almost any graphs you want, just needs the views/templates.

From: Turpinator
but how would users feel about having to enter their villavu info into the script? 'omg i gun get haxd'

User id - you're 76798 (you can see it on your home page), and it could even be done via username resulting in a User ID lookup. It's not hard to see someone's username anyway, and no password may be necessary. Although, if people are worried about others submitting false data in their name, I can create another tool to create a custom password per person, however that would require a registration process and I would prefer this be an easy noob friendly process.

From: Wizzup?
I have a few questions.

First of all, if we are going to start with a new database - then why not use
the system I wrote? There is really very little wrong with it, as I stated
before, it is not very hard to fix, but very annoying considering the huge
database in it. We're talking 10-20GBs of database. So why are you rewriting the
system? What does the database look like? Is it anything like mine – or will it
be back to the very basics – like Dankness' system?

Secondly, does it contain a full history of all the commits?

And finally. what will it be written in?

It's nice to see people wanting to work on this, but I'm saddened that the old
(pretty darn good) work is not being picked up. I'd like to see more details,
because if you're reinventing the wheel, make sure you do it properly.

To clarify some things up front, in no way do I even slightly mean offense to the work you've done. It was great and used for years! However, at it's core, Python isn't a super extensible language, I haven't ran across many people familiar with it, and I haven't seen anyone willing to touch it in over a year. As such, I want to rewrite it in C# - under the .NET framework. Tons of stuff is built into the framework, it's extremely extensible, and there are tons of people who are familiar with the related languages - and even if they aren't, C# is pretty easy for anyone to pick up with OOP experiences in the past.

I would definitely foresee any database involved in this becoming quite large in time, simply due to the amount of traffic incoming and needing to be saved - not to mention any desired administrative information, such as source of commits. As for what was previously all saved and what I save… As of now, it's only the required stats you would want visible on graphs - nothing administrative yet, not to say that won't come, that's just not what I have in place. I don't really see rewriting a database system as that big a problem myself, at least not from the C# and SQL side. But as for why rewrite it? Ease of fixing in the future if anything else comes up. Ease of extending new functionality. It being a more accessible language to more easily be picked up in the future. Besides, it doesn't seem like any tremendously huge undertaking to write up a new database setup and it becomes a nice project to improve familiarity overall on the subject.

Also, doing this, hopefully database management will become a much simpler process, entirely GUI oriented and done completely on permissions of individual users.

Full history of commits is in place already - just not every detail such as source IP and… I'm not sure what else I'm missing actually. Once I setup a basic WebAPI for it, I'm fully willing to OpenSource it publicly for you (or anyone else for that matter) to look over and make any possible recommendations.

From: Dgby714
Current one has support for almost any graphs you want, just needs the views/templates.

I did note that… But I never saw anyone create or work on those, and there could be plenty of other untold functionality that would be more easily implemented if anyone desired (mainly so far would be high-level database management via a simple gui).

From: Kevin
However, at it's core, Python isn't a super extensible language, I haven't ran across many people familiar with it, and I haven't seen anyone willing to touch it in over a year.

Funny you say that, Python, Java, and C are the ones taught in CompSci classes…

From: Dgby714
Funny you say that, Python, Java, and C are the ones taught in CompSci classes…

I've never had Python either. Also I don't like that particular language. However lots of people do. I would love to see what Kevin comes up with in c sharp.

Imo it would be best, if it could retrieve/parse/use the old database.

From: Dgby714
Funny you say that, Python, Java, and C are the ones taught in CompSci classes…

That's odd. I just graduated with a degree in Computer Engineering last year - computer science with less theory and more practice - and I don't recall me or any ComSci student learning Python. Java and C though, we all did learn, as well as a number of other languages. In fact… I'm not sure anyone I work with ever had to learn Python either. And from what I've seen, I don't particularly care for it as a language. I understand many people do, but I don't, and I don't know very many people who do. I would imagine this community has more people familiar with C# than with Python, though.

From: masterBB
I've never had Python either. Also I don't like that particular language. However lots of people do. I would love to see what Kevin comes up with in c sharp.

Imo it would be best, if it could retrieve/parse/use the old database.

I'm not sure entirely, but I think one of the issues was with the old database setup causing an overflow there as well? However, if I had all the connection information, I could set it up to connect to any other database. Right now I'm just doing everything on a local MySQL database as it's freeware and simple to setup.

From: Kevin
That's odd. I just graduated with a degree in Computer Engineering last year

Depends on where you went, tho I have seen some universities teaching Python to students in CompSci.

Anyways, I have no clue what this community has in terms of who knows what…

Anyways, Why don't you like Python?

From: Dgby714
Depends on where you went, tho I have seen some universities teaching Python to students in CompSci.

Anyways, I have no clue what this community has in terms of who knows what…

Anyways, Why don't you like Python?

A few reasons:

• Python is simply known by less people at a start (although it can be granted that in the overall sense more people are familiar with Python than LAPE).
• Python has no multiprocessor support and multithreading isn't a super easy task - whereas in C#, every request for a commit/GET to the database will be inherently multi-threaded with 0 extra effort.
• Python is a notably slow language - not a big deal on smaller scales, but if we were to ever hit a large quantity of users, it could eventually affect a project like this, whereas C# could reasonably support close to 75k or so concurrent committers on a single box. (A scale far beyond what we would ever reach, but still, Python would support a small fraction of that, I'm guessing 2-3k concurrent commiters?)
• And then debuggers - I'm not familiar whether or not Python has debugging capabilities, but the debugging capabilities for C# in Visual Studio (free) is quite nice.

From: Kevin
A few reasons:

• Python is simply known by less people at a start (although it can be granted that in the overall sense more people are familiar with Python than LAPE).
• Python has no multiprocessor support and multithreading isn't a super easy task - whereas in C#, every request for a commit/GET to the database will be inherently multi-threaded with 0 extra effort.
• Python is a notably slow language - not a big deal on smaller scales, but if we were to ever hit a large quantity of users, it could eventually affect a project like this, whereas C# could reasonably support close to 75k or so concurrent committers on a single box. (A scale far beyond what we would ever reach, but still, Python would support a small fraction of that, I'm guessing 2-3k concurrent commiters?)
• And then debuggers - I'm not familiar whether or not Python has debugging capabilities, but the debugging capabilities for C# in Visual Studio (free) is quite nice.

I can't really say anything about more or less people knowing it…

You can use processing module to spawn more processes which will use more cores/processors? Actually here's a link to this same question.

Yes, Python is slow, no clue how slow, but you also have to take into account a lot of the modules are written in C and pretty fast, and the current stats website is Python held up to the constant flow of commits even when our userbase got really big after botany bay.

Python has it's own debugger. With this last question, have you ever even used Python? Afaict, no, or you would know it does…

From: Dgby714
I can't really say anything about more or less people knowing it…

You can use processing module to spawn more processes which will use more cores/processors? Actually here's a link to this same question.

Yes, Python is slow, no clue how slow, but you also have to take into account a lot of the modules are written in C and pretty fast, and the current stats website is Python held up to the constant flow of commits even when our userbase got really big after botany bay.

Python has it's own debugger. With this last question, have you ever even used Python? Afaict, no, or you would know it does…

I absolutely admit that I have not used Python in any legitimate development atmosphere and I thought I admitted that earlier… Etiher way - I will fully admit right now that my Python experience consists of looking at it as a language and trying to see what I could pick up in one day when I heard of the project Unsigned as an open-source Python based project (I think it was Unsigned at least - at least it was a similar project to that concept either way). Never the less - Python is not remotely in any of my areas of expertise. Feel free to completely take that into account with all of this.

Python is a popular language on the scale of languages. However, at least under my experience, I've met far more people who know C#. Maybe I have been under a skewed aspect for that I suppose - but even still, my experience lies far more heavily in C#, and as I'm fully aware of many of the advantages C# has in development as well and noone else is fixing the Python codebase…

I wasn't aware of the multi-processor capabilities of Python. That is kind of neat, however it still remains that each request in C# would inherently have its own thread with no developer needing to trudge through the intricacies of multi-threading in order to use or understand it.

I absolutely believe that Stats has never and would never reach a point when Python wouldn't be fast enough. No offence to us, but our main user-base is Runescape, and Runescape will never have enough popularity that the color bot users will reach a number of active Stats committers that would slow us down even on Python. My comment on speed is merely a random extra plus, with the possible advantage of the same hosting machine being able to support more systems.

Debuggers are always quite nice. That is definitely a plus.

From: Kevin
I absolutely admit that I have not used Python in any legitimate development atmosphere and I thought I admitted that earlier… Etiher way - I will fully admit right now that my Python experience consists of looking at it as a language and trying to see what I could pick up in one day when I heard of the project Unsigned as an open-source Python based project (I think it was Unsigned at least - at least it was a similar project to that concept either way). Never the less - Python is not remotely in any of my areas of expertise. Feel free to completely take that into account with all of this.

Python is a popular language on the scale of languages. However, at least under my experience, I've met far more people who know C#. Maybe I have been under a skewed aspect for that I suppose - but even still, my experience lies far more heavily in C#, and as I'm fully aware of many of the advantages C# has in development as well and noone else is fixing the Python codebase…

I wasn't aware of the multi-processor capabilities of Python. That is kind of neat, however it still remains that each request in C# would inherently have its own thread with no developer needing to trudge through the intricacies of multi-threading in order to use or understand it.

I absolutely believe that Stats has never and would never reach a point when Python wouldn't be fast enough. No offence to us, but our main user-base is Runescape, and Runescape will never have enough popularity that the color bot users will reach a number of active Stats committers that would slow us down even on Python. My comment on speed is merely a random extra plus, with the possible advantage of the same hosting machine being able to support more systems.

Debuggers are always quite nice. That is definitely a plus.

Also note CPython is slow, but you can also use other run-time implementations, like IronPython (Uses .NET, usually close in performance to C#)

On top of that there are other alternatives like pyToC, pyREX and others.

The comment about C# doing it for every call, that creates unneeded overhead, or you have it wrong and a unit/module/include you're using handles it for you.

From: Dgby714
Also note CPython is slow, but you can also use other run-time implementations, like IronPython (Uses .NET, usually close in performance to C#)

On top of that there are other alternatives like pyToC, pyREX and others.

The comment about C# doing it for every call, that creates unneeded overhead, or you have it wrong and a unit/module/include you're using handles it for you.

Most built in frameworks for C# handle all that related logic. They use an existing and dedicated threadpool for all the incoming requests such that each request can have its own unique thread - AND not have to spend any additional time doing normal thread initialization tasks as those were all done upon application start.

I said something wrong in the previous quote, "usually close in performance to C#", read it wrong, should be "uses the .NET and other libraries for C#"

Glad to hear that you will be working on this project :) Good luck mate, and thank you!

I am looking forward to seeing this! Good luck and hope it turns out great!

Update please @Kevin

From: xtrapsp
Update please @Kevin

I have a base setup for charts now - although they aren't very pretty and it's kinda sorta hard-coded at the moment (as in I hard-code a script name / User ID and then it gathers all the information based on that into a chart onto the webpage).

ASP has now been setup to a point such that some endpoints do exist - ie: the chart getter. Adding a JSON based endpoint for submitting new stats shouldn't be a big deal now with other endpoints up and working. And with that up, it would be a valid useable system once scripts/variables are added.

Not yet sure how user based script/variable adding/removal should be handled in a secure manner without having access to villavu cookies. May have some sort of unique registration (only necessary for creation of scripts and variable editing), but not sure how to tie it to villavu with any level of verification yet.

UPDATE:

SVN exists here: https://srl-statistics.googlecode.com/svn/trunk/

Obviously missing a few things - most notably prettiness or the ability to access script information via a gui (but it can be accessed via an endpoint which will then display it in chart format). Just using a MySQL database at the moment - free, public, and easy to use. Commits and gets work via the DBAccessor, and gets can be displayed for the most part. So, at least it's coming along. This weekend should have fully working and versatile commits and either a front page of value or some sort of registration concept - trying to figure out a way to lock a stats account to a villavu account with full verification (for security purposes so you can't just lie and say you're Olly) without being on the villavu domain. Or maybe instead of either of those I'll add some administrative functionality to the database this weekend and gather other information like commiter IP, SMART instance and whatnot. (If you're curious why I'd copy the Smart instance number, it would be to verify that someone isn't using a single instance to commit many fake numbers - whereas a single ip could be running multiple bots on the same script, preventing IP detection alone from covering that).

And the database looks pretty simple as such:

Nice nice, looks like it's coming along ;)

Sorry it took me some time to respond.

Your arguments about Python mostly show a lack of knowledge about how it works.
Python is *very* extensible. Furthermore, the speed of the frontend language has
little to no effect in this kind of program. All the heavy lifting is done in
the database. This meant that effectively Python could be 10 times as slow, and
you still would not notice.

About the multithreading, most frameworks run several processes and equally
spread the requests to the available processes. In fact, SRL Stats as it is now
has 10 backend processes. So it does multiprocess load spreading just fine.

I am not convinced about your argument to do it in C#, and I also do not believe
a redesign / reimplementation is required. The code is really not that bad, and
it's definitely fast. Like I said, if you want to keep the full history, it is
not surprising it becomes slow after 50 MILLION entries. That's what we're at
now, approximately. And the sum of that - that is already incrementally cached -
does no longer fit in a normal sized integer. That is main problem people are
running into.

There is already variable management, etc. There is just not a nice frontend;
about 5 people have access to that, btw.


Regardless, good luck with the project, I'm sure you'll learn alot in the
process – like not using MySQL, and how the frontend speed does not matter
much.

Good luck!

From: Wizzup?
Sorry it took me some time to respond.

Your arguments about Python mostly show a lack of knowledge about how it works.
Python is *very* extensible. Furthermore, the speed of the frontend language has
little to no effect in this kind of program. All the heavy lifting is done in
the database. This meant that effectively Python could be 10 times as slow, and
you still would not notice.

About the multithreading, most frameworks run several processes and equally
spread the requests to the available processes. In fact, SRL Stats as it is now
has 10 backend processes. So it does multiprocess load spreading just fine.

I am not convinced about your argument to do it in C#, and I also do not believe
a redesign / reimplementation is required. The code is really not that bad, and
it's definitely fast. Like I said, if you want to keep the full history, it is
not surprising it becomes slow after 50 MILLION entries. That's what we're at
now, approximately. And the sum of that - that is already incrementally cached -
does no longer fit in a normal sized integer. That is main problem people are
running into.

There is already variable management, etc. There is just not a nice frontend;
about 5 people have access to that, btw.


Regardless, good luck with the project, I'm sure you'll learn alot in the
process – like not using MySQL, and how the frontend speed does not matter
much.

Good luck!

I think the main issue was that nobody (at least so far) has been willing to fix SRLstats. It's been at least 6 months - maybe a year? since commits and updated data were showing up on the stats website. Kevin is willing to fix it, but doesn't know python - thus switching to C. It's less about python and more about how nobody who knows python wants to/or is able to fix it.

Maybe I'm missing it completely but that's how I interpreted it.

Ah, I missed responses on this. I absolutely admit to a lack of Python knowledge, and that is something I have now tried to say a few times over in this thread. I know extremely little about Python. Ashaman is correct in his statements so far, although I was thinking Stats had been down for a year and a half by this point in time. Maybe it does stand that there are people who could fix it and it would not be a terrible process, but none of those people have stood forward to attempt so.

And absolutely, any database would become slow after 50 million entries. That is simply a huge quantity of area that has to be queried. As for the front-end speed boost, as I said before:

I absolutely believe that Stats has never and would never reach a point when Python wouldn't be fast enough. No offence to us, but our main user-base is Runescape, and Runescape will never have enough popularity that the color bot users will reach a number of active Stats committers that would slow us down even on Python. My comment on speed is merely a random extra plus, with the possible advantage of the same hosting machine being able to support more systems.

And yes, whether or not this is a project that gets used in the end or not, at least I will learn from it. Database work isn't something I've touched before very much outside some GET calls, so that's a learning experience.

How has the progress been lately?

From: J J
How has the progress been lately?

It hasn't been progressing for the past 2 weeks. My development machine crashed and I haven't had stable internet at home in 2 weeks, so I've been incapable of installing development software on my other desktop.

It is kinda depressing to notice how much of everything I do at home depends on the internet nowadays…

Computer and internet up and running again. Just committed some changed to the SVN that I wanted to get done.

Commits now work correctly with any type of variable and do a few administrative checks. Graphs are also created upon calls. Can do different graphs on any Script/User and/or combination. For all extensive purposes, this is now a working stats once setup with a host. I'll probably want to test out with some people off my local machine first just to see how everything goes before any sort of dedi, and that'll require me to port forward to my computer, but not much else.

I still want to add a couple more built-in automagic administration to the commits. And I will need to come up with a method of adding scripts as allowed in the database without someone telling me to do it (maybe people message me and I register them? naw, want it to be more automatic than that… e-mail registration maybe?). Still need to add a sorting mechanism if people want to see all-time records. Perhaps add something to graphs for comparative purposes between scripts? I definitely want to make the graphs prettier… But I can't really imagine what prettier looks like exactly… Perhaps if someone wanted to and could draw a better-looking style, I could just emulate that in code? Not really sure what else is missing

Nice work Kevin. Keep up the excellent work! :)

From: Kevin
I definitely want to make the graphs prettier… But I can't really imagine what prettier looks like exactly… Perhaps if someone wanted to and could draw a better-looking style, I could just emulate that in code? Not really sure what else is missing

What do they look like now/what are the drawing capabilities?

From: Clarity
What do they look like now/what are the drawing capabilities?

I'm at work atm and can't screenshot, but remind me later (perhaps via skype) and I'll make sure to add the screenies of the graphs up here.

As for capabilities, probably far more advanced than my imagination goes. Inventing pretty was never quite my thing, but copying pretty isn't too bad. :p

I'm really looking forward to this Kevin, thanks for keeping us updated on your progress!

Agreed, not the prettiest… But besides a table of numbers, I'm not sure how else to make it look in my head.

Just made a check-in this morning that sets up most of the script registration process. By the end of today there should be another check-in that allows the creator of a script full administrative access over scripts they add to Stats. Today that's likely to be through a RESTful interface, tomorrow should be through a GUI I hope.

From: Kevin
Just made a check-in this morning that sets up most of the script registration process. By the end of today there should be another check-in that allows the creator of a script full administrative access over scripts they add to Stats. Today that's likely to be through a RESTful interface, tomorrow should be through a GUI I hope.

good, i think im going to implement this into my slayer script

Just made another check-in. GUI exists to register scripts/add variables. It is now is technically a working project without my intervention (once setup on a server). There are a couple of administrative upgrades almost in place so users can remove variables from their scripts (and even delete the scripts!) without ANY admin intervention, while retaining enough security as should be needed to make sure noone else can delete your stuff without your permission. And then I want to pretty it up some more, and make it a tad more user friendly. But as of now, it is a working project entirely with an interface for your pleasure.

@The Mayor has agreed to help me test it out and give up front feedback before I finalize it and throw it onto a server as well, so hopefully that helps a lot as well.

I assume this has been abandoned?