SRL Archive - SRL Stats

Some kind of upgrade? Or just down?

Up for me :)

Seems not to load for me either.

edit: nvm just loading realy slow.

Up for me.

The stats page goes down/acts slow pretty often for me, it's pretty hit or miss sometimes.

Owww… I was hoping that it was wizzup fixing the issues, oh well it's back for me now too

From: YoHoJo
Up for me.

The stats page goes down/acts slow pretty often for me, it's pretty hit or miss sometimes.

Yes. I think I might just disable caching updating for the time being (until I have time to fix it). So stats will still be logged, but not updated.

From: Wizzup?
Yes. I think I might just disable caching updating for the time being (until I have time to fix it). So stats will still be logged, but not updated.

Any chance you could do a SQL export of the stats so that I could upload them to my sever? Just so I could make pretty pictures etc…? This is provided the stats use SQL I haven't looked into them at all…

From: putonajonny
Some kind of upgrade? Or just down?

stats are "down" approximately every 5 minutes it appears, but it only takes very long to respond most of the time, … but when it comes back, the script totals such as total run time are updated so i assume that may be some task that is kicking off every now and then

some browsers won't wait long enough for response, but i find that if i let my chrome wait about a minute or so, it comes back

Edit if you need to hit me up on some db issues, i'm dealing on every day basis with system that has several tables of 1/2 bil or more rows so i'm not new to dealing with large datasets

I think stats can go down/respond very slow sometimes beacuse of all the loads incomming to the webserver. Think about it, if Simba has 10k+ active botters at all time, and 85% of them using the stats system. It depends on the script but thats like 50000 loads every 5minute and thats very huge data trafic to load for a server ^^ (if you understand what i mean)

Stats are quite buggy for me, it never updates my variables anymore :(

From: Oleaa
I think stats can go down/respond very slow sometimes beacuse of all the loads incomming to the webserver. Think about it, if Simba has 10k+ active botters at all time, and 85% of them using the stats system. It depends on the script but thats like 50000 loads every 5minute and thats very huge data trafic to load for a server ^^ (if you understand what i mean)

I cannot speak to the percentage of people using stats and number of active botters, but seeing that stats has 17.5 mil records, you are suggesting that the stats only contains 29 hours of runtime (29 hours at 50k per 5 minutes = 17.4 mil submissions)

so i looked at actual submissions and as of this message, the last 240 commits came over span of 1 min and 58 seconds, making it about 120 per minute (2 per second) rather than 10k that you suggested (166 per second).

However, seeing that stats are running on postgresql, i am suspecting that auto vacuum is misconfigured or not optimal for the number of records and/or indexes. That would explain the periodic lock out while it restructures tables.

When i get a little bit of time i'll look into the source if i can find it :P hopefully it will include diagram of tables and list of indexes

From: zmon
When i get a little bit of time i'll look into the source if i can find it :P hopefully it will include diagram of tables and list of indexes

Here is some info I found: http://wizzup.org/stats/

And the source: https://github.com/MerlijnWajer/SRL-Stats

Works fine for me, except sometimes it loads slow. But I agree that it needs an upgrade =D

From: zmon
I cannot speak to the percentage of people using stats and number of active botters, but seeing that stats has 17.5 mil records, you are suggesting that the stats only contains 29 hours of runtime (29 hours at 50k per 5 minutes = 17.4 mil submissions)

so i looked at actual submissions and as of this message, the last 240 commits came over span of 1 min and 58 seconds, making it about 120 per minute (2 per second) rather than 10k that you suggested (166 per second).

However, seeing that stats are running on postgresql, i am suspecting that auto vacuum is misconfigured or not optimal for the number of records and/or indexes. That would explain the periodic lock out while it restructures tables.

When i get a little bit of time i'll look into the source if i can find it :P hopefully it will include diagram of tables and list of indexes

No, the issue is simply that my cache currents has to do a full table scan and doesn't run it on a background thread. I'm going to fix this in a rewrite but I need to investigate a bit more on postgresql atomic actions. If you want to work on it, your help is very welcome. We could probably patch the current version as well. (The only issue is that the current version has … somewhat ugly source code compared to the new one. I wrote my own WSGI stuff, new one uses Flask.)

If you wish to work on it please drop in on irc, I pretty much worked out a fix already. :)

E: See comments in https://git.villavu.com/srl-stats.git/tree/query.py?h=flask#n286

It occasionally gets really slow, I think it's because there are like 20 commits per minute. I think it would be faster if the site updates every 5 minutes or so.

http://villavu.com/forum/showthread.php?p=1024487#post1024487