SRL Archive - Web Development

Anyone good with d3js graphing we need something to speed the commits page on stats up
http://stats.grats.pw/commit.php

that's going to destroy laptops in the near future and further down the road desktops.. too much data. Server itself handles it in small fractions of a second.. clients = rip

all the data you'd need is clientside, since it's javascript.. anyone able to help or give an example on speeding it up in any methods?
I've messed with webgl.. not with canvas (no idea)
read canvas probably will speed it up.. probably not enough
progressive loading?
http://syntagmatic.github.io/parallel-coordinates/examples/slickgrid.html
I have no idea on this stuff..

Looks like you've already identified the number of data points (rendering them) as the primary issue. As you mentioned, canvas and webgl are the next step in terms of attempting to render the data speedily.

Personally I wouldn't bother with canvas, webgl, or trying to be fancy with progressive loading. I would go back to basics and think about the story you're trying to tell with your visualisation.

After reviewing the commits page it's pretty clear all you're after is a neat and tidy way of showing the number of commits over time.

To better tell the story and improve performance there are two very basic things we can do to the data; aggregate and limit the range. I'd suggest a combination of the two. Perhaps only show the last 3 months of data? Aggregate it by day? If you want to show the last year, aggregate by week? etc.

I'd recommend crossfilter (http://square.github.io/crossfilter/) to filter the data clientside if you do not wish to do transformations server side.

My experience is primarily is creating dashboards in dc.js (http://dc-js.github.io/dc.js/) but I'd be happy to put a chart together for you if you provide some sample data.

From: honeyhoney
Looks like you've already identified the number of data points (rendering them) as the primary issue. As you mentioned, canvas and webgl are the next step in terms of attempting to render the data speedily.

Personally I wouldn't bother with canvas, webgl, or trying to be fancy with progressive loading. I would go back to basics and think about the story you're trying to tell with your visualisation.

After reviewing the commits page it's pretty clear all you're after is a neat and tidy way of showing the number of commits over time.

To better tell the story and improve performance there are two very basic things we can do to the data; aggregate and limit the range. I'd suggest a combination of the two. Perhaps only show the last 3 months of data? Aggregate it by day? If you want to show the last year, aggregate by week? etc.

I'd recommend crossfilter (http://square.github.io/crossfilter/) to filter the data clientside if you do not wish to do transformations server side.

My experience is primarily is creating dashboards in dc.js (http://dc-js.github.io/dc.js/) but I'd be happy to put a chart together for you if you provide some sample data.

Just to second this opinion, I think it would be much more effective to list the number of commits per day, rather than per hour. Then you could possibly have another query and graph combo for viewing commits per hour averaged out over a period of 30 days? Would allow you to see trends over the day.

From the technical point of view, I really don't know very much D3, but I fancy having a little go anyway, it'll be an extremely basic example, but we'll see! Do you require all of the graphing to be client-side? I'm sure there are some PHP libraries available for this task, but would limit the interactiveness of it all

Top chart is commits grouped by day.

Bottom chart is commits as they come in (I just have 3 commit numbers per day).


As you can see the top chart is a lot less cluttered. There's no real data behind it so the story isn't particularly interesting but an option you have here is you could have multiple linked charts and provide drill-down/interrogation capability like so:

(shows 3 data points that make up the Wed 06 data point)

I'd be interested to see what data you have available if you still want some help on this - but if not I hope I've given you a few ideas on how to approach this.

Edit: just grabbed the data from your stats page:

Been working on this a bit, with the finished result looking like this:


Few minor changes:
- Can't currently zoom (may work on this)
- Highlighted points are hollow, bit easier to see
- Time is rounded to the hour for tidiness
- Time/commits is static (I can make this follow the mouse if you want)
- Much faster mainly because it's simpler
- It now takes the data in JSON format (from an external file, for neatness), the format is as follows:

[{
    "x": "2015-02-17 02:51:24",
    "y": 10
}, {
    "x": "2015-02-17 03:51:24",
    "y": 10
}]

I'll admit it isn't quite as flash as the ones your library generated, with the code being quite messy, but it was my first effort.

Files attached (HTML, CSS + JSON)

That's awesome work @Richard.
Great to see you gave it a shot!

With regards to the code being messy/verbose; that's pretty much d3 for you. For simple charts you want the mock up quickly, it's pretty easy to just append everything to the svg.

I worry that without the use of crossfitler you lose the ability to easily filter the data. I linked it in my previous post - worth checking out if you enjoyed d3 at all.

@grats, have you found any of the above useful?

From: honeyhoney
That's awesome work @Richard.
Great to see you gave it a shot!

With regards to the code being messy/verbose; that's pretty much d3 for you. For simple charts you want the mock up quickly, it's pretty easy to just append everything to the svg.

I worry that without the use of crossfitler you lose the ability to easily filter the data. I linked it in my previous post - worth checking out if you enjoyed d3 at all.

@grats, have you found any of the above useful?

barely even started reading it all and I already know it's all useful lol.. I'll check it out tonight. Had no idea people actually responded since it'd been a while :D

looks really good though

From: Richard
Been working on this a bit, with the finished result looking like this:


Few minor changes:
- Can't currently zoom (may work on this)
- Highlighted points are hollow, bit easier to see
- Time is rounded to the hour for tidiness
- Time/commits is static (I can make this follow the mouse if you want)
- Much faster mainly because it's simpler
- It now takes the data in JSON format (from an external file, for neatness), the format is as follows:

[{
    "x": "2015-02-17 02:51:24",
    "y": 10
}, {
    "x": "2015-02-17 03:51:24",
    "y": 10
}]

I'll admit it isn't quite as flash as the ones your library generated, with the code being quite messy, but it was my first effort.

Files attached (HTML, CSS + JSON)

yea it's not as nice cuz I use c3js+d3js to do the graphics since I'm too retarded to just use d3



looks really nice though super busy during the day so tonight I'll look at it and throw it on there… literally anywhere around the site you guys see fixes just throw them at me. The site isn't open source but the only crap that isn't client side is the database queries, really no reason to be open source imo.. I don't like using github or git at all for that matter and anything anyone wants just lmk, the queries are basic single or double join queries… there's nothing fancy, all the magic is client side.

From: honeyhoney
That's awesome work @Richard.
Great to see you gave it a shot!

With regards to the code being messy/verbose; that's pretty much d3 for you. For simple charts you want the mock up quickly, it's pretty easy to just append everything to the svg.

I worry that without the use of crossfitler you lose the ability to easily filter the data. I linked it in my previous post - worth checking out if you enjoyed d3 at all.

Thanks! Yeah, it's not exactly useful for anything besides viewing the trends over time, or specific commits on a date, even then it can't always pinpoint due to the volume of data. I'll be advancing it so it's much more practical, I'll have a look at those resources you posted.

From: grats
yea it's not as nice cuz I use c3js+d3js to do the graphics since I'm too retarded to just use d3



looks really nice though super busy during the day so tonight I'll look at it and throw it on there… literally anywhere around the site you guys see fixes just throw them at me. The site isn't open source but the only crap that isn't client side is the database queries, really no reason to be open source imo.. I don't like using github or git at all for that matter and anything anyone wants just lmk, the queries are basic single or double join queries… there's nothing fancy, all the magic is client side.

I'll keep at it, produce something which is a big more practical (should still be able to maintain the speed), I've got an idea for the zoom functionality, we'll see where it goes. If there's anything else you'd like doing then let me know, happily to have a go

From: Richard
Thanks! Yeah, it's not exactly useful for anything besides viewing the trends over time, or specific commits on a date, even then it can't always pinpoint due to the volume of data. I'll be advancing it so it's much more practical, I'll have a look at those resources you posted.



I'll keep at it, produce something which is a big more practical (should still be able to maintain the speed), I've got an idea for the zoom functionality, we'll see where it goes. If there's anything else you'd like doing then let me know, happily to have a go

lets make that commits page yours for the most part, I added it to the special thanks lol

I got really sick for 2 months trying to do a lot of work and get my moneys back (yes irl), spent it all. A lot on stats that just stayed in beta… users page is a great example of that with the random list of users that you can't click, lol.


[php]
echo "['x', $v2],";
echo "['Commits', $v1]";
[/php]
as seen in the page source


that's what it looks like right now as far as the dates and values.. it pulls from my cache table for all of the "commits"
that table is how you see it in client side which the x is

'yyyy-mm-dd hh:mm:ss',' etc
and the commits are just 'number'

so yours needs to have a json node per graph node instead it looks like, is this permanent? I can change how the cache database stores info.. I'll change it to be the permanent solution for the commits though rather than change it later again..

When I change that I'll be able to make the commits page with whatever you give me.. it'll be on the same page though not loaded from a .js file, so the disks don't have to be written to extra times.

From: Richard
Just to second this opinion, I think it would be much more effective to list the number of commits per day, rather than per hour. Then you could possibly have another query and graph combo for viewing commits per hour averaged out over a period of 30 days? Would allow you to see trends over the day.

From the technical point of view, I really don't know very much D3, but I fancy having a little go anyway, it'll be an extremely basic example, but we'll see! Do you require all of the graphing to be client-side? I'm sure there are some PHP libraries available for this task, but would limit the interactiveness of it all

I don't want any server-side graphing at all, the less the better. That's all extremely easy and it's also easy to output it as an image etc.. it's the day and age of making clients do their own work!


http://stats.grats.pw/test/graph.html

didn't really look at it, data is undefined, that's just a copy paste into that /test/ from your .zip you gave

From: honeyhoney
Top chart is commits grouped by day.

Bottom chart is commits as they come in (I just have 3 commit numbers per day).


As you can see the top chart is a lot less cluttered. There's no real data behind it so the story isn't particularly interesting but an option you have here is you could have multiple linked charts and provide drill-down/interrogation capability like so:

(shows 3 data points that make up the Wed 06 data point)

I'd be interested to see what data you have available if you still want some help on this - but if not I hope I've given you a few ideas on how to approach this.

Edit: just grabbed the data from your stats page:

I like all of these, for the most part it's easier to make a complex graph that can dynamically change than it is to put multiple graphs on a single page.. so idk how to evolve that.
I really wouldn't say the mouse over I have is that important for the commits page really, either.

From: grats
I like all of these, for the most part it's easier to make a complex graph that can dynamically change than it is to put multiple graphs on a single page.. so idk how to evolve that.
I really wouldn't say the mouse over I have is that important for the commits page really, either.

Multiple charts on the same page is not an issue. (I'm actually a big fan of multiple charts, it helps create a really slick dashboard feel)
The great thing about using crossfilter is that any filters you apply are applied to the data directly. This means it will update any of charts using the same data.

The reason I didn't include the tooltip/mouseover functionality is because I was just demonstrating crossfilter.

From: grats
lets make that commits page yours for the most part, I added it to the special thanks lol

I got really sick for 2 months trying to do a lot of work and get my moneys back (yes irl), spent it all. A lot on stats that just stayed in beta… users page is a great example of that with the random list of users that you can't click, lol.


[php]
echo "['x', $v2],";
echo "['Commits', $v1]";
[/php]
as seen in the page source


that's what it looks like right now as far as the dates and values.. it pulls from my cache table for all of the "commits"
that table is how you see it in client side which the x is

'yyyy-mm-dd hh:mm:ss',' etc
and the commits are just 'number'

so yours needs to have a json node per graph node instead it looks like, is this permanent? I can change how the cache database stores info.. I'll change it to be the permanent solution for the commits though rather than change it later again..

When I change that I'll be able to make the commits page with whatever you give me.. it'll be on the same page though not loaded from a .js file, so the disks don't have to be written to extra times.

Yeah, it does need JSON nodes to function correctly. No idea how your table is made, but the PHP would be something along the lines of:

[php]$return_arr = array();

$fetch = mysql_query("SELECT * FROM table");

while ($row = mysql_fetch_array($fetch)) {
$row_array['x'] = $row['time'];
$row_array['y'] = $row['commits'];

array_push($return_arr,$row_array);
}

echo json_encode($return_arr);[/php]

My other motivation behind using JSON is that it loads the data asynchronously, so everything else on the page also loads, but I understand if you want to keep it on the same page. Would save the JSON fetch.

I don't want any server-side graphing at all, the less the better. That's all extremely easy and it's also easy to output it as an image etc.. it's the day and age of making clients do their own work!


http://stats.grats.pw/test/graph.html

didn't really look at it, data is undefined, that's just a copy paste into that /test/ from your .zip you gave

My mistake, I accidentally zipped an older version with an error when the mouse goes beyond the x bounds. Updated it on both this post and the previous one

From: Richard
Yeah, it does need JSON nodes to function correctly. No idea how your table is made, but the PHP would be something along the lines of:

[php]$return_arr = array();

$fetch = mysql_query("SELECT * FROM table");

while ($row = mysql_fetch_array($fetch)) {
$row_array['x'] = $row['time'];
$row_array['y'] = $row['commits'];

array_push($return_arr,$row_array);
}

echo json_encode($return_arr);[/php]

My other motivation behind using JSON is that it loads the data asynchronously, so everything else on the page also loads, but I understand if you want to keep it on the same page. Would save the JSON fetch.




My mistake, I accidentally zipped an older version with an error when the mouse goes beyond the x bounds. Updated it on both this post and the previous one

lol still all data on the side
I think a get request would be more on the server, and I'll be able to do the php no worries.. I don't ever use mysql or while usually.. but I'll get my db cache exactly how you need it

main thing against a json file is I'd just have to be making a json file instead of a cache table for this in a database.. I guess we could test which works better, I guess I'll do that code real fast. inputting into a file and db isn't that much different

From: grats
lol still all data on the side
I think a get request would be more on the server, and I'll be able to do the php no worries.. I don't ever use mysql or while usually.. but I'll get my db cache exactly how you need it

main thing against a json file is I'd just have to be making a json file instead of a cache table for this in a database.. I guess we could test which works better, I guess I'll do that code real fast. inputting into a file and db isn't that much different

Sorry it's taken me so long, been pretty busy with exams, moving house, summer placement etc.

Little up, the table now follows the mouse like it did before, with pretty similar look-and-feel, the CSS is pretty messy though, so you might want to clean it up a bit to match it on your end if/when you integrate it into the main page. It zooms pretty nicely now, it's click and drag rather than the way it works on your current one. Then double click to reset the zoom (might want to have a little message covering this point).

Floating table:

Zoom selection:

There's probably some redundant code in there that happened when I refactored the commits table, but it works pretty fluidly. Unlike before, it now works on IE9+ and FF (there was a problem with the way it formatted dates in those browsers). That said, still works best in Chrome.

Two versions of the main graph file, one which sources data externally through a PHP file (graph.html), and one with the data included (graph-w-data.html), so you know how to insert your PHP depending on which one you choose.

Finally, I included the font file that you used, so you can edit the CSS accordingly.

From: Richard
Sorry it's taken me so long, been pretty busy with exams, moving house, summer placement etc.

Little up, the table now follows the mouse like it did before, with pretty similar look-and-feel, the CSS is pretty messy though, so you might want to clean it up a bit to match it on your end if/when you integrate it into the main page. It zooms pretty nicely now, it's click and drag rather than the way it works on your current one. Then double click to reset the zoom (might want to have a little message covering this point).

Floating table:

Zoom selection:

There's probably some redundant code in there that happened when I refactored the commits table, but it works pretty fluidly. Unlike before, it now works on IE9+ and FF (there was a problem with the way it formatted dates in those browsers). That said, still works best in Chrome.

Two versions of the main graph file, one which sources data externally through a PHP file (graph.html), and one with the data included (graph-w-data.html), so you know how to insert your PHP depending on which one you choose.

Finally, I included the font file that you used, so you can edit the CSS accordingly.

holy shit that's really nice imo
yea only reason the other one wasn't implemented already is cuz it didn't work lol
My brother is having his wedding in 11 days… AT MY HOUSE so I'm mega busy now, wasn't gonna be here and then the place they were going to have it was like "yea you can't use our pool"… I have a pool so yea

anyway that graph is awesome as hell, definitely going to get it up on that page. Doesn't look like it'll take much I'll probably just use the separate css so I can throw it on there easier for now

thanks a bunch, lmk any updates or tweaks you do in the future..

Idk if you guys have ever seen this http://metricsgraphicsjs.org/examples.htm#addons
Really nice api for this sort of graph.

From: tls
Idk if you guys have ever seen this http://metricsgraphicsjs.org/examples.htm#addons
Really nice api for this sort of graph.

I have a feeling it would get destroyed with this much data ;o

Just to second this supposition, I figure it would be considerably more compelling to list the quantity of submits every day, instead of every hour. At that point, you could have another question and diagram combo for review submits every hour arrived at the midpoint of out over a time of 30 days? Would enable you to see slants throughout the day.

From the specialized perspective, I truly don't know especially D3, yet I extravagant having a little go in any case, it'll be an incredibly essential precedent, yet we'll see! Do you require the majority of the diagramming to be customer side? I'm certain there are some PHP libraries accessible for this errand, however, would restrain the intelligence, all things considered.