SRL Archive - Development

Okay so ive been messing with sound detection.

Before i start, ive not looked into Simba plugins or anything yet, But as the dll will be written in C++ it should be ok.

So, something that intrested me lately is sound detection, Could be quite beneficial in all games aswell as RS (for aninmations and what not).

Ive successfully been able to compare 2 RuneScape sound files (ogg/midi and wav) Generate a signature, and compare the 2, with a given tolerance if you like.

Currently its reading audio from 2 files, Something thats proven to be a little tricky today is instead of reading the audio from a file, Have it capture it from the RS client.. (JVM, Smart, Brower).

The current idea is, basically storing all the pre calculated audio signatures into hashmap or db of some kind.

Since RS only has few hundred sounds this wont be a problem regarding false-positives.

Of course we would need to turn the RS volumes up, But this can probarly be filtered through the dll. and the audio go no further. which would be perfect.

There could be a much easyier way, and reading the RS client to see which sound was playing, but if were going to do that…we may aswell stick with reflection right?

Anyone thats tried/suceeded/failed in this topic, have a little post, and lets try to gather some information together to take this a step further.

oh also, for comparing the sound "Adding money to pouch" (runescape sound) its taking around 200ms, the code isnt even optimised yet, and thats reads 2 files.. dont forget in real life situation were only going to be reading 1 file, then comparing the hash in our db (which should be no more then 50ms)

Peace

Yay I have suggested this feature before. Tell me if I can help ;)

It should be good for detecting different animations.

But what would that help? Detecting that something was opened/ that we are under attack? Detecting what random we are on? Stuff like that?
Hmm, that would be helpful!
There are many ways to help on script! Detecting if your familiar timer went down… That would be a lot better, more accurate … And perhaps a little faster !

From: i3egoHan
Since RS only has few hundred sounds this wont be a problem regarding false-positives.

As far as I can tell, RuneScape has over 16,000 different sound resources. Even taking out ones that I'm pretty sure are music-related, it seems like there are still roughly 14,000 different sound effects and ambient effects for you to work through. That seems like a pretty significant amount of work.

How much memory usage/storage would it take to do such a thing? I think it would be great for everything as a "are you really doing [action here]" type failsafe but personally I hate listening to RS effects and music.

I wonder if there was a way to have the music on but not have to listen to it? (while still being able to hear other things of coarse)

From: Method
As far as I can tell, RuneScape has over 16,000 different sound resources. Even taking out ones that I'm pretty sure are music-related, it seems like there are still roughly 14,000 different sound effects and ambient effects for you to work through. That seems like a pretty significant amount of work.

Of course we dont need all the sounds.

Ill work through the most common like Mining, Woodcutting, Fighting… Just all the basic most used ones.

Yes it would be perfectly for a alternative for animation.. I know people have said sound detection isnt the way forward, but it will be a great addition to simba.

From: johnnimms13
How much memory usage/storage would it take to do such a thing? I think it would be great for everything as a "are you really doing [action here]" type failsafe but personally I hate listening to RS effects and music.

I wonder if there was a way to have the music on but not have to listen to it? (while still being able to hear other things of coarse)

Written in c++ and dealing with such small audio sizes, it should run very well.

Ive developed alot of applications in c++ but sadly none dealing with recording audio, so its something i have to research.

Of course, check my main post, it should be able to filter the clients audio output through and only have the dll read it, and not going through your sound card.

It depends how the JVM handles audio, the closest ive worked to the client is reflection and bcel.

This sounds really cool man! Please let me know if you could use some help collecting sounds or w/e

This is useless if running more than 1 Simba at a time unless you can isolate the sounds that go out of each client.

Question is what is the CPU cost to pay for this, if it increases load too much, its not really worth it, also you can hear other player's sounds like when fighting something. Not just your sounds and cities like varrock now have ambient noise, good luck with this.

From: ashaman88
This sounds really cool man! Please let me know if you could use some help collecting sounds or w/e

Sure thing, thanks.

From: DemiseScythe
This is useless if running more than 1 Simba at a time unless you can isolate the sounds that go out of each client.

Question is what is the CPU cost to pay for this, if it increases load too much, its not really worth it, also you can hear other player's sounds like when fighting something. Not just your sounds and cities like varrock now have ambient noise, good luck with this.

Of course, read the main post, i mentioned it twice about isolating sounds via specific process.

CPU will obviously depend on often its checking and which sounds.

Really? You can here others sounds? Is there no way to turn those volumes off?

I dont think city noise qualifies as music, I think its incorporated into the special effect sounds, stand in middle of varrock and turn your sound up

Some very old but maybe helpful threads:
http://villavu.com/forum/showthread.php?t=15851

A guy named Shocker made a sound detection plugin for SCAR it was in beta stages, but I believe it did work. He has only 3 posts and hasn't' been around in years, but you could try and contact him if you care to:
http://villavu.com/forum/member.php?u=41

Good luck.
And yeah isolating sounds would be nice, also still hearing the sounds even when volume is off. (As in has volume on on runescape, but on windows 7 at least you can disable/mute sounds from certain programs).

Not 100% how useful it will be or if it's worth making (might be some more useful things you can use your time on) but go ahead, it will be nice and interesting!

Seeing I am a soundguy, this will be very useful. Sound signals a lot of processes such as end-of-a-kill, tree-has-fallen, etc.


13ego: How do you compare them? When i was working with Shocker on this, comparing only the first 40 msecs of any sound was enough to distinguish them.

I was thinking about this when writing my fishing script but I though it would be to complex. It would be a great feature though

From: YoHoJo
Some very old but maybe helpful threads:
http://villavu.com/forum/showthread.php?t=15851

A guy named Shocker made a sound detection plugin for SCAR it was in beta stages, but I believe it did work. He has only 3 posts and hasn't' been around in years, but you could try and contact him if you care to:
http://villavu.com/forum/member.php?u=41

Good luck.
And yeah isolating sounds would be nice, also still hearing the sounds even when volume is off. (As in has volume on on runescape, but on windows 7 at least you can disable/mute sounds from certain programs).

Not 100% how useful it will be or if it's worth making (might be some more useful things you can use your time on) but go ahead, it will be nice and interesting!

Great thanks for the information.

From: WT-Fakawi
Seeing I am a soundguy, this will be very useful. Sound signals a lot of processes such as end-of-a-kill, tree-has-fallen, etc.


13ego: How do you compare them? When i was working with Shocker on this, comparing only the first 40 msecs of any sound was enough to distinguish them.

Well im computing a hash for given audio (OGG mainly right?), Then when were trying to compute hashes in real time and compare them, even if the hashes/signatures or somewhat diffrent we can use a tolerance in the algorithms using to compare them.

But last night i rethough it over, since the sounds or so small, maybe we could read so many bytes of the audio, and thus comparing a octet of bytes, trouble with this method is that if we get any abient noise, it may really mess up the match.

How was shocker doing this before? Generating signatures? Comparing bytes?

Peace

We did so by comparing the bytestream. I forgot the exact fileformat, but I believe it came out 8 kHz/8 bits, so 8000 sampes per second at max amplitude of -127 to 128 (or 0 - 256 unsigned). I compared only the first 40 msecs, so 8000/25 = 320 samples. That was sufficient to dicsciminate every sound.

You can turn off all the other sounds such as music and ambient noises.

I don't even think you'd need a dll if you just set those sound settings properly. Is a nice idea, let me know about anything I could potentially help with.

From: WT-Fakawi
We did so by comparing the bytestream. I forgot the exact fileformat, but I believe it came out 8 kHz/8 bits, so 8000 sampes per second at max amplitude of -127 to 128 (or 0 - 256 unsigned). I compared only the first 40 msecs, so 8000/25 = 320 samples. That was sufficient to dicsciminate every sound.

Right ill look into that, Looks like it would be faster, but i always thought there would be multiple matches, i guess we would have a pretty small database though… (small being below 10k which is more then we would need)

Was it being written as a SCAR plugin? Any code around which we could take a look at?

From: Harry
You can turn off all the other sounds such as music and ambient noises.

I don't even think you'd need a dll if you just set those sound settings properly. Is a nice idea, let me know about anything I could potentially help with.

Thanks for the info harry, Yeah exactly, all we need is our players effects volume.

Well the reason for the DLL would be because i dont have the knowledge in any other language apart from C++ to achive this, since one of the most benefical features would be too isolating the sounds from your sound card.

Again it depends on how JVM (or maybe the browser) handles output sounds, which im about to research now.

Oh also just thought of something, Isolating sounds from windows applications is pretty easy, So maybe it would be much easyier to start with the downloadable rs client? Defently alot of research to shed some more light on this.

One of the most common drivers is AC97, though I suspect you will get best results with asio4all, which is opensource afaik, and would provide you an interface to the sound output.

Thinking about it could you not use a private server? forgive me if it sounds nooby but they have all the sounds prebuilt dont they?

Obtaining the sounds isnt really a problem, Could always capture sound going to your audio card.

Okay, im looking at multiple things, comparing the bytestream will only work if they match exactly, if the volume is higher/lower, how every this is much faster.

Still looking into faster algorithms for signatures.

Okay well after a fair bit of research there are a few more methods to detect, obviously we need speed over accuracy (since these methods are being used for hundreds of thousands oif not millions of signatures)

We can convert the audio to a spectogram, and then use computer vision techniques to find simularitys in the audio.

We can read bytes of the audio (as WT stated), this is by far the fastest, but if the audio isnt almost exactly the same it will more then likely fail.

There are a number of signature librarys, which analyse the audio but these require a few seconds to process.

Im looking at computing our own custom signatures, containing some simple properties of the audio. The next step is researching more into the client.. i also see we can choose mono or stereo which makes things some what easyier.

Grab it mono, that's twice as fast. If I grab the audio, it is always exactly the same waveform. Do you want me to provide you with sample accurate sounds? And if so, in what format?

Some sounds are randomly pitched like tree cutting or mining, while others are static. Pitch will be a problem because you will need to upsample the wave, and then perform analysis.

A fourier-analysis would be best in this stage, and although I can prove with stone-old Amiga Basic I once mastered this….I do not remember :smile:

Do you have some of the sound effects? The main ones like cutting, mining etc?

What are the original sounds? OGG for effects?

OGG or WAV is fine.

Depending which route we take, we may need to decode and work with the raw audio, although this isnt really a problem since theres alot out there, (ogg decoder is simple)

Well i could start throwing something together maybe a POC first (for easy testing) then build it into a DLL, the main problem im having is capturing the sound from the JVM and nothing else.

We could anaylise all sounds, but this would be so much better if we could isolate it, whats your thoughts on this?

I still keep meaning to look at the downloadable client as that just may use waveOut().

Ok so i havent been thinking straight, as mentioned in first post at first sight i thought well ok we need to analyze sounds then compare them to our database.. this IS NOT the case is it.

Its actually much simpler, We only need to detect specific sounds.

For example i plan on writing multiple functions into the DLL such as

bool IsMining(); // return true if mining
bool WaitForMining(int timeOut); // returns when mining sound no longer exist, or timeout MS passed.

Then i can start creating universal functions such as, WaitForSound('SOUND_CHARACTERISTICS');

Since ive look into it this way it really isnt very hard, Plan on having a command line app which scripters can use to read characteristics of any audio file, then use these just like you would DTM's/BMP's.

From: i3egoHan
Ok so i havent been thinking straight, as mentioned in first post at first sight i thought well ok we need to analyze sounds then compare them to our database.. this IS NOT the case is it.

Its actually much simpler, We only need to detect specific sounds.

For example i plan on writing multiple functions into the DLL such as

bool IsMining(); // return true if mining
bool WaitForMining(int timeOut); // returns when mining sound no longer exist, or timeout MS passed.

Then i can start creating universal functions such as, WaitForSound('SOUND_CHARACTERISTICS');

Since ive look into it this way it really isnt very hard, Plan on having a command line app which scripters can use to read characteristics of any audio file, then use these just like you would DTM's/BMP's.

but you do need to have some kind of database to compare the sounds, right?

I think something like "SoundPresent" will already be plenty enough for now:)
Comparing sounds might be too hectic.

From: WT-Fakawi
but you do need to have some kind of database to compare the sounds, right?

Not exactly, This is what i thought when the topic came to mind.

The best and fastest way would be to store the signatures/characteristics of the audio inside the DLL.

Pro's: Tad faster.
Con's: Will need higher maintenance. (if sounds change or become unstable)

I could create a hashmap database or something which is insanely fast, which ill probarly do.

But it isnt needed, The plan i have right now, for the most functionality, will be too create a function which gets basic characteristics from the audio, like pitch, length etc, we can then have a tool which 'scripters' use to get these characteristics… Then the dll can be called with something like SoundIsPlaying('audio_data_here')

Of course the audio_data_here will be compared against the playing audio via the DLL.

The DLL will first contain basic skill functions, then a universal function can and will be added.

How much CPU will this use? Hah surprisingly much less the FindColors in a loop, Since we only read audio when theres audio coming from the game, else were just idle. and since the sounds are so small.. it will hardly touch the CPU/Ram.

I should be able to get basic skill functions implemented pretty fast such as Mining, Woodcutting and what ever other skills have effects which can be used to detect animation and what not.

So it works like this?

function WaitWhileMining:Boolean;
var
  MiningSound:TSound;
  WaitTime,PitchTolerance:Integer;
begin
  WaitTime := 1000 + random(200);
  MiningSound := SoundFromString('gaehhuahulrhlarghuli435sr89spog');
  while(WaitForSound(MiningSound,PitchTolerance,WaitTime) do
  begin
    //do stuff
  end;
  FreeSound(MiningSound);
  Result := True;
end;
        

From: masterBB
So it works like this?

function WaitWhileMining:Boolean;
var
  MiningSound:TSound;
  WaitTime,PitchTolerance:Integer;
begin
  WaitTime := 1000 + random(200);
  MiningSound := SoundFromString('gaehhuahulrhlarghuli435sr89spog');
  while(WaitForSound(MiningSound,PitchTolerance,WaitTime) do
  begin
    //do stuff
  end;
  Result := True;
end;
        

Exactly!

Although as mentioned ill first implement and release skill specific functions. Then the universal sound function you demostrated above. Just to see how it handles ambient noises etc, since the sound is hardly going to be exactly the same.

From: i3egoHan
Exactly!

Although as mentioned ill first implement and release skill specific functions. Then the universal sound function you demostrated above. Just to see how it handles ambient noises etc, since the sound is hardly going to be exactly the same.

I had been thinking about it too. It would be so cool if you would create this for simba. I hope it will be opensource, would be nice to see how sound comparison works. Great idea.

I'm not sure if a hash is all you want. Mainly because you're comparing to moving data. I know more about computer vision that sound, so I'd probably suggest a feature vector for sound, but I'm not sure if that's even applicable.

Right now im thinking pitch detections may be a good way to go, Since its much faster.

And when we have specific functions like IsMining(), its easy to check if the mining sound is active in x amount of MS.

It would be much diffrent if we was trying to match 1 sound against 1000+ sounds.. but thats not the case here.

From: i3egoHan
Right now im thinking pitch detections may be a good way to go, Since its much faster.

And when we have specific functions like IsMining(), its easy to check if the mining sound is active in x amount of MS.

It would be much diffrent if we was trying to match 1 sound against 1000+ sounds.. but thats not the case here.

Knowing you (and of coarse trying my hand at encouragement) I want to tell you this: this should come out great. I can't wait! :)

I'm willing to help where I can, I have some spare webdomains I can give you access to if you have any idea on how to add databases online to call from? Unless the DB will be on simba itself? But surely that'l be a huge load when simba is booting up?

I dunno :P

Holy Pizza, this sounds hard to do….

But would be very effective! You have my support! (But Not My Help :p)

From: i3egoHan
Right now im thinking pitch detections may be a good way to go, Since its much faster.

And when we have specific functions like IsMining(), its easy to check if the mining sound is active in x amount of MS.

It would be much diffrent if we was trying to match 1 sound against 1000+ sounds.. but thats not the case here.

Atm PlaySound only supports .wav files, I tried using .ogg, .mp3, .flac, etc. Only .wav works as far as I know. If we could port support for mp3 files that would be great. Iif we can get mp3 support I'll make a music manager.

mmm - http://wiki.lazarus.freepascal.org/ACS

ovoplayer mmmm

Where not trying to play sounds, where trying to detect them.

The only pitfall on this is that without doing driver level code, only certain sound cards are supported.

Your sound card must support a loop back device to capture the sound back.
There are other ways but it's very low level machine code

What you should look into is more complex than a mere hash. Wizzup was on the right track when he discussed feature extraction… what you need is a classifier driven by acoustic signatures. The idea is almost like Shazam, in that you're identifying some audio being played by creating a fingerprint and using this to scan a database. Look into Fourier analysis for some help. Here's a primer that briefly covers Fourier analysis and discrete transforms: http://www.cs.ucl.ac.uk/teaching/GZ05/03-fourier.pdf

Reading: http://en.wikipedia.org/wiki/Acoustic_fingerprint
http://www.music.mcgill.ca/~ich/classes/mumt611_08/fingerprinting/cano05review.pdf
http://ismir2002.ismir.net/proceedings/02-fp04-2.pdf
http://www.ee.columbia.edu/~dpwe/papers/Wang03-shazam.pdf

The main problem I see here is getting the sounds only from RuneScape/Other program. If there is something in WINAPI that covers this, then ok, but how about linux support? I know Windows 7 and Vista have every programs volume controls in Volume Mixer, which would suggest there is some possibility of distinguishing sounds of different programs.

From: Frement
The main problem I see here is getting the sounds only from RuneScape/Other program. If there is something in WINAPI that covers this, then ok, but how about linux support? I know Windows 7 and Vista have every programs volume controls in Volume Mixer, which would suggest there is some possibility of distinguishing sounds of different programs.

Pulseaudio also allows fine grained control of sound per process. I'm pretty sure you can even send the sound of a specific process to your (own!) sink and just analyse is there.

Yep, you can:

pactl move-sink-input #N SINK|SOURCE

And you can create your own pulseaudio sink.

Still, WINAPI has no support for this :) That leaves only one viable option, hooking the waveOut.

EDIT: 2222th post. Yay!

I think you can edit public int write(byte[] b, int off, int len) in com/sun/media/sound/MixerSourceLine the same way smart does with Canvas. Probably a different file for linux and mac though. I actually did this in my loader and that method was being spammed like hundreds of times per second.

This sounds like something you'd have to create a driver for (to listen for sounds and filter them for each process).

This SOUNDS great ;) lol.
I really hope you succeed in doing this as it will be a big help with knowing if a char is doing something.
For example when chopping some trees, it's hard to see the player animating. There for not knowing if it is chopping and having to move the screen around to find if the char is or not. Using a sound to know your char has started chopping and also know if it is still chopping, would be extremely helpful.

Keep up the good work, i3egoHan. :thumbsup:

Should be done from within SMART (would at least be the easiest way). Pretty sure there isn't a simple way on Windows to detect the sound from a specific window, you can just read from the sound buffer which includes all sounds.

Uhm, very interesting… I may need this later