SRL Archive - Reflection (Archive)

Memory management for Reflection


Wizzup?

2011-02-09 16:39:50

Hey there,

In the light of the Simba "code:8" error some people have been getting, I came up with an idea that might help reduce leaks (read: memory leaks) in Reflection. Why? Because I believe that there are memory management issues left in the Reflection include. Code "8" means out of memory; so something must be leaking… And since as far as I know this only happens with Reflection-based scripts, something might just be wrong with our reflection include. I may be completely wrong, but this idea will prove whether I am or not.

As you may (or may not) know, bitmaps (and dtms, but I'll just say bitmaps) in Simba are managed. That means, you can allocate them. (Call CreateBitmap/LoadBitmap/CreateBitmapFromString); and Simba will return you an integer. That's actually pretty weird if you've done coding in an other language, because a bitmap isn't really an integer, right? Normally, you'd expect it to return a "TBitmap" or similar bitmap instance. (For the record: you can actually get the object associated with the bitmap with GetMufasaBitmap) We try to avoid this in Simba for two reasons: To make memory management easier (generally) and to prevent crashes (working with non existent bitmaps) Obviously this only works partially because you'll still have to call FreeBitmap; Simba will only free unfreed bitmaps at the end of the script because it cannot know if you still want to use them before the script stops; but that is another story. It could (in theory, it doesn't do this atm) however, let you know what bitmaps are currently in use. And if you have no bitmaps or dtms; then this number should generally be 0. (Unless you're in a function that uses that bitmap, but you get my point)

To give you an idea on how this is done in Simba: (I'll only include the body of the important functions)


TMBitmaps = class(TObject)
protected
Client : TObject;
FreeSpots : Array of integer;
BmpArray : TMufasaBmpArray;
BmpsCurr,BmpsHigh,FreeSpotsHigh,FreeSpotsLen : integer;
function GetNewIndex : integer;
public
function GetBMP(Index : integer) : TMufasaBitmap;
property Bmp[Index : integer]: TMufasaBitmap read GetBMP; default;
function CreateBMP(w, h: integer): Integer;
function AddBMP(_bmp: TMufasaBitmap): Integer;
function CopyBMP( Bitmap : integer) : Integer;
function CreateMirroredBitmap(bitmap: Integer; MirrorStyle : TBmpMirrorStyle): Integer;
function CreateBMPFromFile(const Path : string) : integer;
function CreateBMPFromString(width,height : integer; Data : string) : integer;overload;
function CreateBMPFromString(BmpName : string; width,height : integer; Data : string) : integer;overload;
procedure FreeBMP( Number : integer);
constructor Create(Owner : TObject);
destructor Destroy;override;
end;

{ snip }

function TMBitmaps.GetNewIndex: integer;
begin
if BmpsCurr < BmpsHigh then
begin;
inc(BmpsCurr);
Result := BmpsCurr;
end else if (FreeSpotsHigh > -1) then
begin;
Result := FreeSpots[FreeSpotsHigh];
dec(FreeSpotsHigh);
end else
begin;
SetLength(BmpArray, BmpsHigh + 6);
BmpsHigh := BmpsHigh + 5;
inc(BmpsCurr);
Result := BmpsCurr;
end;
end;

function TMBitmaps.GetBMP(Index: integer): TMufasaBitmap;
begin
Result := nil;
if (Index >= 0) and (Index <= BmpsCurr) then
if BmpArray[Index] <> nil then
Result := BmpArray[Index];
if Result = nil then
raise Exception.CreateFmt('The bitmap[%d] does not exist',[Index]);
end;

function TMBitmaps.CreateBMP(w,h : integer): Integer;
begin
result := GetNewIndex;
BmpArray[Result] := TMufasaBitmap.Create;
BmpArray[Result].SetSize(w,h);
BmpArray[Result].Index:= Result;
end;

function TMBitmaps.AddBMP(_bmp: TMufasaBitmap): Integer;
begin
Result := GetNewIndex;
BmpArray[Result] := _bmp;
BmpArray[result].Index:= Result;
end;

procedure TMBitmaps.FreeBMP(Number: integer);
var
ToDestroy : TMufasaBitmap;
begin
ToDestroy := GetBMP(Number);
if Number = BmpsCurr then
Dec(BmpsCurr)
else
begin;
inc(FreeSpotsHigh);
if FreeSpotsHigh = FreeSpotsLen then
begin;
inc(FreeSpotsLen);
SetLength(FreeSpots, FreeSpotsLen);
end;
FreeSpots[FreeSpotsHigh] := Number;
end;
//Just for testing purposes
{$ifdef mDebug}
if ToDestroy.BmpName = '' then
mDebug(Format('BMP[%d] has been freed.',[number]))
else
mDebug(Format('BMP[%s] has been freed.',[ToDestroy.BmpName]));
{$endif}
ToDestroy.Free;
BmpArray[number] := nil;
end;

So, simply put: It's an array of the references to the real bitmap instances; indexed by (obviously) an integer. Which is funnily enough also the integer that you use in your script.

I want to propose the same for Reflection. Although I must admit I do not know the complete details as to how the SmartGetFieldObject (or similar) functions work; I think it should be quite easy to add a system like this. I'm willing to write this as well, but I'll have to gain some more understanding as to how SMART objects work. It would also be a big advantage (IMO) if we could (or, must?) ``label'' the objects, much like here: Debugging memory leaks in Simba

Oh, I just looked in the SMART source code and I see all the ``objects'' are just integers as well, which means we should be able to manage them quite well. Perhaps SMART already has a method to manage this, just not for debugging. (If the integers are indices; if they aren't, we can still manage them; but SMART perhaps doesn't. I wouldn't be surprised if it didn't since Java manages memory as well…)

Any thoughts?

mormonman

2011-02-09 17:13:03

I like this idea a lot, and I would love to see it done. I'm not sure if the integers are indices, but it would make sense if they were. Labeling is a good idea, so implement that as well.

Nava2

2011-02-09 18:03:53

I've talked about this a lot actually…

I never got around to writing code for it. I definitely support the idea.

It would involve writing a lot or wrappers though. This is the major downside of this. Unless its implemented internally into SMART.

Wizzup?

2011-02-09 18:05:52

From: Nava2
I've talked about this a lot actually…

I never got around to writing code for it. I definitely support the idea.

It would involve writing a lot or wrappers though. This is the major downside of this. Unless its implemented internally into SMART.


I don't think wrappers would be an issue. Also, why exactly do you think this takes a lot of wrappers? There are only a few methods that actually create or free objects, as far as I could see.

pyroryan

2011-02-09 18:59:23

From: Nava2
I've talked about this a lot actually…

I never got around to writing code for it. I definitely support the idea.

It would involve writing a lot or wrappers though. This is the major downside of this. Unless its implemented internally into SMART.


Wrappers should be standard anyways. I freaking couldn't even code something to handle the new menus until I made a wrapper for nodes. Without a wrapper, the code looks a lot messier and reduces comprehension, etc.

Nava2

2011-02-09 19:58:01

From: pyroryan
Wrappers should be standard anyways. I freaking couldn't even code something to handle the new menus until I made a wrapper for nodes. Without a wrapper, the code looks a lot messier and reduces comprehension, etc.


I think you are missing what a wrapper usually is. :P

Anyways, I'm working on this. I'll have it done for Monday most likely.

https://github.com/Nava2/srl-reflection/tree/smart-memory_management for those who want to keep up with it.

pyroryan

2011-02-09 22:15:26

From: Nava2
I think you are missing what a wrapper usually is. :P

Anyways, I'm working on this. I'll have it done for Monday most likely.

https://github.com/Nava2/srl-reflection/tree/smart-memory_management for those who want to keep up with it.


Perhaps, but I never had a formal programming education. I'd define a wrapper as something that extends the "functionality" of an object. Note that functionality is in quotes because generally the same functionality could be achieved without a wrapper. Its just cleaner with one.
Please educate me on the matter :)

Coh3n

2011-02-10 01:06:16

In case you reflection devs are interested.

http://villavu.com/forum/showpost.php?p=777702&postcount=55

Nava2

2011-02-10 02:27:50

From: pyroryan
Perhaps, but I never had a formal programming education. I'd define a wrapper as something that extends the "functionality" of an object. Note that functionality is in quotes because generally the same functionality could be achieved without a wrapper. Its just cleaner with one.
Please educate me on the matter :)


A wrapper typically denotes just renaming a function by creating another method to call a different one.

In this case, we are doing that, but adding extra functionality as well. You are correct in what we are doing, but a bit off in terminology.
http://en.wikipedia.org/wiki/Wrapper_function

No harm, no foul. :)

Thanks for the info Coh3n, I can look into this this weekend as well.

Markus

2011-02-10 07:56:55

Code 8: out of memory.
That also explains why only reflection scripts are having the problem. There must be some leak in the include.
Didn't Yakman write a SCAR bitmap/dtm leak detector once?

Flight

2011-02-15 05:49:05

Well I've gotten to the point of crashing Simba a few times through unfreed bitmaps and I don't use SMART/Reflection/or even Runescape for that matter. But I think mine leak is something I'm doing wrong rather than Simba itself. I've tried just about everything, and I think I'll take a look at some more example scripts, but if I really can't pinpoint my problem I'll be sure to make a post somewhere with the small script I've created, as I'm sure most people here can easily tell me what I'm doing wrong.

Coh3n

2011-02-15 10:23:27

I feel like the may be a leak in MSI also, but I can't think of where it may be. I can run two reflection only scripts until they stop, but MSI can't run for more than a couple hours before getting the error. Doubt this will help, but here it is.


E: Nvm, just happened with a reflection only script as well.

Wizzup?

2011-02-15 13:24:08

Nearly finished. I just need to decide whether I will force you guys to use a modified SMART (just the names modified) or if I will change all the reflection code. The first one is better, IMO.

I need to improve the performance of remove_object. Probably needs a different data structure…

E: Made it perform a lot faster with the '__U' hack…


{$IFNDEF REFLECTION_MEMORY_DEBUG}
program Refl_memory_test;
{$ENDIF}

(*
Basic memory management system.

TODO:
- Could do with a ``cleanup'' function that truncates the array.
If you use a lot of objects now, it basically ``pollutes'' the system in
the sense that it will be a lot slower as long as it runs. (arrays will
stay large)
- Make it actually use SMART. Either with a SMART method naming hack,
or with a refactor of the reflection include.
- Add support for settting objects names.

XXX DONT NAME YOUR OBJECTS __U XXX

*)


type
tobject_data = record
name: string;
path: string;
objref: integer;
end;
var
objects_free_spots: array of integer;
managed_smart_objects: array of tobject_data;

procedure _SmartFreeObject(ref: integer);
begin end; { XXX: DUMMY FUNCTION }

procedure mem_manage_init;
begin
SetLength(objects_free_spots, 0);
SetLength(managed_smart_objects, 0);
end;

function object_get_new_index(): integer;
begin
if Length(objects_free_spots) > 0 then
begin
Result := objects_free_spots[High(objects_free_spots)];
SetLength(objects_free_spots, High(objects_free_spots));
end
else
begin
SetLength(managed_smart_objects, Length(managed_smart_objects) + 1);
Result := High(managed_smart_objects);
end;
end;

function object_to_string(obj: tobject_data): string;
begin
result := Format('Object< ref:%d, path: %s, name: %s', [obj.objref,
obj.path, obj.name]);
end;

procedure add_object(ref: integer; path: string);
var
index: integer;
begin
index := object_get_new_index();

managed_smart_objects[index].objref := ref;
managed_smart_objects[index].path := path;
managed_smart_objects[index].name := 'No Name';

{$IFDEF REFL_MEMORY_VERBOSE}
writeln('Allocated: ' + object_to_string(managed_smart_objects[index]));
{$ENDIF}
end;

procedure remove_object(ref: integer);
var
i, j: integer;
existant: boolean;
begin
for i := 0 to high(managed_smart_objects) do
begin
if managed_smart_objects[i].name = '__U' then
continue;
{ existant := false;
for j := 0 to high(objects_free_spots) do
if i = objects_free_spots[j] then
begin
existant := true;
break; // Skip ones that don't exist
end;
if existant then
continue; }

if managed_smart_objects[i].objref= ref then
begin
managed_smart_objects[i].objref := 0;
managed_smart_objects[i].path := '';
managed_smart_objects[i].name := '__U';

setlength(objects_free_spots, length(objects_free_spots) + 1);
objects_free_spots[high(objects_free_spots)] := i;

{$IFDEF REFL_MEMORY_VERBOSE}
writeln('Freespot added: ' +
inttostr(objects_free_spots[high(objects_free_spots)]));
writeln('Freeing: ' + object_to_string(managed_smart_objects[i]));
{$ENDIF}

_SmartFreeObject(ref);

// We are done
exit;
end;
end;

{ If we get to this point, you've done something utterly wrong ;-) }
RaiseException(erCustomError, 'Invalid Object Free: ' +
object_to_string(managed_smart_objects[i]));
end;

procedure list_allocated_objects();
var
i, j: integer;
existant, any: boolean;
begin
any := false;
for i := 0 to high(managed_smart_objects) do
begin
existant := false;
for j := 0 to high(objects_free_spots) do
if i = objects_free_spots[j] then
begin
existant := true;
break; { Skip ones that don't exist }
end;
if existant then
continue;

any := true;

writeln('Currently allocated: ' + object_to_string(managed_smart_objects[i]));
end;

if not any then
writeln('Nothing allocated!');
end;

procedure SmartFreeObject(ref: integer);
begin
remove_object(ref);
end;

function SmartGetFieldObject(parent: integer; path: String): integer;
begin
end;

function SmartGetFieldArray3DObject(parent: integer; path: String; x ,y, z: integer): integer;
begin
end;

function SmartGetFieldArray2DObject(parent: integer; path: String; x ,y: integer): integer;
begin
end;

function SmartGetFieldArrayObject(parent: integer; path: String; index: integer): integer;
begin
end;

{$IFNDEF REFLECTION_MEMORY_DEBUG}
var i: integer;
begin
mem_manage_init;

for i := 0 to 1000 do
add_object(i, 'wat');

writeln('done adding');

for i := 0 to 1000 do
remove_object(i);

writeln('done removing');

list_allocated_objects()
end.
{$ENDIF}

Coh3n

2011-02-15 13:28:03

Awesome. I don't care what you need me to do, I'll do it. This is rather annoying when I try to run MSI. I'm just about to run my army with MSI without reflection, so we'll see if there actually is something in MSI as well.

Gu3st...

2011-02-15 13:50:51

The __U hack? Sounds interesting. What's that?

Nava2

2011-02-15 16:51:38

if managed_smart_objects[i].name = '__U' then
continue;

Rather than this, why not set the objref to -1? Integer comparison is much faster than string comparison.

Wizzup?

2011-02-15 16:52:37

From: Nava2
if managed_smart_objects[i].name = '__U' then
continue;

Rather than this, why not set the objref to -1? Integer comparison is much faster than string comparison.


Because that is prone to give false positives? The object reference is an unsigned integer, and we use it as a signed integer. IOW it could be negative.

BenLand100

2011-02-15 17:35:13

From: Wizzup?
Because that is prone to give false positives? The object reference is an unsigned integer, and we use it as a signed integer. IOW it could be negative.

Won't ever be a -1, because its a memory address, and address $ffffffff makes no sense. It is also never 0. (N.B. SMART sometimes returns a 0 or -1 value when an error occurred), you should not free those, obviously.)

Wizzup?

2011-02-16 18:29:38

Untested, but should work…

Save this code as reflection_management.simba; then include it before you include SRL. Also, before you include it:

{$DEFINE REFLECTION_MEMORY_DEBUG}

And you probably don't want to include it always, so, as the top of a script.


{$IFDEF DEBUG_REFLECTION}
{$DEFINE REFLECTION_MEMORY_DEBUG}
{$I reflection_management.simba}
{$ENDIF}

Code here, still lacks actually setting a name (but path is shown); and it's still not super optimised, but it should be fast enough at least for a test run.

{$IFNDEF REFLECTION_MEMORY_DEBUG}
program Refl_memory_test;
{$loadlib libsmart}
{$ENDIF}

(*
Basic memory management system.

TODO:
- Could do with a ``cleanup'' function that truncates the array.
If you use a lot of objects now, it basically ``pollutes'' the system in
the sense that it will be a lot slower as long as it runs. (arrays will
stay large)
- Add support for settting objects names.

XXX DONT NAME YOUR OBJECTS __U XXX

*)


type
tobject_data = record
name: string;
path: string;
objref: integer;
end;
var
objects_free_spots: array of integer;
managed_smart_objects: array of tobject_data;

procedure mem_manage_init;
begin
SetLength(objects_free_spots, 0);
SetLength(managed_smart_objects, 0);
end;

function object_get_new_index(): integer;
begin
if Length(objects_free_spots) > 0 then
begin
Result := objects_free_spots[High(objects_free_spots)];
SetLength(objects_free_spots, High(objects_free_spots));
end
else
begin
SetLength(managed_smart_objects, Length(managed_smart_objects) + 1);
Result := High(managed_smart_objects);
end;
end;

function object_to_string(obj: tobject_data): string;
begin
result := Format('Object< ref:%d, path: %s, name: %s', [obj.objref,
obj.path, obj.name]);
end;

procedure add_object(ref: integer; path: string);
var
index: integer;
begin
index := object_get_new_index();

managed_smart_objects[index].objref := ref;
managed_smart_objects[index].path := path;
managed_smart_objects[index].name := 'No Name';

{$IFDEF REFL_MEMORY_VERBOSE}
writeln('Allocated: ' + object_to_string(managed_smart_objects[index]));
{$ENDIF}
end;

procedure remove_object(ref: integer);
var
i, j: integer;
existant: boolean;
begin
for i := 0 to high(managed_smart_objects) do
begin
if managed_smart_objects[i].name = '__U' then
continue;
{ existant := false;
for j := 0 to high(objects_free_spots) do
if i = objects_free_spots[j] then
begin
existant := true;
break; // Skip ones that don't exist
end;
if existant then
continue; }

if managed_smart_objects[i].objref= ref then
begin
managed_smart_objects[i].objref := 0;
managed_smart_objects[i].path := '';
managed_smart_objects[i].name := '__U';

setlength(objects_free_spots, length(objects_free_spots) + 1);
objects_free_spots[high(objects_free_spots)] := i;

{$IFDEF REFL_MEMORY_VERBOSE}
writeln('Freespot added: ' +
inttostr(objects_free_spots[high(objects_free_spots)]));
writeln('Freeing: ' + object_to_string(managed_smart_objects[i]));
{$ENDIF}

_SmartFreeObject(ref);

// We are done
exit;
end;
end;

{ If we get to this point, you've done something utterly wrong ;-) }
RaiseException(erCustomError, 'Invalid Object Free: ' +
inttostr(ref)));
end;

procedure list_allocated_objects();
var
i, j: integer;
existant, any: boolean;
begin
any := false;
for i := 0 to high(managed_smart_objects) do
begin
existant := false;
for j := 0 to high(objects_free_spots) do
if i = objects_free_spots[j] then
begin
existant := true;
break; { Skip ones that don't exist }
end;
if existant then
continue;

any := true;

writeln('Currently allocated: ' + object_to_string(managed_smart_objects[i]));
end;

if not any then
writeln('Nothing allocated!');
end;

procedure SmartFreeObject(ref: integer);
begin
remove_object(ref);
end;

function SmartGetFieldObject(parent: integer; path: String): integer;
begin
Result := _SmartGetFieldObject(parent, path);
add_object(Result, path);
end;

function SmartGetFieldArray3DObject(parent: integer; path: String; x ,y, z: integer): integer;
begin
Result := _SmartGetFieldArray3DObject(parent, path, x, y, z);
add_object(Result, path);
end;

function SmartGetFieldArray2DObject(parent: integer; path: String; x ,y: integer): integer;
begin
Result := _SmartGetFieldArray2DObject(parent, path, x, y);
add_object(Result, path);
end;

function SmartGetFieldArrayObject(parent: integer; path: String; index: integer): integer;
begin
Result := _SmartGetFieldArrayObject(parent, path, index);
add_object(Result, path);
end;

{$IFNDEF REFLECTION_MEMORY_DEBUG}
var i: integer;
begin
mem_manage_init;

for i := 0 to 1000 do
add_object(i, 'wat');

writeln('done adding');

for i := 0 to 1000 do // this is slow
remove_object(i);

writeln('done removing');

list_allocated_objects()
end.
{$ENDIF}

YOU NEED A MODIFIED SMART: http://vila.villavu.com/libsmart.dll
I didn't get it to work in Wine (SMART, that is), but Coh3n confirmed it loaded for him.

E: Also, nava. If you want to improve performance (although negligible) you could also just add a boolean to the tobject_data. "in_use: boolean;"

E2: Call list_allocated_objects(); in Progress report or so; if it prints something; you probably have a leak.

Wizzup?

2011-02-16 18:37:33

You'll probably want to call {$loadlib smart} before including it, though. (Although I imagine Simba may have already loaded it)
Don't know how this fits in the MSI structure?

Coh3n

2011-02-16 18:41:09

From: Wizzup?
You'll probably want to call {$loadlib smart} before including it, though. (Although I imagine Simba may have already loaded it)
Don't know how this fits in the MSI structure?

I'll make it work.

mormonman

2011-02-16 18:58:33

Unable to name the variables?

Wizzup?

2011-02-17 14:54:42

Fixed a minor bug.

Mormonman, objects. But you can still see the path.

Coh3n

2011-02-18 12:14:06

Soo, I'm kind of extremely mad right now. I just got code:8 twice using MSI without Reflection included. I still think there's a leak in Reflection because I've still gotten the error using Reflection only scripts. This is also the MSI version with SPS so that may have something to do with it also.

E: http://villavu.com/forum/showpost.php?p=780678&postcount=60

E2: So I watched my CPU when running both a reflection and MSI script. The CPU steadily increases with both, but MSI's increases significantly faster. I need to figure this out ASAP.

If you know anything about what could cause a memory leak, please look here.

Wizzup?

2011-02-18 12:54:38

In my opinion, MSI should stop adopting new techniques such as SPS and get their current base stable first. If it works, tag it as stable and move on to the next feature.

Coh3n

2011-02-18 12:59:53

From: Wizzup?
In my opinion, MSI should stop adopting new techniques such as SPS and get their current base stable first. If it works, tag it as stable and move on to the next feature.

The current one is (aside from the memory leak that we didn't know it had). The one with SPS is my repository. I'll be gone most of next week, but I will be putting the SPS version on hold until the memory thing is solved.

E: I've also been following the stats, and it seems the current version is working pretty well for a lot of people, so I dunno.

Sabzi

2011-02-18 16:42:43

What if there is too much un-broke loop started?
What I mean you go in loop 1 then that sends you in loop 2 -> loop 3 -> back to loop 1 and so one, without breaking out those? (pretty much why goto is bad, yeah I know there is no gotos in this case; just mentioned it for example).
Can that cause an out of memory in the processors memory? As far as I know loop variables go in there, no? At least in C, as I was taught.

EDIT: Hmm, I was pretty sure I am in another thread, oh well …

Wizzup?

2011-03-13 07:54:14

Bumping due to the undeserved lack of interest.

Coh3n

2011-03-14 00:56:03

Last time I tried this I was getting runtime errors. I tried to tell you about it, but I must have forgotten. I'll run it right now and post what I get.

E: This is a different thread than I thought it was, sorry.

SundayForever

2011-03-15 23:06:23

I've tried just about everything, and I think I'll take a look at some more example scripts, but if I really can't pinpoint my problem I'll be sure to make a post somewhere with the small script I've created, as I'm sure most people here can easily tell me what I'm doing wrong.

Wizzup?

2011-06-02 05:52:42

Just wondering how memory management is with current Reflection?

Echo_

2011-06-02 10:16:58

From: Wizzup?
Just wondering how memory management is with current Reflection?


When I was helping, we found a few memory leaks with things that needed to be freed (mostly SMART hook references). I can only assume they have found and fixed more since then.

Coh3n

2011-06-02 18:09:46

I haven't had any issues with memory in a very long time.

Wizzup?

2011-06-02 18:22:47

From: Coh3n
I haven't had any issues with memory in a very long time.


Great.

Coh3n

2011-06-03 07:29:26

From: Wizzup?
Great.
Of course as soon as I say that I get an out of memory error in MMWalk.simba. >.<

Nava2

2011-06-03 07:31:01

From: Coh3n
Of course as soon as I say that I get an out of memory error in MMWalk.simba. >.<


There were a lot of memory leaks fixed in Reflection 2. Ones which were not seen previously, so thats definitely good news. :)