SRL Archive - General Botting Help

Where is the memory leak?


Foundry

2014-05-26 02:00:23

If I call this procedure too many times I get an out of memory error. I don't really know how to work with files very well so am I not freeing something?

// By Foundry
{==============================================================================]
Updates the FilteredStrings field of the TLog.
[==============================================================================}
procedure TLog.update();
var
StringArray: TStringArray;
Log, i, j, k, l, LineLimit: Integer;
size: LongInt;
EndPathName, MainString: String;
IndexesToDelete: TIntegerArray;
begin
case LogID of
EVENT: //Event Log
begin
EndPathName := '_Event.2014-05';
end;
PM: //Local Log
begin
EndPathName := 'PM__' + PMName + '.2014-05';
end;
COMBAT: //Combat Log
begin
EndPathName := 'DIRECTORY HERE'; // Not yet needed.
end;
end;
if not fileExists(WurmDirectory + 'Wurm\players\' + Charname +'\logs\' + EndPathName + '.txt') then
begin
writeLn('Failed to find the player folder ' + Charname + ' in directory ' + Wurmdirectory);
writeLn(WurmDirectory + 'Wurm\players\' + Charname +'\logs\' + EndPathName + '.txt');
if LogID = PM then
exit
else
terminateScript();
end;
Log := OpenFile(WurmDirectory + 'Wurm\players\' + Charname +'\logs\' + EndPathName + '.txt', True);
size := fileSize(Log);
ReadFileString(Log, MainString, FileSize(Log));
CloseFile(Log);
if size > 900000 then // If it is over 900000 characters
begin
writeLn('Event log is ' + toStr(size) + ' characters. Quitting Wurm and deleting it.');
quit(false);
wait(2000);
if deleteFile(WurmDirectory + 'Wurm\players\' + Charname +'\logs\' + EndPathName + '.txt') then
writeLn('File successfully deleted, please restart the game.')
else
writeLn('Couldn''t delete the file. Something is wrong...');
terminateScript();
end;
StringArray := StrLines(MainString);
if length(StringArray) <= 50 then
LineLimit := length(StringArray);
if LogID = EVENT then
begin
for i := length(StringArray) - LineLimit to high(StringArray) do
begin
if i < 0 then
continue;
for j := 0 to high(IgnoreNames) do
begin
if (pos(IgnoreNames[j], StringArray[i]) > 0) then
begin
setLength(IndexesToDelete, k + 1);
IndexesToDelete[k] := i;
inc(k);
break;
end;
end;
end;
StringArray.deleteIndexes(IndexesToDelete);
end;
if LinesToLog > length(StringArray) then
LineLimit := length(StringArray)
else
LineLimit := LinesToLog;
setLength(FilteredStrings, LineLimit + 1);
for i := 0 to LineLimit - 1 do
begin
l := high(StringArray) - LineLimit + i;
if l < 0 then
l := 0;
try
FilteredStrings[i] := StringArray[l];
except
writeLn(toStr(l) + ' broke it');
break;
end;
end;
setLength(FilteredStrings, i);
MainString := ''; // I don't know why but I tried this.
setLength(StringArray, 0); // and this.
end;

Brandon

2014-05-26 04:18:22

I don't see anything that could leak in the above. The only dynamically allocated resource is the file opening. The only way that would leak is if ReadFileString throws and you were try catching this function call. In that case, you wouldn't be able to close the file.


There are a couple things you can do to reduce your memory footprint:



for j := 0 to high(IgnoreNames) do
begin
if (pos(IgnoreNames[j], StringArray[i]) > 0) then
begin
setLength(IndexesToDelete, k + 1);
IndexesToDelete[k] := i;
inc(k);
break;
end;
end;

can become:


SetLength(IndexesToDelete, Length(IgnoreNames)); //outside of the outer for-loop.

for j := 0 to high(IgnoreNames) do
begin
if (pos(IgnoreNames[j], StringArray[i]) > 0) then
begin
IndexesToDelete[k] := i;
inc(k);
break;
end;
end;
SetLength(IndexesToDelete, K - 1);

That way it only costs you one resize and one copy compared to a resize and copy every loop (resizing an array to a larger size causes a new array to be created and old elements to be copied to the new one).


I was not aware that copying strings as below:

try
FilteredStrings[i] := StringArray[l];
except
writeLn(toStr(l) + ' broke it');
break;
end;

throws exceptions. Are you sure about this?


Other than the above, you code doesn't have any leaks as far as I can tell.

Foundry

2014-05-26 04:26:30

From: Brandon
I don't see anything that could leak in the above. The only dynamically allocated resource is the file opening. The only way that would leak is if ReadFileString throws and you were try catching this function call. In that case, you wouldn't be able to close the file.


There are a couple things you can do to reduce your memory footprint:



for j := 0 to high(IgnoreNames) do
begin
if (pos(IgnoreNames[j], StringArray[i]) > 0) then
begin
setLength(IndexesToDelete, k + 1);
IndexesToDelete[k] := i;
inc(k);
break;
end;
end;

can become:


SetLength(IndexesToDelete, Length(IgnoreNames)); //outside of the outer for-loop.

for j := 0 to high(IgnoreNames) do
begin
if (pos(IgnoreNames[j], StringArray[i]) > 0) then
begin
IndexesToDelete[k] := i;
inc(k);
break;
end;
end;
SetLength(IndexesToDelete, K - 1);

That way it only costs you two resizes and one copy compared to a resize and copy every loop (resizing an array to a larger size causes a new array to be created and old elements to be copied to the new one).


I was not aware that copy strings as below:

try
FilteredStrings[i] := StringArray[l];
except
writeLn(toStr(l) + ' broke it');
break;
end;

throws exceptions. Are you sure about this? Next is that break statement. There is one thing I must mention here that I saw a lot of developer doing..

This try except statement does not benefit you at all. Why? Because you never catch the exception. As soon as you hit the end, that exception is automatically re-thrown and any statements there-after are never ran. I suspect you might have this behaviour in code segments that might require the freeing of data and due to the nature of such a construct, it never frees.


Other than the above, you code doesn't have any leaks as far as I can tell.


I forgot to take some of the debug out from a while ago, my bad. I had that try..except statement there because sometimes it was trying to access a negative index, but I fixed that already. Thanks for the suggestions for the loop though, I'll put that in.

I really don't understand but when I call this, the amount of memory Simba uses (I watch it in task manager) increases by like 1 or 2MB until it crashes.

Brandon

2014-05-26 04:28:47

From: Foundry
I forgot to take some of the debug out from a while ago, my bad. I had that try..except statement there because sometimes it was trying to access a negative index, but I fixed that already. Thanks for the suggestions for the loop though, I'll put that in.

I really don't understand but when I call this, the amount of memory Simba uses (I watch it in task manager) increases by like 1 or 2MB until it crashes.



I mixed up the try except with try finally. Nothing should be wrong there. However, the try except wouldn't be necessary for the string copy and as you mentioned, you removed it.

I really cannot see anything else that would leak. What does the other code look like? This function itself looks fine. Are you absolutely positive it is this function and only this function that is leaking?

Foundry

2014-05-26 04:40:04

From: Brandon
I mixed up the try except with try finally. Nothing should be wrong there. However, the try except wouldn't be necessary for the string copy and as you mentioned, you removed it.

I really cannot see anything else that would leak. What does the other code look like? This function itself looks fine. Are you absolutely positive it is this function and only this function that is leaking?


Yes, absolutely sure. Figured out that the bigger the file that it reads gets, the more it leaks. So it must have something to do with that? I can't notice an increase with a 1kb file but with a 800kb file it is very noticeable.

E: here is the whole file… I am calling TLog.check() and have ran a test script that ONLY calls it and results in the leak.

{$include_once srlutilities\srlutilities.simba}

// By Janilabo
{==============================================================================]
Explodes str with multiple separators/delimiters (d).
The importance order for d items is from left to right (=>).
So place the important ones first and then less important after those.
[==============================================================================}
function StrExplodeMulti(d: TStringArray; str: string): TStringArray;
var
a, b, i, l, p, r, s, z, g: Integer;
m: Boolean;

begin
SetLength(Result, 1);
l := Length(str);
s := Length(d);
if ((l > 0) and (s > 0)) then
begin
r := 0;
z := 0;
for a := 0 to (s - 1) do
begin
p := Length(d[a]);
if (p > 0) then
if (p <= l) then
begin
if (z = 0) then
z := Length(d[a])
else
if (Length(d[a]) < z) then
z := Length(d[a]);
d[r] := d[a];
Inc(r);
end;
end;
if (r > 0) then
begin
s := r;
a := 1;
p := 1;
r := 0;
while ((a + (z - 1)) <= l) do
begin
m := False;
for i := 0 to (s - 1) do
begin
g := Length(d[i]);
if ((a + (g - 1)) <= l) then
begin
for b := 1 to g do
begin
m := (str[((a + b) - 1)] = d[i][b]);
if not m then
Break;
end;
if m then
Break;
end;
end;
if m then
begin
Result[r] := Copy(str, p, (a - p));
if ((a + (g + 1)) = l) then
Exit;
p := (a + g);
a := ((a + g) - 1);
Inc(r);
SetLength(Result, (r + 1));
end;
Inc(a);
end;
Result[r] := Copy(str, p, ((l - p) + 1));
end else
Result[0] := '';
end else
Result[0] := '';
end;

// By Janilabo
{==============================================================================]
Explodes str with multiple separators/delimiters (d).
The importance order for d items is from left to right (=>).
So place the important ones first and then less important after those.
NOTE: Supports limit.
[==============================================================================}
function StrExplodeMultiEx(d: TStringArray; str: string; limit: Integer): TStringArray;
var
a, b, i, l, p, r, s, z, g: Integer;
m: Boolean;
begin;
SetLength(Result, 1);
l := Length(str);
s := Length(d);
if ((l > 0) and (s > 0)) then
begin
r := 0;
z := 0;
for a := 0 to (s - 1) do
begin
p := Length(d[a]);
if (p > 0) then
if (p <= l) then
begin
if (z = 0) then
z := Length(d[a])
else
if (Length(d[a]) < z) then
z := Length(d[a]);
d[r] := d[a];
Inc(r);
end;
end;
if (r > 0) then
begin
s := r;
a := 1;
p := 1;
r := 0;
while (((r + 1) < limit) and ((a + (z - 1)) <= l)) do
begin
m := False;
for i := 0 to (s - 1) do
begin
g := Length(d[i]);
if ((a + (g - 1)) <= l) then
begin
for b := 1 to g do
begin
m := (str[((a + b) - 1)] = d[i][b]);
if not m then
Break;
end;
if m then
Break;
end;
end;
if m then
begin
Result[r] := Copy(str, p, (a - p));
if ((a + (g + 1)) = l) then
Exit;
p := (a + g);
a := ((a + g) - 1);
Inc(r);
SetLength(Result, (r + 1));
end;
Inc(a);
end;
Result[r] := Copy(str, p, ((l - p) + 1));
end else
Result[0] := '';
end else
Result[0] := '';
end;

// By Janilabo
{==============================================================================]
Returns the lines of str as TStringArray.
[==============================================================================}
function StrLines(str: string): TStringArray;
var
d: TStringArray;
begin
if (str <> '') then
begin
SetLength(d, 3);
d[0] := #13#10;
d[1] := #13;
d[2] := #10;
Result := StrExplodeMulti(d, str);
end else
begin
SetLength(Result, 1);
Result[0] := str;
end;
end;

// By Janilabo
{==============================================================================]
Returns the total count of lines that str contains.
[==============================================================================}
function StrLineCount(str: string): Integer;
var
s: TStringArray;
begin
if (str <> '') then
begin
s := StrLines(str);
Result := Length(s);
end else
Result := 1;
end;

// By Janilabo
{==============================================================================]
Returns the line text from str by number.
NOTE: Line numbers start from 1!
[==============================================================================}
function StrLine(str: string; number: Integer): string;
var
d, r: TStringArray;
begin
if ((str <> '') and (number > 0)) then
begin
SetLength(d, 3);
d[0] := #13#10;
d[1] := #13;
d[2] := #10;
r := StrExplodeMultiEx(d, str, (number + 1));
if (Length(r) >= number) then
Result := r[(number - 1)]
else
Result := '';
end else
Result := '';
end;

// By Foundry
{==============================================================================]
Returns true if any of the strings (strs) are found in the given lines of the
event log.
[==============================================================================}
function isInStrings(Strs, StringsToSearch: TStringArray; LinesFromLast: Integer = -1): Boolean;
var
i, j: Integer = 0;
begin
if LinesFromLast > high(StringsToSearch) then
LinesFromLast := high(StringsToSearch);
for i := high(StringsToSearch) downto high(StringsToSearch) - LinesFromLast do
begin
for j := 0 to high(Strs) do
begin
if pos(Strs[j], StringsToSearch[i]) > 0 then
exit(true);
end;
end;
end;

// By Foundry
{==============================================================================]
Deletes the indexes. Needed to delete multiple ones!
[==============================================================================}
procedure TStringArray.deleteIndexes(indexes: TIntegerArray);
var
i, j: Integer = 0;
begin
for i to high(indexes) do
begin
self.deleteIndex(indexes[i] - j);
inc(j);
end;
end;

function getPMText(str: String): String;
var
i, j: Integer;
read: Boolean;
begin
setLength(result, 200);
for i to length(str) do
begin
if (str[i] = '>') and
(not read) then
begin
inc(i);
read := true;
end;
if read then
begin
result[j] := str[i];
inc(j);
end;
end;
setLength(result, j - 1);
end;

// By Foundry
{==============================================================================]
Type that holds important info used to find information in the various logs.
[==============================================================================}
type
TLog = record
FilteredStrings: TStringArray;
LogID, LinesToLog: Integer;
PMName: String;
end;

var
IgnoreNames: TStringArray;
CharName, WurmDirectory: String;

// Constants for LogID.
const
EVENT = 0;
PM = 1;
COMBAT = 2;

// By Foundry
{==============================================================================]
Updates the FilteredStrings field of the TLog.
[==============================================================================}
procedure TLog.update();
var
StringArray: TStringArray;
Log, i, j, k, l, LineLimit: Integer;
size: LongInt;
EndPathName, MainString: String;
IndexesToDelete: TIntegerArray;
begin
case LogID of
EVENT: //Event Log
begin
EndPathName := '_Event.2014-05';
end;
PM: //Local Log
begin
EndPathName := 'PM__' + PMName + '.2014-05';
end;
COMBAT: //Combat Log
begin
EndPathName := 'DIRECTORY HERE'; // Not yet needed.
end;
end;
if not fileExists(WurmDirectory + 'Wurm\players\' + Charname +'\logs\' + EndPathName + '.txt') then
begin
writeLn('Failed to find the player folder ' + Charname + ' in directory ' + Wurmdirectory);
writeLn(WurmDirectory + 'Wurm\players\' + Charname +'\logs\' + EndPathName + '.txt');
if LogID = PM then
exit
else
terminateScript();
end;
Log := OpenFile(WurmDirectory + 'Wurm\players\' + Charname +'\logs\' + EndPathName + '.txt', True);
size := fileSize(Log);
ReadFileString(Log, MainString, FileSize(Log));
CloseFile(Log);
if size > 900000 then // If it is over 900000 characters
begin
writeLn('Event log is ' + toStr(size) + ' characters. Quitting Wurm and deleting it.');
quit(false);
wait(2000);
if deleteFile(WurmDirectory + 'Wurm\players\' + Charname +'\logs\' + EndPathName + '.txt') then
writeLn('File successfully deleted, please restart the game.')
else
writeLn('Couldn''t delete the file. Something is wrong...');
terminateScript();
end;
StringArray := StrLines(MainString);
if length(StringArray) <= 50 then
LineLimit := length(StringArray);
if LogID = EVENT then
begin
for i := length(StringArray) - LineLimit to high(StringArray) do
begin
if i < 0 then
continue;
for j := 0 to high(IgnoreNames) do
begin
if (pos(IgnoreNames[j], StringArray[i]) > 0) then
begin
setLength(IndexesToDelete, k + 1);
IndexesToDelete[k] := i;
inc(k);
break;
end;
end;
end;
StringArray.deleteIndexes(IndexesToDelete);
end;
if LinesToLog > length(StringArray) then
LineLimit := length(StringArray)
else
LineLimit := LinesToLog;
setLength(FilteredStrings, LineLimit + 1);
for i := 0 to LineLimit - 1 do
begin
l := high(StringArray) - LineLimit + i;
if l < 0 then
l := 0;
try
FilteredStrings[i] := StringArray[l];
except
writeLn(toStr(l) + ' broke it');
break;
end;
end;
setLength(FilteredStrings, i);
MainString := '';
setLength(StringArray, 0);
end;

// By Foundry
{==============================================================================]
Evaluates the LogCheck based on its defined parameters.
[==============================================================================}
function TLog.check(StringsToFind: TStringArray; LinesFromEnd: Integer; UpdateLog: Boolean = false): Boolean;
begin
if UpdateLog then
update();
result := isInStrings(StringsToFind, FilteredStrings, LinesFromEnd - 1);
end;

function TLog.checkRecentPM(var msg: String): Boolean;
var
i: Integer;
begin
update();
for i := high(FilteredStrings) downto 0 do
begin
case between('<', '>', FilteredStrings[i]) of
PMName:
begin
msg := getPmText(FilteredStrings[i]);
exit(true);
end;
'System': exit(false);
capitalize(CharName): exit(true);
end;
end;
end;

function TLog.checkRecentPM(): Boolean; overload;
var
s: String;
begin
result := checkRecentPM(s);
end;

procedure TLog.init(_LogID, _LinesToLog: Integer; _PMName: String = '');
begin
LogID := _LogID;
if _LogID = PM then
PMName := _PMName;
LinesToLog := _LinesToLog;
end;

Brandon

2014-05-26 05:04:58

From: Foundry
Yes, absolutely sure. Figured out that the bigger the file that it reads gets, the more it leaks. So it must have something to do with that? I can't notice an increase with a 1kb file but with a 800kb file it is very noticeable.

E: here is the whole file… I am calling TLog.check() and have ran a test script that ONLY calls it and results in the leak.



I still do not see anything that could leak other than the file reading. I do see a LOT of copying and string manipulation though. Possibly unnecessarily. But it doesn't seem to leak.


I did a test (pumped a file to 3mb) and ran:


Procedure readit;
var
size: Integer;
Log: Integer;
MainString: String;

begin
Log := OpenFile('C:/Users/Brandon/Desktop/Foo.txt', False);
size := fileSize(Log);
ReadFileString(Log, MainString, FileSize(Log));
CloseFile(Log);
end;

var
I: Integer;
begin
For I := 0 to 10000 do
readit;
end.

no leaks. However, I pumped the file to 400mb. I read it once successfully, no leaks. I read it but this time terminated the script. The result is:


no matter how many times I successfully run it after that, the 400+mb is still there and that in itself is a leak that requires you to restart Simba. Again, this is due to premature termination.

If anything, make sure that the script is being terminated safely and that ReadFileString isn't throwing. A single throw will devastate Simba especially if it is a fairly large file. Someone else might have to take a look because right now my eyes cannot find anything wrong with your code other than minor nit-picking details. So that's a good thing from me at least. Hopefully someone else sees it if the above safety precautions doesn't work.

Foundry

2014-05-26 14:42:57

@Brandon; Yeah, I tried seeing if readFileString throws an exception but it doesn't… This is really frustrating because I call this function about once every few minutes in my scripts and I can't go but an hour before running out of memory.

Anyone else see the problem? I must be missing something simple.

E: I figured out it has nothing to do with the opening and closing of the file… I had it exit after it read it and it resulted in no leak.

E2: It's this part:

StringArray := StrLines(MainString);
if length(StringArray) <= 50 then
LineLimit := length(StringArray);

Brandon

2014-05-26 15:26:27

Try swapping out that massive amount of functions and use a TStringList and a regex for splitting a string.. It is more efficient and powerful than what you are running right now. A lot of that code, though useful, is extreme overkill. Especially for delimiting a string.


Procedure EasySplitRegex(const Regex, StringToSplit: String; var Res: TStringArray);
var
I: Integer;
List: TStringList;
Begin
List := TStringList.Create(); //Might be List.Init() in Lape. I didn't test lape.
SplitRegExpr(Regex, StringToSplit, List);
SetLength(Res, List.Count);

For I := 0 To List.Count - 1 Do
Res[I] := List.Strings[I];
List.Free();
End;

Function StringToLines(Str: String): TStringArray;
Begin
EasySplitRegex('\r\n|\r|\n', Str, Result);
End;


var
arr: TStringArray;
str: String;
begin
Str := 'hey' + #13 + 'young' + #10 + 'world' + #13#10 + '.';
arr := StringToLines(Str);
writeln(arr);
end.

Olly

2014-05-26 16:26:45

^ You need a new Simba to do that in Lape.

Foundry

2014-05-26 16:28:25

From: Brandon
Try swapping out that massive amount of functions and use a TStringList and a regex for splitting a string.. It is more efficient and powerful than what you are running right now. A lot of that code, though useful, is extreme overkill. Especially for delimiting a string.

-snip-


E: just saw what you posted Olly -.-

Foundry

2014-05-26 19:52:23

Ok, got the new version of simba and it all compiles except for this:

List.Count

I'm guessing it is different in lape? I tried looking at the srl player form file because I know it uses a TStringList but I couldn't find anything.

Brandon

2014-05-26 19:56:32

From: Foundry
Ok, got the new version of simba and it all compiles except for this:

List.Count

I'm guessing it is different in lape? I tried looking at the srl player form file because I know it uses a TStringList but I couldn't find anything.


Try List.GetCount?

Foundry

2014-05-26 20:41:26

From: Brandon
Try List.GetCount?


Lol, now it's:

List.Strings[I];
Error: Unknown declaration "Strings" at line 309

Maybe it will compile one day…

E: Nvm found this one. It's TStringList.getStrings()

Alright, I am up for anything at this point… Teamviewer anyone? Tried replacing the string splitting function but it doesn't make a difference. It is this line:

StringArray := StringToLines(MainString); 

whenever it gets to this point, Simba starts to eat memory. I tested it by having the procedure exit right before this line and it doesn't leak.

Dgby714

2014-05-27 03:14:55

StrLines? What does this do?

Guessing Explode(#10, …)?

Foundry

2014-05-27 15:54:53

From: Dgby714
StrLines? What does this do?

Guessing Explode(#10, …)?


That fixed it… Apparently that function that I borrowed is causing it. Surprisingly simple, thank you.