SRL Archive - MML Library Development

Due to the growing interest in the MML library I've decided to dedicate a subforum to it - for now.

So what is libMML?

libMML is the library that exports all core Simba functionality to any application that can load the library. (See it as a ``plugin'' for any program.)

It will feature:

• Colour finding methods
• Bitmap finding methods
• DTM finding methods
• OCR methods
• Mouse/Keyboard methods

It may include:

• Methods to manipulate bitmaps. (Aside from data access and set/getpixel)

It will most likely not include:

• Form methods. This will not be included because every language has its own form libraries.
• Internet/file functions. These are also obviously provided by the languages. Unless this proves to be an issue for cross platform compatibility, we will not provide it.

libMML is not complete.

The current libMML builds can use the mouse, do some basic colour finding and partially use the OCR. All the other stuff still has to be implemented in the library.

We do not yet have any binaries for the library either. (You'll have to compile it yourself) I don't even know if it works on Windows or not. I have only tested it on Linux, but I'm sure it will in the near future work on Windows as well.

Why libMML?

The power to use Simba in the language of your choice is an awesome concept.
If we will ever want to switch SRL to another language, this should be what will make the switch simple.


Aside from libMML we're also working on pyMML; a python module that will load and use libMML. Obviously this can be done in other languages too, but we chose python because it is a powerful language.

Small example:

# Create the color object
c = Color(DLL, client)

ret = c.find((0, 0, 100, 100), 0)
print ret

ret = c.findAll((0, 0, 100, 100), 0, tol=100)
print ret

# Mouse object
m = Mouse(DLL, client)

# Position + Button states
print m[(Mouse.Pos, Mouse.Left, Mouse.Right)]

# Set position and right mouse button state
m[(Mouse.Pos, Mouse.Right)] = ((300,300), True)

# Print button states.
print m.getButtonStates()
sleep(0.5)
m.setPos((200,200))

libMML progress:

• Colour finding methods
• Bitmap finding methods
• DTM finding methods
• OCR methods
• Mouse methods
• Keyboard methods

Done
Partially implemented
Not started

If anyone wants to work on it, post here. :)

What's the current license of libMML? LGPL?

Currently, I think GPL. If we change the license of the MMLCore to LGPL, we can make libMML LGPL.

I would love to help, I have very little programming knowledge however. If I can be of assisstance though, I would love to.

What's the main difference between GPL and LGPL at it's current versions?

From: Zyt3x
What's the main difference between GPL and LGPL at it's current versions?

LGPL means you can link against the library without your application being (L)GPL. (So basically, closed source applications can use LGPL libraries, but not GPL libraries)

EL Bad_Boy, I'm afraid it will take some programming skill. Especially when it comes to knowing how to work with pointers & memory.

LGPL would be great =D..
Nice project :). I'm looking forward for a useable .dll =D…

I'm looking forward to pyMML. :p

I have an idea which would benefit greatly from this. As far as I can tell, it only needs mouse (yay done) and bitmap (hopefully soon) functionality.

BTW, the idea would be a certain AI based solver for parts of Puzzle Pirates. It would be equally doable in the present state of Simba I'm sure, I just really really lack experience in Pascal. Plus, certain core AI routines would benefit from being compiled. But that could be solved with a Simba plugin.

MML bindings for something like Java would really help, though.

Also, the current routines seem to be focused on the screen. This makes sense in some ways, but I'm also wondering about the ability to operate on bitmaps. That is, saved past screen state and such.

ForgotMyName, libMML still needs bmp, dtm, ocr, keyboard, and color functionality. And it focuses mainly on the screen because most of what Simba does is visual, I suppose. But saving and manipulating (I assume) will be added as well.

Why Java? Just wondering. I really don't like Java. :/

From: ForgotMyName
Also, the current routines seem to be focused on the screen. This makes sense in some ways, but I'm also wondering about the ability to operate on bitmaps. That is, saved past screen state and such.

Simba itself has several ways to operate with bitmaps. Those could be exported; but I'll make sure the bitmap (pixel) memory is exported.

From: Wizzup?
I'm afraid it will take some programming skill. Especially when it comes to knowing how to work with pointers & memory.

You know I've been wanting to help out somewhere. I know pointers (at least I should :p) and how they work with memory.

Count me in on this. I will obviously have to learn some python.

If I understand this correct, this is pertaining to pyMML that needs the above implemented or does the regular libMML still need them implemented?

From: HyperSecret
You know I've been wanting to help out somewhere. I know pointers (at least I should :p) and how they work with memory.

Count me in on this. I will obviously have to learn some python.

If I understand this correct, this is pertaining to pyMML that needs the above implemented or does the regular libMML still need them implemented?

libMML creates the DLL that pyMML depends upon for usage!

Current MML. Just posting it here because some people think Simba code in general is voodoo, since it is in fact quite easy. Perhaps this sparks some interest… ;)

(Note that they should all be cdecl if exported)

library libmml;

{$mode objfpc}{$H+}

uses
  cmem,Classes,interfaces,graphics,client,sysutils,MufasaTypes,dtmutil;

{$R *.res}

Const
  RESULT_OK = 0;
  RESULT_FALSE = 1;
  RESULT_ERROR = -1;

  MOUSE_UP = 0;
  MOUSE_DOWN = 1;

var
  last_error: String;
  debug: boolean;

function init: integer;
begin
  last_error := '';
  debug := true;
  result := RESULT_OK;
end;

function validate_client(C: TClient): boolean; inline;
begin
  result := Assigned(C);
  if not result then
  begin
    last_error := 'PClient is NULL';
    if debug then
      writeln(last_error);
  end;
end;

function create_client: PtrUInt; cdecl;
var
  C: TClient;
begin
  try
    C := TClient.Create('');
    Result := PtrUInt(C);
  except on e : Exception do
    begin
      writeln('ERROR');
      result := PtrUInt(RESULT_ERROR);
      last_error := e.Message;
    end;
  end;
  writeln(format('C: %d, IOManager: %d', [PtrUInt(C), PtrUInt(C.IOManager)]));
end;

function destroy_client(C: TClient): integer;
begin
  if not validate_client(C) then
  begin
    exit(RESULT_ERROR);
  end;
  C.Free;
end;

procedure set_debug(v: Boolean);
begin
  debug := v;
end;

function get_debug: boolean;
begin
  exit(debug);
end;

{
  VERY IMPORTANT: If you use get_last_error, you must immediately store the
  resulting string somewhere else. As soon as you do other calls, the last error
  may be reset or assigned a different memory position, making your old
  pointer invalid.
}
function get_last_error: pchar;
begin
  exit(@last_error[1]);
end;

function array_to_ptr(ptr: Pointer; size: PtrUInt; objsize: PtrUInt): Pointer;
begin
  result := GetMem(objsize * size);
  Move(ptr^, result^, objsize * size);
end;

function free_ptr(ptr: pointer): boolean;
begin
  result := Assigned(ptr);
  if not result then
  begin
    last_error := PChar('TClient is NULL');
    if debug then
      writeln(last_error);
  end else
    FreeMem(ptr);
end;

function alloc_mem(size, objsize: PtrUInt): Pointer;
begin
  result := GetMem(size * objsize);
end;

function realloc_mem(ptr: Pointer; size, objsize: PtrUInt): Pointer;
begin
  result := ReAllocMem(ptr, size*objsize);
end;

{ Mouse }

function get_mouse_pos(C: TClient; var t: tpoint): integer; cdecl;

begin
  if not validate_client(C) then
  begin
    exit(RESULT_ERROR);
  end;

  try
    C.IOManager.GetMousePos(t.x,t.y);
    result := RESULT_OK;
  except on e : Exception do
    begin
      result := RESULT_ERROR;
      last_error := e.Message;
    end;
  end;
end;

function set_mouse_pos(C: TClient; var t: tpoint): integer; cdecl;
begin
  if not validate_client(C) then
  begin
    exit(RESULT_ERROR);
  end;

  try
    C.IOManager.MoveMouse(t.x,t.y);
    result := RESULT_OK;
  except on e : Exception do
    begin
      result := RESULT_ERROR;
      last_error := e.Message;
    end;
  end;
end;

function ConvIntClickType(Int : Integer) : TClickType; inline;
begin
  case int of
    0 : result := mouse_Left;
    1 : result := mouse_Right;
    2: result := mouse_Middle;
  end;
end;

function get_mouse_button_state(C: TClient; But: Integer): Integer;
begin
  if not validate_client(C) then
  begin
    exit(RESULT_ERROR);
  end;
  writeln(but);

  try
    if C.IOManager.IsMouseButtonDown(ConvIntClickType(But)) then
      result := MOUSE_DOWN;
  except on e : Exception do
    begin
      result := RESULT_ERROR;
      last_error := e.Message;
    end;
  end;
end;

function set_mouse_button_state(C: TClient; But, State, X, Y: Integer): Integer;
begin
  if not validate_client(C) then
  begin
    exit(RESULT_ERROR);
  end;

  try
    if State = MOUSE_UP then
    begin
      C.IOManager.ReleaseMouse(X, Y, ConvIntClickType(But));
      result := RESULT_OK;
    end else if state = MOUSE_DOWN then
    begin
      C.IOManager.HoldMouse(X, Y, ConvIntClickType(But));
      result := RESULT_OK;
    end;
  except on e : Exception do
    begin
      result := RESULT_ERROR;
      last_error := e.Message;
    end;
  end;
end;

function find_color(C: TClient; var x, y: integer; color, x1, y1, x2, y2: integer): integer;
begin
  if not validate_client(C) then
  begin
    exit(RESULT_ERROR);
  end;

  try
    if C.MFinder.FindColor(x, y, color, x1, y1, x2, y2) then
      result := RESULT_OK
    else
      result := RESULT_FALSE;
  except on e : Exception do
    begin
      result := RESULT_ERROR;
      last_error := e.Message;
      writeln('last_error: ' + last_error);
    end;
  end;
end;

function find_color_tolerance(C: TClient; var x, y: integer; color, tol, x1, y1, x2, y2: integer): integer;

begin
  if not validate_client(C) then
  begin
    exit(RESULT_ERROR);
  end;

  try
    if C.MFinder.FindColorTolerance(x, y, color, x1, y1, x2, y2, tol) then
      result := RESULT_OK
    else
      result := RESULT_FALSE;
  except on e : Exception do
    begin
      result := RESULT_ERROR;
      last_error := e.Message;
    end;
  end;
end;

function find_colors(C: TClient; var ptr: PPoint; var len: Integer; color, x1, y1, x2, y2: integer): integer;
var
  TPA: TPointArray;
begin
  if not validate_client(C) then
  begin
    exit(RESULT_ERROR);
  end;

  SetLength(TPA, 0);
  try
    C.MFinder.FindColors(TPA, color, x1, y1, x2, y2);
  except on e : Exception do
    begin
      result := RESULT_ERROR;
      last_error := e.Message;
    end;
  end;

  len := Length(TPA);
  ptr := array_to_ptr(Pointer(@TPA[0]), len, sizeof(TPoint));
  result := RESULT_OK;
  setlength(tpa, 0);
end;

function find_colors_tolerance(C: TClient; var ptr: PPoint; var len: Integer; color, tol, x1, y1, x2, y2: integer): integer;
var
  TPA: TPointArray;
begin
  if not validate_client(C) then
  begin
    exit(RESULT_ERROR);
  end;

  try
    C.MFinder.FindColorsTolerance(TPA, color, x1, y1, x2, y2, tol);
  except on e : Exception do
    begin
      result := RESULT_ERROR;
      last_error := e.Message;
    end;
  end;

  len := Length(TPA);
  ptr := array_to_ptr(Pointer(@TPA[0]), len, sizeof(TPoint));
  result := RESULT_OK;
end;

exports

  init,
  create_client,
  get_last_error,
  get_debug,
  set_debug,
  alloc_mem,
  realloc_mem,
  free_ptr,

  get_mouse_pos, set_mouse_pos,
  get_mouse_button_state, set_mouse_button_state,

  find_color, find_color_tolerance,

  find_colors, find_colors_tolerance;


begin
end.

Added basic DTM methods. (Untested)

What I'm trying to make will require a minor amount of AI. Java is (for better or worse) my most familiar programming environment. Right now, I feel this is going to be enough of a challenge without further burdening myself with other unfamiliarities. Also, I will definitely need to dynamically allocate and deallocate data structures. (I am using a tree search, and possibly other AI techniques for puzzle games.) I'm not even sure if this is possible from SIMBA. (If it is, PLEASE tell me now.)

Yes, the alternative is to make a C plugin with all my computation routines. If anything, that would be the SANE alternative. I could do all the screen-scraping, interacting, and glue in Simba, and just rely on the C plugin for AI solving. This may or may not be easier given my current skillset. Once again, AI is not something I expect to get right the first time through, even the really simple techniques, and I would go a long way to have that hard to debug part take place in a more familiar environment.

I don't actually like Java. Practically speaking, however, it may well be my best option at the moment. Especially given that certain parts of what I may need will likely already be available in the Java community's large supply of existing library code.

I don't think Simba is voodoo. Practically speaking, however, it will cause me additional difficulty from my current lack of knowledge. Also, the dynamic allocate/free concern.

I was always confused why all the methods for finding were screen-reliant. I suppose it's not relevant to RS, but for what I want to make, being able to save essentially a screenshot for later processing could be rather valuable, both to save work until we know we need it, as well as to deal with getting a single consistent frame out of potentially fast animations. While I see it being convenient to just have a FindColorsTolerance (just for example) that directly operates on the screen, it seems it would be more versatile to make a generic version that operates on bitmaps, and then have the direct-screen variation be a wrapper based off that.

Which Simba functions were you referring to for operating directly on searching bitmaps? (Those seem really useful for other purposes.)

As I mentioned on IRC, I am now looking into using JNA (https://jna.dev.java.net/) to make a light wrapper to call libmml from Java, potentially also performing some conversions to and from more Java-friendly objects. (Directly, it would use a generic Java class which represents a C-like struct.) A lot of it seems to be rather nice automagic conversions.

Given my current schedule and my horrid lack of work ethics, this may take a while.

There are methods for searching bitmaps without them being currently on the screen I believe. And for whatever AI you want to do, it should be able to be done in any language.

Have you thought about Python? It's not currently available, but it will be at some point (in Simba).

From: i luffs yeww
There are methods for searching bitmaps without them being currently on the screen I believe. And for whatever AI you want to do, it should be able to be done in any language.

Have you thought about Python? It's not currently available, but it will be at some point (in Simba).

It definitely requires dynamic allocation/deallocation of trees. (There are not really any sane ways around this. Sure you could do the array trick, but all signs point to that being a really bad idea, especially with higherish branching factor.) Otherwise it's pretty generic data manipulation from the programming perspective. Specifically, I'm trying to bot one of several possible minigames in a certain game. They're all puzzle-based, and the way to get good moves is to look ahead a few steps.

Python would be an alternative. Python does not have a just-in-time compiler, however. (Or, I dunno, this may be a complex issue. Not sure what they're up to lately.) What I'm doing may or may not be significantly computationally intensive. (I will need to search 5ply with a branching factor of like 50, but I'm uncertain about how much pruning I could do, nor how expensive node evaluations might be.) If I want JIT, that would involve running pyMML under Jython. This might work. I don't know.

But seriously, what are the Simba methods for finding stuff in an existing bitmap? I've somehow not been able to find them.

In Simba, you can set a bitmap as a target. Then all the Colour/Bitmap/DTM find functions will performed on that bitmap instead.

Any idea when CPython could be used as the interpreter? ^^

From: i luffs yeww
Any idea when CPython could be used as the interpreter? ^^

Embdded in Simba? Unsure.
It'll probably require a full Python install, though.

I'll make sure to talk to you about it later. I just would like to have a somewhat working interpreter so I can run a python script using SPyRL and pyMML and possibly SMART and even more unlikely pyflection. If not it's fine, but even a very ugly hacky pile of crap would suffice. :p I just would like to show some more visual progress.

School's out in about 20 minutes and I'll be home in about an hour to mess around with this if I can. It really seems easy enough to have CPyton embedded, but then again I haven't done any Simba work.

From: i luffs yeww
I'll make sure to talk to you about it later. I just would like to have a somewhat working interpreter so I can run a python script using SPyRL and pyMML and possibly SMART and even more unlikely pyflection. If not it's fine, but even a very ugly hacky pile of crap would suffice. :p I just would like to show some more visual progress.

School's out in about 20 minutes and I'll be home in about an hour to mess around with this if I can. It really seems easy enough to have CPyton embedded, but then again I haven't done any Simba work.

It would be nice so you could do things like colour picking in simba. :)

From: Nava2
It would be nice so you could do things like colour picking in simba. :)

Wait, what? I don't get it? :p

He means that it'd be nice to have CPython as a valid interpreter in Simba so that you could use the color picker. Granted you can use it without CPython, but it would be one less Simba open. :p

Would extensions need to be changed? And all buttons and everything?

Or would everything just always go through libMML and then, depending on the interpreter, have different functionality?

My current idea is rather radical. I'll probably write the MML Daemon and make it spawn processes that start the python interpreter and execute a script. I'll elaborate on this later…
Basic idea: http://villavu.com/wiki/Simba_Daemon

Well that'd be interesting, if it worked correctly. For now I'll just work on lib/pyMML/SPyRL until I can help with the daemon.

libMML. Will it be a dll? or a pascal script file?

A .dll on windows, and a .so on linux.

As stated, it's essentially a "plugin" for simba. It exports all of Simba's functions into a format that can be used by any language.

Has there been much progress in libmml since last year(it doesn't look like it)? Because my holidays started and I want to finish it off while im getting ready for uni. If someone could give me a list of the things that still have to be added to libmml that would be nice.

Nope, it's pretty much unchanged since last year. What needs to be finished in libMML is just the core functions of MML (anything that would be useful for another language to be able to do, like color finding, bitmaps, DTMs, and OCR).

From: i luffs yeww
Nope, it's pretty much unchanged since last year. What needs to be finished in libMML is just the core functions of MML (anything that would be useful for another language to be able to do, like color finding, bitmaps, DTMs, and OCR).

What are those color finding functions that need to be added exactly? Thats what i want to know(eg: add findcolors, finddtmsrotated, etc)

It hasn't seen a whole lot of work, but it's probably not a whole lot of work to finish the initial part. I've been experimenting with sharing the arrays directly, but I don't think it's a good idea, that's partially to blame for the holdup.

If you want to work on it, the #simba IRC channel is where you can find me.