Wizzup?
2010-09-12 15:24:14
Hiya,
I've done some work (and thinking) on Python and libmml(Simba core). This just keeps crossing my mind as we still don't have a (proper) interpreter for Simba. RUTIS and Pascalscript I would classify as OK, but not good. Python is an awesome language and has a very mature / featureful interpreter.
Working with primitive types is trivial, so I'll skip it and go straight to arrays.
There are very important as that's where all the CPU time / processing / speed is spent. (Eg, we don't need Wizzyplugin-power to get the current mouse position, but we do need it when we sort > 50k points.)
Here's how I propose we do it:
Alternative ``copy'':
And if we want to pass an array to libMML from Python, we'd also do a copy. This probably isn't too much of an issue if we don't pass PascalArrays that are too huge. The only problem would be if we had to pass a python list. Python lists are slow compared to simple PascalArrays, and we'll have to turn that list into an Array again. Ultimately,both are possible, but speedwise the PascalArray is very much preferred.
This means that functions like getColors should accept both lists and PascalArrays. Another issue would be all the sorting functions. My suggestion is to simply incorporate these in the PascalArray class (or some class that extends it). This would ultimately provide proper speed (comparable to PascalScript with WizzyPlugin) and also make sure the sorting is accessible in an easy way.
So much for theory. Now a few examples. Please note that the code here is the ``under the hood'' code and is not representive for code in scripts. You could compare it to the code in Simba. (READ THIS PYTHON RANTERS)
My proposal:
Alt. version.
This is code to the findColors(Tolerance) functions. It basically just does a dll call and return the results. It's that simple. One of them constructs a PascalArray class, the other one copies the data into a list and frees the data.
From here I will drop the alternative version in this post as I don't think it is a viable alternative.
A not-finished PascalArray class: (I didn't even test it)
So obviously as soon as Python notices the array class should be deleted as it is no longer used, __del__ is called, and the memory will be freed. So there shouldn't be much memory issues, as long as we don't mix memory with FPC. It can also resize the array, thus providing a SetLength() like Pascal.
A few things are still missing. Loading from a list; exporting itself as a list; and an implementation on how to pass data to libMML/FPC. This probably isn't too hard, but I'll mention it anyway as it isn't incorporated yet in the example.
Sorting this array will probably be done with a class that extends PascalArray. It will probably incorporate most (or all) Wizzyplugin sorting functions.
Called something like this:
Alt:
That's probably the most important part. If this works, everything else is trivial.
What do you guys think? Am I missing something?
This would probably make the Pascal -> Python converter worth trying, mopar.
E: I totally forgot to mention that this PascalArray can be used like normal arrays:
I've done some work (and thinking) on Python and libmml(Simba core). This just keeps crossing my mind as we still don't have a (proper) interpreter for Simba. RUTIS and Pascalscript I would classify as OK, but not good. Python is an awesome language and has a very mature / featureful interpreter.
Working with primitive types is trivial, so I'll skip it and go straight to arrays.
There are very important as that's where all the CPU time / processing / speed is spent. (Eg, we don't need Wizzyplugin-power to get the current mouse position, but we do need it when we sort > 50k points.)
Here's how I propose we do it:
- Python calls the function libMML->FindColors.
- libMML->FindColors copies the TPA returned by the MML to memory. (This means no managed array issues. This involves one copy of the entire array.)
- The copied data is passed to Python. Python can directly access the array data using ctypes. I'll refer to this in Python as PascalArray from now on.
Alternative ``copy'':
- Python can directly access the array data and uses this data to create a list of the data and once the list is created it frees the array data. This can take up to 0.5s for 30k points. (VERY SLOW)
And if we want to pass an array to libMML from Python, we'd also do a copy. This probably isn't too much of an issue if we don't pass PascalArrays that are too huge. The only problem would be if we had to pass a python list. Python lists are slow compared to simple PascalArrays, and we'll have to turn that list into an Array again. Ultimately,both are possible, but speedwise the PascalArray is very much preferred.
This means that functions like getColors should accept both lists and PascalArrays. Another issue would be all the sorting functions. My suggestion is to simply incorporate these in the PascalArray class (or some class that extends it). This would ultimately provide proper speed (comparable to PascalScript with WizzyPlugin) and also make sure the sorting is accessible in an easy way.
So much for theory. Now a few examples. Please note that the code here is the ``under the hood'' code and is not representive for code in scripts. You could compare it to the code in Simba. (READ THIS PYTHON RANTERS)
My proposal:
def findAll(self, box, color, tol = 0):
"""
find all colors in a box, with a specific tolerance.
returned are all the matching points
"""
ptr, _len = PPOINT(), c_int(42)
if tol is 0:
self._mc.dll.find_colors(self._cli, byref(ptr), byref(_len),
color, *box)
else:
self._mc.dll.find_colors_tolerance(self._cli, byref(ptr),
byref(_len), color, tol, *box)
# Construct PascalArray
pa = PascalArray(ptr, _len)
# Do not free ptr!
return pa
Alt. version.
def findAll(self, box, color, tol = 0):
"""
find all colors in a box, with a specific tolerance.
returned are all the matching points
"""
ptr, _len = PPOINT(), c_int(42)
if tol is 0:
self._mc.dll.find_colors(self._cli, byref(ptr), byref(_len),
color, *box)
else:
self._mc.dll.find_colors_tolerance(self._cli, byref(ptr),
byref(_len), color, tol, *box)
# Construct python list
l = [(ptr[x].x, ptr[x].y) for x in range(_len.value)]
# Free ptr!
self._mc.free(ptr)
return l
This is code to the findColors(Tolerance) functions. It basically just does a dll call and return the results. It's that simple. One of them constructs a PascalArray class, the other one copies the data into a list and frees the data.
From here I will drop the alternative version in this post as I don't think it is a viable alternative.
A not-finished PascalArray class: (I didn't even test it)
class PascalArray(object):
"""
PascalArray is a class that allows one to easily access FPC/Pascal
memory in an array like way.
This class is a solution to passing arrays between libMML and Python.
The actual user of pymml should not be bothered by the external memory,
so as soon as there is no reference to this class anymore, Python calls
__del__ and we can safely free the memory.
"""
def __init__(self, pastype, ptr, MC, len):
"""
Set the type of the data we are holding to _pastype_,
save the pointer _ptr_ and store the reference to the MMLCore.
"""
self._type = pastype
self._p = ptr
self._mc = MC
self._len = len
def __del__(self):
"""
Free the array using FPC's FreeMem().
This is required because the memory was allocated using
FPC's AllocMem().
"""
self._mc.dll.fpc_freemem_(self._p)
def __len__(self):
"""
Return the length of the array.
"""
return self._len
def __getitem__(self, pos):
"""
Get an item at a specific position _pos_.
"""
if pos > len(self):
raise MMLException('Out of range')
return cast(self._p, POINTER(self._type))[pos]
def __setitem__(self, pos, item):
"""
Set an item at a specific position _pos_.
"""
if pos > len(self):
raise MMLException('Out of range')
if sizeof(item) != sizeof(self._type):
raise MMLException('Typesize does not match')
cast(self._p, POINTER(self._type))[pos] = item
return True
def resize(self, newsize):
"""
XXX: TODO
"""
So obviously as soon as Python notices the array class should be deleted as it is no longer used, __del__ is called, and the memory will be freed. So there shouldn't be much memory issues, as long as we don't mix memory with FPC. It can also resize the array, thus providing a SetLength() like Pascal.
A few things are still missing. Loading from a list; exporting itself as a list; and an implementation on how to pass data to libMML/FPC. This probably isn't too hard, but I'll mention it anyway as it isn't incorporated yet in the example.
Sorting this array will probably be done with a class that extends PascalArray. It will probably incorporate most (or all) Wizzyplugin sorting functions.
Called something like this:
tpa = Color.findAll((0, 0, 200, 200), 255, tol=25)
tpa.sortFrom((20, 20)
tpas = tpa.split(dist=20)
Alt:
tpa = Color.findAll((0, 0, 200, 200), 255, tol=25)
WizzyPlugin.SortTPAFrom(tpa)
tpas = WizzyPlugin.SplitTPA(tpa)
That's probably the most important part. If this works, everything else is trivial.
What do you guys think? Am I missing something?
This would probably make the Pascal -> Python converter worth trying, mopar.
E: I totally forgot to mention that this PascalArray can be used like normal arrays:
tpa = findAll((0, 0, 100, 100, clWhite)
print tpa[0].x
tpa[2].x = tpa[2].x - 10
# etc...