SRL Archive - S.M.A.R.T.

So, I did some playing around with SMART and it turns out I can get it to run in the other rendering modes, but only when Java is started as it's own process, i.e. not starting the JVM manually via JNI. I successfully got the client to enter Safe Mode, Software, and OpenGL. THAT BEING SAID, Software and OpenGL are both rather useless for the main feature of SMART: minimizing.

First, a bit about how SMART currently works:
When you load the client, you have the initial java.awt.Applet, which does no rendering, and you have a single component in that applet: an RS-specific subclass of java.awt.Canvas where literally everything is rendered and events are processed. In current versions of SMART, I replace the java.awt.Graphics object returned by the getGraphics() method of that canvas with the java.awt.Graphics object of a java.awt.image.BufferedImage, effectively forcing the client to render to that buffer regardless of where it is on the screen, minimized or not. This was designed 6 or so years ago when there was only one rendering mode (the equivalent of Safe Mode now).

Here's what I've found out about the (modern) client:
Basically, you all know there are three main rendering modes: SafeMode, Software, and OpenGL. So the question you're all dying to ask is: why does SMART only work with Safe Mode? The answer is simple: Safe Mode seems to be the only mode that uses Java's rendering framework (the one built into AWT), and hence it's the only one SMART can intercept java-side. As far as I can tell, Software mode renders to the canvas surface, but not from within java. I suspect that they do the rendering using the native-rendering functionality of AWT such that there's no chance for me to intercept the data before it gets to the screen. The end result is that there's nothing I can do to give Simba the color data, however Simba can still scrape the colors off the screen in the traditional manner. That only leaves OpenGL, which is similarly useless as far as SMART is concerned, but further, the rendering is done on the GPU and in the video ram such that it never touches a place I can get the color data from. (Or Simba, for that matter, on Windows, at least. Linux seems to have functionality to get around this.)

How I know this:
Well, for the most part, this is a mixture of fact and educated guesses. I know Safe Mode works and I know why it works. I know Software and OpenGL don't work in the same way, even though I'm positive they are rendering to the same java.awt.Canvas. That is to say, I can still replace the java.awt.Graphics object, but it is not receiving the rendering data and hence Simba would be flying blind in those modes. With software mode enabled there is a horrible flicker of SMART drawing it's buffer to the screen and RS rendering directly to the screen, so that tells me the rendering is still in normal memory using native drawing techniques. For OpenGL there is no flicker, meaning that the video card is completely overwriting that portion outside of regular ram as it is drawn to the screen, hence untouchable.

What it all means:
Basically, if Safe Mode ever goes away, either it's back to browser with mouse/keyboard stealing, or SMART will have to remain visible and you will still have your mouse and keyboard free. Arguably a step down after six years of minimize-and-forget. That being said, there might be hope for OpenGL since I have stumbled upon functionality to grab color data from an OpenGL rendering. I don't know fully how to implement it, but perhaps someone working with OGL 'interception' knows a way to grab a copy of the screen. IFF this works when the program is minimized (it potentially could) then OGL could be a viable solution to the problem.

That being said (and the real reason I wrote all this down) SMART will require a rather major overhaul to get OpenGL or Software mode working as a Simba plugin. It's not something I'm willing to do 1) without assurance that OGL is a viable alternative when minimized OR 2) as long as Safe Mode still exists (I believe Jagex is going to fix the graphical bugs, but I may be proven wrong). If safe mode goes away, I'll make it where SMART provides silent mouse/keyboard but no longer minimizable functionality.

That is all, hope you enjoyed.

Ah shit.. how well do you know OGL And GDI Ben? And did you check your inbox? I messaged you something earlier before you posted this..

Nice read here Ben. Thanks for updating us with what could happen in the future

Ben, I recommend talking to this guy: [email]trilez@tribot.org[/email]

He has created his own OpenGL interception bot client, so I'm pretty sure he has enough knowledge about it to give you the information you need.

From: Trent!
Ben, I recommend talking to this guy: [email]trilez@tribot.org[/email]

He has created his own OpenGL interception bot client, so I'm pretty sure he has enough knowledge about it to give you the information you need.

AFAIK TriBot only uses colour at this stage so I think he is still learning about OGL.

From: [J]ustin
AFAIK TriBot only uses colour at this stage so I think he is still learning about OGL.

He actually made a few scripts using the OpenGL client (all of them are outdated now, API update or something). Besides, if he uses color AND OpenGl then that's probably even more perfect, because he'll know how to combine the two, and hopefully help Ben out with his problem.

TL;DR

Trilez succeeded in making an OpenGL bot client, so he probably would be a good person to talk to

http://villavu.com/forum/member.php?u=58402

Js.

Last I heard from TRileZ is his Tribot had unity between color & OGL. I assume it also uses a silent mouse/keyboard, but does anyone know if it was minimizable also?

Thanks for the simple explanation of everything, as well as future plans, Ben. It's always better this way; letting us know the problem, why it happened, and how it can be fixed.

Another thing, how would DirectX fit into SMART?

@Wizzup:
You're probably right about Aftermath using WINE. If he uses Linux with WINE's Windows API then he gets the best of both worlds. :O I guess that would explain why Simba read color data for him in OGL mode… Too bad that only works for Linux users, no?

Ben, have you tried deleting your runescape cache completely and trying to run safe mode again?

I'm not sure if that helps to remove the rendering issue?

Sounds good.. just an fyi, from what I've seen with 20+ people who have ran bots on my VM's in the past, nobody has minimized, they're just left up overlapping other smarts & simbas while they run.

I assume as long as they're up, you can have a browser or anything on top of them so they're in the "background" correct? If so I don't think it would change the habits of most people

As long as the window is "hidden from view" rather than minimized, i think that shouldn't affected botters too much.

Something like windows aero preview.

I have a simple fix for the hidden from view thing. In Smart's Main, RegisterHotKey(……);

Of course implement a callback. On WParam if it is the hotkey then getwindowlong.. setwindowtransparent.. bam done. On press again, reset transparency to normal.

Either that or hook OGL.

@justin TRiBot is both oGL and Colour. Reason for colour is that oGL on tribot only shows the outer edge of draw objects, so a item say prayer pot will have the same id as a super restore. so what you can do it find all the items with a id then search for the colour, at that id.
Mat

From: Flight
Last I heard from TRileZ is his Tribot had unity between color & OGL. I assume it also uses a silent mouse/keyboard, but does anyone know if it was minimizable also?

Thanks for the simple explanation of everything, as well as future plans, Ben. It's always better this way; letting us know the problem, why it happened, and how it can be fixed.

Another thing, how would DirectX fit into SMART?

@Wizzup:
You're probably right about Aftermath using WINE. If he uses Linux with WINE's Windows API then he gets the best of both worlds. :O I guess that would explain why Simba read color data for him in OGL mode… Too bad that only works for Linux users, no?

Yes. With Linux you can do very neat things, even create multiple virtual mouse pointers and assign them to different virtual or hardware cursors.

Anyway, if someone still wants to run a lot of clients headless, it's very possible, with vnc. Not the usual ``Remote Desktop'' as you know it from Microsoft, but actual vnc, the thing it was invented for. I posted something here about it: http://villavu.com/forum/showpost.php?p=1066251&postcount=2

Ben, I know how to target/get all Direct3D/OpenGL windows on Windows, at least, I know how Fraps used to do it.It's something similar to LD_PRELOAD, in a sense. But you might not like it: http://www.ring3circus.com/gameprogramming/case-study-fraps/

I think on Linux this is all not a problem since XGetImage will return the OpenGL data either way, but I'm not sure how it works with SMART with Linux, I'll just assume Ben tested this.

From: grats
Sounds good.. just an fyi, from what I've seen with 20+ people who have ran bots on my VM's in the past, nobody has minimized, they're just left up overlapping other smarts & simbas while they run.

I'm not sure, I don't think this will work. You can try it though, target SMART with Simba (window picker) and hover something over it, then use SaveScreenshot().

E: I suppose it will work when compositing is turned on.

Offtopic : @Wizzup you double posting noob ;)

Why do you render the buffer when SMART is minimized anyways? wouldn't it be alot better to only to flush the buffer when it's maximized / client disabled? just curious :P

I think you all misunderstand two key things.
1) In order for Simba to 'see' RS must render to the buffer. Otherwise you won't have any color data
2) Generally the window must be fully visible, i.e. on top, in order for color data to be gotten the traditional way.

The whole point of SMART was that you can auto without having to look at it, without safe mode, this will be difficult if not impossible.

E: I didn't mention DirectX because Linux. DirectX is basically a shitty windows version of OpenGL.

From: Wizzup?
I'm not sure, I don't think this will work. You can try it though, target SMART with Simba (window picker) and hover something over it, then use SaveScreenshot().

E: I suppose it will work when compositing is turned on.

I guess benland answered that :p Well I guess that would really suck then… and maybe put us back to 1 simba per VM, hopefully it doesn't happen.

@ben
So you say we need to have rs visible in order to for some modes to work? Am sure we won't mind have smart as a always on top window , cause we can still use the mouse.

how is the BufferedImage made up? a array of bytes?

From: Main
@ben
So you say we need to have rs visible in order to for some modes to work? Am sure we won't mind have smart as a always on top window , cause we can still use the mouse.

^ just throw smart on my extended desktop and im happy :P

Well I got something that might help. I was playing around with the detours I made.. OpenGL when hooked, still renders to the buffer while minimized.. Infact, I can grab anything I want before it even hits the screen.. Meaning Items that are sitting in the buffer waiting to be rendered can be grabbed. So offscreen stuff works still.

Anyway I was able to register a system wide hotkey, minimize the regular RS Client, and take bitmaps of it even while it is minimized..

This is MIGHT be good news for smart since if successful, smart can do the same or perhaps read directly from the buffer. Atm, I'm reading pixels from a DIB using GDI + glReadPixels. The best part is that I do not need to use any device contexts of the window (Only a MemDC is needed since this is GDI) to do it so with the window minimized, everything renders perfectly.

I don't know how to put this crap into smart but if the regular RS client can have it, Smart should be able to as well. DirectX should also work since they both render offscreen first. Now time to go back to reading.

From my Purely-CPlusPlus library modified to work with the OpenGL hooks:

#define GL_BGR 0x80E0  //Define BGR for 24 bits.. BGRA for 32.

typedef union RGB
{
    uint32_t Color;
    struct
    {
        unsigned char B, G, R, A;
    } RGBA;
} *PRGB;

Bitmaps::Bitmaps(int W, int H, uint32_t BitsPerPixel) : Pixels(0), Info(), width(W < 0 ? -W : W), height(H < 0 ? -H : H), size(0), bFileHeader(), TransparentColor(Rgb(0)), DC(0), Image(0)
{
    Pixels.clear();
    memset(&Info, 0, sizeof(BITMAPINFO));
    Info.bmiHeader.biSize = sizeof(BITMAPINFOHEADER);
    Info.bmiHeader.biPlanes = 1;
    Info.bmiHeader.biBitCount = BitsPerPixel;
    Info.bmiHeader.biCompression = BI_RGB;
    Info.bmiHeader.biWidth = width;
    Info.bmiHeader.biHeight = height;
    Info.bmiHeader.biSizeImage = size = ((width * BitsPerPixel + 31) / 32) * 4 * height;
    bFileHeader.bfType = 0x4D42;
    bFileHeader.bfSize = bFileHeader.bfOffBits + size;
    bFileHeader.bfOffBits = sizeof(BITMAPFILEHEADER) + sizeof(Info.bmiHeader);

    std::vector<unsigned char> TEMP(size);
    //Image = CreateDIBitmap(DC, &Info.bmiHeader, CBM_INIT, &TEMP[0], &Info, DIB_RGB_COLORS);
    glPixelStorei(GL_PACK_ALIGNMENT, 4);
    glReadPixels(0, 0, width, height, GL_BGR, GL_UNSIGNED_BYTE, &TEMP[0]);

    unsigned char* BuffPos = &TEMP[0];
    height = (height < 0 ? -height : height);
    //int BytesPerPixel = Info.bmiHeader.biBitCount / 8;
    Pixels.resize(width * height);       //WIDTH * HEIGHT * BYTES-PP

    for (int I = 0; I < height; I++)
    {
        for (int J = 0; J < width; J++)
        {                                                                 //Flips The ScanLines/Rows.
            Pixels[(height - 1 - I) * width + J].RGBA.B = *(BuffPos++);
            Pixels[(height - 1 - I) * width + J].RGBA.G = *(BuffPos++);
            Pixels[(height - 1 - I) * width + J].RGBA.R = *(BuffPos++);
            Pixels[(height - 1 - I) * width + J].RGBA.A = (Info.bmiHeader.biBitCount > 24 ? *(BuffPos++) : 0);
        }
        if(Info.bmiHeader.biBitCount == 24)
            BuffPos += width % 4;
    }
}

Where vector<RGB> Pixels. I only tested 32 and 24 bit Bitmaps. A call was made to that in the glSwapBuffer Detour.

unsigned int NewStride = 12;
bool DrawOverlay = false;
bool LogCalls = false;
bool WireFrame = false;

void Commands()
{
    if (GetAsyncKeyState(VK_F11) & 1)
    {
        WireFrame = !WireFrame;
        if (WireFrame)
            glPolygonMode(GL_FRONT_AND_BACK, GL_LINE);
        else
            glPolygonMode(GL_FRONT_AND_BACK, GL_FILL);
    }

    if (GetAsyncKeyState(VK_F12) & 1)
    {
        Bitmaps X(765, 565, 24);
        X.Save("C:/Users/Brandon/Desktop/Meh.bmp");
    }
}

GL_EXPORT __stdcall BOOL GLHook_wglSwapBuffers(HDC hdc)
{
    Commands();
    return (*optr_wglSwapBuffers) (hdc);
}

Problems.. Since I am actually reading a Dib and saving it, it takes approx 1 second to save which is a bit slow. Rendering and saving directly from the buffer is as good as it will get. That's another topic though. I haven't figured out how to do that yet but it will be extremely fast! Maybe even a quick Bitblt to smart's canvas will be faster than saving it to the desktop like I demonstrated just for testing purposes. Note that the code above also creates the bitmap on the fly every time I press F12 so that is slow too.

Well, that is fantastic news I can work with. One question… does this only need to be run from the same process, or is there some special OGL setup I need to do?

E: Also, I'll let you in on a little secret, if you make the height negative in the bitmap the origin will be at the top left and everything will work as expected.

EE: I'll respond to that PM eventually, we should chat on IRC sometime.

From: BenLand100
Well, that is fantastic news I can work with. One question… does this only need to be run from the same process, or is there some special OGL setup I need to do?

E: Also, I'll let you in on a little secret, if you make the height negative in the bitmap the origin will be at the top left and everything will work as expected.

Ahh I left the height as is for a reason. My library took care of that stuff with other calls. I basically wrote that entire thing from scratch before I posted it so yeah i know it's a bit bad ={ Then again I am sorta new to GDI since I only learned GDI after trying to mod games.

Uhh what do you mean ran from the same process? The official RS client loads my modded opengl32.dll and that dll saves the bitmap on keypress. I can always make it save automatically without keypress but that means it'll do it every time even 1 pixel is rendered to the screen. The DLL is here if you want to take a look: https://github.com/Brandon-T/GLHook

It's a simple dll that wraps all calls to OGL. My ridiculous explanation here: http://villavu.com/forum/showthread.php?t=86230

I'll talk with you about it some time.