Anyone got a python module for Fujitsu ATLAS?

[gandalfrockman]

I'm writing a quick and dirty (probably an understatement) python tool to do rapid machine translations of rpg maker patches made using Habisain's AWESOME RPGTrans tool.

Does anyone have a python module to tie into fujitsu ATLAS?

I'm using google translate right now, but I'm already running into their bot detection just testing the damned thing, and frankly a properly configured ATLAS does a much better translation with the right dictionaries plugged into it.

I can probably write something to tie python into atlas myself if I have to, but it will take me MUCH longer and it will be ugly. Anyone got a python module they can toss me?

 

[habisain]

Re: Anyone got a python module for Fujitsu ATLAS?

Hey, you know what? I do have a Python module for using ATLAS! What are the odds?

You must be registered to see the links

I was originally planning to add this as a feature to RPGMaker Trans, but I ended up deciding against it due to feature creep. That and it turns out that it's quite a lot of effort to do... Anyhow, feel free to use this. It's Python2 and fairly well documented, requires the win32api module (

You must be registered to see the links

), and I think it should work with the current version of ATLAS (not sure, can't test it at the moment as I'm using my Linux machine).

 

[gandalfrockman]

Re: Anyone got a python module for Fujitsu ATLAS?

Thanks that's awesome! This is going to save me a LOT of time...just looking at it...way more time than I actually thought....
I wasn't even sure I would get a response at all, and then the man himself drops in and hands me exactly what I'm looking for on a silver platter...

I'm working in python 3 atm, but it isn't much trouble for me to switch, or maybe rewrite the atlas module for python 3....

 

[habisain]

Re: Anyone got a python module for Fujitsu ATLAS?

Well, not entirely certain on this, but running the

You must be registered to see the links

over it should probably get you a version that runs on Python 3 (using the unicodeTranslate method for strings and translate for encoded byte strings). Sorry about it being Python 2, but this is from the days when RPGMaker Trans was written in Python 2.

 

[gandalfrockman]

Re: Anyone got a python module for Fujitsu ATLAS?

Well, not entirely certain on this, but running the [ over it should probably get you a version that runs on Python 3 (using the unicodeTranslate method for strings and translate for encoded byte strings). Sorry about it being Python 2, but this is from the days when RPGMaker Trans was written in Python 2.

Will do, tried going back to python2, holy crap is unicode ever a pain in the ass in python 2! every carriage return is a potential crash!

 

[habisain]

Re: Anyone got a python module for Fujitsu ATLAS?

Eh, I had a free moment so here's a quick port to Python3:

You must be registered to see the links

It was actually a little more involved than I thought, but then again most things involving porting 2 to 3 and bytes/strings are.

And yes, working with Unicode is a pain in Python 2. Bugs in Python 2's handling of Unicode file names under some circumstances is what forced me to migrate RPGMaker Trans.

 

[gandalfrockman]

Re: Anyone got a python module for Fujitsu ATLAS?

HAH
attached is my port to python 3.
Ill probably use yours though since mine causes access violations every time you translate something

Good learning experience though, I never realized some of the ctypes REQUIRED ascii for compatibility, I'll probably need the knowledge of the encode function I gained later on anyway.

edit: I face palmed when I realized you just used b values instead of re-encoding to the expected inputs... I should have thought of that!

 

[habisain]

Re: Anyone got a python module for Fujitsu ATLAS?

Hmm. A bit wierd, because for me your port does work (using Python 3.3). My suspicion is that there may be a race condition between Atlas and Pythons garbarge collector during initialisation, because the bytes object returned by genString.encode on line 81 immediately falls out of scope, so the c_char_p wrapping it may be pointing at memory already freed. (a c_char_p is not enough to keep memory alive)

Also, protip: The line "if __name__ == '__main__':" in mine stops the test case code from being executed when you import the file as a module. It's also very important on Windows if you ever look into threading/multiprocessing.

Also: theoretically, everything that you encode with ascii could also have been encoded with cp932, as ascii is a subset of cp932. It's a bit weird that you said ascii was required...

 

[gandalfrockman]

Re: Anyone got a python module for Fujitsu ATLAS?

Hmm. A bit wierd, because for me your port does work (using Python 3.3). My suspicion is that there may be a race condition between Atlas and Pythons garbarge collector during initialisation, because the bytes object returned by genString.encode on line 81 immediately falls out of scope, so the c_char_p wrapping it may be pointing at memory already freed. (a c_char_p is not enough to keep memory alive)

Also, protip: The line "if __name__ == '__main__':" in mine stops the test case code from being executed when you import the file as a module. It's also very important on Windows if you ever look into threading/multiprocessing.

Also: theoretically, everything that you encode with ascii could also have been encoded with cp932, as ascii is a subset of cp932. It's a bit weird that you said ascii was required...

Mine works on my side, it returns the translated text, but then it throws an exception afterwards.

Yeah I realized after I saw yours I didn't actually HAVE to use ascii, but the first documentation I found on c_char_p said it was expecting ascii bytecodes, I have worked with ascii and and unicode quite a bit, codepage stuff? not so much.

The test case stuff was just me being stupid sloppy, I'm mostly a systems guy, some of the more complex oop stuff, particular in python, still feels pretty foreign to me, but I'm getting there.

 

[habisain]

Re: Anyone got a python module for Fujitsu ATLAS?

That's really weird. I honestly can't see why your version would have a problem. I even tried running it under Python 3.4 and still no problems. Unless you're invoking the translator in a very strange way, I really have no idea why it would crash.

Oh well. Important thing is that you've got something that works.

 

[gandalfrockman]

Re: Anyone got a python module for Fujitsu ATLAS?

Agreed, the only other complex issue I have to deal with now is preserving RPGMakers control codes. I'd say I should be able to knock the rest of it out in under a week, but I know better than that....

 

[habisain]

Re: Anyone got a python module for Fujitsu ATLAS?

I would offer the code for my translator class (which handles loading/saving translatables), but a) no documentation as such b) it's somewhat gnarly, and c) it's pretty heavily tied into the RPGMaker Trans framework now (which is pretty big).

If it helps, when you're outputting the patch, you don't have to worry about any ADVICE field, as they're completely ignored by RPGMaker Trans. Obviously, you'll want to read advice lines in so you can split the translated text appropriately.

(Incidentally, if you have a string, x, representing the contents of a begin/end block, you could just use x.replace('# TRANSLATION', '# TRANSLATION\n%s' % translatedString.strip()) which would preserve all control codes intact)

 

[gandalfrockman]

Re: Anyone got a python module for Fujitsu ATLAS?

Useful info, but Ill probably take a slightly different approach at first.
I'm treating each file as a cursor table, and blowing through it one line at a time, it just copies everything except Japanese text which it still copies , but accumulates into a single string value, translates, and then drops in after the #TRANSLATION line. Then it trucks along to the next line.

It's not the 'correct' way to do it. It wont be efficient, and big games (Im using the monstrous 'BFRogue SAITAMA!' as a test case and the RPGTrans patch is over 10MB empty) will probably take some time. HOWEVER, it's dead simple to implement, and I should be able to deal with the control characters via a custom atlas dictionary initially.

It will still be faster than a google translate based system, especially now that those have to deal with the bot detection crap.

The correct way to do it would be the way vkozyrev's system worked, with a proper database, a verified translation cache instead of atlas's translation memory system, and what have you, I might do that at some point, but it would be a huge project, and I have plenty of those right now.

edit:Im not even planning a gui for this thing yet.

 

[reddo]

Re: Anyone got a python module for Fujitsu ATLAS?

Useful info, but Ill probably take a slightly different approach at first.
I'm treating each file as a cursor table, and blowing through it one line at a time, it just copies everything except Japanese text which it still copies , but accumulates into a single string value, translates, and then drops in after the #TRANSLATION line. Then it trucks along to the next line.

It's not the 'correct' way to do it. It wont be efficient, and big games (Im using the monstrous 'BFRogue SAITAMA!' as a test case and the RPGTrans patch is over 10MB empty) will probably take some time. HOWEVER, it's dead simple to implement, and I should be able to deal with the control characters via a custom atlas dictionary initially.

It will still be faster than a google translate based system, especially now that those have to deal with the bot detection crap.

The correct way to do it would be the way vkozyrev's system worked, with a proper database, a verified translation cache instead of atlas's translation memory system, and what have you, I might do that at some point, but it would be a huge project, and I have plenty of those right now.

edit:Im not even planning a gui for this thing yet.

How hard would it be to create a few more threads for each .txt? (I'm not familiar with python)

That would probably make it good enough for projects of any size, no optimizations needed. I'm guessing giving each thread a different file would also not be too tough to implement, as they'd not need to really speak with each other.

 

[gandalfrockman]

Re: Anyone got a python module for Fujitsu ATLAS?

How hard would it be to create a few more threads for each .txt? (I'm not familiar with python)

That would probably make it good enough for projects of any size, no optimizations needed. I'm guessing giving each thread a different file would also not be too tough to implement, as they'd not need to really speak with each other.

It wouldn't be that hard, but there's a couple problems:

1.I came across a note in the source code for translation aggregator stating that Atlas isn't thread safe, so it wouldn't work with atlas. I suspect powertranslate is multithreaded, but have no idea if there are python hooks for it, or if it's even any good outside of basic business use cases.
ATLAS does way better translations than people give it credit for, its just that most people never bother to configure it properly.

2.How long a game takes to translate is now processor dependent since you aren't dealing with a network resource. On a newer cpu I can't imagine anything would take more than an hour or two to translate. That's still totally reasonable for any size project; we're building patches here so you only have to do it once. Besides, I may be seriously overestimating this particular problem.

On that note: Getting the control codes through atlas intact was WAY easier than I thought, I didn't even need a custom dictionary. There are translation style options in atlas that normally turn brackets into reserved characters by default, which of course strips them out of your string on the way in. Shuffle some radio buttons around, and it passes bracketed content without alteration. It already passes most special characters normally. It still has issues with exclamation points and question marks but so far that hasn't been much of an issue. Would be cool if I could get it to treat \n[1] as the potential subject of a sentence, but I'm not going to worry about that right now.

I should be doing some test inserts and stuff tonight/tomorrow then its just down to string-post-processing, and some polish.

 

[habisain]

Re: Anyone got a python module for Fujitsu ATLAS?

The bigger problem with Python and multithreading comes in the form of the GIL (Global Interpreter Lock) - you can only have a single thread accessing any python object at a single time. If you want a parallelisation speedup, you'll need to look at the multiprocessing module (incidentally, this is why RPGMaker Trans spawns so many processes), which should be easy enough to use (it pretty much uses a threading API). Also, as it's multiple processes, even if ATLAS isn't thread safe you should probably be fine (provided that you create the translator object inside the function that you dispatch with multiprocessing).

My only concern with how you're parsing the translation files might end up causing problems in ATLAS (as well as, you know, being slow). I've had ATLAS crash with certain inputs.

Incidentally, if I recall right, Vkozyrevs patcher worked on names by replacing \n[1] etc with actual names, and then swapping back after translation.

 

[gandalfrockman]

Re: Anyone got a python module for Fujitsu ATLAS?

The bigger problem with Python and multithreading comes in the form of the GIL (Global Interpreter Lock) - you can only have a single thread accessing any python object at a single time. If you want a parallelisation speedup, you'll need to look at the multiprocessing module (incidentally, this is why RPGMaker Trans spawns so many processes), which should be easy enough to use (it pretty much uses a threading API). Also, as it's multiple processes, even if ATLAS isn't thread safe you should probably be fine (provided that you create the translator object inside the function that you dispatch with multiprocessing).

My only concern with how you're parsing the translation files might end up causing problems in ATLAS (as well as, you know, being slow). I've had ATLAS crash with certain inputs.

Incidentally, if I recall right, Vkozyrevs patcher worked on names by replacing \n[1] etc with actual names, and then swapping back after translation.

Yeah I have stumbled across some unusual ways to crash atlas, initially I was initializing and deconstructing the translator for each translation, just for debugging some stuff. I knew I would need to move to a single instance at some point. .. But starting and stopping it over and over introduced some signifigant instability. THEN I found some character codes that CRASH ATLAS! fortunately they crash rpgmaker as well,and it turns out the only reason they showed up at all is from where someone had typed them into the ability descriptions in the rpgmaker editor for some unused abilities.

As far as vkozyrev's \n[1] solution, he was using google translate. No custom dictionaries, so it was really his only option. I should be able to do it with a dictionary or translation rule entry. Just as an FYI your python module uses whatever translation settings are on the atlas clipboard translator, changing settings anywhere else has no affect on translations.

 

[habisain]

Re: Anyone got a python module for Fujitsu ATLAS?

That makes sense, because I'm using the same API as ChiiTrans (I think... not actually sure). My suspicion is that one of the Unknown integers passed in at Atlas engine initialisation would control which set of options is loaded, but I'm not in a position to test that hypothesis quite yet.

I know for certain that some inputs which are valid with RPGMaker will crash ATLAS, but they're fortunately quite rare. If I recall right, they were mostly some edge cases involving some grammar. There's also some much more common unfortunate cases where it doesn't translate everything you pass in (i.e. passing in a paragraph and getting a couple of words), again due to problems with grammar.

 

[gandalfrockman]

Re: Anyone got a python module for Fujitsu ATLAS?

That makes sense, because I'm using the same API as ChiiTrans (I think... not actually sure). My suspicion is that one of the Unknown integers passed in at Atlas engine initialisation would control which set of options is loaded, but I'm not in a position to test that hypothesis quite yet.

I know for certain that some inputs which are valid with RPGMaker will crash ATLAS, but they're fortunately quite rare. If I recall right, they were mostly some edge cases involving some grammar. There's also some much more common unfortunate cases where it doesn't translate everything you pass in (i.e. passing in a paragraph and getting a couple of words), again due to problems with grammar.

I think I have some insight into whats going on with the incomplete translations. Its not a character input issue exactly.

When...atlas...does...this...it means... that...advanced...sentence composition...is...failing.

I've worked with ATLAS in a corporate setting (its better than google at pretty much everything if you setup your environment right, but it DESTROYS every other automated translation system for patents and business documentation), and we had a similar issue when one of ATLAS's other .dll files was blocked by our security solution. ATLAS has a two tier approach to sentence identification and subject/object organization, it trys the tier 1 (simple method) first, if that fails ( I have no idea what the success/failure criteria is) it has a more rigorous, but more resource intensive tier 2 method, in my business scenario, we were denying it access to a dll function it needed to execute the more advanced tier 2 sentence structure stuff. The results were the same as I am seeing here, if it cant get to its better sentence analyzer it gives up, and switches to the EVEN SIMPLER word unit translation. Which is a real pita because it doesn't use spaces, which really scares the bjeesus out of my carriage return insertion logic in post processing!

NOTE:We had to get atlas support on the line to sort this out when it happened, even they were initially confused.

As further indication that this is a similar issue,
I've got my pre and post processing far enough along that I can strip out punctuation and the proper pronoun indicator characters. Sometimes this fixes the problem, but I think it only works because I'm dramatically simplifying the sentence structure.

Feeding the same string unaltered (ie complete, complex sentence structure) through either Translation Aggregator or atlas clipboard translates them properly.

Of course the python code we are working with atm only loads up atlecont.dll and feeds it very specific instructions this is awesome because we aren't loading up a bunch of crap we wont use, but problematic because we are missing something we want.

SO my current plan of action is to identify the necesarry dll's by blocking atlas's access to their functions and running some test translations until it goes...all...brain...damaged...slow..talk...on...me. Fortunatley we've got translation aggregators source code. Then I port whatever functions and ctype garbage I need from translation aggregator. I could just port all of it, but that would be silly, there's lots of stuff it does with atlas that only really makes sense for real time translations, and a bunch more stuff that's just there to provide faster access to stuff buried in ATLAS's config menu.

I'm going into a little more depth here than I need to in case any other developers stumble-across/are-reading this later and find it useful.

This is probably the most technical post I have ever made on a pornography related forum.... probably....

 

[habisain]

Re: Anyone got a python module for Fujitsu ATLAS?

Hmm. You sure on that? Number 1) I've had the x...y...z problem happen with Atlas clipboard translate and Number 2) TA basically calls the same functions as my module, although it does do a substantial amount of preformatting. In particular, I notice it replaces all \t, \r, \n characters with a space, and converts all Shift-JIS punctuation to ASCII punctuation.

Now, TA does load a couple more DLLs, but I can't see any uses of them. I'd like to believe that means they're not important, but I guess they might be... Anyhow, the extra libraries are awdict and awuenv, should be easy enough to add loads for these.