[TOOL] MechaTranslator

[reddo]

This is a Python application that uses Atlas to translate RPG Maker Trans patch folders made by Habisain's RPG Maker Trans. It also has some partial support for Wolf Trans patch folders and is able to translate RPG Maker MV games directly.

It also uses Habisain's ATLAS python code to translate sentences.

Now on GitHub!
Sources:

You must be registered to see the links

Binary:

You must be registered to see the links

By downloading the Binary version, the only thing you need to install is ATLAS. If you instead use the Source, here's what you need:

1. Install
   You must be registered to see the links
2. Install
   You must be registered to see the links
3. Install pypiwin32 for Python ("pip install pypiwin32" or "python -m pip install pypiwin32" at command prompt)
4. Extract MechaTranslator
5. Run MechaTranslator.py

So for now, bugs:

• Some games will crash because one of their scripts was touched while it shouldn't, if that happens disable inline Scripts translating or Scripts.txt translating
• Some games will crash because they used Actor names in scripts, if that happens disable Actors.txt translating
• Version 0.2.1 is probably not very friendly towards Wolf games. It's better to use 0.14 if you're working with Wolf Trans patches.

About MV Games translating (new to 0.2.1):

1. Experimental. Do backup your games, as it's possible it'll stop working. MechaTranslator will make backups of the files it touches, but I hope you're not going to re-rename 180 files manually if something goes wrong.
2. As far as I'm aware, MV translation should be perfectly stable, as only known objects get translated.

I'll stick the Changelog straight to the GitHub page, as will every future release. Please download from there.

 

[habisain]

Re: [TOOL] MechaTranslator for RPG Maker Trans

I don't really know how I did the ATLAS script either - mine is a port of some C# bindings I found ages back. Speaking of which, it will fail on a handful of games - those which use Japanese or other characters not in cp932. That's my fault really, so I'll see about fixing that when I can. This is pretty rare though (I've seen it once or twice), so hopefully it won't be an issue.

One thing I'd recommend: for distribution, grab a freezing script. I use cx_freeze myself, which works well and is easily installed via pip install cx_Freeze. Trust me that people will appreciate having a binary rather than having to install Python and Python dependencies. Otherwise this looks pretty good.

 

[reddo]

Re: [TOOL] MechaTranslator for RPG Maker Trans

I don't really know how I did the ATLAS script either - mine is a port of some C# bindings I found ages back. Speaking of which, it will fail on a handful of games - those which use Japanese or other characters not in cp932. That's my fault really, so I'll see about fixing that when I can.

One thing I'd recommend: for distribution, grab a freezing script. I use cx_freeze myself, which works well and is easily installed via pip install cx_Freeze. Trust me that people will appreciate having a binary rather than having to install Python and Python dependencies. Otherwise this looks pretty good.

Hey, thanks for the help!

I'll have to look into cx_freeze. I have to admit I had never touched Python until today, so I was completely lost and mostly googling everything.

EDIT:

Added a link to a binary on the first post, it has like a bajillion files, though, which can get messy. Should work without needing Python, however. (It was too big for the forums)

 

[rizo]

Re: [TOOL] MechaTranslator for RPG Maker Trans

some screen shots of dialogue would be cool.
But this seems like a helpful tool to use.

 

[reddo]

Re: [TOOL] MechaTranslator for RPG Maker Trans

some screen shots of dialogue would be cool.
But this seems like a helpful tool to use.

It's Atlas.

The dialogue only looks a tad better because I'm fixing up things like quotes and splitting sentences, and Atlas has an easier time with shorter sentences.

Right now I've managed to make it translate sentences with script calls without touching the script calls themselves, also taught it how to translate Scripts.txt, but touching Scripts.txt still breaks some games.

Also tried to have it translate inline scripts that don't mention filenames, but those are still a bit iffy. I'll have to make it so it only ever touches inline scripts if they are composed of japanese characters only, it's also been breaking some games.

(These were made with the version I'm currently making, the one in the OP doesn't take care around quotes and stuff yet, I should upload it as soon as I find a way of not breaking most games while still translating the Scripts.txt file, at least the Vocabs)

 

[reddo]

Re: [TOOL] MechaTranslator for RPG Maker Trans

I have cleaned up the OP and uploaded a new version.

If anyone's knowledgeable about these things, there are two problems I'm currently having:

• ATLAS refuses to multi-thread (it crashes if two threads ask anything of it at the same time)
• I have so far managed to initialize LEC translator basing on habisain's Atlas script and Translation Aggregator's usage of LEC, but I have failed to actually have it translate things

I kind of prefer ATLAS to LEC, but can't really say LEC isn't convenient.

Having ATLAS work in multiple threads would allow us to translate multiple times faster (my code already supports different threads doing different files), so this would be amazing.

 

[Rep]

Re: [TOOL] MechaTranslator for RPG Maker Trans

Having ATLAS work in multiple threads would allow us to translate multiple times faster (my code already supports different threads doing different files), so this would be amazing.

Maybe launch multiple atlas copies? You probably don't need to translate everything in one window line by line...

 

[reddo]

Re: [TOOL] MechaTranslator for RPG Maker Trans

Maybe launch multiple atlas copies? You probably don't need to translate everything in one window line by line...

Running them through multiple threads wasn't working at all, despite creating new instances of the AtlasTranslator and even making multiple copies of the ATLAS directory so that each of them was running their very own Atlas. If it didn't crash, they'd taint each other or try to access memory which isn't theirs, etc. A mess.

Switching from Threads to Processes did the trick, though. I've updated the OP with one creating 8 processes. I don't know if it's Python or whatever, but these processes never run at 100% of a cpu core, so that number should be increased if someone has 8 cores, 8 processes appears ideal for 4 cores, though. Now it translates huge games like XVI in a mere 40 seconds (with i5 2500k).

Unfortunately, processes are not compatible with cx_freeze, I'll have to find another one, until then the .py version is fast and the Binary version isn't.

EDIT: Updated to always create cpu_count() * 2 processes.

EDIT 2: And one more update. Now the little hearts will no longer render files untranslatable, nor will they render whole sentences untranslatable! They'll simply get ignored.

 

[Rep]

Re: [TOOL] MechaTranslator for RPG Maker Trans

Some makers use characters which can't be encoded for Atlas nor read by my own code, mostly ♥ and ♡. You need to remove these before the file can be translated.

Maybe just convert them into Unicode (or whatever else code) before it goes into atlas, then convert them back after translation is done? For example,
ありがと♥
ありがと♡
atlas removing hearts or show them as "?". On the other hand
ありがとU+2665
ありがとU+2661
shown as
Thank you U+2665
Thank you U+2661

upd: lol, nevermind.

 

[reddo]

Re: [TOOL] MechaTranslator for RPG Maker Trans

Maybe just convert them into Unicode (or whatever else code) before it goes into atlas, then convert them back after translation is done? For example,
ありがと♥
ありがと♡
atlas removing hearts or show them as "?". On the other hand
ありがとU+2665
ありがとU+2661
shown as
Thank you U+2665
Thank you U+2661

upd: lol, nevermind.

We have to encode to cp932 for Atlas, it doesn't support UTF-8. For now I just have them ignored and replaced with nothing, the random ?'s confuse Atlas and make translations worse.

I think I'll try to hardcode the little hearts and have them as untranslatable symbols so that they remain on the translations. They're not important, but they do serve a purpose, after all.

 

[Rep]

Re: [TOOL] MechaTranslator for RPG Maker Trans

We have to encode to cp932 for Atlas, it doesn't support UTF-8.

It doesn't need to support anything. I have only vague idea how all this work, but if your script instead of replacing them with nothing replace them with simple numbers and eng alphabet, atlas will just ignore those altogether. Then script replace them back.

 

[habisain]

Re: [TOOL] MechaTranslator for RPG Maker Trans

Unfortunately, processes are not compatible with cx_freeze, I'll have to find another one, until then the .py version is fast and the Binary version isn't.

They certainly are compatible with cx_freeze! It's how RPGMaker Trans works. They just need a little support on Windows. Specifically:

1) from multiprocessing import freeze_support
2) Immediately after the line 'if __name__ == '__main__': freeze_support()

(Ref:

You must be registered to see the links

)

As to heart symbols etc: try using the string method isalnum to break a string into groups of letters and groups of symbols, then translate the letters. Or wait until I can clean up my translator layer, but that might take a bit longer.

(Note: This method *still* isn't foolproof! There are Japanese characters not encoded in cp932. So still wrap the translate function with try/except UnicodeEncodeError so that you can fail nicely rather than crashing out. Incidentally, this is one of the reasons I would like to tidy up my implementation before releasing it!)

 

[reddo]

Re: [TOOL] MechaTranslator for RPG Maker Trans

They certainly are compatible with cx_freeze! It's how RPGMaker Trans works. They just need a little support on Windows. Specifically:

1) from multiprocessing import freeze_support
2) Immediately after the line 'if __name__ == '__main__': freeze_support()

(Ref:

You must be registered to see the links

)

As to heart symbols etc: try using the string method isalnum to break a string into groups of letters and groups of symbols, then translate the letters. Or wait until I can clean up my translator layer, but that might take a bit longer.

(Note: This method *still* isn't foolproof! There are Japanese characters not encoded in cp932. So still wrap the translate function with try/except UnicodeEncodeError so that you can fail nicely rather than crashing out. Incidentally, this is one of the reasons I would like to tidy up my implementation before releasing it!)

Ty, I just reinstalled Windows so it'll take a little while before I can rebuild.

Currently, each line is translated separately inside a try/except, if it fails it just skips translating it, but keeps going for the rest. The issue I was having with the whole files not working was because it turns out hearts can't be printed() either, so my exception handling would in turn create a new exception and that made it go into my other exception handler which would then skip the whole file.

I changed your encode to cp932 line to simply ignore any and all characters it can't encode, so it'll mostly work with everything (though if it misses japanese characters, some sentences might mistranslate).

Thanks for the help.

EDIT: I've tried that, but no dice. It seems the newest version of cx_Freeze is from Jan 7th and there's a GitHub issues page about Processing not working with it. Some guy made his own fork which supposedly fixes it, but trying to use his cx_Freeze module inside my project resulted in an "can't import utils".

It doesn't need to support anything. I have only vague idea how all this work, but if your script instead of replacing them with nothing replace them with simple numbers and eng alphabet, atlas will just ignore those altogether. Then script replace them back.

There are no guarantees towards the positioning of the english symbols after translation. Sometimes Atlas also mangles them for some reason or I guess depending on what they are.

But I already am splitting sentences into multiple instances of a TranslationSymbol class, some of which are translatable and some of which aren't, which then are put back together at the end in the same order. All I need to do is make the processor consider hearts as untranslatable symbols, just as it does quotes. In fact, it'd probably be good to do the same for any symbol which can't be converted.

Replacing them with nothing is just a temporary measure with easy implementation. I just added an "errors = ignore" to the encode function.

(My initial idea was to make a string like SSS0SSS[japanese]SSS1SSS, but Atlas wasn't working very well with that, so I ended up legit splitting it and the symbols don't even get processed by Atlas)

EDIT: 0.6 Version maintains symbols in the final result.

 

[reddo]

Re: [TOOL] MechaTranslator for RPG Maker Trans

I just noticed that WolfTrans makes patches very similar to RPG Maker Trans, so I changed a few things and now they are supported as well! I think! Haven't tested thoroughly yet. Other than that, made a bunch of QoL changes.

Here's the few last versions for which I didn't make any posts for:

Version 0.9:

• Will now request game-specific settings from the user.

Version 0.8:

• Added support for WolfTrans patch folders since they have the same syntax as RPG Maker Trans. I don't know the names of the untranslatabled files yet (like Scripts.txt), so if it has anything of the sort it'll still be getting translated and that'd be a problem. Unfortunately, I don't have many interesting WolfRPG games to test this on right now.
• Improved sentence split on highlighted text (between quotes of some kind).
• Added sentence split on all sentence finishers (dots, exclamation marks, etc). These will be retained on the translation. Still unsure if I should remove any sentence finishers added by ATLAS itself, though.

Version 0.7:

• Will now request for patch folder when ran. Can select another folder on completion. Cancelling the folder request ends the application.

Version 0.6:

• If a translated line would begin with ">" and therefore crash RPG Maker Trans, a backslash will be added before it
• Characters which can't be encoded for Atlas will be treated as untranslatable symbols and remain on the final translation
• Scripts.txt vocabulary should now correctly translate things after %s. Oops.
• Careful translation of inlineScripts reactivated, will probably not work with every game. I really need to make a GUI to let these become optional.

I still can't make the binary with multiprocessing on. I might make some non-multiprocessing binaries, though. Is that acceptable? It's much, much slower (around 8 times slower on 4 CPUs, even slower at more CPUs).

I also still can't properly use LEC translator, so I have mostly given up on both of those. Help would be appreciated.

 

[reddo]

Re: [TOOL] MechaTranslator for RPG Maker Trans

You know, I really hate Python...

Managed to have the Binary version work, link is at the first post. Every error I kept getting always googled to a page saying "Oh, this is because that Python version has this bug" or "That cx_freeze version has a bug" and even "Well, that cx_freeze doesn't work with that python", but it eventually worked out with python 3.3.4 and older versions of pipwin32 and cx_freeze. Took a while to go through multiple python versions until I found one that worked, though.

Anyway, now there are also options on it for certain problem-files if you want them not to be translated, you can opt in to InlineScripts/Scripts Vocabulary and you also get to choose the length of each sentence (there are the suggested default values, but it really depends on the game, some games use smaller fonts, some games show faces, etc, you can test it until you find a length that works well).

Also, now translates files from biggest to smallest, so it's less likely for threads to remain idle at the end (occasionally would have some big files left for the end and then you'd only have one thread working on them for a long time).

EDIT: Here's some pics of how translations look like right now.

 

[Rep]

Re: [TOOL] MechaTranslator for RPG Maker Trans

I think it would be for the better if program had some script/dictionary where would be already translated phrases for certain files. Like

Enemies.rvdata2
スライム = Slime
System.rvdata2
の攻撃!=  attacks!
%1に %2 のダメージを与えた! = %1 took %2 damage!
Armors.rvdata2
レザーアーマー = Leather Armor
Classes.rvdata2
剣士 = Swordsman
Items.rvdata2
ポーション = Potion
Weapons.rvdata2
ショートソード = Short Sword

et cetera. This way things that already were decently translated wouldn't need to be translated every time and would look nice. Or make some option to use/import 3rd party scripts/dictionaries for such purpose.

 

[reddo]

Re: [TOOL] MechaTranslator for RPG Maker Trans

I think it would be for the better if program had some script/dictionary where would be already translated phrases for certain files.

This way things that already were decently translated wouldn't need to be translated every time and would look nice. Or make some option to use/import 3rd party scripts/dictionaries for such purpose.

I updated it to create a JSON file that can be edited/re-used. I wasn't going to since this was outside of the scope, but since it doesn't really slow down initial translation while speeding repeated translations immensely it was pretty good to have (repeated translations being fast means it's easy to change maximum line length for each game through trial and error, and translating from the dictionary is about a hundred times faster than translating through ATLAS).

Right now it dumps all translations it does, though, which brings a couple issues:

1. If I improve sentence translation for a new version, older dictionaries will become worse than new translations
2. One's personal translations will be put in the same file as a lot of machine translation, meaning that if one wants to have their own private "best translations" dictionary, they'd have to create a second file to safe-keep them

One other thing is that the translations in the dictionary are not yet "cleaned up", which allows one to take advantage of line-splitting in MechaTranslator. You can just translate without a care and it'll take care of it. But since it's not yet "clean", you'll see things such as "！" rather than "!" in the translations. Don't worry about those - MechaTranslator would clean them up before using them for real.

EDIT:
Updated to make it not use old MechaTranslator's dictionary. Also updated for it to always use human-made translations.

I plan on having it also include Mecab/JParser information as comments if possible. This way translators-in-learning could use MechaTranslator as a first-pass while they go through each file.

 

[meh65]

Re: [TOOL] MechaTranslator for RPG Maker Trans

So i tried using this on a WOLF RPG game.

But uh...all it does is let me translate the map info.

I can't translate dialogue or anything.

*edit*

The map info is apparently the dialogue...but you can't simply overwrite the old files, since you have text and they are .mds

Not sure how you apply translation to wolf games.

P.S FOUND DEAD could be a good test game for WOLF translating.

 

[reddo]

Re: [TOOL] MechaTranslator for RPG Maker Trans

So i tried using this on a WOLF RPG game.

But uh...all it does is let me translate the map info.

I can't translate dialogue or anything.

*edit*

The map info is apparently the dialogue...but you can't simply overwrite the old files, since you have text and they are .mds

Not sure how you apply translation to wolf games.

P.S FOUND DEAD could be a good test game for WOLF translating.

You need to run MechaTranslator on the patch folder created by

You must be registered to see the links

. There's a tutorial here on ULMF for it.

Right now I'm not certain if MechaTranslator is looking into every folder created by WolfTrans (making it work with it was sort of a side project so I didn't test it thoroughly), but the translations I've been getting with a few Wolf games I tried on were somewhat random. Some games would end up mostly translated while others would have huge chunks left untranslated, etc.

I'll check FOUND DEAD out.


EDIT:
Apparently for FOUND DEAD the dialogue gets translated just fine, but something about the Wolf-equivalent to Actors.txt is getting translated and that seems to be causing some errors with MISSING PARTY which come up every step you take. Also there were 100+ files, but MechaTranslator only touched 45 of them, so definitely not getting all the files.

I'll take a better look at this game's patch files later.

 

[meh65]

Re: [TOOL] MechaTranslator for RPG Maker Trans

You need to run MechaTranslator on the patch folder created by

You must be registered to see the links

. There's a tutorial here on ULMF for it.

Right now I'm not certain if MechaTranslator is looking into every folder created by WolfTrans (making it work with it was sort of a side project so I didn't test it thoroughly), but the translations I've been getting with a few Wolf games I tried on were somewhat random. Some games would end up mostly translated while others would have huge chunks left untranslated, etc.

I'll check FOUND DEAD out.


EDIT:
Apparently for FOUND DEAD the dialogue gets translated just fine, but something about the Wolf-equivalent to Actors.txt is getting translated and that seems to be causing some errors with MISSING PARTY which come up every step you take. Also there were 100+ files, but MechaTranslator only touched 45 of them, so definitely not getting all the files.

I'll take a better look at this game's patch files later.

Ya, i was going to mention that after I noticed it. Also I used your program on zombie town gunsweeper and got an error after battle.

It was similar to the error i got in another game using the 8chan automated translator tool that also runs ruby and translates scripts.

Do you know what I am referring to or should i post a pic? I assume these errors have something to do with a conflict the translation provides when trying to execute something.

Oddly enough, the game I translated with the 8chan tool was ghost HOUSE gunsweeper, which gave a game crashing error after battles, but when I translated the same game with YOUR translator, no issue.

BUT when doing the zombie town gunsweeper (essentially a sequel), it would give a similar game crashing error fairly similar to the 8chan version of the ghost HOUSE gunsweeper game. Found that to be odd.