Improving the Android build environment

Chronoboy · Post by **Chronoboy** » Sun Jan 18, 2015 12:50 am

I was on IRC for a bit today as I was researching how the current Android build system for OHR works. I was patiently waiting for either Bob or TMC to arrive for I could ask them some questions. So, first let me iterate how the current Android build system for OHR is currently implemented...

In order to use the current build system, you must install the Android NDK, and have a working cross-platform build target working for the ARM process. This is used to actually recompile FreeBASIC and then GAME to an Android binary. Only once that is done, can you compile the APK with your RPG data embedded.

In contrast to a similar build environment, Python subset for Android, you actually do not need to have the Android NDK installed and working. All that is required is a working Android SDK, and Ant, nothing more. They are able to do this because they actually ship the Python subset for Android with pre-compiled ARM binaries.

I'd like to see about replicating a similar environment for OHR Android compiling. Whereas, Bob and/or TMC would compile a release binary of GAME for ARM using the current release version of OHRRPGCE. They would then place the compiled .so library files which were compiled from the NDK into a package which can be downloaded onto any platform of choice, Linux, OS/X, and Win. I believe I can easily fork the Python subset for Android to be compatible with OHR project building. Then, as a result, all you need to do is the following commands to install the SDK/Ant:

Code: Select all

./android.py installsdk

I will use the code basically unmodified from the Python subset for Android for this part. Then when you want to prep your game for deployment on Android and/or OUYA, you just need to configure it as such:

Code: Select all

./android.py configure

Which will ask you some questions about your RPG, like it's title, an icon, and other relevant information. Similar to what Python subset for Android asks. Then finally to build the actual APK with your RPG file:

Code: Select all

./android.py build

Thus, streamlining the entire process. I can start the process of making this possible now, however I am struggling to figure out how the game uses gamedata.zip[00-99]. I need to know how this process works in order to make it possible to build APKs with embedded RPG data easily. I already have access to precompiled .so ARM binaries by opening up an existing APK package.

http://pygame.renpy.org/android-packaging.html

Basically, my point here being, is that there should be no need to download and install the NDK + cross-compiling toolchain. It isn't needed with PyGame subset for Android, as they found an easy way to make PyGame accessible on Android. I would like to make OHR more accessible on Android, and I don't think it would take any more than a couple days of hard work and forking to make this a reality for this community.

Oh, and don't worry, this new method will still allow for custom keypad skins specific for your game, a custom icon, etc... The Java classes and the APK are still being compiled in this process, it's just the SDL and GAME binary code that will be pre-compiled(just like how it is when you download CUSTOM/GAME for Win/Lin/X).

Post by **TMC** » Sun Jan 18, 2015 2:14 am

This is great news.

I caught Chronoboy on IRC.

Chronoboy · Post by **Chronoboy** » Sun Jan 18, 2015 4:02 am

Here is what I got so far, tons of work, and is coming along. Currently, the only issues I need to look into are how to get Ant to see the Google play services being used here.

https://bitbucket.org/kveroneau/ohrrpgce-android

Hopefully I can get this done within the next couple of days. I haven't updated the install_sdk scripting yet, which fully automates the downloading and installing of both the Android SDK and Ant. That will come shortly.

Chronoboy · Post by **Chronoboy** » Mon Jan 19, 2015 4:09 am

SUCCESSFUL APK COMPILE!
Just made a commit, which I was able to build an APK successfully! I still have to test it on my phone(it's currently charging in my room), I will update this post as soon as I am able to properly test it. Looking inside the APK itself, it looks proper, just like one James would have built himself.

How to use...
Download a package from here: https://bitbucket.org/kveroneau/ohrrpgc ... fb79cb.zip

Install Python and Java onto your operating system of choice...
Enter the directory with android.py and run the following:

Code: Select all

python android.py installsdk

This will take care of downloading the Android SDK and installing everything you need easily for you. No need to read about how to use the SDK or any thick manuals here.

Once, the SDK is downloaded and installed, create a new directory in the same directory as android.py for your specific game you wish to build into an APK. The current process here is now fully automated, so I don't apologize. Just place your .RPG game file in that directory.

Once all that is finally done, you are ready to configure your Android project, so run python android.py configure <project directory name, it will ask you a series of questions relating to the Android port of your game. Answer them as best you can, feel free to post any questions you may have here.

You only need to run configure once really, unless you want to update your version number, which you will need to do if you plan on uploading to the Android market.

Once configure is all done, you are now ready to build your APK, so run

Code: Select all

python android.py build <project> release

, you can also add an install option if you have an Android emulator running or your Android phone connected to auto-install and launch the APK right on your device. This feature is great for testing out the built APK as fast as you can say spiffy.

Example run to build an APK for Walthros:

Code: Select all

$ python android.py installsdk
$ mkdir walthros
$ cp $HOME/RPGs/walthros.rpg walthros
$ python android.py configure walthros
$ python android.py build walthros release

Enjoy! The resulting APK is Signed, when you install the SDK, it should ask you about your Android keystore for signing APKs. So this is an all inclusive process with minimal headaches involved.

Results may currently vary, as I haven't completely tested a built APK on my phone, I believe there may still be work needed with the Globals.java. But this isn't much work.

Spoonweaver · Post by **Spoonweaver** » Mon Jan 19, 2015 4:22 am

You can literally have all my Slimebucks now.

Post by **Bob the Hamster** » Mon Jan 19, 2015 4:31 am

Cool! I'll be testing this out this week. Does it do any signing yet, or is it only intended to go as far as making the unsigned apk?

Chronoboy · Post by **Chronoboy** » Mon Jan 19, 2015 6:08 am

Bob the Hamster wrote:Cool! I'll be testing this out this week. Does it do any signing yet, or is it only intended to go as far as making the unsigned apk?

It's all inclusive, the final APK is fully signed using the key created during the installsdk process.

Here is the listing of the output directory if your curious what you get:

Code: Select all

total 201M
-rw-r--r-- 1 kveroneau kveroneau 1.5K Jan 18 22&#58;53 AndroidManifest.xml
-rw-r--r-- 1 kveroneau kveroneau  104 Jan 18 22&#58;53 AndroidManifest.xml.d
-rw-r--r-- 1 kveroneau kveroneau  182 Jan 18 22&#58;54 build.prop
drwxr-xr-x 3 kveroneau kveroneau 4.0K Jan 18 22&#58;53 classes
-rw-r--r-- 1 kveroneau kveroneau 4.2M Jan 18 22&#58;53 classes.dex
-rw-r--r-- 1 kveroneau kveroneau  21K Jan 18 22&#58;53 classes.dex.d
drwxr-xr-x 2 kveroneau kveroneau 4.0K Jan 18 22&#58;53 dexedLibs
-rw-r--r-- 1 kveroneau kveroneau  120 Jan 18 22&#58;53 jarlist.cache
-rw-r--r-- 1 kveroneau kveroneau  109 Jan 18 22&#58;53 proguard.txt
drwxr-xr-x 5 kveroneau kveroneau 4.0K Jan 18 22&#58;53 res
-rw-r--r-- 1 kveroneau kveroneau  17K Jan 18 22&#58;53 R.txt
-rw-r--r-- 1 kveroneau kveroneau  48M Jan 18 22&#58;53 Walthros-1.ap_
-rw-r--r-- 1 kveroneau kveroneau  58K Jan 18 22&#58;53 Walthros-1.ap_.d
-rw-r--r-- 1 kveroneau kveroneau  50M Jan 18 22&#58;54 Walthros-1-release.apk
-rw-r--r-- 1 kveroneau kveroneau  50M Jan 18 22&#58;54 Walthros-1-release-unaligned.apk
-rw-r--r-- 1 kveroneau kveroneau  50M Jan 18 22&#58;53 Walthros-1-release-unsigned.apk
-rw-r--r-- 1 kveroneau kveroneau 1.6K Jan 18 22&#58;53 Walthros-1-release-unsigned.apk.d

Post by **TMC** » Mon Jan 19, 2015 6:38 pm

Chronoboy and I spent last night on this and found that there were still a number of things that were broken, but in the end I managed to package a working .apk for a game other than Walthros. The biggest problem was that the sdl1.2-android build system patches Globals.java according to AndroidAppSettings.cfg, but this RenPy-based build system needs to do its own patching of Globals.java. And I wasn't aware that I actually had to edit AndroidAppSettings.cfg

James, did you document the steps you make when you package a game as an apk anywhere? I see a few comments scattered about the Android Port page about a few changes that need to be made to AndroidAppSettings.cfg, but it doesn't look complete. What about the icons?

Also, it turns out that it's not actually necessary to compile any Java files when building an .apk, provided that we modify it to read a fews settings from a file rather than hardcode code them in Globals.java. So it seems like we might need rather little of the android SDK.

Post by **Bob the Hamster** » Tue Jan 20, 2015 1:00 am

I do need to write down exactly what I do, just been super busy this week

Chronoboy · Post by **Chronoboy** » Tue Jan 27, 2015 6:10 am

Hey you guys. So TMC put in lots of great commits. I just wanted to check and see if Bob had some time to go through quality assurance and confirm that the build system is in tip-top shape and can be considered suitable for everyday use by the community?

Is there any work that is left to do TMC that I can help out with?

Post by **Bob the Hamster** » Tue Jan 27, 2015 2:16 pm

I haven't yet, thank you for reminding me! So busy lately :)

Post by **Bob the Hamster** » Tue Jan 27, 2015 7:03 pm

Testing!

Looks awesome so far, but some things will need to change before I can use it to replace the process I am currently using.

I notice that it creates one single keystore to be shared between all projects. I will definitely need the ability to use different keystores for each rpg. Right now I make a unique one for every game. (when I want to share between two games from the same author I just copy the keystore file)

Why is VIBRATE permission default? We don't support that yet (or ever?) anyway. The packages I have been currently building have INTERNET and WRITE_EXTERNAL_STORAGE permissions, although I suspect that they never actually use either of them for anything. (that is a topic for later discussion)

After I ran ./android.py build proj/baconthulhu release I was briefly baffled as to where to find my apk file. Oh! There it is inside bin/

After installing the resulting apk on my phone, I tried testing it. I started out in the filebrowser pointing to /sdcard/Android/data/com.hamsterrepublic.baconthulhu/files/baconthulhu.rpgdir

I backed up one level, and tried to select the rpgdir and got an error message saying that general data is missing.

I did adb shell to examine the files, and discovered that although my baconthulhu.rpg was unlumped to baconthulhu.rpgdir, all the files inside had uppercase names. I think that no matter what the case of the filename in the rpg file is, it should be forced to lowercase when unlumped. (and the rpg file format specification should be updated to reflect this. I know that all lumps are always accessed via lowercase names, and the implementation in lumpfile.bas strictly enforces this, but I don't think we ever said it anywhere in the docs)

We will also have to deal with icon files, button image files, and possiby a few other things, but overal my impression of this tool so far is really good.

This is definitely on the right track, and I think it is the future of ohrrpgce android packaging :)

If I get some more time later today, I will post some more about how I am currently building packages, which should shed some more light on the icons and button images

Post by **Bob the Hamster** » Tue Jan 27, 2015 8:33 pm

Here is an example of one of the project directories that I am currently using with sdl-android

http://hamsterrepublic.com/tmp/baconthu ... mplate.zip

So first I have a copy of my patched sdl-android environment checked out

Code: Select all

 git clone git&#58;//github.com/bob-the-hamster/commandergenius.git ohrrpgce

the baconthlhu project directory goes in sdl-android/project/jni/application/baconthulhu

The stuff near the top of AndroidAppSettings.cfg is the only stuff I usually change:

Code: Select all

AppName="Crypt of Baconthulhu"
AppFullName=com.hamsterrepublic.baconthulhu
# Increment the following everything time you change any settings in this file,
# otherwise they won't change! &#40;which is super annoying&#41;
AppVersionCode=21
AppVersionName="1.0.21"

ohrrpgce-android already seems to handle all those settings very nicely.

icon.png is the icon image. ouya_icon.png is optional. Just putting these files in this location is enough to tell sdl-android to use them.

The buttonhack.sh script is something I run to copy the button and dpad images into the (strange) places where sdl-android expects them to be.

The ohrrpgce actually supports up to 6 touchscreen buttons, even though by default only 2 are used, and I think the most any one game uses is 3 (Motrya, battles only)

srcdata/rezip.sh is a script I use for updating AndroidData/gamedata.zip

ohrrpgce-android's handling of rpg files already seems simpler and more generic than mine (pending a fix for the uppercase unlump problem)

the tmp folder with the ohrrpgce source in it is actually a symlink on my system, pointing to the appropriate place in my ohrrpgce source tree.

After running rezip.sh and making sure my version number is updated in AndroidAppSettings.cfg, I make sure that the sdl-android/project/jni/application/src symlink is pointing to baconthulhu and then from the sdl-android directory I run:

Code: Select all

./build.sh release
src/ohrsign.sh

That ohrsign.sh does the signing with baconthulhu.keystore (which is not actually included in my example zip)

I have been creating my keystore files with -validity 36540 so they are good for 100 years, but in restrospect, I can't remember why I didn't choose an even longer time.

So as you can see, my system is just a few crude shell scripts cobbled on top of the sdl-android build system. I am sure my system is not doing anything that ohrrpgce-android can't do much better.

Post by **TMC** » Wed Jan 28, 2015 10:58 am

Oh, I got distracted by moving house, and didn't even have my computer for most of a week. Have a bunch of stuff to catch up on now. I agree that while it works, it needs polish and documentation.

The INTERNET permission is only needed if the game data files are downloaded from an external source. Assuming that your game isn't more than 50MB, it's not needed. I wonder whether that permission is needed if you download an .obb data file from the Play store. I assume that WRITE_EXTERNAL_STORAGE is needed to access the app's (non-private) storage space under Android/ on the sdcard.

In never realised that it had its own unlump routine. The code is so short that I didn't recognise it :)

Is Motrya's third button the run button? Why don't all games using the battle system need that?

(BTW there are 365.24 days in a year, not 365.4)

Post by **Bob the Hamster** » Wed Jan 28, 2015 4:48 pm

TMC wrote:I assume that WRITE_EXTERNAL_STORAGE is needed to access the app's (non-private) storage space under Android/ on the sdcard.

I originally assumed the same thing, but now I am not so sure. I think apps might *ALWAYS* get write access to /sdcard/Android/data/com.appname.whatever/files no matter what, and that WRITE_EXTERNAL_STORAGE actually means it can do whatever the heck it wants in /sdcard/*

Further research required, but I just haven't gotten around to it for 4 months :)

TMC wrote:Is Motrya's third button the run button? Why don't all games using the battle system need that?

Yeah, it is for run, but Motrya needs it because it uses active-time battles in a way that sometimes relies on being able to hold down the "skip turn" button while you wait for other hero's ready meters to fill up.

I think the Plip Tamer Jumping Plip battle in Wandering Hamster used to have the same issue, before I retooled it and added "Defend"

TMC wrote:(BTW there are 365.24 days in a year, not 365.4)

Oops! My secret has been discovered! I am actually a time-traveller from 1.38 million years in Earth's future, and I accidentally used the year length from my home-era. Please don't tell anyone else!