How To Compile Your Own Kernels, Modules, Tinboot (NAND boot), and More for Beginners

[natemcnutty]

This is a quick guide on getting started with compiling Android for the Touch Pro 2 from the git repositories. I am going to try to keep this short and simple, so I will be using my personal preference of software though others will work fine. This was made at the request of few people in the NAND testing thread, but I will try to include information for the non-NAND people too.

Required Tools
Virtualbox
Ubuntu (32bit is fine) Note: I recommend 10.04 for better stability with Virtualbox, but 10.10 does work just fine.
10 GB of disk space
High Speed Internet Connection
2 hours of time (about 30 minutes of actual interaction)

Installing Linux
First, install VirtualBox leaving everything as default. Now, click New and type Ubuntu 10.4 (or 10.10) for the name. Make sure Ubuntu is in the name so it auto-configures some options for us. For memory, I recommend 1 GB unless you have more than 4GB of RAM. For drives, leave the defaults and just keep clicking Next until it finishes.

Before you click Start, we want to optimize the settings and attach the ISO so it will boot from “CD”. To do this, Select Ubuntu 10.4 (or 10.10) and click Settings. Here are the settings you need to change:
Under the System section on the Motherboard tab, check Enable IO APIC, and on the Processor tab, Set Processor Count to the number of cores on your processor.

Under the Storage section, click on IDE Controller and change Type to ICH6. Now, click on the CD labeled Empty, then click the Folder with the green up arrow to open the Virtual Media Manager. Click Add and browse to the ISO you downloaded, click Open, click Select, and then click OK.
Now we can start the VM and install Ubuntu. Click Start and wait for the plum background with the keyboard at the bottom, press the down arrow and select Install Ubuntu. Keep clicking Forward until you can click Install Now. Follow the rest of the prompts while Ubuntu installs.

Preparing the Environment
Once Ubuntu is up and running, we need to install three software packages: Virtualbox Additions, git-core and wine. To install the VBox Additions, click Devices-CD/DVD Devices, and select VBoxGuestAdditions.iso. It never auto-runs for me, so I have to click Places, VBOXADDITIONS, then run autorun.sh, install and restart so we can copy/paste between Windows and Ubuntu

Terminal is where we spend the rest of our time, so click Applications-Accessories, and drag Terminal to the taskbar. Now open Terminal and copy/paste the following into it:

sudo apt-get install git-core
sudo apt-get install wine
mkdir android
cd android
git clone git://android.git.kernel.org/platform/prebuilt.git

Note: If you prefer codesourcery's toolchain, you should download this one: Link

NAND users will also need the following:

git clone git://gitorious.org/~acl/linux-on-qualcomm-s-msm/linux-msm-rhod-nand.git
git clone git://gitorious.org/tinboot-for-linux-msm/tinboot-linux-msm.git
git clone git://gitorious.org/tinboot-for-linux-msm/bootenv.git

Haret users will need the following:

git clone git://gitorious.org/linux-on-qualcomm-s-msm/linux-msm.git
git clone git://gitorious.org/xdandroid/rootfs.git

Cloning these could take a while, especially for the prebuilt toolchain from Google. Just copy/paste and go make dinner

Compiling Kernel and Modules
Now that you have everything downloaded, we are ready to start compiling. The developers have done an awesome job with the Makefiles, so we only need to worry about the toolchain (to compile ARM code using an x86 processor) and the defconfig (configuration that determines what gets built).

If you wanted to simply compile the zImage and modules, using Terminal, copy/paste the following:

cd android
mkdir modules
cd linux-msm-rhod-nand
make clean
make ARCH=arm htc_msm_nand_defconfig
make ARCH=arm CROSS_COMPILE=~/android/prebuilt/linux-x86/toolchain/arm-eabi-4.4.0/bin/arm-eabi- INSTALL_MOD_PATH=~/android/modules zImage modules modules_install

Your zImage will be located under ~/android/linux-msm-rhod-nand/arch/arm/boot/ and the modules will be output to ~/android/modules (or wherever specified by INSTALL_MOD_PATH).

Haret users should use the following:

cd android
mkdir modules
cd linux-msm
make clean
make ARCH=arm htc_msm_android_defconfig
make ARCH=arm CROSS_COMPILE=~/android/prebuilt/linux-x86/toolchain/arm-eabi-4.4.0/bin/arm-eabi-  INSTALL_MOD_PATH=~/android/modules zImage modules modules_install

On NAND, we are now putting all of the modules into a squash compressed file. This takes a little bit of setup, but it allows us to support multiple kernels. Keep in mind that this requires squashfs tools version 3.4. The modules need to be in the root of the squash file, and the name must be 2.6.27.46-01192-version.sqsh.

To set upsquashfs tools, download squashfs tools from here: ftp://ftp.slax.org/source/slax/sqlzm...shfs3.4.tar.gz
Now, extract the files, then browse into the squashfs3.4/squashfs-tools folder, then type make, then sudo make install.

On Haret, they load the modules from a tar.gz on in /sdcard/andboot. The name will be modules-2.6.27.46-01192-version.tar.gz. To get the version, try running:

cat ~/android/linux-msm-rhod-nand/include/config/kernel.release

androidupdate.tgz
Since the modules are now applied through androidupdate.tgz, you will need to make a directory structure in ~/android to look like this:

~/android
|--- androidupdate
|--- data
|--- system
|--- lib
|--- modules

The modules squash file should be put under /androidupdate/system/lib/modules. Also, any files that you want updated in either data or system can be placed in here. For example, if you want to include certain APK's, you can put them under /system/app or /data/app.

Compiling Initrd.gz
For NAND, we have our own initrd.gz in the bootenv git that we cloned. We try to rebuild these and commit them every time we update bootenv, but sometimes the change doesn't affect enough rebuild and upload. So, I always recommend rebuilding initrd.gz fresh and copying that into ~/android/tinboot-linux-msm/kernel. You can build this very easily by simply browsing to ~/android/bootenv and running sh buildInitrd.sh

Compiling Tinboot
Tinboot has been changed significantly to only use 1 cmdline for all devices . If you are compiling for NAND, you will need to use the initrd.gz from the bootenv git we cloned. If you are compiling for Haret, you will need the initrd.gz from Main Page - The XDAndroid Project. Just grab the latest and copy that along with the zImage into ~/android/tinboot-linux-msm/kernel. If you want to overclock the processor, change the lcd density, or change anything else, you can modify ~/android/tinboot-linux-msm/tinboot/tinboot2.S

Changing this file is extremely simple, but you can really mess things up if you change some of the code. I would only recommend changing the line that starts with .asciz. So, let's open tinboot2.S in gedit, find .asciz, and make the changes to the command line. If you want to overclock, add acpuclock.oc_freq_khz=716800 (change number to whatever overclock your phone can handle). If you are cooking a ROM, you can change the mtdparts to change the size of the partitions. All values are in HEX, and I will have a separate post for NAND chefs that will explain this much better.

We are now ready to compile. Unfortunately, the scripts on git require quite a bit of tweaking to get working. For now, try using the full build or tinboot script that I have posted in the second post.

Making the RootFS (Haret Only)
The RootFS is a little different since there is no simple make command. The current way to "compile" the RootFS is through the script located here: generate-rootfs.sh

Attempting to modify this script always ended up giving me weird genext2fs warnings, and I found it to be much cleaner to just use the script as is. Instead, I choose to inject my modified init file right before genext2fs creates rootfs.img. Also, note that the script will throw a warning about .git not found or something similar. That is something from the gitclean.sh and can be ignored.

cd "${TMPDIR}"/"${REPODIR}"
./scripts/gitclean.sh
cp -a . "${TMPDIR}"/rootfs

# copy modified init file
rm "${TMPDIR}"/rootfs/init
cp "${ANDROID}"/rootfs/init "${TMPDIR}"/rootfs

chmod u+s "${TMPDIR}"/rootfs/bin/pppd "${TMPDIR}"/rootfs

The only catch here is that you have to keep an eye out for updates to init as you will be overwriting any future updates. I used to keep a simple text file with my script that keeps track of what changes I made to init so I could easily add them back in the future if init was updated.

[natemcnutty]

Below is a copy of my full build script for NAND. Haret users will have to make a few changes which I'll explain later. Just copy/paste it into gedit and save as fullbuild.sh and run it from terminal. You may need to change the directory paths at the beginning if you don't put them where I recommended.

#!/bin/sh
#-------------------------------------------------------------------------
# Build Script for linux-msm-rhod-nand (Android on HTC) kernel, modules, XIP, and NBH
#-------------------------------------------------------------------------
#
# Set the following variables
# ANDROID -- Root director for Android files
ANDROID=~/android
# TINBOOT -- tinboot directory
TINBOOT=$ANDROID/tinboot-linux-msm
# KERNEL_PATH -- directory containing the linux-msm kernel source
KERNEL_PATH=$ANDROID/linux-msm-rhod-nand
# TOOLCHAIN_PATH -- directory containing the arm toolchain
TOOLCHAIN_PATH=$ANDROID/prebuilt/linux-x86/toolchain/arm-eabi-4.4.0/bin/arm-eabi
# MODULES_PATH -- directory for modules
MODULES_PATH=$TINBOOT/modules
#-------------------------------------------------------------------------

# do some cleanup before building if zImage already exists
echo "Removing previous zImage"
if [ -f $TINBOOT/kernel/zImage ] ; then
     rm $TINBOOT/kernel/zImage
fi

# build kernel and modules
#export ARCH=arm
cd $KERNEL_PATH
make clean
make ARCH=arm htc_msm_nand_defconfig
make ARCH=arm CROSS_COMPILE=$TOOLCHAIN_PATH- INSTALL_MOD_PATH=$MODULES_PATH zImage modules modules_install
[ $? -eq 0 ] || fail "Kernel compilation failure"

# copy all drivers to single level directory for compression
cd $MODULES_PATH
KERNMODS=$(find -name "*.ko")
if [ ! -d $ANDROID/androidupdate/data/modules ] ; then
     mkdir -p $ANDROID/modules
fi
for i in $KERNMODS ; do
     cp $i $ANDROID/modules
done

# make squash files
KER_VER="$(cat $KERNEL_PATH/include/config/kernel.release)"
mkdir -p $ANDROID/androidupdate/system/lib/modules
mksquashfs $ANDROID/modules $ANDROID/androidupdate/system/lib/modules/$KER_VER.sqsh

# compress modules to a tar.gz and move to output
echo "Outputting modules to Desktop"
cd $ANDROID/androidupdate
tar -cvzf $ANDROID/androidupdate.tgz data system
rm -rf $ANDROID/modules
rm -rf $ANDROID/androidupdate/system/lib/modules

# clean up modules
rm -Rf $MODULES_PATH
mkdir $MODULES_PATH
rm -Rf $KERNEL_PATH/lib/modules

# move new kernel
mv $KERNEL_PATH/arch/arm/boot/zImage $TINBOOT/kernel

#-----start build XIP-----
# remove old XIP if it exists
cd $TINBOOT
if [  -f xip/rhod ]; then
    rm xip/rhod
    echo Deleting old XIP
fi

# make XIP folder if it does not exist
if [ ! -d "$TINBOOT/xip" ] ; then
mkdir $TINBOOT/xip
fi

# compile tinboot XIP
echo "Start compiling tinboot/xip for rhod....."
$TOOLCHAIN_PATH-as  tinboot/tinboot2.S -o tinboot.o --defsym rhod=1 --defsym MTYPE=2292
$TOOLCHAIN_PATH-objcopy tinboot.o -O binary tinbootxip
mv tinbootxip xip/rhod

# cleanup tinboot
rm tinboot.o

# make sure XIP build was successful
if [ ! -f xip/rhod ] ; then
    fail "XIP build failed. XIP file was not found."
fi

#-----start build NBH-----
# clean up previous log file
if [ -e "tools/log" ]; then
rm tools/log
fi

# compile tinboot
cp tools/rhod_payload2 os.nb.payload

echo "Inserting tinboot into payload"
wine tools/osnbtool -c os.nb.payload 1 xip/rhod >> tools/log
mv os.nb.payload.NEW os.nb.payload >> tools/log

echo "Inserting blank imgfs into payload"
wine tools/ImgfsToNb.exe  tools/imgfs.bin os.nb.payload os-new.nb.payload >> tools/log

echo "Creating os.nb portion of nbh"
tools/nbmerge < os-new.nb.payload > os-new.nb

echo "Creating NBH"
wine tools/yang.exe -F RHODIMG.NBH -f os-new.nb -t 0x400 -s 64 -d RHOD****0 -c 11111111 -v Tinboot -l WWE >> tools/log

# clean up payloads
mv RHODIMG.NBH $ANDROID/RHODIMG.NBH
rm os.nb.payload
rm os-new.nb.payload
rm os-new.nb

# clean up tinboot directory
rm -rf xip

What the heck is zImage, modules, tinboot, rootfs, initrd, and System.ext2?
This section will be updated slowly as I have time.

The zImage is the kernel (which is required to run the operating system). We currently use the linux kernel 2.6.27.46.

The modules are your drivers. These are a matched pair with the zImage/kernel. It is also important to clean up the modules periodically when building the kernel/modules or you will start running out of space

Awesome explanation of the kernel and its modules here: http://www.howtogeek.com/howto/31632...at-does-it-do/

Tinboot is the bootloader for running Android from the NAND. Currently, we put initrd.gz and the zImage on the NAND while the zImage, rootfs, and System.ext2 are pulled from the SD Card.

RootFS is a special type of ramfs file system. It basically lets us use part of the memory as a partition for caching files. This gives us a nice performance boost

Initrd is an initial RAM disk that is temporary and used only for booting. Generally it uses a slightly different scheme than rootfs does.

System.ext2 is your file system. This is technically your hard drive where all of the Android files and applications are stored. It uses the ext2 file system which is an extended file system designed specifically for linux.

[Lmiller1708]

Thanks for the info on this Nate!

[coolsilver]

I've compiled for HARET before once. This really helps though.

Thanks!

[Lmiller1708]

Can't seem to get past this step:

make ARCH=arm linux-msm-nand-defconfig

What I see:

~/Desktop/Android/linux-msm-rhod-nand$ make ARCH=arm linux-msm-nand-defconfig
  HOSTCC  scripts/basic/fixdep
scripts/basic/fixdep.c: In function ‘traps’:
scripts/basic/fixdep.c:377: warning: dereferencing type-punned pointer will break strict-aliasing rules
scripts/basic/fixdep.c:379: warning: dereferencing type-punned pointer will break strict-aliasing rules
  HOSTCC  scripts/basic/docproc
make[1]: *** No rule to make target `linux-msm-nand-defconfig'.  Stop.
make: *** [linux-msm-nand-defconfig] Error 2

Everything else went good. Any Ideas?

Thanks!

[natemcnutty]

Originally Posted by Lmiller1708

Can't seem to get past this step:

make ARCH=arm linux-msm-nand-defconfig

What I see:

~/Desktop/Android/linux-msm-rhod-nand$ make ARCH=arm linux-msm-nand-defconfig
  HOSTCC  scripts/basic/fixdep
scripts/basic/fixdep.c: In function ‘traps’:
scripts/basic/fixdep.c:377: warning: dereferencing type-punned pointer will break strict-aliasing rules
scripts/basic/fixdep.c:379: warning: dereferencing type-punned pointer will break strict-aliasing rules
  HOSTCC  scripts/basic/docproc
make[1]: *** No rule to make target `linux-msm-nand-defconfig'.  Stop.
make: *** [linux-msm-nand-defconfig] Error 2

Everything else went good. Any Ideas?

Thanks!

Nice catch. It should have been htc_msm_nand_defconfig. Updating now

[Lmiller1708]

Originally Posted by natemcnutty

Nice catch. It should have been htc_msm_nand_defconfig. Updating now

That seemed to work.
But the Next Step:

make ARCH=arm  CROSS_COMPILE=~/android/prebuilt/linux-x86/toolchain/arm-eabi-4.4.0/bin/arm-eabi- INSTALL_MOD_PATH=~/android/modules zImage modules modules_install

I get this:

make: /home/lance/android/prebuilt/linux-x86/toolchain/arm-eabi-4.4.0/bin/arm-eabi-gcc: Command not found
  CHK     include/linux/version.h
make[1]: `include/asm-arm/mach-types.h' is up to date.
  CHK     include/linux/utsrelease.h
  SYMLINK include/asm -> include/asm-arm
ln: creating symbolic link `include/asm': Operation not permitted
make: *** [include/asm] Error 1

[natemcnutty]

Originally Posted by Lmiller1708

That seemed to work.
But the Next Step:

make ARCH=arm  CROSS_COMPILE=~/android/prebuilt/linux-x86/toolchain/arm-eabi-4.4.0/bin/arm-eabi- INSTALL_MOD_PATH=~/android/modules zImage modules modules_install

I get this:

make: /home/lance/android/prebuilt/linux-x86/toolchain/arm-eabi-4.4.0/bin/arm-eabi-gcc: Command not found
  CHK     include/linux/version.h
make[1]: `include/asm-arm/mach-types.h' is up to date.
  CHK     include/linux/utsrelease.h
  SYMLINK include/asm -> include/asm-arm
ln: creating symbolic link `include/asm': Operation not permitted
make: *** [include/asm] Error 1

Try make clean, then do the defconfig one more time, then the last make. Also, double check and make sure the arm-eabi-gcc is there. If not, do a git pull in the prebuilt directory.

[[ACL]]

FYI this "how to" is ACL approved

[InvincibleLiving]

This is a GREAT thread, thanks Nate