Touch Pro / WM6 file system caching and performance discussion

[FormerPalmOS]

EDIT - for these changes to be effective on the root file system, they have to be made to boot.hv which must be cooked into the ROM - they will not be effective if made to the registry you can edit after the system boots.

Mods - I am not sure if this applies equally to GSM and CDMA TPs or just CDMA. I am working with a CDMA version. If someone determines that this info is relevant to CDMA only, not also to GSM, please move this to the CDMA sub-forum.

After spending some time learning the WM6 file system and digging through a lot of (incorrect) info available about performance tweaks, I decided to post what I've discovered works and doesn't work specific to this hardware. Much of the WM6 specific stuff comes from MSDN. Some of this was finished off at 3am so anyone feel free to correct me.


Part 1 - How caching works on WM6
There are three types of cache - a disk cache, a file filter cache and a replicated storage cache. The disk cache sits between a file system (such as FAT) and caches disk metadata (the file allocation table itself and directory metadata). By caching this, accesses to the storage hardware are minimized - access to the RAM cache is much faster. The file filter cache understands how files are accessed and performs read-ahead of the file (much more effective than a dumb read-ahead on a disk cache) and keeps most frequently accessed files in the cache. It uses the cached meta data from the disk cache in its operation. Note however, that this cache is used only for data, not for static files (like executables and DLLs). The replicated storage cache is used for static files - it loads a copy of a ROM-based static file to a replicated storage folder and caches a pointer to this called an Object Identifier or OID. When the OS requests the OID, the replicated storage filter retrieves it fromt he RAM-based copy, not the ROM (which is faster). Note - this cache can only be applied to the root filesystem, not to SD cards. The other two can also be applied to SD cards.

The disk cache and replicated storage sizes can be configured in the registry - the parameters are described in part 3. The file system filter cache uses part of the pagepool and can only be changed in the ROM. This is unfortunate since IMHO this is where the biggest bang for the buck may be, but changing it involves a re-flash.


Part 2 - File system and filters
The HTC Touch Pro uses only two file systems - FAT and IMGFS. FAT is what we as users interact with - FAT lives on both the internal flash storage and on the SD Card. IMGFS appears to be used only for updating the flash (ROM) image.

Filters sit between the actual storage hardware and the OS in a stack. The actual storage hardware is at the bottom of the stack and the filter on the top of the stack is what is presented to the OS. This allows actions such as encryption, caching or compression to be performed transparently to the user/OS. The file cache and replicated storage cache are both filters. Encryption for the data stored on the SD card is also done via an encrypting filter. Finally, HTC implements an HTCFSFILTER - I have no idea what this does - just that it is given with order 0, meaning it is loaded last (filters are loaded based on an order key from highest to lowest, where highest is closest to the hardware and lowest is closest to the user/OS. I haven't tried, but someone willing to hard reset their device if this experiment fails could try bypassing this filter to see if anything breaks.

The combination of a file system and filters is a storage profile. WM6 includes a number of default storage profiles like CD/UDFS, USB drives, FlashDrive, etc. The HTC Touch Pro implements many of these (though many are also unused), plus a couple of added ones. The only used profiles are FLASHDRV, SDMemory and BTDISK (Bluetooth disk). GSM devices may also use the MegaSIM profile. The hardware driver for a storage device tells the storage manager what profile it is using.


Part 3 - Registry keys and potential tweaks
The registry keys that control the file system are contained in HKLM\System\StorageManager (I'll just use ... from here out but this is the parent key I'm referring to). There is a key for each file system (...\BTFS, ...\FATFS and ...\IMGFS) and for the filters (...\Filters - the individual filter global parameters can be applied there). Parameters applied globally apply to any instance of the file system or filter, unless the same parameter exists in the storage profile. The storage profiles are contained under the ...\Profiles key. All new keys are type DWORD unless stated otherwise.

Looking at a storage profile (such as FLASHDRV), you will see an entry for two file systems (FATFS and IMGFS). The cache parameters specific to the on-board storage are configured here. You will also see a Filters key which calls out the file system replication filter (fsreplxfilt) - recall this filter can ONLY be applied to the root file system (that's why it's not applied globally). You will see that FATFS is also the only applied file system under SDMemory.

Looking at the parent FATFS file system entry, you will see that the CacheFilt (file cache filter), ENCFilt (encryption filter) and HTCFSFILTER are all applied globally. Thus the on-board storage using FAT uses all four filters (the above three plus file replication) and the SD FAT uses only the above three. Note - I do not believe that the encrypting filter should be applied here - I believe it should be applied only to the SDMemory profile, but if you remove it here and add it to the SDMemory profile, it reappears. It may be an interesting experiment to adjust the rom-based boot registry to "fix" this - I don't know if data stored in the on-board flash device is encrypted or not. Maybe it is - if so, this filter must remain. Removing it may yield a slight performance increase though if someone stole your device they could potentially access the on-board device via JTAG or some other hardware means and retrieve data. Encryption would prevent this.

So - here are the keys that are relevant and some that are not (but are widely discussed). If I say a parameter is applied globally that means it is not applied in a profile. Profile means it is:

FATFS
CacheSize (global - obsolete) - this is a common "tweak" that is called out, but this value is deprecated in WM6. This value did serve a purpose in WM5 so tweaks here are likely leftovers. But changing this value in WM6 does not seem to do anything. Note - Diamond Tweak and Advanced Config both change this value. See below for the WM6 values that replaced this. I just eliminated this key.
EnableCache (global) - enables the disk cache
Flags and FriendlyName (global) - deprecated (left-over from WM5 - separately named flags now apply. Though I haven't experimented here, the MS documentation would suggest that changes to the Flags value would have no effect)
Paging (global) - determines if files can be loaded to the file filter cache in pages (more efficient) or must be loaded all at once. Should be 1 - I haven't tried, but I would assume setting this to 0 would slow things down.

Filters applied to FATFS (...\FATFS\Filters)
CacheFilt - This is the file system filter. Keys are:
LockIOBuffers (new key) - set to dword:00000001 - this is likely the default if this key isn't specified since it sounds like things would break if it were not. Specifies that the critical I/O buffers for the file cache filter should be locked. I added this key, but again I don't think it's necessary - I assume it defaults to locked.
Order - should be 2 (this filter is loaded between the HTC filter and the file replication filter (if present)

ENCFilt - This is the encryption filter. Keys are:
Order - should be 5, which puts this closest to the hardware

HTCFSFILTER - No idea what this does
Dll - htcfsfilter.dll
Order - 0 which puts this closest to the user/OS

Global Filters (...\Filters)
ENCFilt - settings for the Encryption system. Note - you will see a key called DeviceGUID - this seems to change with every reflash / hard reset. Could this be (or drive) the key for encryption? I assume this isn't changeable, but an interesting experiment would be to hard-code a value here into a ROM and see if the value persists with hard resets or re-flashes of that same ROM. If so, perhaps you wouldn't lose what's on your SD card if it is encrypted...

fsdspy - this seems to be a sample. I tried removing it but it comes back at soft reset. Something else that could be potentially removed in a custom ROM.

fsreplxfilt - settings for the file replication filter. Keys are:
ReplStoreCacheSize - this is the number of OID entries that are cached. The tweaking utilities change this, however they refer to this as a number of sectors and this is inaccurate. This value controls the number of file mappings that are cached. What I don't know is if there is one OID per file or multiple. If one, then 8192 seems like a reasonable number here. If more than one, then a larger number like 32768 would seem prudent. I'm currently running with 16384. This is zero by default; changing this will improve performance.
ReplStoreDoImmaculate - if set, basically flushes the cache at boot, eliminating possibly stale mappings. Will increase boot time slightly (have not quantified) but may(?) speed up the system early on. Not likely to have a lingering effect so may not be worth any longer boot time. May increase boot time significantly according to MS.

LsFilter - LockStream DRM filter. Not sure what it does beyond DRM, but I didn't mess with it.

Storage Profiles (...\Profiles)
FLASHDRV - this is the profile reported for the on-board storage. FATFS and IMGFS file systems.
...\Profiles\FLASHDRV\FATFS
DataCacheSize - sets the size of the data cache. Default is 2048 sectors (1MB). Microsoft recommends NOT using data cache if you are using the file filter cache (CacheFilt). Provided the CacheFilt is added to the \Profiles\FLASHDRV\FATFS\Filters key, then you should set DataCacheSize to 0 (system managed). This key and FatCacheSize replace the obsolete CacheSize key.
EnableFatCacheWarm - pre-loads the file allocation table from the storage device at boot, up to the limit set by the cache size. Slight increase in boot time, but slight speed-up right after boot.
EnableWriteBack - set to one for write back mode, set to zero for write-through (slower). This is set to 1 by default.
FatCacheSize - sets the size of the file allocation table and metadata cache. Default is 256 sectors (128KB). If you set it to zero, the system will choose the best cache size, up to a limit of 512 sectors. I am running with this set at zero and it seems to be working well.
UpdateAccess - if set, updates the access time stamp on files (slower). If cleared, does not. This is a new key - I'm not sure what the default may be if this key is not specified but I assume it is zero. If I'm correct, then adding this key with a value of zero should have no effect, but adding it with a value of 1 would slow things down slightly.

SDMemory - this is the profile reported for the micro-SD card. FATFS file system.
..\Profiles\SDMemory\FATFS
Same keys as above apply. I set UpdateAccess = 0, DataCacheSize = 512 and FatCacheSize=0. Note - MS recommends a data cache for the SD card to prevent performance impact from the encrypting filter. If you aren't going to encrypt your storage card, you can set DataCacheSize to 0. Otherwise go with some low number. Changes here will only speed up access to the SD card. Note - there is a separate set of SDHC tweaks that should improve performance on SDHC cards by tweaking the driver. It is not clear if those tweaks help plain ole SD cards are not but they shouldn't hurt, and can be done in parallel with settings here as this tweaks the file system, not the driver.


Part 4 - Page pool
As I said above, IMHO the most effective of the caches is the file system filter cache, but unfortunately the only way to change the size of that cache is to change the ROM and re-flash. There are four particular parameters in the page pool configuration that matter. There are threads on changing the page pool size in a .nbh ROM file with a HEX editor - I haven't put this info together with that to come up with a strategy - if someone else does, that would be a good addition to this thread.

The page pool is divided into a loader pool that contains RAM copies of ROM EXE and DLL files, and a file cache (for data files) and memory-mapped files. The size of the two pools can be controlled independently, though all of these configurations must be done at compile time - so we may be stuck finding the resulting constants and changing them with a hex editor. Here is the declaration from the header for the configuration parameters:

const volatile DWORD LoaderPoolTarget;
const volatile DWORD LoaderPoolMaximum;
const volatile DWORD LoaderPoolReleaseIncrement;
const volatile DWORD LoaderPoolCriticalIncrement;
const volatile DWORD LoaderPoolNormalPriority256;
const volatile DWORD LoaderPoolCriticalPriority256;
const volatile DWORD FilePoolTarget;
const volatile DWORD FilePoolMaximum;
const volatile DWORD FilePoolReleaseIncrement;
const volatile DWORD FilePoolCriticalIncrement;
const volatile DWORD FilePoolNormalPriority256;
const volatile DWORD FilePoolCriticalPriority256;

The parameters that are of interest are the Target and Maximum parameters. The way it works is that the memory management system reserves the target amount of memory at initialization for each pool. The pool may not be full and is wasting some of that space. Once the target is exceeded, a low-priority thread starts flushing in an attempt to reach the target memory size. But since the thread is low priority, if the system is active (such as at boot time or when you are doing something file-intensive) the pool may grow faster than the low-priority thread releases unused pages. Each pool can grow up to its maximum size. As it grows it consumes additional memory. When it gets close to maximum a critical-priority thread takes over and starts flushing. The total page pool memory will never be less than the sum of the two target parameters or more than the sum of the two maximum parameters.

Experimentation would be necessary to find the optimum combination of loader and file pool sizes, but I suspect they are too small by default. Not knowing what the defaults are for these four parameters, I'm not sure what to suggest. However it seems to me that you want a fairly large target for both - probably in the 4-6MB range, with an additional 2MB for each for the maximum. This would put the target somewhere in the 8-12MB range and the maximum in the 12-16MB range. Sprint users can go higher.

Attached is an export of my current registry settings for the storage manager.

[Decidion]

Lots of good detailed technical info here!

However, for the layman, such as myself who isn't going to be hex editing my ROM, what registry keys should I be changing and what would suggest as new values. I realize you probably answered this in various parts of your post, but can you break it down into an easy to understand list?

Thanks for compiling all of this!

[FormerPalmOS]

Originally Posted by Decidion

Lots of good detailed technical info here!

However, for the layman, such as myself who isn't going to be hex editing my ROM, what registry keys should I be changing and what would suggest as new values. I realize you probably answered this in various parts of your post, but can you break it down into an easy to understand list?

Thanks for compiling all of this!

The basic changes I suggest in the XIP are (all numbers are decimal and all added keys are type DWORD):
HKLM\System\StorageManager\FATFS\CacheFilt - add value LockIOBuffers = 1
HKLM\System\StorageManager\Filters\fsreplxfilt - change value ReplStoreCacheSize from 0 to 4096, change value ReplStoreDoImmaculate from 0 to 1
HKLM\System\StorageManager\Profiles\FLASHDRV\FATFS - change value DataCacheSize from 2048 to 0 (system managed), change value FatCacheSize from 256 to 0, add UpdateAccess = 0, add EnableFatCacheWarm = 1
HKLM\System\StorageManager\Profiles\SDMemory\FATFS - add values DataCacheSize = 0, EnableFatCacheWarm = 1, EnableWriteBack=1, FatCacheSize = 0

That should give you some boost without taking up too much memory.

[bullet2300]

I don't seem to have the StorageManager folder... I am using Total Commander. Am I missing something?

[speck]

Originally Posted by bullet2300

I don't seem to have the StorageManager folder... I am using Total Commander. Am I missing something?

HKLM\System\StorageManager

[FormerPalmOS]

Originally Posted by bullet2300

I don't seem to have the StorageManager folder... I am using Total Commander. Am I missing something?

Nope - I am - specifically System from the key name. Pust updated. Sorry about that.

[PaulB007]

And what will these key changes do? Will it speed up the device, and will it potentially cause lock ups/other issues?

[bullet2300]

Ah, there it is! Thank you!

[FormerPalmOS]

Originally Posted by PaulB007

And what will these key changes do? Will it speed up the device, and will it potentially cause lock ups/other issues?

LockIOBuffers may do nothing - I assume it defaults to 1 but added it just to be sure. If it defaults to zero, this would reduce the chance of a file-system related crash. Since that hasn't happened to me (ever) I therefore assume this is already locked. According to MS, you have to use this when using the file cache filter with disk caching.

Changing the file replication cache size will increase the amount of memory that filesys.exe uses, but with this value, not by much (not more than a few tens of KB). It will speed up file access for all but the first time a file in the cache is accessed.

Changing the DataCacheSize and FatCacheSize keys to zero will allow WM to determine the optimum cache size. In general this will be the smaller of the entire FAT or 512 sectors (if your FAT is more than 512 sectors you have waaaay to many files on your storage card or device). YOu don't want a big data cache - instead you want a big file filter cache. YOu need a bigger page pool to enable that. When the OS needs a file, it first looks for it in the directory - a flash access if not cached. Once it gets the file it gets a pointer to the FAT for that file and reads that - another flash access if not cached. Then it gets the file by sectors in the FAT. Now if the file itself is in the file system filter cache - it just retrieves the file from the file cache with no flash accesses (very fast).

EnableFatCacheWarm just pre-populates the FAT cache at file system start-up (boot) - speeds up first accesses at the expense of longer boot time. No lasting performance impact beyond the first access.

UpdateAccess = 0 disables updating the "last accessed" time stamp in the file system. This prevents an access to the flash every time you touch a file. I believe this is disabled by default because this would be a big performance hit if this was enabled - I didn't see a significant boost from just adding this key which leads me to believe it is disabled by default. So this key probably does nothing.

It's time-consuming to quantify the performance impact from each key alone so I haven't done that, but as a whole, this group of edits definitely makes my device much more responsive, yet still leaves it with plenty of memory, and I have yet to have a crash.

Finally, reducing the number of files in the most commonly used directories will increase performance. Every file in your ROM is stored in \Windows - it's just hidden. Only way to reduce the number of files there is to cook a new ROM with fewer files. But there's a big bang for the buck here.

[FormerPalmOS]

I learned a couple of new items that mostly render this discussion moot. First and foremost, for the changes to be effective on the FLASHDRV FAT file system, they must be made to the boot hive that is loaded by the SPL. The registry that you can edit after the system is booted is loaded by the file system - these changes need to be made to the minimal registry (boot.hv) that is loaded in order to load the file system. That means they have to be cooked into the ROM.

That said, you can apply these changes to the SDMemory file system.

I also learned that the system-manged settings actually appear to do a pretty good job. I am currently running with DataCacheSize and FatCacheSize both set to 0 (system managed) on my SD card and it seems to be faster (but then again I also thought the machine was faster with the changes made in the first thread and that may not have been the case).

I have updated the first post to reflect the new learning - the remaining posts have not been updated.