Askemos Forum

Background

Two recent projects, the BMWi aided analysis of scalability related problems and deployment in an e-Paper project (converting InDesign®-crafted newspapers to visually simillar web presentations), we spotted urgent need to modify storage adaptor handling.

The first issue is related to the pstore facility of RScheme. It's designed to work with repositories of up to 4GiB, which already places a limit. However recently discovert issues, according to Donovan probably a signess-bug, restricts it even further to 2GiB repository size.

The newspaper conversion depends on the caching of intermediate results for reasonable speed. Thus switching it off and work only from the file system is no option.

Possible Options

Two solutions come to mind:

• Split the pstore in several volumes.
• Store large objects (BLOB's) in file system only, while pointer intensive data is kept in pstore

The former would be better done tightly integrated at RScheme level and will incure considerable effort. When it's going to be implemented we should also consider splitting the "monotones" (which apparently are not) of the pstore commit operation into smaller chunks. ((At least that's my theory, why long running commit operations to heavily slow down other treads. Even those running at higher priority.))

The second solution appears to be implementable in short term, but still useful if any progress on the longer term option no 1 is made.

Proposed Solution

There's a single property (the body slot of the place), which is often quite a large contiguous memory block.

An new configuration variable $pstore-body-treshold of type integer is introduced.

If a BLOB larger then this treshold, the slot is removed from the pstore-stored frame and only kept in other storage adaptors, like the file system mirror.

This will asure that large objects, like PDF documents, pictures etc. consume only disk space, but leave the memory load alone.

As future improvements it's suggested to use mmap'ed areas for those large objects. Synchronisation of those things should be done using rsync or bitorrent protocols.