Accounting API considerations

[Joar Wandborg]

On 12/04/2013 10:22 AM, Chris Travers wrote:
> On Dec 3, 2013 11:41 PM, "Joar Wandborg" <joar at wandborg.se> wrote:
>> [snip] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Alternative 2: Leverage
>> Ledger ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> 
>> Should we build on top of a pre-existing codebase?
>> 
>> Ledger as in http://ledger-cli.org is very flexible, basic and
>> uses simple text files and command line arguments for all
>> operations, it would be feasible to build an application on top
>> of ledger-cli to create an accounting API application.
>> 
>> Some caveats:
>> 
>> - The ledger-cli program uses <8M of heap memory in total on a 
>> minimal balance report run. - Unless you use a caching mechanism,
>> most reporting requests that comes in would require a subprocess
>> of ledger-cli to be spawned. - ledger-cli reads a "ledger file"
>> on startup. It cannot as far as I know load or reload a "ledger
>> file" once started. - ledger-cli takes command line arguments in
>> order to produce the requested reports. This means that
>> ledger-cli will need to be spawned once for each report unless a
>> caching mechanism is used.
> 
> Almost any system is going to require frequent respawning of
> processes or at least threads for concurrent reports. The larger
> question I have there is whether it reads all info on every report.
> If so you will have issues scaling to larger data sets.  This is
> not fatal to a reference implementation and if it is the case it
> would just mean your reference implementation would be limited to
> smaller NPOs. It would seem ok to expect other projects to make
> things work in a compatible way.

Testing with Conservancy's books, which use ledger although I supect
in slightly a more advanced degree than the accounting API will:

31 files, >100k lines in total.

`valgrind ledger -f MAIN_LEDGER_FILE bal` reports 102,003,202 bytes of
heap allocated, that's >100MiB.

`time ledger -X USD -f MAIN_LEDGER_FILE bal` reports 7.39s user, 0.04s
system, 99% cpu, 7.447 total. This on an i7 3770k.

I believe this is tolerable for this kind of dataset, but it's still
something that I hope we can improve.

>> [snip]