shared API for double-entry accounting, treating it as a 'math' library.

Fri Nov 15 22:10:32 EST 2013

On Fri, Nov 15, 2013 at 8:48 AM, Bradley M. Kuhn <bkuhn at ebb.org> wrote:

> Chris,
>
> Your 20,000 invoice/week example reads to me like a classic scaling
> issue that could be addressed in various ways, but certainly is going to
> require specialized software development to handle in any event.
>

Actually, part of is is classic scaling.  You also have optimizing full
workflow efficiency vs responsiveness, and how you address concurrency
issues.  In other words it isn't just scaling the amount of data, it's also
scaling the the workflow enforcing human elements.

At any rate it was the one case where I had seen seconds of useful db time
turn into minutes of computing time.  Given the volume, the 5 min wait was
deemed acceptable and it still allowed one person to process these 20k
invoices in one day.

Even there however, moving to a workflow where they get the full
transaction information back to them after posting didn't pose anywhere
near the same problems.  But if you are worried about volume of information
causing something that takes seconds to take minutes, that's an actual case
which can be discussed and learned from.

>
> Meanwhile, when I think about the full space of users for a possible NPO
> accounting software project, the bulk of them are going to be at small
> non-profits with 1-5 bookkeepers and no more than a 100-200 invoices a
> week.

This is true.  But long-term the question is (and I think this is what you
are getting at in this email):

Do you want to limit yourself to this niche?  Or do you want to possibly
allow similarly situated larger organizations (particularly governmental
organizations which may have closely analogous accounting restrictions) to
be able to build on your work?

Now I see two directions:

1.  You could start another open source project solely to meet the needs of
these smaller NPO's which I think is where you started out.  Or

2.  You could start an open source project to meet the needs of smaller
NPO's, but separating the accounting engine off so that it could be more
generally applicable (what I think you were discussing with the API
proposal).  Long-run if you could pull that off, it would mean a lot more
help on the accounting side allowing you to focus on the small NPO market a
lot more.

>  As you say, these sort of high-volume situations come in up heavy
> inventory based accounting for large-scale retail or manufacturing is
> surely a niche.  I can imagine very large non-profits (say, the Red
> Cross) having to deal with this kind of volume, but I don't think we
> have the expertise nor the resources assembled here to figure out how
> handle that sort of situation.
>

Many of them are generalized problems though.  Manufacturing is a niche.
 Inventory, while a widespread need is probably a niche within NPO space
(albeit a big one-- not-for-profit hospitals come to mind).

> As we've been doing these codebase evals, I continually come back to the
> conclusion that everyone who has approached this problem in the Free
> Software community (including us!)  had a small-scale, niche market in
> mind.  Ours at the moment is non-profits -- and in particular fund-based
> fiscal sponsorship accounting with 30 or more (up to a thousand, I
> suspect, when we consider an org like Fractured Atlas) pools of
> temporarily restricted assets.
>

Right.

>
> We basically have two choices of approach to this situation (and I don't
> think this is a false dichotomy but if it is, let me know):
>
>   (a) We can do the same that everyone else did: start a project, and/or
>       bend an existing codebase, to address our niche, and have yet
>       another specialized version of one of the existing accounting
>       systems, or
>
>   (b) try to build a toolkit useful in the creation of small business
>       accounting applications (and reusing code from whereever seems
>       prudent, when that code is flexible enough to be adapted into a
>       toolkit).
>

> I'll try to anticipate that you might be about say that your 20k
> invoices/week *is* a small business, and thus this project won't even be
> able to please small businesses in your niche if we design the
> infrastructure "too flexibly" such that it can't do what you might
> consider a basic operation.
>

First, nope on both counts.  They are  a midsized financial services firm
which also runs (and the figures include) an EU subsidiary which has
separate books but the same staff.

Secondly....  the big question is what is in scope vs not in scope and
whether we can use it depends on how we would use it ;-)  If the underlying
API is a specification we can offer then obviously we can offer to back-end
it (and if you offer a project which back-ends it against ledger-cli then
there is an upgrade path for those who need it).  From our perspective
though there are two relatively non-negotiable constraints on our use.
 These are:

1.  We are fully invested in PostgreSQL and would need to be able to
process back-end information in that context, and

2.  For the reasons mentioned think that tying everything to a
fundamentally stateless interface can be asking for scaling problems down
the road.

>
> But, here's the thing: people keep solving this problem over and over
> again in slightly different ways to address specific niches.  For
> example, I'd bet LedgerSMB is the only Free Software accounting project
> who has even tried to solve the batch 20k invoices/week problem.
>

We were the third open source accounting system they tried.   So I know
that two others did try unsuccessfully to solve that problem.

>
> In other words, I think we have settle for the idea that we're not going
> to *revolutionize* accounting software systems -- not even non-profit
> accounting ones.

I think it is honestly unlikely that we will ever see major revolutions to
this industry.  What we can hope to see is gradual evolution.

Keep in mind that double entry accounting has been documented back into the
15th century and some vestiges of it go back at least 4-5 centuries prior
to that (btw, I don't think that double entry accounting systems derived
from single entry, but rather from something entirely independent).  I
think the idea that the situation today is the product of a thousand years
of really organic evolution of debt tracking encourages some humility in
such things.

 We're going to help by making things for non-profits a
> bit better, and hopefully the code this project does will be at least
> marginally more reusable than the code already out there for other
> accounting needs.  I'm somewhat inspired by CiviCRM here -- they started
> focusing on a niche (contact database for community organizing) but
> build a flexible enough codebase that they've been able to add many
> modules.
>
> Meanwhile, if the re-usability angle means we made it impossible for the
> users who want to get into the
> tens-of-thousands-of-transactions-per-week to reuse our code, I think
> that's a trade-off I'm likely willing to make for now.
>
> Chris, however, if you have a specific idea for a design of an API that
> would handle both, I'm open-minded to it.  I'm not trying to cover my
> eyes and pretend the problem doesn't exist; I'm trying to find a project
> that we can make real progress on with the meager resources we have.
>

I think the point of an API is that it forms a contract.  In theory anyone
is capable of writing software to either side of that contract and that's
where the value lies.

It seems to my mind that your API framework is sufficient as long as
back-end independence is a goal.  In other words, the specification of an
API is valuable *particularly* if you don't expect everything to go through
that API.

I am also going to make a second suggestion here, which is that you may
want to make the API transport agnostic.  It may be possible to implement
the API at least partially over web services, through UNIX sockets, etc.
 This allows cases where state is important to address that in a reasonable
way.  Your implementation could assume text files and shell scripts, for
example, but if someone wants to set that up over web services or web
sockets (or even just "use a tcp connection like a UNIX socket!") it should
be easy enough to handle that.

In fact if you build with shell scripts and UNIX sockets in mind, it would
be pretty easy to wrap in some other format (including other platforms).

>
> BTW, if they pay 20k invoices per week in a batch, and *are* a small
> business, how in the world do they review all of them, and do internal
> approvals, etc.?
>

I wouldn't call them "small" but they aggressively automate, cross-check,
and review.  The actual invoice entry to the books and payments for clients
payable invoices are managed by two-three people including review for
numeric accuracy.  There are other checks on what gets paid both automated
and in a few cases manual.  Their accounting staff is much smaller than
you'd expect.  Routine payment involves two people (four eyes), and more
complex cases may involve up to 4, and their IT staff is significantly
larger than their accounting staff (of course they have to deal with
PCI-DSS compliance in a big way).

Best Wishes,
Chris Travers

> --
>    -- bkuhn
> _______________________________________________
> npo-accounting mailing list
> npo-accounting at sfconservancy.org
> http://lists.sfconservancy.org/mailman/listinfo/npo-accounting
>

-- 
Best Wishes,
Chris Travers

Efficito:  Hosted Accounting and ERP.  Robust and Flexible.  No vendor
lock-in.
http://www.efficito.com/learn_more.shtml
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sfconservancy.org/pipermail/npo-accounting/attachments/20131115/3624aed3/attachment-0001.html>