<div dir="ltr"><br><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Nov 15, 2013 at 8:48 AM, Bradley M. Kuhn <span dir="ltr"><<a href="mailto:bkuhn@ebb.org" target="_blank">bkuhn@ebb.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Chris,<br>
<br>
Your 20,000 invoice/week example reads to me like a classic scaling<br>
issue that could be addressed in various ways, but certainly is going to<br>
require specialized software development to handle in any event.<br></blockquote><div><br></div><div>Actually, part of is is classic scaling. You also have optimizing full workflow efficiency vs responsiveness, and how you address concurrency issues. In other words it isn't just scaling the amount of data, it's also scaling the the workflow enforcing human elements.</div>
<div><br></div><div>At any rate it was the one case where I had seen seconds of useful db time turn into minutes of computing time. Given the volume, the 5 min wait was deemed acceptable and it still allowed one person to process these 20k invoices in one day.</div>
<div><br></div><div>Even there however, moving to a workflow where they get the full transaction information back to them after posting didn't pose anywhere near the same problems. But if you are worried about volume of information causing something that takes seconds to take minutes, that's an actual case which can be discussed and learned from.</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Meanwhile, when I think about the full space of users for a possible NPO<br>
accounting software project, the bulk of them are going to be at small<br>
non-profits with 1-5 bookkeepers and no more than a 100-200 invoices a<br>
week.</blockquote><div><br></div><div>This is true. But long-term the question is (and I think this is what you are getting at in this email):</div><div><br></div><div>Do you want to limit yourself to this niche? Or do you want to possibly allow similarly situated larger organizations (particularly governmental organizations which may have closely analogous accounting restrictions) to be able to build on your work?</div>
<div><br></div><div>Now I see two directions:</div><div><br></div><div>1. You could start another open source project solely to meet the needs of these smaller NPO's which I think is where you started out. Or</div><div>
<br></div><div>2. You could start an open source project to meet the needs of smaller NPO's, but separating the accounting engine off so that it could be more generally applicable (what I think you were discussing with the API proposal). Long-run if you could pull that off, it would mean a lot more help on the accounting side allowing you to focus on the small NPO market a lot more.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> As you say, these sort of high-volume situations come in up heavy<br>
inventory based accounting for large-scale retail or manufacturing is<br>
surely a niche. I can imagine very large non-profits (say, the Red<br>
Cross) having to deal with this kind of volume, but I don't think we<br>
have the expertise nor the resources assembled here to figure out how<br>
handle that sort of situation.<br></blockquote><div><br></div><div>Many of them are generalized problems though. Manufacturing is a niche. Inventory, while a widespread need is probably a niche within NPO space (albeit a big one-- not-for-profit hospitals come to mind).</div>
<div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
As we've been doing these codebase evals, I continually come back to the<br>
conclusion that everyone who has approached this problem in the Free<br>
Software community (including us!) had a small-scale, niche market in<br>
mind. Ours at the moment is non-profits -- and in particular fund-based<br>
fiscal sponsorship accounting with 30 or more (up to a thousand, I<br>
suspect, when we consider an org like Fractured Atlas) pools of<br>
temporarily restricted assets.<br></blockquote><div><br></div><div>Right. </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
We basically have two choices of approach to this situation (and I don't<br>
think this is a false dichotomy but if it is, let me know):<br>
<br>
(a) We can do the same that everyone else did: start a project, and/or<br>
bend an existing codebase, to address our niche, and have yet<br>
another specialized version of one of the existing accounting<br>
systems, or<br>
<br>
(b) try to build a toolkit useful in the creation of small business<br>
accounting applications (and reusing code from whereever seems<br>
prudent, when that code is flexible enough to be adapted into a<br>
toolkit).<br></blockquote><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
I'll try to anticipate that you might be about say that your 20k<br>
invoices/week *is* a small business, and thus this project won't even be<br>
able to please small businesses in your niche if we design the<br>
infrastructure "too flexibly" such that it can't do what you might<br>
consider a basic operation.<br></blockquote><div><br></div><div>First, nope on both counts. They are a midsized financial services firm which also runs (and the figures include) an EU subsidiary which has separate books but the same staff.</div>
<div><br></div><div>Secondly.... the big question is what is in scope vs not in scope and whether we can use it depends on how we would use it ;-) If the underlying API is a specification we can offer then obviously we can offer to back-end it (and if you offer a project which back-ends it against ledger-cli then there is an upgrade path for those who need it). From our perspective though there are two relatively non-negotiable constraints on our use. These are:</div>
<div><br></div><div>1. We are fully invested in PostgreSQL and would need to be able to process back-end information in that context, and</div><div><br></div><div>2. For the reasons mentioned think that tying everything to a fundamentally stateless interface can be asking for scaling problems down the road.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
But, here's the thing: people keep solving this problem over and over<br>
again in slightly different ways to address specific niches. For<br>
example, I'd bet LedgerSMB is the only Free Software accounting project<br>
who has even tried to solve the batch 20k invoices/week problem.<br></blockquote><div><br></div><div>We were the third open source accounting system they tried. So I know that two others did try unsuccessfully to solve that problem. </div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
In other words, I think we have settle for the idea that we're not going<br>
to *revolutionize* accounting software systems -- not even non-profit<br>
accounting ones.</blockquote><div><br></div><div>I think it is honestly unlikely that we will ever see major revolutions to this industry. What we can hope to see is gradual evolution.</div><div><br></div><div>Keep in mind that double entry accounting has been documented back into the 15th century and some vestiges of it go back at least 4-5 centuries prior to that (btw, I don't think that double entry accounting systems derived from single entry, but rather from something entirely independent). I think the idea that the situation today is the product of a thousand years of really organic evolution of debt tracking encourages some humility in such things.</div>
<div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> We're going to help by making things for non-profits a<br>
bit better, and hopefully the code this project does will be at least<br>
marginally more reusable than the code already out there for other<br>
accounting needs. I'm somewhat inspired by CiviCRM here -- they started<br>
focusing on a niche (contact database for community organizing) but<br>
build a flexible enough codebase that they've been able to add many<br>
modules.<br>
<br>
Meanwhile, if the re-usability angle means we made it impossible for the<br>
users who want to get into the<br>
tens-of-thousands-of-transactions-per-week to reuse our code, I think<br>
that's a trade-off I'm likely willing to make for now.<br>
<br>
Chris, however, if you have a specific idea for a design of an API that<br>
would handle both, I'm open-minded to it. I'm not trying to cover my<br>
eyes and pretend the problem doesn't exist; I'm trying to find a project<br>
that we can make real progress on with the meager resources we have.<br></blockquote><div><br></div><div>I think the point of an API is that it forms a contract. In theory anyone is capable of writing software to either side of that contract and that's where the value lies.</div>
<div><br></div><div>It seems to my mind that your API framework is sufficient as long as back-end independence is a goal. In other words, the specification of an API is valuable *particularly* if you don't expect everything to go through that API.</div>
<div><br></div><div>I am also going to make a second suggestion here, which is that you may want to make the API transport agnostic. It may be possible to implement the API at least partially over web services, through UNIX sockets, etc. This allows cases where state is important to address that in a reasonable way. Your implementation could assume text files and shell scripts, for example, but if someone wants to set that up over web services or web sockets (or even just "use a tcp connection like a UNIX socket!") it should be easy enough to handle that.</div>
<div><br></div><div>In fact if you build with shell scripts and UNIX sockets in mind, it would be pretty easy to wrap in some other format (including other platforms).</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
BTW, if they pay 20k invoices per week in a batch, and *are* a small<br>
business, how in the world do they review all of them, and do internal<br>
approvals, etc.?<br></blockquote><div><br></div><div>I wouldn't call them "small" but they aggressively automate, cross-check, and review. The actual invoice entry to the books and payments for clients payable invoices are managed by two-three people including review for numeric accuracy. There are other checks on what gets paid both automated and in a few cases manual. Their accounting staff is much smaller than you'd expect. Routine payment involves two people (four eyes), and more complex cases may involve up to 4, and their IT staff is significantly larger than their accounting staff (of course they have to deal with PCI-DSS compliance in a big way).</div>
<div><br></div><div>Best Wishes,</div><div>Chris Travers</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<span class="HOEnZb"><font color="#888888">--<br>
</font></span><div class="HOEnZb"><div class="h5"> -- bkuhn<br>
_______________________________________________<br>
npo-accounting mailing list<br>
<a href="mailto:npo-accounting@sfconservancy.org">npo-accounting@sfconservancy.org</a><br>
<a href="http://lists.sfconservancy.org/mailman/listinfo/npo-accounting" target="_blank">http://lists.sfconservancy.org/mailman/listinfo/npo-accounting</a><br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr">Best Wishes,<div>Chris Travers</div><div><br></div><div>Efficito: Hosted Accounting and ERP. Robust and Flexible. No vendor lock-in.</div>
<div><a href="http://www.efficito.com/learn_more.shtml" target="_blank">http://www.efficito.com/learn_more.shtml</a></div></div>
</div></div>