<div dir="ltr">A couple clarifications in response.<div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Nov 14, 2013 at 6:02 AM, Bradley M. Kuhn <span dir="ltr"><<a href="mailto:bkuhn@sfconservancy.org" target="_blank">bkuhn@sfconservancy.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Chris Travers wrote at 05:49 (EST):<br>
<br>
> 1. Complexity of input data. Financial transactions are complex, and<br>
> they get more complex in certain kinds of environments. In the<br>
> for-profit world, I would expect a grocery store receipt to add up to<br>
> maybe a hundred of gl line items.<br>
<br>
So, I agree that the overall picture of the financial transactions are<br>
complex, but think about how much of that is *not* the actual<br>
double-entry accounting data. Considering your grocery store receipt<br>
example, if we ignore the other parts of the question, it's just (I<br>
apologize for the Ledger-CLI-y syntax):<br>
<br>
2013-11-14 John Q. Customer<br>
Income:Gross Sales $-2.99<br>
Income:Sales Tax Collected $-0.26<br>
Expenses:Sales Tax $0.26<br>
Inventory 1 BoxedCereal {=$2.99}<br>
<br>
...repeated for every item bought....<br>
<br>
This is a lot of data, no question, but it's not complex. Obviously,<br>
I've eliminated much of the detail that might want to be kept. That<br>
John Q. Customer maybe should be a customer id that (in SQL land) would<br>
index to a record with lots more data. BoxedCereal might be an<br>
inventory code instead of a moniker like it is in Ledger-CLI.<br></blockquote><div><br></div><div>This is true, but you still have the issue of reporting, and the ability to run reports in the future based on criteria that may not have occurred to you yet. I think, reading your API specification, that you try to get around this by tagging, and that's not a bad way to go. So I think you'd have a sku tag for the part, and maybe a till tag, and/or maybe a salesperson tag, etc on every line. </div>
<div><br></div><div>Again your specification handles that side and does so pretty well. (I am not sure how much payee information needs to go into the API? That's a topic for another time perhaps :-).</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
The point is that the *just* the computation of double-entry accounting<br>
can be separated into something more basic.<br>
<br></blockquote><div>To some extent until you want to ask, "how has our gross margin on boxed serial changed in the last five years? I want month by month figures. And rank specific products by year by gross margin." Once you want the ability to even be able to ask these questions you have to be capturing all the links to all the data.</div>
<div><br></div><div>I guess what I am getting at is that I am not sure how far the accounting side can be separated from everything else.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
> 2. Pervasiveness of data. Financial transactions touch everything<br>
> and they tend to be connected to everything. Managing that<br>
> connectedness is a significant challenge of a sort of loosely coupled<br>
> approach. An API would need to be able to handle that.<br>
<br>
I'm not convinced on that. Plenty of places, these things would be<br>
identifiers that would have to index somewhere else, but the double<br>
entry library just doesn't care about that.<br></blockquote><div><br></div><div>Right. Again you have tags on one side, but I don't think that is quite far enough. You would have to also support the other side being able to take the data stored and store links on its side.</div>
<div><br></div><div>In other words, so we have a boxed cereal purchase. Not only do you need to be able to store tags on the transactions but you may need the other side of the system (the inventory tracking system) to be able to link lines in for FIFO inventory tracking purposes. So I think what's missing is that the return values from the API would need to include what was submitted plus id's for indexing on the other side.</div>
<div><br></div><div>Once you have that bilateral communication, then I think the rest of the problems are ok.</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
I realize that the annoying nature of having to talk through an API to<br>
get to double-entry data....<br></blockquote><div><br></div><div>I don't know you will always be able to force folks to go through the API. As the volume gets even mildly large (let's say a few million lines), the ability to run declarative queries becomes important, which is why I would assume that larger volume users will probably put the data in an RDBMS. </div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
> 3. Integration of data. Often times it is important to be able to do<br>
> complex reporting Often times one needs to tie line items of a<br>
> transaction back to some other data. For example, you might want to<br>
> find out "how has our spending on scalpals we ship to clinics in<br>
> Africa compared to our spending on insulin?"<br>
<br>
... and this problem is a real one. If you "know" it's a database<br>
underneath, why not just to a join instead of going through the API.<br>
So, I can imagine that keeping people "to" the API would be the toughest<br>
problem.<br></blockquote><div><br></div><div>I am not sure that's a problem. I think that if you have an API rich enough for the basic common cases (including, for example, a trial balance and funds report), I don't know that it matters if people use the API for everything.</div>
<div><br></div><div>The big problem that you have with API's and set-based data is that a row-by-row analysis is slow and it is, quite frankly, a pain to process. If you have a database underneath you are probably going to be pulling the data from the API into the db periodically for reporting purposes for that reason.</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
<br>
> Having said all of this, if you had an API that submitted a JSON<br>
> document and received one back with line identifiers, etc. populated,<br>
> then it should be manageable enough. There are naturally limits to<br>
> this kind of modularity due to the complexity of the data being stored<br>
> and retrieved on both sides.<br>
<br>
See, I think the issue is *volume*, not complexity. You can do this<br>
complex queries, get a big-honking-JSON back, and then parse through it.<br>
Queries that would take seconds now take minutes. That's the real issue<br>
I see with the idea. I don't have an idea of how to solve that.</blockquote><div><br></div><div>I have seen things like this be an issue, but I think that where you see them, you are going to see them first in areas where that is not so easily avoided. At any rate you do raise a good point.</div>
<div><br></div><div>I am going to think about this and the times I have seen volume raise problems. To be fair, it is rarely just volume. One often wants to mark invoices as locked for payment so they can't be double-paid for example, and when paying a few thousand invoices at a time, this can introduce performance issues.</div>
<div><br></div><div>At any rate, from my experience I would expect the bottlenecks to be in trying to manage state over a stateless exchange if that is supported, and figuring out how to interact with the user if it is a lot of data presented for review (bulk payment of invoices is a key issue there).</div>
<div><br></div><div>Speaking of bulk/batch workflows, this is one thing I didn't see in your API so far: separation of duties/four eyes principle. How would you expect that would be handled?</div><div><br></div></div>
-- <br><div dir="ltr">Best Wishes,<div>Chris Travers</div><div><br></div><div>Efficito: Hosted Accounting and ERP. Robust and Flexible. No vendor lock-in.</div><div><a href="http://www.efficito.com/learn_more.shtml" target="_blank">http://www.efficito.com/learn_more.shtml</a></div>
</div>
</div></div>