<div dir="ltr">Having had the benefit of a night's sleep on the issues Bradley mentioned with my feedback, I think this rabbit hole goes a bit further than it may be apparent and more issues that need to be discussed.<div>
<br></div><div>I want to discuss the performance issues I have seen with volume transactions and where the likely solutions are. I don't actually think that the interface format is a big problem (I favor JSON because there are fewer ambiguity issues than there are with XML on the parsing side), but there are several other bottlenecks that need to be considered also.</div>
<div><br></div><div>The biggest issue is the question of state vs stateless interface. If this is done over HTTP and something like a web service, then certain kinds of workflows become very problematic performance-wise because tracking state across requests imposes significant overhead.</div>
<div><br></div><div>Typically what we found in LedgerSMB is that most workflows are perfectly manageable in terms of response times, assuming a reasonable platform underneath that can handle concurrency, but that bulk payment operations are the big problem. Our discussion of inventory is interesting but probably a niche issue for NPO's, but bulk payments may be more common. The performance issues we have seen here are worth sharing.</div>
<div><br></div><div>One of our users of LedgerSMB pays about 20000 invoices through the system every week. The largest workflows run about 5000 invoices through the system at a time, payable to about 200 payees. Obviously this poses some performance challenges and we have done a significant amount of profiling about where the time and effort is spent. This is all done through a web interface. Because of concurrency issues we write to a lock table and locks are expired with session logouts. This is to prevent invoices from being double-paid.</div>
<div><br></div><div>Now as far as performance what we found (on PostgreSQL here) was that selecting this against a table with around 10 million line items shows roughly the following performance breakdowns:</div><div><br>
</div><div>1. A database query to just pull the open invoices runs in about 2 sec.</div><div><br></div><div>2. Adding the locking logic slows it from 2 sec. to 49 sec in part because one goes from pure reads to a mixture of reads and writes.. </div>
<div><br></div><div>3. Generating a user interface (using Template Toolkit) takes about 4 minutes.</div><div><br></div><div>So there you have going from seconds to minutes, which I presume is the issue mentioned. Now in this case it isn't a deal-killer because this is about 5 minutes in between a longish workflow but it could be optimized a lot.</div>
<div><br></div><div>Here are the lessons I draw from this. Other interpretations are welcome.</div><div><br></div><div>The first is that a stateless API poses very significantly costs regarding payment workflows. If we are worried about performance at volume that's a big thing to consider there.</div>
<div><br></div><div>The second thing is that at volume there are huge limits to what a web-based UI can accomplish. I have been building a point of sale system based on wxperl with most of the common modules, and compared to fast-cgi (with cached code, for most of it), the wxwidget-based client is vastly more responsive (guessing 2 orders of magnitude faster than a comparable web interface).</div>
<div><div class="gmail_extra"><br></div><div class="gmail_extra">So if we are worried about performance at scale I think the API needs to assume the possibility of a stateful interface with a "pure data" exchange. json or xml would work for that and there are good parsers for them in many languages. My concern over XML though is that while the schema documents tighten up some checks for strongly typed languages there are ambiguity in semantics that can make it difficult to ensure compatibility between implementations. With JSON, we'd have to dispense with those checks (or rather push them to the client) but there are fewer traps regarding inconsistent implementations. If there are established specifications output can be tested. This doesn't solve everything though. But hey, at least it isn't X12......</div>
<div class="gmail_extra"><br></div><div class="gmail_extra">The other option is this: Decide that these workflows aren't necessarily within the scope of this API but this doesn't preclude another one that would handle these other issues differently.</div>
<div class="gmail_extra"><br></div><div class="gmail_extra">Anyway, hope this is helpful.</div><div class="gmail_extra"><div><br></div>-- <br><div dir="ltr">Best Wishes,<div>Chris Travers</div><div><br></div><div>Efficito: Hosted Accounting and ERP. Robust and Flexible. No vendor lock-in.</div>
<div><a href="http://www.efficito.com/learn_more.shtml" target="_blank">http://www.efficito.com/learn_more.shtml</a></div></div>
</div></div></div>