Unable to integrate Celery upon windows using Kallithea 0.2.1 on Win2k12

Sun Apr 19 10:27:48 EDT 2015

On 04/19/2015 09:11 AM, Todd Morgan wrote:
> Hi,
>  I'm trying to set up a new Kallithea 0.2.1 installation upon a 
> Win2k12 box (clean - nothing else installed).
>
> I am able to get Kallithea working AOK without Celery and RabbitMQ - 
> but I have a large number of repos (100+) to support so I need the 
> scaling Celery provides.
>
> When I enable celery I get a message in the celeryD log indicating 
> that "*Task of kind 'kallithea.lib.celerylib.tasks.create_repo' is not 
> registered, please make sure it's imported."*
>
> [2015-04-19 22:13:48,183: DEBUG/MainProcess] Consumer: Ready to accept 
> tasks!
> [2015-04-19 22:20:14,302: INFO/MainProcess] Got task from broker: 
> kallithea.lib.celerylib.tasks.create_repo[65d4b6a4-8ef8-41d0-9f75-32479394e32e]
> [2015-04-19 22:20:14,332: DEBUG/MainProcess] Mediator: Running 
> callback for task: 
> kallithea.lib.celerylib.tasks.create_repo[65d4b6a4-8ef8-41d0-9f75-32479394e32e]
> [2015-04-19 22:20:14,348: DEBUG/MainProcess] TaskPool: Apply <function 
> execute_and_trace at 0x00000000053A6E48> 
> (args:('kallithea.lib.celerylib.tasks.create_repo', 
> '65d4b6a4-8ef8-41d0-9f75-32479394e32e', ({'repo_type': u'git', 
> 'repo_name': u'automation', 'repo_enable_downloads': False, 
> '_authentication_token': u'85022200563145360862159926964161116912', 
> 'repo_copy_permissions': False, 'repo_enable_locking': False, 
> 'repo_landing_rev': u'rev:tip', 'group_name': '', 'clone_uri': u'', 
> 'add': u'Add', 'repo_name_full': u'automation', 'repo_group': None, 
> 'group_path': '', 'repo_private': False, 'repo_enable_statistics': 
> False, 'repo_description': 
> u'http://littleone:8088/scm/git/automation.git' 
> <http://littleone:8088/scm/git/automation.git%27>}, 2), {}) 
> kwargs:{'hostname': 'WIN-PF6LKO8R2ST', 'request': {'retries': 0, 
> 'loglevel': 10, 'delivery_info': {'consumer_tag': u'2', 'routing_key': 
> u'celery', 'exchange': u'celery'}, 'is_eager': False, 'taskset': None, 
> 'logfile': 'celeryd.log', 'id': '65d4b6a4-8ef8-41d0-9f75-32479394e32e'}})
> [2015-04-19 22:20:14,380: DEBUG/MainProcess] Task accepted: 
> kallithea.lib.celerylib.tasks.create_repo[65d4b6a4-8ef8-41d0-9f75-32479394e32e] 
> pid:2884
> [2015-04-19 22:20:14,473: ERROR/MainProcess] Task 
> kallithea.lib.celerylib.tasks.create_repo[65d4b6a4-8ef8-41d0-9f75-32479394e32e] 
> raised exception: *Task of kind 
> 'kallithea.lib.celerylib.tasks.create_repo' is not registered, please 
> make sure it's imported.*
> NotRegistered: 'kallithea.lib.celerylib.tasks.create_repo'
> None

As you suggest, this seems to mean that celeryd runs in a way where it 
doesn't see your Kallithea instance and it can thus not launch the 
create_repo task.

>
> I have followed the instructions here 
> http://kallithea.readthedocs.org/en/latest/installation_win.html
>
> My installation includes:
>
>   * The win2k12 box is clean (VM) setup for testing this - all patches
>     - UAC disabled
>   * Python 2.7.9 (python-2.7.9.amd64.msi) +
>     pywin32-219.win-amd64-py2.7.exe
>   * Postgres support ( I had to add this pip install psycopg2 to get
>     the postgres support)
>   * I installed the latest version of Erlang otp_win64_17.5.exe
>   * Latest version of Postgres (postgresql-9.4.1-3-windows-x64.exe)
>   * Latest version of Rabbit rabbitmq-server-3.5.1.exe
>

I wasn't aware that celery was an option on windows ... but it totally 
makes sense it is. It seems like you have gotten further than the 
documentation covers - it would be great if you can help extend the 
documentation ... especially when you get it working in the end ;-)

> The post suggests it's not including the relevant "virtual 
> environment" in the path (I'm a java dev by profession rather than a 
> Python dev) so I'm not sure if that's right ...
>
> I did try using a fully qualified path for the Paster.exe
>
> (Env) C:\Kallithea\Bin>C:\Kallithea\Env\Scripts\paster.exe celeryd 
> production.ini &
> (Env) C:\Kallithea\Bin>C:\Kallithea\Env\Scripts\paster.exe serve 
> production.ini

Is the & a typo? It is sh notation for running a process in the 
background, but AFAIK it means "command sequence" on windows? Are you 
sure you actually are launching celeryd here?

It also seems wrong to use a paster.exe directly this way. On unix the 
virtualenv has a bin/paster which launches a 
pkg_resources.load_entry_point - it seems magic if this paster.exe does 
the same ... but perhaps it does.

> Given that the machine is solely dedicated to this purpose I also 
> tried NOT using virtualenv - that way it would be impossible??? to get 
> an incorrect context for Celeryd (as there would only be 1 python) and 
> the result was the same - celery task not registered

Yes, that might be simpler in this case. Windows has a stronger 
tradition for installing Python whereever you want, while Unix systems 
are more like only having one global installation in a fixed location.

In that case I suggest verifying that you really don't use virtualenv 
and that Kallithea is installed globally ... and perhaps restart the 
machine to make sure everything is running in the right environment.

Also, you might consider starting out without celeryd. That might be ok 
for your actual work load - mainly depending on how often you make big 
clones or have a slow mail server.

/Mads
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sfconservancy.org/pipermail/kallithea-general/attachments/20150419/be733daf/attachment-0001.html>