[PATCH 00 of 17 v3] Introduction of kallithea-cli command

[Thomas De Schampheleire]

,
El lun., 22 oct. 2018 a las 1:11, Mads Kiilerich
(<mads at kiilerich.com>) escribió:
>
> On 10/18/2018 10:49 PM, Thomas De Schampheleire wrote:
> > This series introduces a kallithea-cli command to replace our custom gearbox
> > commands.
> > At the end of this series, the help text will show:
> >
> >
> > Usage: kallithea-cli [OPTIONS] COMMAND [ARGS]...
> >
> >    Various commands to set up a Kallithea instance.
> >
> > Options:
> >    --help  Show this message and exit.
> >
> > Commands:
> >    cache-cleanup-keys  Clean up existing cache keys.
> >    cache-show-keys     Show existing cache keys with their status.
>
> These two commands are really not so much about some kind of "keys".
> They are about database entries that indicate that some worker process
> has a repo cached in memory, so the cache can be invalidated and flushed
> when necessary. It would thus perhaps be better to name these commands
> so they are grouped with other repo commands, like:
>
> repo-cache-cleanup   Flush in-memory repository caches.
> repo-cache-show    Show current in-memory repository caches.
>
> >    celery-run          Start Celery worker(s) for asynchronous tasks.
> >    config-create       Create a new configuration file.
> >    db-create           Initialize the database.
> >    extensions-create   Write template file for extending Kallithea in Python.
> >    front-end-create    Create the front-end.
>
> The "create" naming consistency is generally nice, but for this one,
> "build" seems much more descriptive and correct.
>
> >    iis-install         Install into IIS using isapi-wsgi.
> >    index-create        Create or update full text search index
> >    ishell-run          Interactive shell for Kallithea.
> >    repo-purge-deleted  Purge backups of deleted repositories.
> >    repo-scan           Scan filesystem for repositories.
> >    repo-update-cache   Update repo cache after external modification.
>
> This one is perhaps better described as: Update database cache of latest
> repository change.

Regarding the command names, we now have suggested:

repo-cache-cleanup
repo-cache-show
repo-update-cache

There is some inconsistency here that needs to be solved IMO.
The 'update-cache' not only handles the latest repository change, it
also invalidates the database cache.

For repo-cache-cleanup/show: under which circumstances would a user
execute any of these commands? Are these commands actually necessary?
Or can they be combined with repo-update-cache, possibly renamed?

>
> > Some notes:
> > - In v1 and v2, each topic was a separate word in the CLI, e.g. 'repo
> >    update-cache' rather than 'repo-update-cache'. There were two problems
> >    with that approach:
> >    1. to know which commands are available, you'd use 'kallithea-cli --help'
> >       to discover the topics, then 'kallithea-cli topic --help', and then
> >       you'd still need another 'kallithea-cli topic command --help' to
> >       get a list of the actual options.
>
> Hmm. I see. Good point. Annoying how it seems to force us to use a
> not-so-elegant solution with the "object" and the "method" tied together
> in the command name. But it also gives a more flat namespace that is
> more explorable ...
>
> In a way, it could perhaps be more elegant with more specific top level
> command names, something like
>    kallithea-repo update-cache my.ini foorepo barrepo
> but that would have some of the same problems ... and perhaps give too
> many commands, which might be a problem or a solution ...

It's technically possible, sure. But we'd have quite a large amount of
top-level commands, I'd say a bit too much to make it obvious.
But see also further in this mail...

>
> Also related to that: when thinking about a consistent solution, we
> should perhaps also consider where the existing kallithea_api.py
> kallithea_backup.py  kallithea_gist.py could fit in.

Of these, I only used kallithea_api.py.

With the current scheme, we could have following?
kallithea-cli api-call
kallithea-cli gist-create
kallithea-cli backup-create

>
> >    2. some command-lines would become quite long, and distinction between
> >    elements of the command itself, and its arguments, were hard to see. For
> >    example, to update repository caches:
> >       kallithea-cli repo update-cache my.ini foorepo barrepo
> >       vs.
> >       kallithea-cli repo-update-cache my.ini foorepo barrepo
>
> Also, looking at the examples (both commands in this patch series an
> examples from other tools), it seems less elegant when commands have
> different "positional parameters", like how parameter 1 is the .ini file
> and parameter 2 and up are repository names. I kind of like how the
> gearbox solution consistently use -c for these config files. The general
> rule for command line design might be that un-named parameters only
> should be used to name the main things we operate on.

I was following the guideline that mandatory arguments should not be
passed via options. Options should be optional.
But I could not find an authoritative source stating this guideline. I
looked at:
http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap12.html  (POSIX)
https://click.palletsprojects.com/en/7.x/
http://docopt.org/

I'm fine with reintroducing '-c <config>' instead.

>
> > - We need to find a better place for template.ini.mako which is now the sole
> >    inhabitant of kallithea/lib/paster_commands/.
>
> kallithea/templates/ is mostly html templates for the front-end, but we
> also have kallithea/templates/email_templates/ , so perhaps this one
> could be placed as something like kallithea/templates/config.ini .
>
>
> soooo ...
>
>
> With these nice experiments and overviews, it is interesting to take a
> step back and look at them and other complex and more or less elegant
> commands with sub-commands. Here are some I know about:
> https://docs.docker.com/engine/reference/commandline/cli/
> https://docs.docker.com/engine/reference/commandline/docker/
> https://docs.ansible.com/ansible/2.4/command_line_tools.html
> https://dnf.readthedocs.io/en/latest/command_ref.html
> https://linux.die.net/man/1/fedpkg
> https://www.mercurial-scm.org/doc/hg.1.html
> http://man7.org/linux/man-pages/man1/git.1.html
> https://www.openssl.org/docs/man1.1.1/man1/openssl.html
>
> One thing I conclude is that it indeed only is common to use one level
> of sub commands. But it varies whether sub-sub commands are implemented
> by having verbose sub command names (seems quite common for example in
> git) or using additional flags and having other options that only works
> together with some flags (for example like openssl or hg). I agree with
> the longer sub-command names without nesting here in v3.

This question shows the two approaches you refer to, right?
https://softwareengineering.stackexchange.com/questions/366218/positional-arguments-vs-options-in-a-command-line-interface

In most of the examples you linked to, subcommands are actions. e.g.
'pull, push, revert, resolve, stash, verify'.
This makes perfect sense to me.

With the approach in v2, where the subcommands where nouns like db,
config, repo, ..., that principle was violated.
In this v3, we again have actions with a classifier as to what the
action refers to, e.g. db-create vs config-create.
An alternative would be to swap noun and action, doing: create-db ,
create-config, ...
I would say neither is ideal. It stems from the fact that the CLI is
not very crisp, quite scattered.
Looking back at the list of commands in this v3 (without your
suggestions yet, just for simplicity):

  cache-cleanup-keys  Clean up existing cache keys.
  cache-show-keys     Show existing cache keys with their status.
  celery-run          Start Celery worker(s) for asynchronous tasks.
  config-create       Create a new configuration file.
  db-create           Initialize the database.
  extensions-create   Write template file for extending Kallithea in Python.
  front-end-create    Create the front-end.
  iis-install         Install into IIS using isapi-wsgi.
  index-create        Create or update full text search index
  ishell-run          Interactive shell for Kallithea.
  repo-purge-deleted  Purge backups of deleted repositories.
  repo-scan           Scan filesystem for repositories.
  repo-update-cache   Update repo cache after external modification.

There are commands that you need to set up Kallithea initially,
commands that you need to run Kallithea in operation and for
maintenance and then there are more debug-type of commands.

For initial set up, you need config-create, db-create, front-end-create.
You may also need iis-install and extensions-create in this category.

For standard operation, there is celery-run, kallithea_api, kallithea_gist
For maintenance there is cache-cleanup-keys, cache-show-keys,
index-create, repo-purge-deleted, repo-scan, repo-update-cache,
kallithea_backup.

And for debug there is ishell-run.


Like we also touched upon at
https://lists.sfconservancy.org/pipermail/kallithea-general/2018q3/002726.html
 I think it would be better to simplify the setup process.
We could have one command that combines config-create, db-create and
front-end-create. It could e.g. be 'kallithea-setup' or 'kallithea-cli
setup', to be decided. By default it does all these things, but via
--options similar to how I handled front-end-create, you could
restrict the actions that are performed. We'll have to change the
behavior of config-create so that the db settings are required, and
thus that you can create the config file and the database in one go,
and later tweak other settings in the config file.

But even if we do that, we still are left with the other commands that
seem difficult to group.
You can indeed do as you hinted earlier: create separate top-level
commands kallithea-index, kallithea-cache, etc. and each of these
could more naturally take an action as argument.
With config-create/db-create/front-end-create out of the way,  this
would already be more realistic.

>
> I do however think that the config file as first unnamed parameter was
> another thing that contribute equally much to the hard-to-read commands
> from v2. That remains. What are your thoughts about re-introducing -c as
> a (mandatory) parameter?

As said above, I'm fine with that.

>
> Another observation from looking for examples, is a concern whether my
> preference for "kallithea-cli" over just "kallithea" is justified. I
> would still say a vague "yes", but I am less confident. (For the
> examples I listed above, the command line tool comes closer to being the
> tool itself. For Kallithea, the tool is only for admin stuff - it is not
> Kalllithea itself.)

The biggest disadvantage of kallithea-cli is that it is more typing.
But depending on the debate above regarding multiple top-level
commands, there may be no issue.

Ok, many open questions to discuss further on... :-)

Best regards,
Thomas