Integrating npm front-end with Kallithea

[Mads Kiilerich]

Hello, Community

Big parts of Dominik's front-end work around introducing Bootstrap has 
landed. Thanks!

The next step require a bit of an architectural decision. I would like 
to share it here to get feedback/awareness from the rest of the community.

It is about how we combine the use of the npm package manager and 
packages with Kallithea primarily being distributed on pypi as GPL 
application. It might be an annoying problem, but we just have to find a 
solution to it.

I have a proposal that describe the problem and some solutions ... and 
implement one that seems to work. It is available on 
https://kallithea-scm.org/repos/kallithea-incoming/changeset/1b9cb678344f80221f14bdb9b779d5a4823d829a 
for testing and experimenting, but I will also paste it here.

Does it work for you, or what do you suggest?

/Mads


# HG changeset patch
# User Mads Kiilerich <mads at kiilerich.com>
# Date 1510108450 -3600
#      Wed Nov 08 03:34:10 2017 +0100
# Node ID 1b9cb678344f80221f14bdb9b779d5a4823d829a
# Parent  6bef1d7bafa6b1743ad10f644d72139e10b95b76
setup: install and run npm commands when installing Kallithea with pip

Kallithea is under the GPL license, and we can thus only distribute any
generated code if we also ship the corresponding source.

We are moving towards a web front-end that use npm to download and compile
various open source components. The components might not be GPL, but if we
distribute any parts of their code (compiled or converted to other
representation), then we also must distribute the corresponding source under
the GPL.

It doesn't seem feasible for us to distribute the source of everything 
that npm
downloads and includes when we are building. It thus also doesn't seem 
feasible
for us to build and ship the compiled (possibly minified) front-end code.
Instead, we have to make it as smooth as possible for our users to get 
up and
running.

It doesn't seem feasible for us to ship or install npm. We must assume it is
available. That requirement must be documented clearly, and we must 
recommend
how to install npm for the most common platforms.

We could perhaps just document what manual steps to run. Kallithea 
doesn't work
out of the box anyway - it has to be configured and initialized. Extra steps
might not be a big problem. We could perhaps also have a custom gearbox 
command
to run the commands. But running it on demand when launching the editor 
would
probably scare most system administrators ... and would also create 
different
problems with the different ways of running Kallithea.

But it would be nice if we could call out to npm while pip is installing
Kallithea and download the requirements and build the files. It can be 
done by
customizing setuptools commands in setup.py . That is what is done here.

Python packaging is fragile. Even though we only support pip, it really 
isn't
built for things like this. Custom output is muted and buffered and only 
shown
if running with -v or the command fails. And pip and setup.py can be used to
build and install in so many ways that we probably can't make it work 
reliably
with all ways of installing Kallithea. But we can try.

The npm invocation can be disabled by setting the environment variable
KALLITHEA_INSTALL_RUN_NPM=NO .

The custom 'install' method is used by a plain 'pip install' form pypi 
or tar,
while 'develop' is used when running 'pip install -e' from source. While
developing, it might also be convenient to just run the 'less' command
directly, or to use 'nodemon' to run it automatically when the source files
change.

For now, this will just create/update style.css ... but currently probably
without any actual changes. The files created by npm (and the node_modules
directory) must *not* be a part of the release package made with 'setup.py
sdist'.

diff --git a/docs/overview.rst b/docs/overview.rst
--- a/docs/overview.rst
+++ b/docs/overview.rst
@@ -69,6 +69,10 @@ installed.
    (``pip install kallithea`` from a source tree will do pretty much 
the same
    but build the Kallithea package itself locally instead of 
downloading it.)

+Kallithea also include some front-end code that need npm_ to download 
packages
+and build the front-end. The ``npm`` binary must thus be available, and 
it will
+be invoked automatically and invisibly when installing with ``pip``.
+

  Web server
  ----------
@@ -138,3 +142,4 @@ continuous hammering from the internet.
  .. _WSGI: http://en.wikipedia.org/wiki/Web_Server_Gateway_Interface
  .. _HAProxy: http://www.haproxy.org/
  .. _Varnish: https://www.varnish-cache.org/
+.. _npm: https://www.npmjs.com/
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -3,6 +3,8 @@
  import os
  import sys
  import platform
+import subprocess
+

  if sys.version_info < (2, 6) or sys.version_info >= (3,):
      raise Exception('Kallithea requires python 2.6 or 2.7')
@@ -122,6 +124,31 @@ class sdist_new(sdist_org):
          self.owner = self.group = 'root'
  sdist.sdist = sdist_new

+
+from setuptools.command.develop import develop
+from setuptools.command.install import install
+
+class DevelopNpm(develop):
+    def run(self):
+        develop.run(self)
+        install_npm()
+
+class InstallNpm(install):
+    def run(self):
+        install.run(self)
+        install_npm()
+
+def install_npm():
+    if os.environ.get('KALLITHEA_INSTALL_RUN_NPM') != 'NO':
+        import kallithea # has just been installed
+        rootdir = 
os.path.dirname(os.path.dirname(os.path.abspath(kallithea.__file__)))
+        srcdir = os.path.join(rootdir, 'kallithea', 'public', 'less')
+        sys.stderr.write("\nRunning 'npm install' in %s to install 
packages from package.json\n" % srcdir)
+        subprocess.check_call(['npm', 'install'], cwd=srcdir)
+        sys.stderr.write("\nRunning 'npm run less' in %s to build css 
from package.json\n" % srcdir)
+        subprocess.check_call(['npm', 'run', 'less'], cwd=srcdir)
+
+
  packages = setuptools.find_packages(exclude=['ez_setup'])

  setuptools.setup(
@@ -140,6 +167,9 @@ setuptools.setup(
      data_files=data_files,
      packages=packages,
      include_package_data=True,
+    cmdclass={
+        'install': InstallNpm,
+        'develop': DevelopNpm},
      message_extractors={'kallithea': [
              ('**.py', 'python', None),
              ('templates/**.mako', 'mako', {'input_encoding': 'utf-8'}),