Initial attempt to make local docker dev setup (#981)

* Initial attempt to make local docker dev setup

* Docker setup improvements

* Create project in the root
* Allow specifying directory parameter in misago-start
* Clean up project files when starting container
* Minor changes from review
* Added project files to .gitignore
* Removed docker volume for Postgres data

* Comment wait_for_postgres.sh

* Ignore project files only in root

* Append console email backend to settings

* Minor improvements

* Create superuser on deploy configurable from compose
* Rename forum -> deform
* Empty out STATICFILES_DIRS in settings

* Remove small derp

* Remove theme and cron.txt from .gitignore

* Delete theme and cron.txt on deployment

* Empty template dirs in settings

* Create project named 'devforum` by default

* Development docs - take 1

* Include log command

* Added script for easy postgres/psql access

* Moving devforum setup to a separate init.sh file

* Moved init script too root + doc update

* Update admin user in docs

* Working tests

* Extending the posters image to create a second test database

We avoid running to postgres instances this way

* Don't include virtualenvs in the docker context

This speeds up the build time tremendouly

* Pull test database name from compose setup

* Include example of running tests in docs

* Minor tweaks to README

.dockerignore

@@ -0,0 +1,14 @@
+# Speed up bulding by excluding files from the docker context
+
+# Project files we don't need in docker
+postgres/
+
+# Local development stuff
+dev/
+env/
+venv/
+venv2/
+venv3/
+venv34/
+venv35/
+venv36/

.gitignore

@@ -47,6 +47,10 @@ pylint.txt
 .project
 .pydevproject
 
+# IDEs
+.idea/
+.vscode/
+
 # Rope
 .ropeproject
 
@@ -65,7 +69,7 @@ _book/
 # Local development stuff
 dev/
 dev-manage.py
-.vscode/
+env/
 venv/
 venv2/
 venv3/
@@ -75,3 +79,9 @@ venv36/
 future-manage.py
 misago-admin.py
 db.sqlite3
+
+# Local development files
+/manage.py
+/devforum/
+/avatargallery/
+/static/

Dockerfile

@@ -0,0 +1,28 @@
+# This dockerfile is only meant for local development of Misago
+# If you are looking for a proper docker setup for Misago look elsewhere
+FROM python:3
+
+ENV PYTHONUNBUFFERED 1
+ENV PATH "$PATH:/srv/misago"
+
+# Install dependencies in one single command/layer
+RUN apt-get update && apt-get install -y \
+    vim \
+    libffi-dev \
+    libssl-dev \
+    sqlite3 \
+    libjpeg-dev \
+    libopenjpeg-dev \
+    locales \
+    cron \
+    postgresql-client
+
+# Add requirements and install them. We do this unnecessasy rebuilding.
+ADD requirements.txt /
+RUN pip install -r requirements.txt
+
+WORKDIR /srv/misago
+
+EXPOSE 8000
+
+CMD python manage.py runserver 0.0.0.0:8000

README.rst

@@ -82,44 +82,62 @@ If you are looking into using Misago to run live forum, you are absolutely invit
 Development
 ===========
 
-To start Misago site locally, first setup and activate virtual environment for it and then fire following commands::
+To start Misago site locally, first make sure you have `docker <https://www.docker.com/community-edition#/download>`_ installed.
 
-    python setup.py install
-    misago-start.py testforum
+To get a basic empty forum up simply run::
 
-This will install Misago and its dependencies in your virtual environment and will make pre-configured Misago site for you named ``testforum``::
+   docker-compose build
+   docker-compose run --rm misago initdev
+   docker-compose up -d
 
-    testforum
-      + avatar_store
-      + media
-      + testforum
-        * __init__.py
-        * settings.py
-        * urls.py
-        * wsgi.py
-      + static
-      + theme
-      + cron.txt
-      + manage.py
+A Django developer server will start, enabling you to visit ``127.0.0.1:8000``
+in your browser and see the forum index. You should now be able to sign in with the superuser account.
 
-Now  edit ``settings.py`` file in your editor of choice in order to set up basic settings like database connection, default timezone or interface language.
+Admin Control Panel available under ``127.0.0.1:8000/admincp/`` url.
 
-Next, initialize database by using migrate commands provided by ``manage.py`` admin utility that you'll find in directory up one level from where ``settings.py`` is::
+The ``initdev`` script prepares everything you need:
 
-    python manage.py migrate
+* requirements are installed
+* ``devforum`` project is created in the misago project root
+* ``settings.py`` is modified with variables from the ``docker-compose.yaml`` file
+* database migrations runs
+* superuser is created
 
-Then, call ``createsuperuser`` command to create super admin in database::
+The default env vars passed in ``docker-compose.yml`` is:
 
-    python manage.py createsuperuser
+.. code-block:: yaml
 
-Finally start development server using ``runserver`` command::
+    environment:
+      # Postgres
+      - POSTGRES_USER=misago
+      - POSTGRES_PASSWORD=misago
+      - POSTGRES_DB=misago
+      - POSTGRES_HOST=postgres
+      - POSTGRES_TEST_DB=misago_test
+      # Superuser
+      - SUPERUSER_USERNAME=Admin
+      - SUPERUSER_EMAIL=admin@example.com
+      - SUPERUSER_PASSWORD=password
 
-    python manage.py runserver
+Some useful commands during development:
 
-If nothing is wrong with your setup, Django developer server will start, enabling you to visit ``127.0.0.1:8000`` in your browser and see the forum index. You should now be able to sign in to user account that you have created ealier.
+.. code-block:: bash
 
-You will likely want to customize your site via changing settings and creating categories. You can do this with Admin Control Panel available under ``127.0.0.1:8000/admincp/`` url.
+    # Enter the running misago container
+    docker-compose exec misago bash
 
+    # Manually run the misago container. Run ``python manage.py runserver 0.0.0.0:8000``.
+    # This can be useful when debugging.
+    docker-compose run --rm --service-ports misago bash
+
+    # View container logs
+    docker-compose logs -f --tail 100
+
+    # Enter psql so you can inspect or modify the database
+    docker-compose run --rm misago extras/psql.sh
+
+    # Runnning tests
+    docker-compose run --rm misago python runtests.py
 
 Frontend
 --------

docker-compose.yaml

@@ -0,0 +1,35 @@
+# This compose setup is only meant for local development of Misago itself
+# This is not for running your Misago site in docker
+version: "3.0"
+services:
+  postgres:
+    build: postgres
+    environment:
+      - POSTGRES_USER=misago
+      - POSTGRES_PASSWORD=misago
+      - POSTGRES_DB=misago
+  misago:
+    build: .
+    command: python manage.py runserver 0.0.0.0:8000
+    volumes:
+      # Map in the entire project into the container
+      # This makes sure files in the container updates on the fly as we were working locally
+      - .:/srv/misago
+    environment:
+      # Postgres
+      - POSTGRES_USER=misago
+      - POSTGRES_PASSWORD=misago
+      - POSTGRES_DB=misago
+      - POSTGRES_HOST=postgres
+      - POSTGRES_TEST_DB=misago_test
+      # Superuser
+      - SUPERUSER_USERNAME=Admin
+      - SUPERUSER_EMAIL=admin@example.com
+      - SUPERUSER_PASSWORD=password
+    ports:
+      # Map port 8000 in the container to port 8000 on the host
+      # This way we can access the forum through http://localhost:8000
+      - "8000:8000"
+    depends_on:
+      - postgres
+    tty: true

extras/createdevproject.py

@@ -0,0 +1,59 @@
+"""
+Creates a test project for local development
+"""
+
+import os
+import sys
+
+from misago.core import setup
+
+
+BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+
+
+def main():
+    project_name = 'devforum'
+
+    # Allow for overriding project name
+    if len(sys.argv) > 1:
+        project_name = sys.argv[1]
+    else:
+        sys.argv.append(project_name)
+
+    settings_file = os.path.join(BASE_DIR, project_name, 'settings.py')
+
+    # Avoid recreating if already present
+    if os.path.exists(settings_file):
+        return
+
+    setup.start_misago_project()
+    fill_in_settings(settings_file)
+
+
+def fill_in_settings(f):
+    with open(f, 'r') as fd:
+        s = fd.read()
+
+        # Postgres
+        s = s.replace("'NAME': '',", "'NAME': os.environ['POSTGRES_DB'],")
+        s = s.replace("'USER': '',", "'USER': os.environ['POSTGRES_USER'],")
+        s = s.replace("'PASSWORD': '',", "'PASSWORD': os.environ['POSTGRES_PASSWORD'],")
+        s = s.replace("'HOST': 'localhost',", "'HOST': os.environ['POSTGRES_HOST'],")
+
+        # Specify console backend for email
+        s += "\nEMAIL_BACKEND = 'django.core.mail.backends.console.EmailBackend'\n"
+
+        # Empty the contents of STATICFILES_DIRS (STATICFILES_DIRS = [])
+        pos = s.find('STATICFILES_DIRS')
+        s = s[:s.find('[', pos) + 1] + s[s.find(']', pos):]
+
+        # Remote theme dir from template dirs
+        pos = s.find("'DIRS': [")
+        s = s[:s.find('[', pos) + 1] + s[s.find(']', pos):]
+
+    with open(f, 'w') as fd:
+        fd.write(s)
+
+
+if __name__ == '__main__':
+    main()

extras/createsuperuser.py

@@ -0,0 +1,21 @@
+"""
+Create superuser for the devforum
+"""
+
+import os
+import django
+
+os.environ['DJANGO_SETTINGS_MODULE'] = 'devforum.settings'
+django.setup()
+
+from django.contrib.auth import get_user_model
+
+User = get_user_model()
+
+
+if User.objects.count() == 0:
+    User.objects.create_superuser(
+        os.environ['SUPERUSER_USERNAME'],
+        os.environ['SUPERUSER_EMAIL'],
+        password=os.environ['SUPERUSER_PASSWORD']
+    )

extras/psql.sh

@@ -0,0 +1,2 @@
+#!/usr/bin/env bash
+PGPASSWORD=$POSTGRES_PASSWORD psql --username $POSTGRES_USER --host $POSTGRES_HOST $POSTGRES_DB

extras/wait_for_postgres.sh

@@ -0,0 +1,10 @@
+#!/usr/bin/env bash
+# Sometimes postgres is not ready before django attempts to connect.
+# This script waits until we can do a basic select before continuing.
+export PGPASSWORD=$POSTGRES_PASSWORD
+RETRIES=10
+
+until psql -h $POSTGRES_HOST -U $POSTGRES_USER -d $POSTGRES_DB -c "select 1" > /dev/null 2>&1 || [ $RETRIES -eq 0 ]; do
+  echo "Waiting for postgres server, $((RETRIES--)) remaining attempts..."
+  sleep 5
+done

initdev

@@ -0,0 +1,15 @@
+#!/usr/bin/env bash
+# This script is a shortcut for configuring newly-build docker container for Misago development
+python setup.py develop
+
+# Create new project
+python extras/createdevproject.py devforum /srv/misago
+
+# Clean up unnecessary project files
+rm -rf theme
+rm cron.txt
+
+# Database
+./extras/wait_for_postgres.sh
+python manage.py migrate
+python extras/createsuperuser.py

misago/core/setup.py

@@ -37,16 +37,20 @@ def get_misago_project_template():
 
 
 def start_misago_project():
-    parser = OptionParser(usage="usage: %prog project_name")
+    parser = OptionParser(usage="usage: %prog project_name [directory]")
     _, args = parser.parse_args()
 
-    if len(args) != 1:
+    if len(args) < 1:
         parser.error("project_name must be specified")
 
+    dir = None
+    if len(args) == 2:
+        dir = args[1]
+
     project_name = validate_project_name(parser, args[0])
 
     argv = [
-        'start-misago.py', 'startproject', project_name,
+        'start-misago.py', 'startproject', project_name, dir,
         '--template=%s' % get_misago_project_template()
     ]
 

postgres/Dockerfile

@@ -0,0 +1,3 @@
+FROM postgres:9
+
+COPY docker-entrypoint-initdb.d /docker-entrypoint-initdb.d

postgres/docker-entrypoint-initdb.d/create-test-db.sh

@@ -0,0 +1,7 @@
+#!/bin/bash
+set -e
+
+psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" <<-EOSQL
+    CREATE DATABASE misago_test;
+    GRANT ALL PRIVILEGES ON DATABASE misago_test TO "$POSTGRES_USER";
+EOSQL

runtests.py

@@ -122,14 +122,14 @@ TEST_NAME = 'travis_ci_test'
 DATABASES = {
     'default': {
         'ENGINE': 'django.db.backends.postgresql_psycopg2',
-        'NAME': 'misago_test',
-        'USER': '%s',
-        'PASSWORD': '',
-        'HOST': '',
-        'PORT': '',
+        'NAME': os.environ['POSTGRES_TEST_DB'],
+        'USER': os.environ['POSTGRES_USER'],
+        'PASSWORD': os.environ['POSTGRES_PASSWORD'],
+        'HOST': os.environ['POSTGRES_HOST'],
+        'PORT': 5432,
     }
 }
-""" % pwd.getpwuid(os.getuid())[0]
+"""
 
     with open(settings_path, "w") as py_file:
         py_file.write(settings_file)