D2.1 Documentation

D2.1 - general

The deliverable report describes the whole picture of the NOMAD database development status as of month 6.

Technical details going beyond this can be found here. This is supplementary information, thus this page is not exhaustive.

Deliverable report

... report will be downloadable here as soon as submitted

Archive and Parsers (Base Layer) setup

Git Repositories

You can get the main git repository with

git clone --recursive git@gitlab.mpcdf.mpg.de:nomad-lab/nomad-lab-base.git

the metadata description repository is at

git clone git@gitlab.mpcdf.mpg.de:nomad-lab/nomad-meta-info.git

The main git repository (nomad-lab-base) contains submodules (nomad-meta-info etc.), hence the --recursive option is required during clone. If you clone the main git repository without the --recursive option, then you can initialize the submodules with

 git submodule update --init

This command initializes your local configuration file and then fetches the submodule. More information on submodule can be found on Git Tools: Submodules

Setup

submodules

Get the git submodules in this repo:

git submodule update --init

scala

Scala by itself is not strictly required, but it ensures that all dependencies are there (mainly java).

#wget http://downloads.typesafe.com/scala/2.11.5/scala-2.11.5.deb?_ga=1.172612385.307956976.1430825833
wget http://downloads.typesafe.com/scala/2.11.6/scala-2.11.6.deb
sudo dpkg -i scala-2.11.6.deb

sbt installation

if [ ! -e "/etc/apt/sources.list.d/sbt.list" ] ; then
   echo "deb http://dl.bintray.com/sbt/debian /" | sudo tee -a /etc/apt/sources.list.d/sbt.list
   sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 642AC823
fi
sudo aptitude update
sudo aptitude install sbt

hdf5 / netCDF installation

read & execute (line by line if you have problems/worry about security)

hdf5/getAndBuild.sh

compilation

just use sbt normally:

$ sbt
> compile
> test
> testOnly *MyTests*
> myParser/test
> tool/run
> re-start
> re-stop
...

this interactive use keeps things cached and gives faster compilation/test, console will start an interactive scala environment where you can import and play with all the infrastructure (i.e. scala REPL + all dependencies and compiled code).

Encyclopedia infrastructure setup

Connecting to NOMAD virtual machine

$ ssh USER@con01.rzg.mpg.de
$ ssh root@enc-devel-nomad

Add a new user

the new user will have access to our Nomad database and run the python application:

user: nomadapp pass: NoMaD_App

enc-devel-nomad:~ # useradd -m nomadapp
enc-devel-nomad:~ # passwd nomadapp

test to see if user was created and we can login

enc-devel-nomad:~ # su - nomadapp
nomadapp@enc-devel-nomad:~> whoami
nomadapp
nomadapp@enc-devel-nomad:~> pwd
/home/nomadapp
nomadapp@enc-devel-nomad:~>

PostgreSQL installation:

let's install the necessary packages

enc-devel-nomad:~ # zypper install postgresql94-server postgresql94 postgresql94-contrib
Loading repository data...
Reading installed packages...
Resolving package dependencies...

The following 5 NEW packages are going to be installed:
  libpq5 postgresql-init postgresql94 postgresql94-contrib postgresql94-server

5 new packages to install.
Overall download size: 5.2 MiB. Already cached: 0 B. After the operation, additional 23.2 MiB will be used.
Continue? [y/n/? shows all options] (y): y
[ ... ]
enc-devel-nomad:~ #

check to see if the packages are installed and running properly

enc-devel-nomad:~ # postgres --version
postgres (PostgreSQL) 9.4.6

let's start PostgreSQL server once and check if its status

enc-devel-nomad:~ # service postgresql start
enc-devel-nomad:~ # service postgresql status
postgresql.service - LSB: Start the PostgreSQL master daemon
   Loaded: loaded (/etc/init.d/postgresql)
   Active: active (exited) since Fri 2016-03-04 16:25:29 CET; 6s ago
  Process: 29987 ExecStart=/etc/init.d/postgresql start (code=exited, status=0/SUCCESS)

enable PostgreSQL to start at boot time (after a restart):

enc-devel-nomad:~ # chkconfig postgresql on
enc-devel-nomad:~ # chkconfig --list
postgresql                0:off  1:off  2:off  3:on   4:off  5:on   6:off
enc-devel-nomad:~ # systemctl is-enabled postgresql

a postgres user is automatically created when the packages were installed, so, let's see if is working

enc-devel-nomad:~ # su - postgres
postgres@enc-devel-nomad:~> whoami
postgres
postgres@enc-devel-nomad:~> psql
psql (9.4.6)
Type "help" for help.

postgres=# \conninfo
You are connected to database "postgres" as user "postgres" via socket in "/var/run/postgresql" at port "5432".
postgres=#

postgres-# \l
                                  List of databases
   Name    |  Owner   | Encoding |   Collate   |    Ctype    |   Access privileges
-----------+----------+----------+-------------+-------------+-----------------------
 postgres  | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 |
 template0 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +
           |          |          |             |             | postgres=CTc/postgres
 template1 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +
           |          |          |             |             | postgres=CTc/postgres
(3 rows)

postgres-#
postgres=# SELECT * FROM pg_catalog.pg_tables;
     schemaname     |        tablename        | tableowner | tablespace | hasindexes | hasrules | hastriggers
--------------------+-------------------------+------------+------------+------------+----------+-------------
 pg_catalog         | pg_statistic            | postgres   |            | t          | f        | f
 pg_catalog         | pg_type                 | postgres   |            | t          | f        | f

postgres-# \q

let’s create a new user role for our database, using the same login name as the user created for running python app:

enc-devel-nomad:~ # su - postgres
postgres@enc-devel-nomad:~> createuser --interactive
Enter name of role to add: nomadapp
Shall the new role be a superuser? (y/n) y
postgres@enc-devel-nomad:~>

test the new user/role for database

enc-devel-nomad:~ # su - nomadapp
nomadapp@enc-devel-nomad:~> psql -d repodump
repodump=# \conninfo
You are connected to database "repodump" as user "nomadapp" via socket in "/var/run/postgresql" at port "5432".
repodump=#
repodump=# \password nomadapp

Note: the password added is Nomad_Flask

in order to have authentifcation enabled for our webapp is needed to replace ident with md5 in /var/lib/pgsql/data/pg_hba.conf config file. The important section should look like (after our changes):

# IPv4 local connections:
host    all             all             127.0.0.1/32            md5
# IPv6 local connections:
host    all             all             ::1/128                 md5

and then restarting Postgres is needed to reread the config file

enc-devel-nomad:~ # service postgresql restart

Note: PostgreSQL LOG files are in: /var/lib/pgsql/data/pg_log/

Nginx installation

Nginx is not present in the default SLES12 repository, we need to add it's repository to our system

enc-devel-nomad:~ # zypper addrepo -G -t yum -c 'http://nginx.org/packages/sles/12' nginx
Adding repository 'nginx' ......................................................................................[done]
Warning: GPG checking is disabled in configuration of repository 'nginx'. Integrity and origin of packages cannot be verified.
Repository 'nginx' successfully added
Enabled     : Yes
Autorefresh : No
GPG Check   : No
Priority    : 99
URI         : http://nginx.org/packages/sles/12

let's install the package

enc-devel-nomad:~ # zypper install nginx

start the webserver once, to check if it is installed and working

enc-devel-nomad:~ # systemctl start nginx
enc-devel-nomad:~ # systemctl status nginx
nginx.service - nginx - high performance web server
   Loaded: loaded (/usr/lib/systemd/system/nginx.service; disabled)
   Active: active (running) since Wed 2016-03-09 10:14:21 CET; 2s ago

check if Nginx is enabled at boot time

enc-devel-nomad:~ # systemctl list-unit-files | grep nginx
nginx.service                           disabled

is not enabled, so we have to enable it

enc-devel-nomad:~ # systemctl enable nginx
ln -s '/usr/lib/systemd/system/nginx.service' '/etc/systemd/system/multi-user.target.wants/nginx.service'

check again if indeed is enabled

enc-devel-nomad:~ # systemctl list-unit-files | grep nginx
nginx.service                           enabled

add/prepare a config file for our application

enc-devel-nomad:~ # cd /etc/nginx/conf.d
enc-devel-nomad:/etc/nginx/conf.d # pwd
/etc/nginx/conf.d

enc-devel-nomad:/etc/nginx/conf.d # mv default.conf default.conf.orig

enc-devel-nomad:/etc/nginx/conf.d # cat nomadapp.conf
server {
    listen       80;
    server_name  localhost;

    location / {
        proxy_pass http://127.0.0.1:5000;
    }
}

restart the webserver to read the new config file

enc-devel-nomad:/etc/nginx/conf.d # systemctl restart nginx

all right, our webserver is up and running

Note: The Nginx config file is very minimal but for first Deliverable D2.1 is sufficient

Python and Flask environment setup

install the necessary python packages:

enc-devel-nomad:~ # zypper install python-setuptools
enc-devel-nomad:~ # easy_install pip
enc-devel-nomad:~ # pip install virtualenv

in order to use psycopg2 (to connect to database using python, we don't use SQLAlchemy right now) we have to install postgresql94-devel package:

enc-devel-nomad:~ # zypper install postgresql94-devel

let’s install supervisord package which will monitor our python app. (not used for Deliverable D2.1)

enc-devel-nomad:~ # pip install supervisor
Collecting supervisor
  Downloading supervisor-3.2.2.tar.gz (410kB)
    100% |...........................| 419kB 2.1MB/s

let’s generate the supervisor config file and place it in nomadapp user’s home folder (we can also put it in /etc/)

enc-devel-nomad:~ # su - nomadapp
nomadapp@enc-devel-nomad:~> echo_supervisord_conf > supervisord.conf

nomadapp@enc-devel-nomad:~> ls -ltr supervisord.conf
-rw-r--r-- 1 nomadapp users 8658 Mar  9 14:31 supervisord.conf

we need to create a virtual environment for python where to install all necessary modules for our app (Flask, Werkzeug, gunicorn, ...) and to run the app in this virtualenv

nc-devel-nomad:~ # su - nomadapp
nomadapp@enc-devel-nomad:~> pwd
/home/nomadapp

nomadapp@enc-devel-nomad:~> virtualenv nomadvenv
New python executable in /home/nomadapp/nomadvenv/bin/python
Installing setuptools, pip, wheel...done.

nomadapp@enc-devel-nomad:~> source nomadvenv/bin/activate
(nomadvenv) nomadapp@enc-devel-nomad:~>

add the python modules to our virtualenv

(nomadvenv) nomadapp@enc-devel-nomad:~> pip install flask gunicorn
Collecting flask
  Using cached Flask-0.10.1.tar.gz
Collecting gunicorn
  Downloading gunicorn-19.4.5-py2.py3-none-any.whl (112kB)
    100% |...........................| 122kB 2.4MB/s
Collecting Werkzeug>=0.7 (from flask)
  Downloading Werkzeug-0.11.4-py2.py3-none-any.whl (305kB)
    100% |...........................| 307kB 1.9MB/s
Collecting Jinja2>=2.4 (from flask)
  Downloading Jinja2-2.8-py2.py3-none-any.whl (263kB)
    100% |...........................| 266kB 6.1MB/s
Collecting itsdangerous>=0.21 (from flask)
  Downloading itsdangerous-0.24.tar.gz (46kB)
    100% |...........................| 51kB 10.6MB/s
Collecting MarkupSafe (from Jinja2>=2.4->flask)
  Downloading MarkupSafe-0.23.tar.gz
Building wheels for collected packages: flask, itsdangerous, MarkupSafe
  Running setup.py bdist_wheel for flask ... done
  Stored in directory: /home/nomadapp/.cache/pip/wheels/d2/db/61/cb9b80526b8f3ba89248ec0a29d6da1bb6013681c930fca987
  Running setup.py bdist_wheel for itsdangerous ... done
  Stored in directory: /home/nomadapp/.cache/pip/wheels/97/c0/b8/b37c320ff57e15f993ba0ac98013eee778920b4a7b3ebae3cf
  Running setup.py bdist_wheel for MarkupSafe ... done
  Stored in directory: /home/nomadapp/.cache/pip/wheels/94/a7/79/f79a998b64c1281cb99fa9bbd33cfc9b8b5775f438218d17a7
Successfully built flask itsdangerous MarkupSafe
Installing collected packages: Werkzeug, MarkupSafe, Jinja2, itsdangerous, flask, gunicorn
Successfully installed Jinja2-2.8 MarkupSafe-0.23 Werkzeug-0.11.4 flask-0.10.1 gunicorn-19.4.5 itsdangerous-0.24
(nomadvenv) nomadapp@enc-devel-nomad:~>

for Deliverable D2.1 we run the python app only using gunicorn, without configuring supervisord

(nomadvenv) nomadapp@enc-devel-nomad:~/app> pwd
/home/nomadapp/app
(nomadvenv) nomadapp@enc-devel-nomad:~/app> gunicorn -c gunicorn.conf app:app -b localhost:5000
(nomadvenv) nomadapp@enc-devel-nomad:~/app>

check if our application is running:

enc-devel-nomad:~ # ps aux | grep gunicorn
nomadapp 13510  0.0  0.1  93044 13060 ?        S    13:31   0:00 /home/nomadapp/nomadvenv/bin/python /home/nomadapp/nomadvenv/bin/gunicorn -c gunicorn.conf app:app -b localhost:5000
nomadapp 13515  0.0  0.2 140168 19588 ?        S    13:31   0:00 /home/nomadapp/nomadvenv/bin/python /home/nomadapp/nomadvenv/bin/gunicorn -c gunicorn.conf app:app -b localhost:5000
root     13560  0.0  0.0   9236   928 pts/1    S+   13:33   0:00 grep --color=auto gunicorn
enc-devel-nomad:~ #

All DONE

Notes:

there is no automatic deployment for miniAPI and miniGUI
the miniAPI is in /home/nomadapp/app/ and the miniGUI is in /home/nomadapp/app/static/
gunicorn's config file:

daemon = True
accesslog = "logs/gunicorn_access.log"
errorlog = "logs/gunicorn_error.log"

nginx config file (really minimal file, nginx is acting as a reverse proxy for our python app):

server {
    listen       80;
    server_name  localhost;

    location / {
        proxy_pass http://127.0.0.1:5000;
    }
}

Mini-API

The API exposes to the GUI the following properties:

List of data to be shown in the mini-GUI for the database - deliverable 2.1

Technologies used:

Python programming language with Flask microframework is used to develop the RESTful API interface
Nginx is used as web server. It is designed to serve high-load webservices and it also includes handy features like reverse-proxy, load-balancing, basic authentication which we will use/need in the future
Gunicorn (Green Unicorn) is used as a WSGI server to serve our main Python application. It is the recommended choice for Python web applications
Supervisor is used as a process manager. It provides a consistent interface through which long-running programs can be monitored and controlled

How to use the minimal API:

Request: GET

URI: /materials/search?element=ELEMENTS&page=PAGE-NUMBER&limit=ITEMS-PER-PAGE

Parameters:

element - the elements for searching, separated by comma: eg. element=NH,C,Ba
page - the number of page, 1 being the first page: eg. page=1
limit - the number of entries shown on one page: eg. limit=25

Note: all parameters are necessary!!!

Request example:

enc-devel-nomad:~ # curl -i http://localhost/materials/search?element=C'&'page=1'&'limit=3

Response example:

HTTP/1.1 200 OK
Server: nginx/1.8.1
Date: Wed, 20 Apr 2016 09:54:50 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 791
Connection: keep-alive
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: HEAD, GET
Access-Control-Max-Age: 21600
Access-Control-Allow-Credentials: true
Access-Control-Allow-Headers: Origin, X-Requested-With, Content-Type, Accept, Authorization

[
  {
    "xc_treatment_basis": "PBE+vdW",
    "space_group": 1,
    "nr_atoms": 26,
    "code_name": "FHI-aims 031011",
    "basis_set_type": "numeric AOs",
    "formula": "C<sub>6</sub>N<sub>4</sub>H<sub>14</sub>O<sub>2</sub>",
    "nr_results": 883
  },
  {
    "xc_treatment_basis": "PBE+vdW",
    "space_group": 1,
    "nr_atoms": 37,
    "code_name": "FHI-aims 031011",
    "basis_set_type": "numeric AOs",
    "formula": "CdC<sub>9</sub>N<sub>5</sub>H<sub>20</sub>O<sub>2</sub>",
    "nr_results": 883
  },
  {
    "xc_treatment_basis": "PBE+vdW",
    "space_group": 1,
    "nr_atoms": 28,
    "code_name": "FHI-aims 031011",
    "basis_set_type": "numeric AOs",
    "formula": "CaC<sub>6</sub>N<sub>4</sub>H<sub>15</sub>O<sub>2</sub>",
    "nr_results": 883
  }
]