POI Data Provider - Installation and Administration Guide

Introduction

The purpose of this document is to provide the required information for a system administrator in order to install and configure the POI (Points of Interest) Data Provider generic enabler reference implementation. The POI GE is implemented as a RESTful web service using PHP programming language. It is described in more detail in FIWARE.OpenSpecification.MiWi.POIDataProvider.

Document Releases

This document is associated to the latest release of the POI Data Provider. Links to versions related to earlier software releases are in the table below.

Release Date Description
r3.3 2014-09-17 Original release - a POI belongs to exactly one category

System Requirements

Hardware Requirements

The POI Data Provider should work on any modern PC computer that is capable of running Ubuntu Server 14.04. Therefore, the bare minimum requirements are: 300 MHz x86 processor 192 MiB of system memory (RAM) (256 MiB for a virtual installation) * 1.4 GB of disk space

For a small practical deployment, the recommended system is:

  • 1 GHz Dual core CPU
  • 4 GB of system memory (RAM)
  • 40 GB of disk space

The hardware needs of the POI Data Provider are mainly dominated by the databases (PostgreSQL and MongoDB), and as such the two most important factors are:

  • memory size (the bigger the better)
  • disk size
  • disk speed (e.g. using SSD drives)

You can have a rough estimate for the required disk space by using the following formula: 

size_on_disk = number_of_pois * 10KB

So, for example for one million (1000000) POIs, you get the following estimate for data size on disk: 

size_on_disk = 1000000 * 10KB = 10000000 KB ~ 10GB

However, this estimate may be inaccurate for many cases, as the mean POI size can be much smaller or larger than 10 kilobytes.

Operating System Support

The implementation and this installation guide have been tested with Ubuntu 14.04. Other Linux distribution may need modifications in installation procedure and configuration files.

Software Requirements

In order to have the POI Data Provider up and running, the following software is required:

  • PostgreSQL 9.3
  • PostGIS 2.1.x spatial database extension
  • MongoDB 2.0.x
  • Apache HTTP Server 2.2.x
  • (Or basically any HTTP server with PHP support)
  • PHP 5.x
  • PostgreSQL module
  • MongoDB module
  • JSON Schema for PHP 1.4.3 [1]
  • pecl_http module

Software Installation and Configuration

Update OS if applicable

You may run to problems with old OS image. The software packages may assume some new OS corrections. So you may need to update the OS first:

$ sudo apt-get update

Installing required packages

The required software packages can be installed using the 'apt-get' command-line package installation tool:

$ sudo apt-get install -y postgis postgresql-9.3-postgis-2.1
$ sudo apt-get install mongodb
$ sudo apt-get install apache2
$ sudo apt-get install php5 php5-pgsql
$ sudo apt-get install git

The installation of the MongoDB module for PHP5. :

$ sudo apt-get install php-pear php5-dev gcc make
$ sudo pecl install mongo

The installation of the Pecl_HTTP module for PHP5. This enables use of HTTP requests to obtain dynamic data from other sites.

$ sudo apt-get install libcurl3-openssl-dev
$ sudo pecl install pecl_http

Add these lines to /etc/php5/apache2/php.ini:

extension=mongo.so
extension=raphf.so
extension=propro.so
extension=http.so

Enable access control in per-directory basis. The POI DP uses .htaccess file to protect external access keys for dynamic POIs.

Change the following line in the file /etc/apache2/apache2.conf within <Directory /var/www> section:

AllowOverride None

---> Change to --->

AllowOverride All

Restart Apache web server:

$ sudo /etc/init.d/apache2 restart

Configuring PostGIS

  1. Create GIS database user:

    $ sudo -u postgres createuser gisuser

    Answer "n" to all questions.

  2. Create database owned by that user

    $ sudo -u postgres createdb --encoding=UTF8 --owner=gisuser poidatabase

  3. Activate PostGIS on the created database:

    $ sudo -u postgres psql -d poidatabase -f /usr/share/postgresql/9.3/contrib/postgis-2.1/postgis.sql
$ sudo -u postgres psql -d poidatabase -f /usr/share/postgresql/9.3/contrib/postgis-2.1/spatial_ref_sys.sql
$ sudo -u postgres psql -d poidatabase -f /usr/share/postgresql/9.3/contrib/postgis-2.1/postgis_comments.sql
$ sudo -u postgres psql -d poidatabase -c "GRANT SELECT ON spatial_ref_sys TO PUBLIC;"
$ sudo -u postgres psql -d poidatabase -c "GRANT ALL ON geometry_columns TO gisuser;"

  4. Enable UUID functions for that database:

    $ sudo -u postgres psql -d poidatabase -c 'create extension "uuid-ossp";'

  5. Grant local access to the database:

    Before you can access the database, you must edit PostgreSQL configuration to allow local unix socket connections (from the same computer where the database is running) without password.

    Edit the file /etc/postgresql/9.3/main/pg_hba.conf and change the following line:

    # "local" is for Unix domain socket connections only
local   all             all                                     peer

    ---> Change to --->

    # "local" is for Unix domain socket connections only
local   all             all                                     trust

  6. Restart PostgreSQL

    $ sudo /etc/init.d/postgresql restart

Installing POI Data Provider

  1. Fetch the POI Data Provider from GitHub:

    $ git clone https://github.com/Chiru/FIWARE-POIDataProvider.git

  2. Create required database tables using the provided script:

    $ cd FIWARE-POIDataProvider/install_scripts
$ ./create_tables.sh
$ cd ../..

  3. Copy the folder FIWARE-POIDataProvider/php from the cloned project under the current working directory, e.g. to /var/www/html/poi_dp

    $ sudo cp -r FIWARE-POIDataProvider/php /var/www/html/poi_dp

Installation of JSON Schema for PHP

$ wget http://getcomposer.org/composer.phar
$ php composer.phar require justinrainbow/json-schema:1.4.3
$ sudo cp -r vendor /var/www/html/poi_dp/

More information about the JSON Schema for PHP implementation can be found at [3].

Enable Handling of Cross-origin Resource Sharing and URL Rewrite in Apache

Cross-origin Resource Sharing (CORS) is required if the POI-DP client is a web application that is hosted on a different domain than the POI-DP backend. In practice this means that the POI-DP Apache server needs to add the following HTTP header for each response:

Access-Control-Allow-Origin "*"

Rewrite is used to default the .php extension from service requests. E.g. http//www.example.org/poi_dp/radial_search -> http//www.example.org/poi_dp/radial_search.php .

Enable mod_headers and mod_rewrite Apache modules:

$ sudo a2enmod headers
$ sudo a2enmod rewrite
$ sudo service apache2 restart

Sanity check procedures

The Sanity Check Procedures are the steps that a System Administrator will take to verify that an installation is ready to be tested. This is therefore a preliminary set of tests to ensure that obvious or basic malfunctioning is fixed before proceeding to unit tests, integration tests and user validation.

End to End testing

You can do a quick test to see if everything is up and running by accessing the following URL:

http://hostname/poi_dp/radial_search?lat=1&lon=1&category=test_poi

You should get a JSON structure representing a test POI as a response.

List of Running Processes

You can use the following command to check if Apache HTTP server, PostgreSQL and MongoDB are running:

$ ps ax | grep 'postgres\|mongo\|apache2'

The output of the command should be something like the following: 

 8404 ?        Ssl   37:07 /usr/bin/mongod --config /etc/mongodb.conf
12380 ?        S      0:00 /usr/sbin/apache2 -k start
12381 ?        S      0:00 /usr/sbin/apache2 -k start
12382 ?        S      0:00 /usr/sbin/apache2 -k start
12383 ?        S      0:00 /usr/sbin/apache2 -k start
12384 ?        S      0:00 /usr/sbin/apache2 -k start
17966 ?        Ss     0:20 /usr/sbin/apache2 -k start
18845 ?        S      0:00 /usr/sbin/apache2 -k start
21262 ?        S      0:00 /usr/sbin/apache2 -k start
21263 ?        S      0:00 /usr/sbin/apache2 -k start
27956 ?        S      0:00 /usr/lib/postgresql/9.1/bin/postgres -D /var/lib/postgresql/9.1/main -c config_file=/etc/postgresql/9.1/main/postgresql.conf
27958 ?        Ss     0:00 postgres: writer process
27959 ?        Ss     0:00 postgres: wal writer process
27960 ?        Ss     0:00 postgres: autovacuum launcher process
27961 ?        Ss     0:00 postgres: stats collector process
28100 pts/0    R+     0:00 grep --color=auto postgres\|mongo\|apache2

Network interfaces Up & Open

The only required port open to the Internet is TCP port 80, used by HTTP protocol.

Databases

The POI Data Provider utilizes two database systems:

  • PostgreSQL/PostGIS for storing and accessing data components with spatial data (fw_core data component)
  • MongoDB for storing and accessing all other data components that do not require spatial searches

PostgreSQL

PostgreSQL has a database named 'poidatabase'. It contains a table called 'fw_core' and it contains the core information, such as name and location, about the POIs.

You can test if this table is succesfully created with the following commands:

$ psql -U gisuser poidatabase
poidatabase=> SELECT count(*) FROM fw_core;

If the table was created succesfully, this query should return '4', as there sould be four test POI entries created by the installation.

To exit PostgreSQL use:

poidatabase=> \q

MongoDB

MongoDB should also contain a database named 'poi_db'. It should contain a collection named 'testData' containing a single test POI entry, created by the installation.

You can test if MongoDB was succesfully configured with the following commands:

$ mongo
> use poi_db
> show collections

The show collections command should list five POI data component collections created by the installation: fw_contact, fw_marker, fw_media, fw_relationships and fw_time.

To exit MongoDB use:

> exit

Diagnosis Procedures

The Diagnosis Procedures are the first steps that a System Administrator will take to locate the source of an error in a GE. Once the nature of the error is identified with these tests, the system admin will very often have to resort to more concrete and specific testing to pinpoint the exact point of error and a possible solution. Such specific testing is out of the scope of this section.

Resource availability

The amount of available resources depends on the size of the database and the usage rate of the service. The minimum recommended available resources are:

  • Available memory: 4 GB
  • Available disk space: 40 GB

Resource consumption

The load value reported e.g. by the 'top' utility should not exceed the number of CPU cores in the system. If this happens, the performance of the system can dramatically drop.

Remote Service Access

Check that the HTTP port (80) is open and accessible from all the networks from which POI-DP will be used.

I/O flows

All the incoming and outgoing data of the POI Data Provider will go through TCP port 80. The size of the flow is entirely dependant on the usage of the service, e.g. number of users.

Updating Database to R5.1

The language key for the non-language-specific strings is changed from "" to "__" (two underscore characters. The reason is that the MongoDB database does not like zero-length keys. The keys in databases that are created with earlier versions of the POI-DP can be updated using the following command:

$ psql -U gisuser -d poidatabase -a -f FIWARE-POIDataProvider/install_scripts/update_fw_core_intl_to_5.1.sql