Installing PyAMS

PyAMS default installation is based on Buildout utility. It’s not mandatory to use a virtual environment, but it allows you to have a better control over your Python resources.

Current PyAMS version is based and validated for Python 3.5; your Python environment must also include a C compiler as well as development headers for Python, libjpeg, libpng, libfreetype, libxml2, libxslt and eventually libldap or libzmq.

PyAMS default components configuration also pre-suppose that the following external tools are available:

  • a Memcached or Redis server, to store sessions and cache (can be changed through Beaker configuration)

Optional tools also include:

  • an LDAP server for authentication
  • an ElasticSearch server for full text indexing (see PyAMS_content_es package)
  • a WebSockets server using AsyncIO. This is used to manage notifications (see PyAMS_notify and PyAMS_notify_ws packages). An out of the box environment can be built using pyams_notify scaffold.

PyAMS also needs that you use a ZEO server, as several background processes needing a concurrent access to ZODB are started by PyAMS main process. See Creating a ZEO server to know how to create a ZEO server with the help of PyAMS tools.

Creating initial buildout

PyAMS provides a new Pyramid scaffold, called pyams, generated via a cookiecutter template.

A simple option to install PyAMS is to create a buildout environment including Pyramid and all PyAMS packages:

# mkdir /var/local/
# pip3 install virtualenv
# virtualenv --python=python3.5 env
# cd env
# . bin/activate
(env) # pip3.5 install cookiecutter
(env) # cookiecutter hg+http://hg.ztfy.org/pyams/scaffolds/pyams

CookieCutter will ask you for a small set of input variables that you can change or not:

  • pyams_release: version of PyAMS configuration file to use. “latest” (default value) will point to last release; you can also choose to point to a given release (“0.1.4” for example)
  • project_name: current environment name in “human form”
  • project_slug: “technical” package name, based on project name
  • virtual_hostname: Apache virtual-host name
  • webapp_name: web application package name (“webapp” as default)
  • webapp_port: TCP/IP port to use when running application outside Apache (“6543” as default)
  • eggs_directory: relative or absolute path to directory containing downloaded eggs; this directory can be shared with other projects (“eggs” as default)
  • logs_directory: absolute path to directory containing Apache’s log files
  • run_user: user name under which Apache process will run (“www-data” as default)
  • run_group: group name under which Apache process will run (“www-data” as default)
  • beaker_backend: name of Beaker backend to use to store sessions and cache data (“redis” as default)
  • beaker_server: IP address and port of Beaker backend server (“127.0.0.1:6379” as default)
  • zeo_server: IP address and port of ZEO server (“127.0.0.1:8100” as default); WARNING: ZEO server installation is not part of application installation; another “zeo_server” cookiecutter receipe is available for that.
  • zeo_storage: name of ZEO storage to use
  • zeo_username: ZEO user name
  • zeo_password: ZEO password
  • zeo_realm: ZEO authentication realm
  • zeo_blobs_dir: local directory to use to store cache of ZODB blobs; cache size is limited to 1GB as default
  • use_postgresql: specify if PostgreSQL access is required; if so, please check that PostgreSQL development files are available to compile PsycoPG2 extension
  • use_oracle: specify if Oracle access is available; if so, please check that Oracle development files are available to compile cx_Oracle extension, and that ORACLE_HOME environment variable is correctly defined
  • use_ldap: specify if LDAP access will be required for authentication
  • use_elasticsearch: specify if an ElasticSearch server will be used for indexation
  • elasticsearch_server: URL used to access Elasticsearch server (“http://127.0.0.1:9200” as default); this URL can include login and password (“http://login:password@127.0.0.1:9200”), if required…
  • elasticsearch_index: name of Eslasticsearch index to use (“pyams” as default)
  • create_elasticsearch_index: specify if Elasticsearch index should be created after installation is complete
  • define_elasticsearch_mappings : specify if Elasticsearch mappings should be defined after installation is complete
  • smtp_server: DNS name of SMTP server (“localhost” as default)
  • smtp_server_name: “human” name given to SMTP server (“pyams” as default)
  • pyams_scheduler: TCP/IP address and port to use to access PyAMS tasks scheduler process (“127.0.0.1:5555” as default)
  • pyams_medias_converter: TCP/IP address and port to use to access PyAMS medias converter process (“127.0.0.1:5556” as default)
  • pyams_es_indexer: TCP/IP address and port to use to access PyAMS Elasticsearch indexer process (“127.0.0.1:5557” as default)
  • use_notifications: specify if PyAMS notifications services are to be used
  • pyams_ws_notify: TCP/IP address and port of PyAMS websockets server managing notifications service (“127.0.0.1:8081” as default)
  • lexicon_languages: NLTK lexicon languages to use (“en:english fr:french” as default)
  • extension_package: name of a PyAMS extension package to include in environment configuration
  • need_pyams_gis: specify if PyAMS GIS features are to be used by given extension package; if so, please check that libgdal development files are available and on Debian, you have to specify environment variables for C_INCLUDE_PATH and CPLUS_INCLUDE_PATH pointing to /usr/include/gdal directory. WARNING: you have to check also that your libgdal release is matching “GDAL” release given in PyAMS configuration file (actually 2.1.0).

You can then check, and eventually update, the proposed Buildout configuration file buildout.cfg, to add or remove packages or update settings to your needs. Then finalize Bootstrap initialization:

(env) # python3.5 bootstrap.py
(env) # ./bin/buildout

This last operation can be quite long, as many packages have to downloaded, compiled and installed in the virtual environment. If you encounter any compile error, just install the required dependencies and restart the buildout.

Environment settings

The project generated from pyams scaffold is based on default Pyramid’s zodb scaffold, but it adds:

  • a custom application factory, in the webapp directory (see PyAMS site management)
  • a set of directories to store runtime data, in the var directory; each directory contains a README.txt file which should be self-explanatory to indicate what this directory should contain, including a ZEO cache
  • a set of configuration files, in the etc directory; here are standard development.ini and production.ini configuration files, a ZODB configuration files (zodb-zeo.conf) for a ZEO client storage and two Apache configurations (for Apache 2.2 and 2.4) using mod_wsgi.

Once the project have been created from the scaffold, you are free to update all the configuration files.

If you need to add packages to the environment, you have to add them to the buildout.cfg file AND to the INI file (in the pyramid.includes section) before running the buildout another time; don’t forget to add the requested version at the end of buildout.cfg file, as Buildout is not configured by default to automatically download the last release of a given unknown package.

development.ini and production.ini files contain many commented directives related to PyAMS components. Read and update them carefully before initializing your application database!

Initializing the database

When you have downloaded and installed all required packages, you have to initialize the database so that all required components are available.

From a shell, just type:

(env) # ./bin/pyams_upgrade etc/development.ini

This process requires that every package is correctly included into pyramid.includes directive from selected configuration file.

Starting the application

When database upgrade process has ended, you can start the web application process with the standard Pyramid’s pserve command line tool:

(env) # ./bin/pserve etc/development.ini

In standard debug mode, all registered components are displayed in the console, until the final line (here using ZEO):

2016-12-28 10:58:46,347 INFO  [ZEO.ClientStorage][MainThread] [('localhost', 8100)] ClientStorage (pid=30133) created RW/normal for storage: 'pyams'
2016-12-28 10:58:46,349 DEBUG [ZODB.blob][MainThread] (30133) Blob directory `/var/local/env/pyams/db/blobs` has layout marker set. Selected `bushy` layout.
2016-12-28 10:58:46,349 WARNI [ZODB.blob][MainThread] (30133) Blob dir /var/local/env/pyams/db/blobs/ has insecure mode setting
2016-12-28 10:58:46,349 INFO  [ZEO.cache][MainThread] created temporary cache file 5
2016-12-28 10:58:46,373 BLATH [ZEO.zrpc][MainThread] (30133) CM.connect(): starting ConnectThread
2016-12-28 10:58:46,374 BLATH [ZEO.zrpc][Connect([(<AddressFamily.AF_INET: 2>, ('localhost', 8100))])] (30133) CT: attempting to connect on 1 sockets
2016-12-28 10:58:46,380 BLATH [ZEO.zrpc][Connect([(<AddressFamily.AF_INET: 2>, ('localhost', 8100))])] (30133) CW: attempt to connect to ('127.0.0.1', 8100)
2016-12-28 10:58:46,380 BLATH [ZEO.zrpc][Connect([(<AddressFamily.AF_INET: 2>, ('localhost', 8100))])] (30133) CW: connect_ex(('127.0.0.1', 8100)) returned EINPROGRESS
2016-12-28 10:58:46,381 BLATH [ZEO.zrpc][Connect([(<AddressFamily.AF_INET: 2>, ('localhost', 8100))])] (30133) CT: select() 0, 1, 0
2016-12-28 10:58:46,381 BLATH [ZEO.zrpc][Connect([(<AddressFamily.AF_INET: 2>, ('localhost', 8100))])] (30133) CW: connect_ex(('127.0.0.1', 8100)) returned 0
2016-12-28 10:58:46,382 INFO  [ZEO.ClientStorage][Connect([(<AddressFamily.AF_INET: 2>, ('localhost', 8100))])] [('localhost', 8100)] Testing connection <ManagedClientConnection ('127.0.0.1', 8100)>
2016-12-28 10:58:46,383 INFO  [ZEO.zrpc.Connection(b'C')][[('localhost', 8100)] zeo client networking thread] (127.0.0.1:8100) received handshake b'Z3101'
2016-12-28 10:58:46,483 INFO  [ZEO.ClientStorage][Connect([(<AddressFamily.AF_INET: 2>, ('localhost', 8100))])] [('localhost', 8100)] Server authentication protocol None
2016-12-28 10:58:46,484 INFO  [ZEO.ClientStorage][Connect([(<AddressFamily.AF_INET: 2>, ('localhost', 8100))])] [('localhost', 8100)] Connected to storage: ('localhost', 8100)
2016-12-28 10:58:46,485 INFO  [ZEO.ClientStorage][Connect([(<AddressFamily.AF_INET: 2>, ('localhost', 8100))])] [('localhost', 8100)] No verification necessary -- empty cache
2016-12-28 10:58:46,486 DEBUG [ZEO.ClientStorage][Connect([(<AddressFamily.AF_INET: 2>, ('localhost', 8100))])] pickled inval None b'\x03\xbc>\x1a\x166f"'
2016-12-28 10:58:46,488 BLATH [ZEO.zrpc][Connect([(<AddressFamily.AF_INET: 2>, ('localhost', 8100))])] (30133) CM.connect_done(preferred=1)
2016-12-28 10:58:46,489 BLATH [ZEO.zrpc][Connect([(<AddressFamily.AF_INET: 2>, ('localhost', 8100))])] (30133) CT: exiting thread: Connect([(<AddressFamily.AF_INET: 2>, ('localhost', 8100))])
2016-12-28 10:58:46,676 DEBUG [config][MainThread] include /var/local/env/pyams/eggs/pyams_template-0.1.0-py3.4.egg/pyams_template/configure.zcml
2016-12-28 10:58:46,677 DEBUG [config][MainThread] include /var/local/env/pyams/eggs/pyramid_zcml-1.0.0-py3.4.egg/pyramid_zcml/configure.zcml
2016-12-28 10:58:46,678 DEBUG [config][MainThread] include /var/local/env/pyams/eggs/pyramid_zcml-1.0.0-py3.4.egg/pyramid_zcml/meta.zcml
2016-12-28 10:58:46,684 DEBUG [config][MainThread] include /var/local/env/pyams/eggs/pyams_template-0.1.0-py3.4.egg/pyams_template/meta.zcml
2016-12-28 10:58:46,710 DEBUG [config][MainThread] include /var/local/env/pyams/eggs/pyams_viewlet-0.1.0-py3.4.egg/pyams_viewlet/configure.zcml
2016-12-28 10:58:46,711 DEBUG [config][MainThread] include /var/local/env/pyams/eggs/pyramid_zcml-1.0.0-py3.4.egg/pyramid_zcml/configure.zcml
2016-12-28 10:58:46,711 DEBUG [config][MainThread] include /var/local/env/pyams/eggs/pyramid_zcml-1.0.0-py3.4.egg/pyramid_zcml/meta.zcml
2016-12-28 10:58:46,746 DEBUG [config][MainThread] include /var/local/env/pyams/eggs/zope.browserpage-4.1.0-py3.4.egg/zope/browserpage/meta.zcml
2016-12-28 10:58:46,825 DEBUG [config][MainThread] include /var/local/env/pyams/eggs/pyams_template-0.1.0-py3.4.egg/pyams_template/meta.zcml
...
2016-12-28 11:02:01,354 DEBUG [PyAMS (utils)][MainThread] Registering adapter <function WfSharedContentIndexInfo at 0x7f506aa9a0d0> for (<InterfaceClass pyams_content.shared.common.interfaces.IWfSharedContent>,) providing <InterfaceClass pyams_content_es.interfaces.IDocumentIndexInfo>
2016-12-28 11:02:01,370 DEBUG [PyAMS (utils)][MainThread] Registering utility <class 'pyams_content_es.site.ContentIndexerGenerationsChecker'> named 'PyAMS content indexer' providing <InterfaceClass pyams_utils.interfaces.site.ISiteGenerations>
2016-12-28 11:02:01,383 DEBUG [PyAMS (pagelet)][MainThread] Registering pagelet view "test-indexer-process.html" for <InterfaceClass pyams_content_es.interfaces.IContentIndexerUtility> (<class 'pyams_content_es.zmi.ContentIndexerProcessTestForm'>)
2016-12-28 11:02:01,387 DEBUG [PyAMS (pagelet)][MainThread] Registering pagelet view "properties.html" for <InterfaceClass pyams_content_es.interfaces.IContentIndexerUtility> (<class 'pyams_content_es.zmi.ContentIndexerUtilityPropertiesEditForm'>)
2016-12-28 11:02:01,400 DEBUG [PyAMS (utils)][MainThread] Registering adapter <class 'pyams_default_theme.configuration.StaticConfiguration'> for (<InterfaceClass pyams_utils.interfaces.site.IStaticConfigurationManager>, <InterfaceClass pyams_default_theme.layer.IPyAMSDefaultLayer>, <InterfaceClass zope.interface.Interface>) providing <InterfaceClass pyams_skin.interfaces.configuration.IStaticConfiguration>
2016-12-28 11:02:01,403 DEBUG [PyAMS (pagelet)][MainThread] Registering pagelet view "" for <InterfaceClass zope.interface.Interface> (<class 'pyams_default_theme.page.BaseIndexPage'>)
2016-12-28 11:02:01,410 DEBUG [PyAMS (utils)][MainThread] Registering utility <class 'pyams_default_theme.skin.PyAMSDefaultSkin'> named 'PyAMS default skin' providing <InterfaceClass pyams_skin.interfaces.ISkin>
2016-12-28 11:02:01,411 DEBUG [PyAMS (utils)][MainThread] Registering adapter <class 'pyams_default_theme.skin.ResourcesAdapter'> for (<InterfaceClass zope.interface.Interface>, <InterfaceClass pyams_default_theme.layer.IPyAMSDefaultLayer>, <InterfaceClass zope.interface.Interface>) providing <InterfaceClass pyams_skin.interfaces.resources.IResources>
Starting server in PID 30235.
serving on http://0.0.0.0:6543

From this point, you can launch a browser and open URL http://127.0.0.1:6543/admin to get access to PyAMS management interface; default login is “admin/admin”, that you may change as soon as possible…