The infrastructure consists of 5 applications in two groups (publishing and curation) that interact. Each application has its own GitHub repository (or two in the case of Rhakhis) with associated README.md documentation describing how to to install it. This page gives an overview of the commonalities between the applications, where they might be installed and how they interact.

Publishing applications

  • Website - Publishes the public facing view. A PHP application that queries a Apache SOLR index for data (LAMP without MySQL). Repository: wfo-p2 (p2 originally named for “Portal 2.0”)
  • List API - that publishes the public APIs. A PHP application that queries a Apache SOLR index for data (LAMP without MySQL). Repository: wfo-plant-list (Named for its original purpose was to server the plant list alone.)

Curation applications

  • Rhakhis - for editing the backbone. Repositories: wfo-backbone-management. This is a full LAMP (Linux, Apache, MySQL, PHP) application that provides a management interface and a private API for the front end which is a Bootstrap-React Javascript application wfo-backbone-ui.
  • Fyllo - for binding text content stored in GitHub to the portal. A full LAMP (Linux, Apache, MySQL, PHP) application. Repository: wfo-fyllo.
  • Apache Airflow - Orchestrates workflows that move data between applications. A Java application framework with workflows written in Python. Repository: wfo-airflow.

Design pattern

Apart from Apache Airflow and the Rhakhis UI the applications follow roughly the same design pattern.

  • WFO HOME <- a base directory in which to install the applications. This will typically be /var/wfo or similar
    • application <- the application directory. This is typically a direct clone of the GitHub repository created with git clone <url>.
      • www <- web root for the application. There should be a virtual host set up in Apache config (/etc/apache2/sites-enabled) to serve this directory to the web on a subdomain e.g. list.worldfloraonline.org.
        • style <- css stuff
        • scripts <- javascript used in page rendering
        • data or downloads <- data files that need to be downloadable. This directory will frequently need to be web writeable so that downloadable and/or cached files can be created by the user. Something like sudo chown -R ubuntu:www-data downloads/ on install.
      • include(s) <- code used in www but not served directly
      • scripts <- code run on the command line and by cron jobs
    • application_secrets.php <- configuration variables that shouldn’t be checked into GitHub like passwords and API keys. There should be a template version of this file in the application root that you can copy down a layer and edit.

Locations

Data curation applications and the data publishing applications should never be on the same machine. They function entirely separately and have different requirements.

Publishing applications locations.

There are multiple ways the Website and List API could be deployed. They both need to interact with an Apache SOLR index which should be on the same machine or LAN in a production environment. (In development/testing environments they can talk to SOLR over the internet but this will be much slower.)

  1. All on one machine: This is the simplest approach and is probably the one to start with. A single Linux box with SOLR 8., PHP 8. and two Apache virtual hosts, one for the website and one for the List API.
  2. Split out the index: Apache SOLR is given its own machine. (This could be a Docker container instance if the infrastructure supports that.) A second machine hosts the Website and List API.
  3. Parallelise the front end: An enhancement of 2 would be to have multiple machines serving the website and api applications but all talking to the same SOLR index.
    • A three machine set up would have SOLR on one machine, Website on another and List API on a third. This might give the best robustness to complexity ratio as issues can be isolated to individual machines.
    • If multiple instances of the website are deployed then something like a hardware based round robin DNS load balancer that maintained user sessions would needed. These assign users randomly to machines but all a user’s calls go to the same machine. Probably beyond what we need.
  4. Parallelise the back end: There are many options to scale SOLR index performance using SolrCloud. Unlikely to be necessary.

There could be multiple instances of the publishing applications deployed around the world. An institution could, for example, have its own instance of the List API locally for performance and stability. There is only one issue with this approach. The taxonomic backbone is imported as a six monthly JSON dump file but the text content is pushed to the live index by the Airflow application from Fyllo. Currently the workflow can only target one instance at a time but methods for cloning between SOLR indexes could be developed if the functionality was needed.

Curation applications locations

Ideally these should all be on the same machine or same LAN. Airflow needs to access the MySQL database behind Rhakhis but not Fyllo. Airflow also needs to be able access the publishing server over HTTPS for API calls.