*[Overview](https://osf.io/mqk45/wiki/home/) • [Bibliography](http://osf.io/mqk45/wiki/Bibliography/) • [Design](https://osf.io/mqk45/wiki/Design/) • [Project](https://osf.io/mqk45/wiki/Project/) • [Community](https://osf.io/mqk45/wiki/Research%20Community/) • [Team](https://osf.io/mqk45/wiki/Team/) • [Timeline](https://osf.io/mqk45/wiki/Timeline/)* # Design Approach Even though FAIMS Mobile design decisions were sound, new development is now required because the software components used to build FAIMS Mobile have become obsolete (some are end-of-life), making ongoing maintenance unsustainable, or have been surpassed by new technologies. As a result, FAIMS Mobile will be decommissioned in 2020. This project, however, does not merely replace FAIMS Mobile. The proposed Electronic Field Notebooks would surpass it, using new technologies and approaches to support fieldwork-based research while dramatically simplifying customisation. ## Adopt The proposed Platform concentrates on a single part of the data lifecycle - data capture and management in the field - for a wide range of fieldwork disciplines. In order for such a platform to serve researchers, it must work with platforms and services that can manage data across the remainder of the data lifecycle. As such, the proposed FAIMS 3.0 Electronic Field Notebooks Platform will be able to exchange data via API with existing platforms to for the management and analysis of ‘active’ data, as well as repositories for the archiving and dissemination of datasets. Thanks to new partnerships with AARNet, the Platform will export data to Cloudstor for active data storage, sharing, and cleaning / analysis. At minimum, SWAN will be available for cleaning and analysis, and during technical elaboration we will explore the deployment of containerised applications to Cloudstor for non-tabular data (e.g., QGIS or GRASS for geospatial data). Also during elaboration we will explore the viability of a data ‘round-trip’ between our Platform and Cloudstor, where data can be captured in FAIMS 3.0, exported to Cloudstor, edited, and then returned to our Platform for further corrections in the field. This ‘round-trip’ was one of the major requests made by existing FAIMS users. The Platform will also export data to domain-specific platforms and repositories, including AVRE Geo DeVL, Alveo Human Communication Science VL, Open Context, and the Digital Archaeological Record (tDAR). This export will be accomplished either via API or a custom exporter (the latter being the approach taken by the previous version of FAIMS Mobile, which currently supports export to Alveo and Open Context). Thus, the FAIMS 3.0 Platform will adopt existing solutions for later stages of the data lifeycle, including Cloudstor / SWAN, AVRE Geo DeVL, Alveo Human Communication Science VL, Open Context, and the Digital Archaeological Record (tDAR). This interoperability will also serve as a model for federation with additional platforms and services in the future. ## Adapt This proposal aims to re-engineer FAIMS 2.x (designed 2012-2014) from the ground up using 2020 technologies to make it more sustainable by facilitating uptake, e.g., through easier self-customisation and deployment. In most cases, technologies will be adapted rather than adopted wholesale or built from scratch - but it is sometimes difficult to draw a bright line between ‘adapt’ and ‘build’. As such, a coherent overview of the core of the Platform (the electronic field notebook applications themselves) is presented here. It will be built using widely-used open-source components and based on core web (W3C) standards as far as possible to avoid dependencies and lock-in. Specific technology choices would be finalised during technological elaboration (those discussed below should be considered indicative). New parts of the platform that did not previously exist (e.g., a web application for GUI-driven customisation and orchestrated deployment to public cloud infrastructure) will be discussed under ‘build’. Like our existing software, FAIMS 3.0 will implement a Model-View-Controller design to abstract specific data capture tasks from core data and UI management software. The ‘core’ software will remain distinct from customisations. Customisation will still be accomplished using a Domain Specific Language (DSL) to specify data schema, user interface, and automation (updated from XML in FAIMS 2.x to JSON in FAIMS 3.0). Using code to specify each customisation offers many advantages: improved transparency, reproducibility, and reusability, including the ability to leverage distributed version control systems like git / GitHub to share and modify customisations. It also intrinsically documents field workflows, processes, and data structures (FAIMS 2.x, for example, automatically generates PDF wireframes of all data entry screens and their relationships to one another, a feature that will be retained, while implicit ontologies could be extracted from the data structures and vocabularies defined in this document). FAIMS 2.x used a runtime interpreter to instantiate customisations from XML configuration files. We now propose that the core application should be a Javascript based Progressive Web Application that can run on various platforms. Customisations would be instantiated by loading a JSON configuration file into a generic application that interprets it to display the custom UI. The application could be optionally compiled into an installable application using a technology such as Capacitor or Apache Cordova (recognising that these technologies introduce a significant dependency). See ‘build’ below for more detail. Regarding network architecture, during elaboration we will explore mesh networking, which would allow clients to synchronise directly with one another without the need for a server. This approach promises 'Google Drive-like’ editing capability while completely isolated from a local or online server. We note, however, that earlier tests of this topology were not successful, so we reserve a client-server architecture as a secondary synchronisation strategy. In either case, the server will still be needed as an ultimate source of truth for synchronisation (even if it is only one node in a mesh), as well as for user management, backup, export, and compiling binaries (see above). The server would also expose data using the server's API for properly credentialed users. This architecture assumes network-degraded or offline environments with opportunistic synchronisation when a connection is present. FAIMS Mobile v2.5 was built using an append-only datastore implemented in a relational DBMS. FAIMS 3.0 will instead, preferentially, use a graph database implemented in a native JSON noSQL datastore (e.g., Apache CouchDB / PouchDB). This approach offers a number of advantages, including the ability to maintain a complete data version history without the performance problems (high-costs reads) of an append-only DKNF relational database (or, conversely, the need to develop materialised views in SQLite, the only viable open-source, relational DBMS for offline mobile data collection). A noSQL datastore supports a flexible approach to data schemas, without requiring ‘tables of columns’ and ‘tables of rows’ (such a highly abstracted key-value store increases query complexity and imposes a performance penalty in exchange for its flexibility). Multi-device asynchronous editing with fork detection can be problematic. Elaboration will determine if this challenge can be overcome, with two fall-back approaches: (1) combining a OLTP DBMS and native XML/JSON datastores as a redo archive log, or (2) implementing a DVCS repository and commit log for the JSON stored on the mobile device. Mapping and GIS will be accomplished using a tiler with a local cache for offline viewing and editing on devices. We have investigated the use of a full-featured mobile GIS (QField, an Android port of QGIS), but it is poorly documented C++ code implemented using the Android NDK. Adaptation of this code would be expensive, and it would produce an Android-only mobile application. QField performance on even a fast tablet was also frustrating. Tiler-based solutions have fewer inherent GIS operations than QField / QGIS, but needed functionality (e.g., computational or manual creation and editing of geometries) can be built at a lower cost than QGIS Android NDK / C++ adaptation. If these tiler solutions cannot perform geometric computations, we would implement a local tileserver on the device (e.g., using Node and Node-GDAL as a geometry and tile bridge and cache between the server and the instance on the mobile device). The second fallback would use a relational DBMS with spatial extensions (e.g., SQLite / SpatiaLite) as a geometry computation engine (but not necessarily as the principal datastore). For connections to external sensors and other devices, we believe that we will still need to (1) trap input events from devices that emulate an HID over USB, and (2) exchange ASCII over a simulated serial connection over Bluetooth and USB. This capability may require custom extensions to be built for each target platform although we note the recent release of a draft WebUSB API specification, currently limited to Chrome and Opera browsers. Data ingest to, and export from the server will use the same generalised shell-script plugin architecture used in FAIMS v2.x. As with customisation specification files, these plugins can be shared and modified leveraging a DVCS (like git / GitHub). We would improve our current deployment by running the plugins in Docker containers to improve security and isolation. The server will also provide a complete RESTful API for data interactions. ## Build The principal parts of the Platform described above that needs to be built from scratch are the interpreter and the compiler for instantiating custom electronic field notebook applications from the JSON customisation files. These replace the monolithic Android only application of the previous implementation and enable the software to run on any web-enabled platform as well as providing installable applications on Android and IOS. The components listed above (or similar) will be used to reengineer an updated version of FAIMS Mobile (developed since 2012). Seven years' experience with FAIMS, plus user feedback and customer research (via CSIRO ON Prime customer interviews) gives us a clear idea of what to build to drive the uptake necessary to produce a commercially viable product. Key to this uptake is easier, self-service customisation and deployment. This capability will be supported by a new JSON based configuration format that encodes the details of all fields being collected and the order and organisation of the collection forms.  While this could be written by hand, we will implement a graphical, web-based interface to allow users to develop their own customisation. This application will: - Allow the user to configure their data collection application by selecting or entering the appropriate fields/vocabulary items and arranging them into pages. - Allow selection and fine-tuning of data entry widgets - e.g., numerical ranges, date selection, map based entry, etc. - Provide constraints on data entry that must be met - e.g. required fields, sequencing of fields, conditional fields (show this field if value entered in XX is YY). - Allow drag-and-drop reordering of fields within a form and pages within the application. - Provide a preview of the application on a simulated mobile device. - Configure style and branding of the application. - Generate the JSON code necessary to instantiate a customisation, either interpreted using a generic application or compiled into a custom application (see 'Adapt' above). The custom JSON configuration will then be either interpreted by a generic application (similar to FAIMS 2.5 but implemented using web technologies) or compiled into a custom application for multiple platforms. Depending on the approach taken (see the discussion above of mesh networking and syncing) we may also at this point generate a generic or custom server application to support the mobile application and act as a central data store or configuration host.   This server could then be deployed to a dedicated hosting provider or on the clients own facilities using standard components (e.g. Docker containers). The overall scope of this development is well-defined. The goal of our initial beta release will be the ability to redeploy the customisations that exist for CSIRO (environmental sampling), UNSW (oral history), and LTU (survey archaeology) in FAIMS 3.0, reimplementing the necessary functionality of FAIMS 2.x while adding (1) cross-platform deployability (iOS, Android, desktop) and (2) the ability to create and deploy customisations of low to moderate complexity via a web app with a GUI (requirements which were gathered through our ON Prime interviews). Little additional business analysis or requirements gathering needs to be undertaken, so the Project's technical staff (Abed, Ballun-Stanton, Cassidy) can begin technical elaboration immediately. ## Reuse The proposed Platform is generalised and deeply customisable, making it reusable across field research domains. FAIMS 2.x started in archaeology but has now been used for geochemical sampling, avian and marine ecology, oral history, ethnography, linguistics, and archaeology (2014-2019). FAIMS 3.0 will be equally flexible. Participants on this Proposal have new use cases (e.g., metadata capture for geophysical remote sensing instruments), and additional reuse in other domains, like agriculture and soil sciences, are imminently feasible. Thus, the Platform itself is reusable across domains. To make deployments easier and encourage the emergence of communities of practice, configuration files, custom exporters, external device support modules, and other extensions to the 'core' platform, will be amalgamated and documented in a library on GitHub. In this way, customisations of the Platform will be reusable. Key parts of the software will be open source, meeting transparency and openness requirements and allowing others to reuse or repurpose them, including JSON configuration files. Some components will be closed, supporting our business model, including the web application for customisation. The licensing of other components of the system will be determined during technical elaboration and business plan development.