Project: https://github.com/crim-ca/weaver License: https://github.com/crim-ca/weaver/blob/master/LICENSE.txt Documentation: https://pavics-weaver.readthedocs.io/ ## Description ### Implementations - OGC API - Processes (WPS-REST bindings) - Web Processing Services (WPS) - Earth System Grid Federation (ESGF) processes - Common Workflow Language (CWL) for OGC Application Package - Execution Management Service (EMS) for Workflows - Application, Deployment and Execution Service (ADES) ### Summary Weaver is an OGC-API flavored Execution Management Service (EMS) that allows the execution of workflows chaining various applications and Web Processing Services inputs and outputs. Remote execution is deferred by the EMS to one or many Application, Deployment and Execution Service (ADES) or remote service providers, and employs Common Workflow Language (CWL) configurations to define an OGC Application Package deployed for each process. ### Details Weaver is primarily an Execution Management Service (EMS) that allows the execution of workflows chaining various applications and Web Processing Services inputs and outputs. Remote execution of each process in a workflow chain is dispatched by the EMS to one or many registered Application, Deployment and Execution Service (ADES) by ensuring the transfer of files accordingly between instances when located across multiple remote locations. Weaver can also accomplish the ADES role in order to perform application deployment at the data source using the application definition provided by Common Workflow Language (CWL) configuration. It can then directly execute a registered process OGC Application Package with received inputs from a WPS request to expose output results for a following ADES in a EMS workflow execution chain. Weaver extends OGC API - Processes by providing additional functionalities such as more detailed job logs endpoints, adding more process management and search request options than required by the standard, and supporting remote providers registration for dynamic process definitions, to name a few. Because of this, not all features offered in Weaver are guaranteed to be applicable on other similarly behaving ADES and/or EMS instances. The reference specification is tracked to preserve the minimal conformance requirements and provide feedback to Open Geospatial Consortium (OGC) in this effect. Weaver can be launched either as an EMS, an ADES or an HYBRID of both according to its configuration. ## Applications The video shares the fundamental ideas behind the architecture, illustrates how application stores for Earth Observation data processing can evolve, and illustrates the advantages with applications based on machine learning. https://www.youtube.com/watch?v=no3REyoxE38 ## Platforms The video shares the fundamental ideas behind the architecture, illustrates how platform managers can benefit from application stores, and shows the potential for multidisciplinary workflows in thematic platforms. https://www.youtube.com/watch?v=QkdDFGEfIAY ## Extra Details & Sponsors The project was initially developed upon *OGC Testbed-14 – ESA Sponsored Threads – Exploitation Platform* findings and following improvements. It is also advanced with sponsorship from the *U.S. Department of Energy* to support common API of the [Earth System Grid Federation](https://esgf.llnl.gov/) (ESGF). The findings are reported on the [OGC Testbed-14](https://www.ogc.org/initiatives/testbed-14/) thread, and more explicitly in the [ADES & EMS Results and Best Practices Engineering Report](http://docs.opengeospatial.org/per/18-050r1.html). The project has been employed for [OGC Testbed-15 - ML Thread](https://www.ogc.org/initiatives/testbed-15/#MachineLearning) to demonstrate the use of Machine Learning interactions with OGC web standards in the context of natural resources applications. The advancements are reported through the [OGC Testbed-15: Machine Learning Engineering Report](http://docs.opengeospatial.org/per/19-027r2.html). Developments are continued in [OGC Testbed-16](https://www.ogc.org/initiatives/t-16/) to improve methodologies in order to provide better interoperable geospatial data processing in the areas of Earth Observation Application Packages. Findings and recommendations are presented in the [OGC Testbed-16: Data Access and Processing Engineering Report](http://docs.opengeospatial.org/per/20-016.html). Videos and more functionalities were introduced in *Weaver* following [OGC Earth Observation Applications Pilot](https://www.ogc.org/initiatives/eoa-pilot/). Corresponding developments are reported in the [OGC Earth Observation Applications Pilot: CRIM Engineering Report](http://docs.opengeospatial.org/per/20-045.html). *Weaver* has been used to participate in interoperability testing effort that lead to [OGC Best Practice for Earth Observation Application Package](https://docs.ogc.org/bp/20-089r1.html) technical report. This resulted, along with previous efforts, in the definition of [OGC API - Processes - Part 2: Deploy, Replace, Undeploy](https://github.com/opengeospatial/ogcapi-processes/tree/master/extensions/deploy_replace_undeploy) (DRU) extension backed by validated test cases using [Common Workflow Language](https://www.commonwl.org/) (CWL) as the representation method for the deployment and execution of [OGC Application Package](https://github.com/opengeospatial/ogcapi-processes/blob/master/openapi/schemas/processes-dru/ogcapppkg.yaml) close to the data. The project is furthermore developed through the *Data Analytics for Canadian Climate Services* ([DACCS](https://app.dimensions.ai/details/grant/grant.8105745)) initiative. *Weaver* is implemented in Python with the [Pyramid](http://www.pylonsproject.org) web framework. It is part of [PAVICS](https://ouranosinc.github.io/pavics-sdi/index.html) and [Birdhouse](http://bird-house.github.io/) ecosystems and is available within the [birdhouse-deploy](https://github.com/bird-house/birdhouse-deploy) server stack.