Climate Data Access: Re-thinking our Data Analysis Workflows
Christian Pagé (1), Maarten Plieger (2), Wim Som de Cerff (2), Alessandro
Spinuso (2), Ernst de Vreede (2), Niels Drost (3), Iraklis Klampanos (4),
Vangelis Karkaletsis (4), Malcolm Atkinson (5)
(1) CERFACS, (2) Royal Netherlands Meteorological Institute (KNMI), R&D
Observations and Data Technology, De Bilt, Netherlands (plieger@knmi.nl),
(3) Netherlands eScience Center, (4) National Centre for Scientific
Research Demokritos, (5) The University of Edinburgh
Users of Climate Data are facing challenge to analyze all the data they
need. Data volumes are sharply increasing, and the ability to download all
needed data is most of the time not a possibility anymore. A platform
scientific gateway called climate4impact (C4I) has been designed and
developed to enhance the use of research data. This platform supports
researchers in getting access to data and in doing data processing. It is
currently under development within the European Project IS-ENES3 and builds
on previous developments from previous IS-ENES projects, CLIPC and
C3S-Magic. C4I offers a front-end with guidance and documentation along
with standard services (APIs) on top of the climate data infrastructure,
but also using available European e-infrastructure services. It can be
visited at https://climate4impact.eu/.
The current version provides processing services that also include climate
indicator calculations, country based statistics and polygon extraction.
C4I makes use of the DKRZ Birdhouse framework, which is an extendable and
modular processing framework based on PyWPS. Data is obtained from various
climate infrastructure data nodes using secure communications. C4I provides
a personal basket where users can upload their own data and do research
with the provided tools. The software is open, reusable, modular and
packaged. Components are available via docker containers to enable easy
re-use.
In the current C4I architecture, on-demand calculations are taking place on
the front-end server. This is not scalable and can lead to performance
problems as well as front-end disruptions. Within the DARE project,
delegation of the calculations on the DARE Platform using the DARE API has
been implemented and tested in a prototype, using EUDAT CDI B2DROP as an
intermediate storage service. Via the DARE API, the platform allows users
to register and retrieve processing elements and workflows, and to execute
them exploiting the elasticity of the cloud. It is worth mentioning that
the DARE Platform as well as the EUDAT B2 Services are interoperable with
EOSC. This prototype service delegation will be made operational during the
upcoming year.
In the IS-ENES3 project, the web portal will be redesigned with a
completely new architecture using a micro-services and containerized
approach, building on experience gained during the previous projects. The
next version of the portal will be built using the React framework, which
allows for creating large web applications which can change data, without
reloading the page. We are actively seeking input from current as well as
potential users at this time, to make the next version of C4I useful to as
many people as possible.
The IS-ENES3 project has received funding from the European Union’s Horizon
2020 research and innovation programme under grant agreement N°824084, and
the DARE project under N°777413.
