How SciDAP works

Overview

SciDAP consists of a central server (https://scidap.com/) that provides user interface and satellites that process and store the data. The satellite is typically installed on your hardware or cloud service. This can be a desktop, laptop, server, cluster or cloud running MacOs X, Linux or Windows operating systems. Alternatively, we can provide users with access to our satellite to try out SciDAP.

This design allows us to provide users with the most up-to-date interface and to quickly deploy new pipelines through the central server. At the same time, to reduce HIPAA and data security concerns, the data are kept on satellites that are controlled by users and do not leave user's network. The users can access SciDAP from any computer by logging into (https://scidap.com/).

SciDAP serves as a data processing and exchange center for the whole laboratory or institution. All authorised laboratory members can add data, execute analyses and view the results. Do not invite strangers to your laboratory.

scidap-satellite explained

Technical details:

SciDAP-satellite is the back-end data analysis and storage component of SciDAP platform. It downloads requested data files e.g. from GEO or other urls with aria2c and executes CWL pipelines with CWL-Airflow. It provides authorized access to data (raw and analyzed) via the interface provided by SciDAP central server and authorized by secure JWT tokens. In order to receive commands from the master server your satellite needs access to the Internet. However, the satellite does not need to be accessible from the Internet but in order to work with a satellite, the user needs to be in the same local network.

Satellite software

To perform the analysis scidap.com uses a private scidap-satellite installation. There are two types of scidap-satellites: single laboratory and institutional. See instructions for a single laboratory installation on the tutorials page. We recommend at least 32GB per processing unit. For institutional installation please contact us

Datirium provides scidap-satellite bundle to simplify setup process on Mac OS X. On other OSes all its components have to be installed and configured manually. Scidap-satellite pckage includes the following pre-configured open-source software packages:

What data is exchanged between SciDAP.com and satellites

By logging into scidap.com, you can start a pipeline from anywhere. However, to access all the results a direct connection from your computer to to the satellite is required. Only limited information is transferred between a scidap-satellite installation and scidap.com:

  • Sample's metadata
  • Sample's CWL pipeline file (CWL files can be checked at https://github.com/datirium/workflows)
  • Sample's status like running, failed, finished
  • Sample's mapping statistics for basic analysis