Structured data processing fit for the Life Science use cases
AgileData comprises our data science and data engineering services. Our team of scientists and clinicians turned data scientists and engineers understands the complex nature of scientific experiments, data, and biological models, allowing us to execute on the development of custom, production-ready pipelines suited for scientific data science and analytics. AgileData also includes our Algorithm development and analytics asset packaging, which ensures our clients can fully take ownership of our custom solutions.
We combine the right skills with the agility required to address the rapidly evolving analytical needs of the pharma/biotech industry. Based on our clients’ projected data growth and performance needs, we have implemented analytic pipelines that directly perform extract-transform-load (ETL) operations and update data lakes or data warehouses in real time. Some projects benefit from distributed computing open source frameworks such as Apache Spark, while others leverage container orchestration through Kubernetes and containerization of individual pipeline components.
We support our clients in transforming raw data into insightful information more efficiently. These libraries are implemented using a variety of languages such as Scala (for Spark), Java, Python (PySpark for python-based Spark), and R. The targets of these artifacts and modules are containerized processes that can be implemented as application-level components of Bioinformatics pipelines, drastically increasing efficiency and promoting reusable components.
In sum, we implement multi-omics analysis pipelines which require a great deal of computing resources, are production-ready, and can process millions of samples. This facilitates the extraction of information from vast multidimensional datasets enabling all areas of informatics research.
We leverage the value of existing data sources, both internal and external, by developing ontologies, appropriate data models, data management, and implementing relevant data processing tools. Our bioinformatics services include data curation, informatics ontologies (i.e. Gene Ontology, ChEBI, etc…) We have extensive know-how loading and utilizing large public data sources such as 1000 Genomes, TCGA, CCLE, and others.
Handling heterogenous data for scientific analytics is a notable challenge in the life sciences, pharma, and biotech. Our data engineering teams facilitate the standardization of scientific and healthcare data, which allows scientific and clinical users to perform analytics in a self-guided manner. This eliminates the need for significant latencies between requesting data and having the requests filled by data stewards. The automated methods resulting from this platform approach ensure reproducibility in data-driven analytics and support a more automated, user-driven functionality.
Many complex situations and relationships can be modelled in data tables and columnar formats that are flexible across domains. We have experience implementing different normalized formats of databases. Leveraging formats such as PostgreSQL’s binary-json (jsonb) column helps achieve flexible schemas without overcomplicating a database with table permutations modelling a high-level concept. From Infrastructure to data model flexibility, our team has experience implementing the right technology according to the use case at hand.
Algorithm Development
Arrayo has supported many experimental and academic algorithm optimization projects. Traditional bioinformatics techniques are typically not scalable, efficient, or practical in a commercial setting. By taking a modular, application-based approach, we assist in optimizing and scaling these algorithms. Bioinformatics modules we have written in the past leverage open-source, community-vetted tools . There are multiple, robust open-source biological programming frameworks such as BioPython and BioJava that can enhance the processing of sample data; this by integrating with relational databases to construct arguments and interpret data. This moves away from traditional methods where the contents of a directory are iterated through, and little to no error data / quality metrics are persisted in a usable source.
Our pipeline and integration of individual modules follow the pattern of “dockerization”. This allows our clients to package analytical assets, making them more scalable and more efficient to run.
