software engineering Archives - Ger Inberg

FeatureExtraction on CRAN

Ger — Tue, 03 Sep 2024 12:40:01 +0000

Feature engineering is a crucial step in the data science process, often making the difference between a good model and a great one. It involves transforming raw data into meaningful features that can improve the performance of predictive models. For those working in R, the FeatureExtraction package on CRAN offers a powerful and flexible toolset for automating and streamlining this process.

Originally developed as part of the OHDSI (Observational Health Data Sciences and Informatics) ecosystem, FeatureExtraction is particularly well-suited for working with large-scale observational data. In this post, we’ll explore the package in detail, focusing on its core features, practical applications, and a step-by-step example to help you get started.

Key Features and Capabilities

1. Automated Feature Generation

FeatureExtraction excels at automatically generating a wide range of features from raw data. These features include basic demographic variables like age and gender, as well as more complex attributes derived from longitudinal data, such as the frequency of medical visits or the presence of certain conditions over time.

2. Temporal Features

Temporal data, such as patient histories or time-dependent events, are common in many fields, especially healthcare. FeatureExtraction handles temporal data adeptly, allowing users to define time windows relative to key events (e.g., diagnosis dates). This feature is crucial for creating time-sensitive covariates that capture trends and patterns in data over specified periods.

3. Custom Feature Extraction

While the package offers extensive automated capabilities, it also allows for custom feature extraction. Users can define custom covariates and specify how these should be generated from the underlying data, incorporating domain-specific knowledge into the feature engineering process.

4. Scalability

Feature engineering can become computationally intensive, particularly with large datasets. FeatureExtraction is designed for scalability, leveraging parallel processing and optimized algorithms to ensure that feature extraction remains efficient even with big data.

5. Integration with OHDSI Tools

As part of the OHDSI ecosystem, FeatureExtraction integrates seamlessly with other tools like PatientLevelPrediction and CohortMethod, enabling a smooth workflow from data extraction to model building and analysis.

Getting started

Installing the FeatureExtraction package is straightforward. You can install it directly from CRAN using the following command:

install.packages("FeatureExtraction")

Load the package in your R session:

library(FeatureExtraction)

Practical Example: Creating Covariates Based on Other Cohorts

To illustrate how FeatureExtraction can be applied, let’s walk through an example where we create covariates based on the presence of patients in other cohorts. This is particularly useful in studies where the relationship between different conditions or treatments over time is of interest.

Step 1: Setting Up the Database Connection

First, we need to define the connection to our CDM-compliant database:

connectionDetails <- createConnectionDetails(dbms = "postgresql",
server = "your_server",
user = "your_username",
password = "your_password")
cdmDatabaseSchema <- "your_cdm_schema"
cohortDatabaseSchema <- "your_cohort_schema"

Step 2: Define the Cohorts of Interest

Assume we have a cohort of patients with diabetes and another cohort with a history of cardiovascular disease. We want to create a feature that indicates whether a patient in the diabetes cohort has a prior history of cardiovascular disease.

# Define cohort IDs (these would be predefined in your database)
diabetesCohortId <- 1
cvdCohortId <- 2

Step 3: Create the Feature Extraction Settings

Next, we define the feature extraction settings, specifying that we want to create covariates based on the presence of patients in the cardiovascular disease cohort:

covariateSettings <- createCohortBasedCovariateSettings(useDemographicsGender = TRUE,
useDemographicsAge = TRUE,
cohortId = cvdCohortId,
startDay = -365,
endDay = 0)

In this example, the startDay and endDay parameters define a time window of one year prior to the cohort’s index date. This means the feature will reflect whether a patient was in the cardiovascular disease cohort within one year before the index date.

Step 4: Extract the Features

Now, we extract the features for the diabetes cohort using the settings we defined:

covariateData <- getDbCovariateData(connectionDetails = connectionDetails,
cdmDatabaseSchema = cdmDatabaseSchema,
cohortDatabaseSchema = cohortDatabaseSchema,
cohortTable = "cohort",
cohortId = diabetesCohortId,
covariateSettings = covariateSettings)

This function retrieves the covariate data for the specified cohort, based on the feature extraction settings we provided.

Step 5: Use the Extracted Features

The extracted features are now available in the covariateData object, which can be used for further analysis, such as model building or cohort characterization.

# Explore the covariate data
summary(covariateData)

This simple example demonstrates how FeatureExtraction can be used to create meaningful features based on different cohorts. The package’s flexibility and scalability make it a powerful tool for a wide range of applications, from small-scale studies to large observational databases.

The post FeatureExtraction on CRAN appeared first on Ger Inberg.

DrugExposure Diagnostics

Ger — Sat, 01 Apr 2023 09:32:00 +0000

DrugExposureDiagnostics: A Comprehensive R Package for Assessing Drug Exposure in Clinical Research

Drug exposure is an essential aspect of clinical research, as it directly affects the efficacy and safety of drugs. Measuring drug exposure accurately and understanding the factors that influence it is crucial for clinical decision-making. This is where the R package DrugExposureDiagnostics comes in handy.

As the author of this R package, I am excited to introduce you to this powerful tool for analyzing drug exposure data. Before delving into the package, let’s first understand what drug exposure is and why it is crucial in clinical research.

Drug exposure refers to the extent to which a drug enters and stays in the body, thereby producing its intended therapeutic effects. Measuring drug exposure accurately involves capturing key metrics, such as drug concentrations, AUC, Cmax, and Tmax. By doing so, researchers can evaluate drug efficacy and safety and make informed decisions regarding dosing and administration.

One way to capture drug exposure data is through the Observational Medical Outcomes Partnership (OMOP) Common Data Model (CDM), developed by the Observational Health Data Sciences and Informatics (OHDSI) community. The OMOP CDM standardizes and integrates data from various sources, allowing for large-scale observational studies and analysis.

This is where the R package DrugExposureDiagnostics comes in. It is a comprehensive tool for analyzing drug exposure data in the OMOP CDM format. The package includes functions for calculating various exposure metrics, handling missing data, and summarizing data at different levels, such as by subject or visit. Additionally, it provides tools for identifying outliers and comparing exposure between groups.

DrugExposureDiagnostics has been extensively tested and validated, ensuring that it produces accurate results. The package has been released on the Comprehensive R Archive Network (CRAN), making it easily accessible to R users worldwide. To use the package, simply install it using the install.packages() function in R and load it using the library() function.

If you are interested in learning more about DrugExposureDiagnostics or trying it out for yourself, visit the package github

The post DrugExposure Diagnostics appeared first on Ger Inberg.

Web scraping: R vs python

Ger — Tue, 09 Feb 2016 08:08:48 +0000

For my last post, I used a python script to scrape the data from a website. I used Python..just because I am used to do webscraping in Python. But I heard R also got better at scraping, so I rewrote my script in R.

The package rvest is the equivalent of BeautifulSoup in python. It is available since 2014 and created by Hadley Wickham. Underneath it uses the packages ‘httr’ and ‘xml2’ to easily download and manipulate html content.

You can use rvest in the following way:

[code language=”r” wraplines=”true” collapse=”false”]
# install and load package
install.packages(“rvest”)
library(rvest)

url <- “http://live.ultimate.dk/desktop/front/?eventid=2021049&language=nl”
data <- read_html(url)
resultsTable <- data %>% html_nodes(“table.leaderboard_table_results”)
rows <- resultsTable %>% html_nodes(“tr”)
for(i in 1:length(rows)){
tds <- rows[i] %>% html_nodes(“td”)
print(tds[4] %>% html_text)
print(tds[10] %>% html_text)
}
[/code]

A couple of things are good to know:

get the website content with read_html(): this will return an xml document
select content from certain nodes with html_nodes(.)
get attribute content from a node with html_attr()
the pipe operator “%>%” can be used to chain operations. Use it, it’s very convenient.

When you are used to BeautifulSoup, it is easy to learn rvest, because it has a similair syntax.

You can find the R and python scripts that I wrote for webscraping below. I am wondering what language you prefer for webscraping? Please let me know in a comment below.

python script vs R script

The post Web scraping: R vs python appeared first on Ger Inberg.

Internet of Things developersday

Ger — Sun, 19 Apr 2015 20:08:57 +0000

Last week, I have been to the developersday in the Jaarbeurs in Utrecht. The theme of the meeting was Internet of Things. PKN has accounced it’s IOT academy today.
Some interesting speakers and food for thought. The internet of things is not really new, it is more an umbrella term for Ambient Intelligence, Ubiquitous Computing, Sensor Networks, Grid Computing, Service-oriented architectures and Mobile Communications to name a few
It is also mentioned a lot in combination with big data and data analysis because if we connect al those devices to the internet and put sensors in it, we will have a lot of data to process.
Why do we want to connect all our devices to the internet? There must be some added value for us (and not only feeding data to the big comppanies like google, facebook, etc)
A nice example of the added value of connected devices is the control of street lighting in Los Angeles by Philips

During they day it was also possible to do some workshops. I did a workshop to connect a temperature sensor to an arduino. By connecting the arduino to our labtop, we were able to display the temperature and humidity.

The post Internet of Things developersday appeared first on Ger Inberg.