Novel Transformer based methods for Automatic Classification and Quantification of Stromule Dynamics from Microscopy Images
Project Information
bioinformaticsProject Status: Complete
Project Region: CAREERS
Submitted By: Anita Schwartz
Project Email: chandrak@udel.edu
Project Institution: University of Delaware
Anchor Institution: CR-University of Delaware
Project Address: Newark, Delaware. 19716
Mentors: Jeffrey Caplan
Students: Huining Liang
Project Description
Our work involves developing an image processing pipeline for automatically classifying and quantifying the dynamics of chloroplasts, stromules, and the plant’s cytoskeleton to better understand the function, relationships, and movement characteristics of these intracellular structures. Our current pipeline consists of fast, automatic segmentation of microscopy images, active contour-based tracking, and unsupervised movement classification based on a U-Net, a convolutional neural network (CNN) for segmentation, and Computer vision methods for tracking.For the CAREERS project, we propose to undertake and complete a transformer based pipeline, more specifically, TransUNet to leverage the power of CNNs and transformers for the image segmentation task. The proposed TransUNet involves CNN-Transformer Hybrid Encoder, Patch Embedding and Cascaded Upsampler. For tracking, we propose to use the tracking-by-attention paradigm which not only applies attention for data association but jointly performs tracking and detection, using TrackFormer, an end-to-end trainable MOT (multi-object tracking) approach based on an encoder-decoder Transformer architecture.
Additional Resources
Launch Presentation:Wrap Presentation: 6 months
Project Information
bioinformaticsProject Status: Complete
Project Region: CAREERS
Submitted By: Anita Schwartz
Project Email: chandrak@udel.edu
Project Institution: University of Delaware
Anchor Institution: CR-University of Delaware
Project Address: Newark, Delaware. 19716
Mentors: Jeffrey Caplan
Students: Huining Liang
Project Description
Our work involves developing an image processing pipeline for automatically classifying and quantifying the dynamics of chloroplasts, stromules, and the plant’s cytoskeleton to better understand the function, relationships, and movement characteristics of these intracellular structures. Our current pipeline consists of fast, automatic segmentation of microscopy images, active contour-based tracking, and unsupervised movement classification based on a U-Net, a convolutional neural network (CNN) for segmentation, and Computer vision methods for tracking.For the CAREERS project, we propose to undertake and complete a transformer based pipeline, more specifically, TransUNet to leverage the power of CNNs and transformers for the image segmentation task. The proposed TransUNet involves CNN-Transformer Hybrid Encoder, Patch Embedding and Cascaded Upsampler. For tracking, we propose to use the tracking-by-attention paradigm which not only applies attention for data association but jointly performs tracking and detection, using TrackFormer, an end-to-end trainable MOT (multi-object tracking) approach based on an encoder-decoder Transformer architecture.
Additional Resources
Launch Presentation:Wrap Presentation: 6 months