Computer Vision Bootcamp

Computer Vision will be abbreviated as CV in the course content.

Understand the scope of CV, how images are digitally represented, and perform basic image processing and feature extraction using classical techniques.

What is Computer Vision (CV)?

• Definition, Goals (Understanding, Reconstruction, Manipulation)
• Historical Context (Brief: From Marr's framework to Deep Learning)
• Key Application Areas (Autonomous Driving, Medical Imaging, AR/VR, Security, Robotics)

Digital Image Representation:

• Pixels as Building Blocks
• Image Resolution & Aspect Ratio
• Color Spaces: Grayscale, RGB, HSV, BGR (OpenCV default)
• Image File Formats (JPG, PNG, BMP, TIFF) - Compression concepts

Basic Image Manipulation:

• Pixel Access and Modification
• Image Cropping, Resizing (Interpolation methods: Nearest, Bilinear, Bicubic)
• Geometric Transformations: Translation, Rotation, Scaling, Affine, Perspective

Image Filtering & Enhancement:

• Concept of Kernels/Filters
• Linear Filtering: Blurring (Average, Gaussian), Sharpening (Laplacian)
• Non-Linear Filtering: Median Filter (Noise Reduction)
• Histogram Equalization (Contrast Enhancement)

Feature Extraction (Classical):

• Concept of Image Features (Edges, Corners, Blobs)
• Edge Detection: Sobel, Canny algorithms (Gradient magnitude/direction, Non-maximum suppression, Hysteresis thresholding)
• Corner Detection (Harris), Feature Descriptors (SIFT, SURF - concept only)
• Loading, Displaying, Saving Images (OpenCV imread, imshow, imwrite, Matplotlib pyplot)
• Converting Color Spaces (OpenCV cvtColor)
• Accessing Pixel Values, Basic Arithmetic on Images (NumPy)
• Applying Transformations (OpenCV resize, warpAffine, warpPerspective)
• Applying Filters (OpenCV filter2D, GaussianBlur, medianBlur)
• Applying the Canny Edge Detector (OpenCV Canny)

Image Processing Toolkit (Lab 1)

• Objective: Implement a set of functions to load, process, and extract basic features from images.
• Tasks:
  1. Load an image and convert it between RGB, Grayscale, and HSV. Display the results.
  2. Implement functions for rotation and resizing with different interpolation methods. Visually compare the outputs.
  3. Apply Gaussian blur with varying kernel sizes and observe the smoothing effect.
  4. Apply Canny edge detection and experiment with threshold parameters.
  5. Combine multiple operations (e.g., resizing, blurring, followed by edge detection).
• Deliverables: Python script/notebook containing the functions and visualized outputs.
• Tools: Python, OpenCV, NumPy, Matplotlib
• Dataset: Standard test images (e.g., Lena, Baboon) or custom images.

Grasp the core concepts of neural networks, how they learn via gradient descent, and implement a basic training pipeline using PyTorch.

From Traditional ML to Deep Learning:

• Limitations of Hand-crafted Features
• Representation Learning

Artificial Neural Networks (ANNs):

• The Neuron Model (Inputs, Weights, Bias, Activation)
• Activation Functions: Sigmoid, Tanh, ReLU (Properties, Pros/Cons), GELU (Used in Transformers)
• Feed-Forward Networks (Layers: Input, Hidden, Output)

Training Neural Networks:

• Loss Functions: Mean Squared Error (MSE) for Regression, Cross-Entropy Loss for Classification
• Optimization Problem: Finding Optimal Weights
• Gradient Descent: Concept, Learning Rate
• Backpropagation: How Gradients Are Efficiently Computed
• Stochastic Gradient Descent (SGD) and Mini-batches
• Optimizers: Momentum, RMSprop, Adam, AdamW (Adaptive Learning Rates)

Overfitting and Regularization:

• Bias-Variance Tradeoff
• Detecting Overfitting (Train vs. Validation Loss)
• Regularization Techniques: L1/L2 Weight Decay, Dropout
• Early Stopping

Introduction to PyTorch:

• Tensors: The Fundamental Data Structure
• Automatic Differentiation (Autograd)
• Defining Network Layers and Models (torch.nn.Module)
• Common Layers (torch.nn.Linear)
• Creating and Manipulating PyTorch Tensors
• Using Autograd to Compute Gradients (tensor.backward())
• Defining a Simple FFN Model Structure Using nn.Module and nn.Linear
• Defining Loss Functions (torch.nn.CrossEntropyLoss, torch.nn.MSELoss) and Optimizers (torch.optim module: Adam, SGD, AdamW)
• Implementing a Basic Training Loop (Forward pass, Loss calculation, Zero gradients, Backward pass (loss.backward()), Optimizer step (optimizer.step()))
• Monitoring Training with Print Statements or Basic Plots

Your First Neural Network (Non-Vision) (Lab 2)

• Objective: Build and train a simple FFN for classification on a tabular or standard ML dataset using PyTorch.
• Tasks:
  1. Load a dataset (e.g., Iris, Wine, or flattened MNIST). Preprocess if necessary.
  2. Define an FFN architecture using PyTorch's nn.Module.
  3. Select an appropriate loss function (torch.nn) and optimizer (torch.optim).
  4. Implement the training loop including forward/backward passes and weight updates.
  5. Split the data into training and validation sets.
  6. Train the model for several epochs, monitoring training and validation loss/accuracy.
  7. Evaluate the final model on a held-out test set.
• Deliverables: Python script/notebook containing the trained model, training plots, and evaluation results.
• Tools: Python, PyTorch, NumPy, Scikit-learn (for data splitting/metrics)
• Dataset: MNIST, Iris, Wine, or a similar non-vision dataset.

Understand the motivation and mechanisms of CNNs, build standard CNN architectures, and apply them to image classification including data augmentation using PyTorch.

Why CNNs for Image Processing?

• Parameter Sharing & Sparsity of Connections
• Translation Equivariance (and Invariance via Pooling)
• Handling High-Dimensional Image Data

Core CNN Layers:

• Convolution Layer: Kernels/Filters, Feature Maps, Stride, Padding ('valid' vs 'same')
• Pooling Layer: Max Pooling, Average Pooling (Downsampling, Invariance)
• Activation Functions in CNNs (ReLU is common)

Building CNN Architectures:

• Typical Structure: CONV -> ACT -> POOL stacks
• Concept of Receptive Field Size
• Evolution of Architectures (Brief): LeNet-5, AlexNet (Deeper, ReLU, Dropout)
• VGGNet (Smaller filters, Deeper stacks)

Modern CNN Innovations:

• Residual Networks (ResNet): Skip Connections, Identity Mapping, Solving Vanishing Gradients
• Bottleneck Blocks in ResNet
• Batch Normalization: Stabilizing Training, Regularization Effect
• (Brief Mention) Inception Networks (GoogLeNet), DenseNet

Data Augmentation for Images:

• Importance for Generalization and Data Scarcity
• Common Techniques: Flipping (Horizontal/Vertical), Rotation, Scaling, Cropping (Random/Center), Translation (Shifting), Color Jitter (Brightness, Contrast, Saturation, Hue), Gaussian Noise
• Advanced Techniques: Mixup, Cutmix, Random Erasing
• Implementing Convolution (nn.Conv2d) and Pooling (nn.MaxPool2d, nn.AvgPool2d) Layers in PyTorch
• Building a Sequential CNN Model (nn.Sequential)
• Building a ResNet-style Model with Skip Connections (nn.Module subclassing)
• Using torchvision.transforms for Data Augmentation
• Setting up a torch.utils.data.Dataset and torch.utils.data.DataLoader
• Training a CNN for Image Classification using the standard PyTorch training loop
• Using Experiment Tracking (W&B/TensorBoard) to log metrics and images

CNN for Image Classification (Lab 3)

• Objective: Build, train, and evaluate a CNN to classify images from a standard benchmark dataset using PyTorch.
• Tasks:
  1. Load and preprocess an image dataset (e.g., CIFAR-10, CIFAR-100, Fashion-MNIST) using torchvision.datasets.
  2. Implement a robust data augmentation pipeline using torchvision.transforms.
  3. Build a CNN architecture (e.g., a simplified ResNet or VGG) using PyTorch nn.Module.
  4. Set up the training loop with loss, optimizer, and DataLoaders.
  5. Integrate W&B or TensorBoard to log metrics, parameters, and sample predictions.
  6. Train the model while monitoring validation performance to prevent overfitting (use early stopping if needed).
  7. Evaluate the final model on the test set and analyze the confusion matrix.
• Deliverables: Python script/notebook, trained model weights, link to W&B/TensorBoard dashboard, evaluation report.
• Tools: Python, PyTorch, NumPy, Matplotlib, Scikit-learn, W&B/TensorBoard, torchvision.
• Dataset: CIFAR-10/100, Fashion-MNIST, or Imagenette/Imagewoof (using torchvision.datasets or Hugging Face datasets).

Understand the paradigm shift toward Transformers in computer vision, the architectures of ViTs and Swin Transformers, and how modernized CNNs like ConvNeXt compete. Learn to use these models as backbones using PyTorch.

Re-evaluating the Limitations of CNNs:

• Fixed, Local Receptive Fields
• Difficulty in Modeling Long-Range Dependencies

The Transformer Architecture (Review & Vision Focus):

• Self-Attention Mechanism (Query, Key, Value, Scaled Dot-Product Attention)
• Multi-Head Self-Attention (MHSA)
• Transformer Encoder Block (MHSA + Feed-Forward Network + Add & Norm)

Vision Transformer (ViT):

• Core Concept: Treating Images as Sequences of Patches
• Patch Embedding (Linear Projection)
• Positional Encodings (Learnable vs. Fixed)
• [CLS] Token for Classification
• Inductive Bias vs. Data Hunger (Need for Large-Scale Pre-training)
• SOTA Performance Scaling (ViT-G, ViT-H)
• Key Variants: DeiT (Data-efficient training, Distillation Token)

Hierarchical Vision Transformers:

• Motivation: Need for Multi-Scale Features (Detection, Segmentation)
• Swin Transformer: Hierarchical Structure (Patch Merging), Shifted Window MHSA (Efficiency, Cross-Window Connection), Relative Positional Biases
• Other Hierarchical Variants (PVT - Pyramid Vision Transformer - brief)

Modernized Convolutional Networks (ConvNeXt):

• Motivation: Can CNNs match ViTs with modern designs/training?
• ConvNeXt V1: Modernizing ResNet (Training techniques, Macro/Micro design changes inspired by Swin - larger kernels, inverted bottlenecks, LN, GELU, fewer activations/norms)
• ConvNeXt V2: Co-design with Self-Supervision (FCMAE, Feature Collapse Problem, GRN Layer)
• Performance & Characteristics (Competitive with ViTs, potential efficiency gains)

Hybrid CNN-Transformer Models:

• Rationale: Combining Local Strength (CNN) with Global Context (Transformer)
• Integration Taxonomy (Brief overview: Early, Lateral, Sequential, Parallel, Block Integration)
• Examples (MobileViT for efficiency, CMT, MaxViT)

Using Pre-trained Backbones:

• Concept of Foundational Models / Backbones
• Model Hubs (timm (PyTorch Image Models), Hugging Face transformers)
• Loading pre-trained ViT, Swin, ConvNeXt models (timm, Hugging Face transformers for PyTorch)
• Running Inference with these models in PyTorch
• Inspecting model architectures (layers, parameters) in PyTorch
• Extracting intermediate features from different layers/stages using PyTorch hooks
• Visualizing Attention Maps in ViT (using hooks or PyTorch-compatible libraries)

Feature Extraction with SOTA Backbones (Lab 4)

• Objective: Extract meaningful features from images using pre-trained SOTA models (PyTorch versions) and compare their representational power.
• Tasks:
  1. Select a diverse set of images (e.g., different objects, scenes, styles).
  2. Load pre-trained weights for ViT, Swin Transformer, and ConvNeXt (PyTorch models from timm or HF).
  3. Pass the images through each model (in evaluation mode - model.eval()) and extract the final feature vector (e.g., before the classification head) or pooled output.
  4. Compute pairwise cosine similarity between feature vectors for all images using features from each backbone.
  5. Visualize the similarity matrices (heatmaps).
  6. Analysis: Do similar images have higher similarity scores? Are there differences across backbones?
  7. (Optional) Extract and compare features from intermediate layers using PyTorch hooks.
• Deliverables: Python script/notebook, similarity matrix visualizations, analysis of results.
• Tools: Python, PyTorch, timm, Hugging Face transformers, NumPy, Matplotlib/Seaborn.
• Dataset: Custom diverse image set or a small labeled dataset.

Master fine-tuning techniques to adapt large pre-trained PyTorch models to specific classification tasks, understand advanced training strategies, and rigorously evaluate performance.

Transfer Learning Strategies:

• Motivation: Leveraging knowledge from large datasets (ImageNet, JFT)
• Feature Extraction: Using the backbone as a fixed feature extractor, training a new classifier head.
• Fine-tuning: Unfreezing some/all backbone layers and training end-to-end with a smaller learning rate.
• Layer-wise Learning Rate Decay

Fine-tuning Best Practices:

• Choosing the Right Backbone
• Dataset Size Considerations
• Learning Rate Selection (Lower LR for backbone, higher for head)
• Learning Rate Schedulers (Cosine Annealing, Warmup)
• Optimizer Selection (AdamW is generally preferred for Transformers/modern training)
• Regularization During Fine-tuning

Review & Application of Advanced Training Techniques:

• Revisiting Mixup and Cutmix in the context of fine-tuning
• Label Smoothing
• Stochastic Depth (used in ConvNeXt/Swin)

Deep Dive into Evaluation Metrics:

• Limitations of Accuracy (Imbalanced datasets)
• Confusion Matrix Analysis
• Precision, Recall, F1-Score (Per-class and macro/micro/weighted averages)
• ROC Curve and AUC (Area Under Curve)

Introduction to Self-Supervised Learning (SSL) for Pre-training:

• Concept: Learning from unlabeled data via pretext tasks
• Contrastive Learning (SimCLR, MoCo - brief idea)
• Masked Image Modeling (MAE, BEiT - brief idea)
• Impact: How SSL pre-training (e.g., MAE pre-trained ViT, FCMAE ConvNeXt V2) provides strong starting points for fine-tuning.
• Loading a pre-trained SOTA backbone (timm/HF for PyTorch) and replacing the classification head.
• Implementing different fine-tuning strategies (freezing/unfreezing layers by setting param.requires_grad = False/True).
• Setting different learning rates for backbone and head (using optimizer parameter groups).
• Using learning rate schedulers (torch.optim.lr_scheduler).
• Applying advanced augmentations and label smoothing.
• Using scikit-learn or torchmetrics for comprehensive evaluation.
• Logging detailed metrics and confusion matrices to W&B/TensorBoard.

Fine-tuning SOTA Models for Custom Classification (Lab 5)

• Objective: Fine-tune a SOTA pre-trained PyTorch model (ViT, Swin, ConvNeXt) on a specific downstream classification task and achieve high performance.
• Tasks:
  1. Select a relevant image classification dataset (e.g., Oxford Pets, Food-101, EuroSAT, medical dataset subset, domain-specific dataset).
  2. Prepare the dataset (train/val/test splits, Dataset/DataLoader).
  3. Choose a pre-trained PyTorch backbone and implement a transfer learning strategy (start by fine-tuning the head, then unfreeze more layers).
  4. Apply appropriate data augmentation, learning rate scheduling, and potentially label smoothing.
  5. Train the model using W&B/TensorBoard for tracking. Experiment with hyperparameters (LR, batch size, epochs, optimizer).
  6. Rigorously evaluate on the test set with accuracy, precision, recall, F1-score, and confusion matrix using torchmetrics or scikit-learn.
  7. If time permits, compare results from different fine-tuning strategies or backbones.
• Deliverables: Python script/notebook, fine-tuned model weights, W&B/TensorBoard link, detailed evaluation report comparing strategies/results.
• Tools: Python, PyTorch, timm/HF transformers, NumPy, Scikit-learn/torchmetrics, W&B/TensorBoard, torchvision/HF datasets.
• Dataset: Specific downstream classification dataset (more complex than CIFAR).

Understand the object detection task, learn about dominant architectures (one-stage, two-stage, Transformer-based), and gain hands-on experience training and evaluating SOTA detectors like YOLOv8 and DETR variants using PyTorch.

Task Definition:

• Classification + Localization (Bounding Boxes)
• Input/Output Format (Image -> List of [Box Coordinates, Class Label, Confidence Score])

Core Concepts & Metrics:

• Bounding Box Representations (xyxy, xywh)
• Intersection over Union (IoU)
• Evaluation Metrics: Average Precision (AP), mean Average Precision (mAP) - explaining calculation nuances (IoU thresholds, PR curve).

Evolution of Object Detectors:

• Two-Stage Detectors: R-CNN family (R-CNN, Fast R-CNN, Faster R-CNN - Region Proposal Network - RPN)
• One-Stage Detectors: YOLO (You Only Look Once - Grid system, direct prediction), SSD (Single Shot MultiBox Detector - Multi-scale features), RetinaNet (Focal Loss for class imbalance)
• Anchor Boxes: Concept, role, limitations.

Transformer-based Detectors (DETR & Variants):

• DETR: Detection as direct set prediction, End-to-End (No NMS/Anchors), CNN Backbone + Transformer Encoder-Decoder, Object Queries, Bipartite Matching Loss (Hungarian Algorithm)
• Challenges: Slow convergence, complexity.
• Improvements: Deformable DETR (Efficient attention), DINO (Improved denoising anchor boxes, better performance), Co-DETR (Collaborative training, auxiliary heads for richer supervision), RT-DETR (Real-time performance focus)

Overview of SOTA Models:

• YOLO Family (YOLOv5, YOLOv7, YOLOv8 - practical SOTA for speed/accuracy tradeoff, PyTorch-based)
• DETR Variants (DINO, Co-DETR - often topping leaderboards for accuracy, typically PyTorch)
• EfficientDet (Scalable CNN-based)

Open-Vocabulary Detection (OVD):

• Concept: Detecting objects from classes unseen during training using text prompts.
• Methods (Brief): Grounding DINO, OWL-ViT (leveraging VLMs like CLIP).
• Using the ultralytics library for YOLOv8 (PyTorch backend): Installation, inference on images/videos, training/fine-tuning via command line and Python API.
• DETR/RT-DETR using Hugging Face transformers for PyTorch: Loading models, preprocessing images, running inference, decoding predictions (boxes, labels).
• Visualizing bounding boxes and confidence scores (OpenCV).
• Understanding dataset formats (COCO JSON, YOLO TXT).
• Evaluating model performance using standard libraries/tools (e.g., pycocotools).

Training and Evaluating an Object Detector (Lab 6)

• Objective: Fine-tune a SOTA PyTorch object detector (YOLOv8 or RT-DETR) on a custom dataset and evaluate its performance.
• Tasks:
  1. Select an object detection dataset (e.g., a subset of COCO, Pascal VOC, or a custom dataset like thermal imaging or retail products). Ensure it is in (or convert it to) the proper format.
  2. Choose a detector model (YOLOv8 recommended for ease of use, RT-DETR via HF to explore Transformer-based detectors).
  3. Set up the training environment using the selected library (ultralytics or HF transformers + accelerate, all PyTorch-based).
  4. Fine-tune the pre-trained model on the custom dataset. Monitor training with W&B/TensorBoard.
  5. Experiment with training parameters (epochs, batch size, learning rate, augmentation levels).
  6. Evaluate the trained model on the validation/test set using the standard mAP metric.
  7. Run inference on sample images/videos and visualize the detected bounding boxes.
• Deliverables: Training scripts, fine-tuned model weights, W&B/TensorBoard link, evaluation results (mAP scores), sample inference visualizations.
• Tools: Python, ultralytics or Hugging Face (transformers, datasets), OpenCV, W&B/TensorBoard, PyTorch.
• Dataset: COCO subset, Pascal VOC, Global Wheat Detection, custom dataset.

Understand different segmentation tasks, learn about key architectures (FCN, U-Net, Mask R-CNN, Transformer-based), and use SOTA models including the Segment Anything Model (SAM) using PyTorch.

Task Definitions & Applications:

• Semantic Segmentation: Pixel-level classification (Background vs. Foreground, Multi-class such as road, sky, car).
• Instance Segmentation: Pixel-level classification + distinguishing instances (Person 1 mask vs. Person 2 mask).
• Panoptic Segmentation: Unified task - semantic for "stuff" (sky, road) + instance for "things" (car 1, person 2).

Evaluation Metrics:

• Semantic: Pixel Accuracy, Mean Intersection over Union (mIoU).
• Instance: Mask Average Precision (Mask AP - similar to detection AP but over masks).
• Panoptic: Panoptic Quality (PQ = SQ * RQ).

Architectural Evolution:

• Fully Convolutional Networks (FCNs): Replacing FC layers with Convs for pixel output. Upsampling strategies (Transpose Conv - nn.ConvTranspose2d, Bilinear Upsampling - nn.functional.interpolate).
• U-Net: Encoder-Decoder structure, Skip Connections (critical for preserving detail), dominant in medical imaging.
• DeepLab Family: Atrous (Dilated) Convolutions (larger receptive field without downsampling), Atrous Spatial Pyramid Pooling (ASPP - multi-scale context).
• Instance Segmentation: Mask R-CNN (extends Faster R-CNN with a mask prediction head).

Transformer-based Segmentation:

• Motivation: Leveraging global context.
• Mask Classification Paradigm: MaskFormer (predicting a set of masks + classes), Mask2Former (improved with masked attention).
• Unified Frameworks: Mask DINO (extends DINO detector with a mask head, SOTA unified performance). ONE-PEACE, BEiT-3. (Typically PyTorch implementations)

Foundation Models for Segmentation:

• Segment Anything Model (SAM): Architectural overview (ViT Encoder, Prompt Encoder, Mask Decoder), training on the SA-1B dataset, Promptable Segmentation (Points, Boxes, Text), Zero-Shot Generalization. (PyTorch implementation)
• Using SAM: Strengths and weaknesses (e.g., class-agnostic, granularity). Fine-tuning SAM (MedSAM).

Open-Vocabulary Segmentation:

• Concept: Segmentation based on arbitrary text descriptions.
• Methods (Brief): ODISE (using Diffusion + CLIP), combining SAM with VLMs.
• Using pre-trained semantic segmentation models (e.g., DeepLabv3, U-Net from torchvision.models.segmentation).
• Using pre-trained instance segmentation models (e.g., Mask R-CNN from torchvision.models.detection).
• Using Mask2Former/Mask DINO via Hugging Face transformers for PyTorch or Detectron2 (PyTorch-based).
• Interacting with the Segment Anything Model (SAM) using official PyTorch code or HF implementation: Loading models, creating prompts (points, boxes), generating masks.
• Visualizing segmentation masks (overlaying on images).
• Calculating mIoU using libraries (e.g., torchmetrics).

Semantic & Promptable Segmentation (Lab 7)

• Objective: Apply pre-trained PyTorch models for semantic segmentation and experiment with the capabilities of the Segment Anything Model (SAM).
• Tasks:
  1. Load a pre-trained semantic segmentation model (e.g., DeepLabv3+ from torchvision). Run inference on sample images (e.g., Cityscapes demo, ADE20K demo, or custom) and visualize the predicted class masks.
  2. (Optional) Evaluate the semantic segmentation model on a small validation set using the mIoU metric (torchmetrics).
  3. Set up the Segment Anything Model (SAM) using its PyTorch implementation.
  4. Use SAM with various prompts on different images:
     • Single-point prompts.
     • Multi-point prompts (foreground/background).
     • Bounding box prompts.
     • (If using a text-prompt variant) Text prompts.
     • Automatic mask generation (no prompts).
  5. Visualize the masks generated by SAM for different prompts and analyze its effectiveness and limitations (e.g., object boundaries, multiple objects, ambiguity).
  6. (Bonus) Try fine-tuning a semantic segmentation model on a small custom dataset using PyTorch.
• Deliverables: Python script/notebook with inference results, visualizations of semantic masks and SAM-generated masks for various prompts, analysis of SAM's behavior.
• Tools: Python, PyTorch, torchvision/HF transformers/Detectron2, segment-anything library, OpenCV, Matplotlib, torchmetrics.
• Dataset: Demo images from standard datasets (Cityscapes, ADE20K, COCO) or custom images. Optionally, a small labeled semantic segmentation dataset for fine-tuning.

Mini Project 1: Integrated Image Processing System (Week 8-9)

• Objective: Combine detection and segmentation/classification skills using PyTorch models.
• Sample Task: Build a system that detects objects in an image (e.g., cars, pedestrians using YOLOv8) and then classifies an attribute (e.g., car color using a fine-tuned classifier) or precisely segments the detected object using SAM or a fine-tuned segmentation model.
• Focus: Pipeline building, model integration, processing the outputs of one stage as inputs to the next stage within a PyTorch ecosystem.

Understand the challenges of video data and learn SOTA techniques for action recognition, object tracking, and temporal reasoning, primarily using PyTorch frameworks.

Challenges of Video Data:

• Temporal Dimension, Motion Information
• Computational Cost (Processing Frames)
• Data Volume

Action Recognition / Classification:

• Task Definition: Classifying actions in trimmed video clips.
• Evolution: 2D CNNs (frame pooling), 3D CNNs (C3D, I3D - spatio-temporal kernels), SlowFast Networks (two pathways for spatial/temporal), Video Transformers (ViViT, TimeSformer - factorized attention), VideoMAE (SSL pre-training). (Most SOTA models are in PyTorch)
• Temporal Action Localization (TAL): Finding start/end times (ActionFormer).
• Benchmarks (Kinetics, Something-Something, EPIC-Kitchens, Ego4D).

Video Object Tracking:

• Task Definition: Tracking objects over time (Single Object Tracking - SOT, Multi-Object Tracking - MOT).
• Classical Approaches (Brief): Correlation Filters.
• Deep Learning Approaches: Siamese Networks (SiamRPN, SiamMask), Transformer-based trackers (TransT). (Typically PyTorch)
• SOTA Trends: Incorporating Video-Level Context (MCITrack - Mamba), Multi-Modal Tracking (RGB+Depth/Thermal/Events - Un-Track, SUTrack - unified frameworks).
• Benchmarks (LaSOT, GOT-10k, TrackingNet, MOTChallenge).

Temporal Reasoning in Videos:

• Higher-Level Understanding (Order, Duration, Causality).
• Tasks: Video Question Answering (VideoQA), Video Grounding (Text-to-Moment), Summarization.
• Dominant Approach: Video Large Language Models (Vid-LLMs) - integrating Video Encoders with LLMs. (Most open models use PyTorch)
• Techniques: Dynamic Prompting (TSADP), Modeling Temporal Consistency (T3T), Integrating Structure (Video-of-Thought - VoT), Preference Learning (TEMPLE).
• Benchmarks (TemporalVQA, NExTQA, REXTINE).
• Using pre-trained Action Recognition models (e.g., VideoMAE, SlowFast via PyTorchVideo, HF transformers for PyTorch, MMAction2 (PyTorch)). Running inference on video clips.
• Using pre-trained SOT trackers (e.g., from the pysot library, Stark, TransT implementations - check PyTorch versions). Running tracking on a sequence.
• (Conceptual) Overview of how a Vid-LLM can be used for VideoQA (using libraries like Video-LLaVA (PyTorch) or demonstrating API calls).
• Libraries for loading and processing video (decord, OpenCV VideoCapture, PyTorchVideo).

Action Recognition & Tracking Fundamentals (Lab 8)

• Objective: Use pre-trained PyTorch models to perform action classification on video clips and track objects in video sequences.
• Tasks:
  1. Load a pre-trained action recognition model (e.g., VideoMAE or TimeSformer from HF/PyTorchVideo - PyTorch-based).
  2. Download sample video clips representing different actions (e.g., from Kinetics dataset samples).
  3. Preprocess the video clips as required by the PyTorch model (sampling frames, normalization).
  4. Run inference and classify the actions in the clips.
  5. Load a pre-trained Single Object Tracker (find a PyTorch implementation from a standard library or GitHub repo).
  6. Select a video sequence with a clear object to track. Define the initial bounding box for the target object in the first frame.
  7. Run the tracker over the sequence and visualize the tracked bounding box overlaid on the video frames.
  8. Qualitatively evaluate the tracking: Does it track the object accurately? When does it fail?
• Deliverables: Python script/notebook for action classification and tracking, visualization of tracking results.
• Tools: Python, PyTorch, Hugging Face/PyTorchVideo/MMAction2, OpenCV, decord, Tracking library (PyTorch-compatible).
• Dataset: Sample action clips (Kinetics, UCF101 samples), standard tracking sequences (e.g., from the VOT benchmark website).

Understand different 3D data representations and learn SOTA techniques for 3D reconstruction, point cloud analysis, and novel view synthesis (NeRF/Gaussian Splatting), focusing on PyTorch implementations.

Introduction to 3D Vision:

• Importance (Robotics, AV, VR/AR, Medical)
• Challenges (Data Acquisition, Representation Diversity, Computational Cost)

3D Representations:

• Meshes (Vertices, Faces), Voxels (Grids), Point Clouds (Unordered Sets), Implicit Functions (SDF, Occupancy), Radiance Fields.

3D Reconstruction:

• Goal: Building 3D models from sensor data (Images, Depth).
• Classical Methods (Brief: Structure from Motion - SfM, Multi-View Stereo - MVS).
• Neural Implicit Surfaces: DeepSDF, VolSDF, NeuS (Combining implicit functions with volume rendering for image-based reconstruction). Instant-NGP, Neuralangelo (Faster/Higher Quality). (Typically PyTorch)
• Gaussian Splatting (3DGS): Explicit representation (3D Gaussians), Fast training/rendering, SOTA quality. (Official implementation is PyTorch)
• Diffusion Models for 3D Reconstruction (CDM - Conditional Diffusion).

Point Cloud Analysis:

• Properties of Point Clouds (Unordered, Variable Density).
• Pioneering Architectures: PointNet (using Symmetric Functions for Permutation Invariance), PointNet++ (Hierarchical feature learning).
• Advanced Architectures: Graph CNNs, Point Transformers (Point-BERT, Point-MAE), Mamba3D. (Most research uses PyTorch)
• Tasks: Classification (ModelNet), Part/Semantic Segmentation (ShapeNetPart, S3DIS, ScanNet), Detection.
• SSL for Point Clouds: Contrastive (PointContrast), Masked Reconstruction (Point-MAE), Geometric Shortcut Problem (Sonata).

Novel View Synthesis (NeRF Focus):

• Neural Radiance Fields (NeRF): Core Concept (MLP mapping 5D coordinates to density/color), Volume Rendering Equation, Positional Encoding. (Many PyTorch implementations)
• Advances: Efficiency (Instant-NGP, Plenoxels), Realism (Mip-NeRF, Ref-NeRF), Dynamic Scenes (D-NeRF), Large Scenes (Block-NeRF), Editing NeRFs.
• Comparison: NeRF vs. Gaussian Splatting (Implicit vs. Explicit, Speed vs. Memory).

3D Scene Understanding:

• Goal: Holistic Interpretation (Objects, Relationships, Semantics).
• Tasks: 3D Semantic/Instance Segmentation, 3D Object Detection, 3D Scene Question Answering (3D SQA - ScanQA), Visual Grounding.
• Loading and visualizing point clouds (Open3D library, works with PyTorch tensors).
• Using a pre-trained PointNet/PointNet++ model for classification or segmentation (e.g., via PyTorch Geometric or model zoos).
• (Conceptual) Setting up and running a simple NeRF or Gaussian Splatting training using existing libraries (e.g., nerfstudio (PyTorch), official 3DGS repo (PyTorch)) - focusing on usage.
• Running inference with a pre-trained NeRF/3DGS model to render novel views.
• Using Open3D for basic point cloud processing (downsampling, registration - brief).

Point Cloud Classification & Novel View Synthesis (Lab 9)

• Objective: Classify 3D objects represented as point clouds using PyTorch models and render novel views from a pre-trained NeRF or 3DGS model (PyTorch-based).
• Tasks:
  1. Load point cloud data from a standard dataset (e.g., ModelNet10/40).
  2. Visualize sample point clouds using Open3D.
  3. Load a pre-trained point cloud classification model (e.g., PointNet++ implemented in PyTorch Geometric).
  4. Preprocess the point cloud data as required by the PyTorch model.
  5. Run inference to classify point cloud objects. Evaluate accuracy on the test set.
  6. Download pre-trained NeRF or Gaussian Splatting data for a sample scene (e.g., from nerfstudio demos or 3DGS samples).
  7. Use the corresponding library/viewer (likely PyTorch-based) to load the pre-trained model.
  8. Render novel views of the scene by interactively changing the virtual camera pose or defining a camera trajectory.
  9. Observe the quality and realism of the synthesized images.
• Deliverables: Python script/notebook for point cloud classification, evaluation results, screenshots/video of novel views rendered from NeRF/3DGS.
• Tools: Python, PyTorch, Open3D, PyTorch Geometric (optional), NeRF/3DGS library (nerfstudio, gaussian-splatting - PyTorch-based).
• Dataset: ModelNet10/40, Pre-trained NeRF/3DGS scene data.

Understand the principles behind Diffusion Models and GANs, and gain experience using SOTA models for text-to-image generation and basic image editing using PyTorch libraries.

Generative Modeling Goals: Learning data distributions, synthesizing new samples.

Generative Adversarial Networks (GANs):

• Core Concept: Generator vs. Discriminator min-max game.
• Training Dynamics & Challenges (Mode Collapse, Stability).
• Advanced Architectures (Brief): StyleGAN (Style-based generator, ADA - Official implementation is PyTorch), Conditional GANs (cGAN).
• Pros/Cons: Fast sampling, good for specific domains (faces), generally harder to train/control than diffusion.

Diffusion Probabilistic Models:

• Core Concept: Forward Process (adding noise), Reverse Process (learning to denoise).
• Key Formulations: DDPM, Score-Based Models (SGM/SDE).
• Denoising Network Architecture: U-Net (common), Diffusion Transformers (DiTs - Sora, SD3 - scalability).
• Conditioning: Text (CLIP/T5 embeddings), Class Labels, Images.
• Guidance: Classifier-Free Guidance (CFG - improving sample quality/adherence), Model-guidance (MG).
• Latent Diffusion Models (LDMs - Stable Diffusion): Diffusion in a compressed latent space (VAE). (Stable Diffusion uses PyTorch)

Key Applications & SOTA Models:

• Text-to-Image (T2I): Stable Diffusion, DALL-E 3, Midjourney, Imagen, SD3, PixArt. (Many open models/libraries are in PyTorch)
• Video Generation: Sora, Latte, DDT (using DiTs).
• Image Editing/Translation: Inpainting (RePaint), Instruction-based editing (InstructPix2Pix), Style Transfer, Super-Resolution using Diffusion. ControlNet (adding spatial control). (PyTorch implementations available).

Efficiency Challenges & Solutions:

• Slow Sampling (Many steps): Faster solvers (DDIM), Distillation (Progressive Distillation), Latent Space.
• Training Cost.
• Using the Hugging Face diffusers library (PyTorch-first):
  • Loading pre-trained Diffusion pipelines (Stable Diffusion).
  • Generating images from text prompts (T2I).
  • Exploring parameters (guidance scale, steps, negative prompts).
  • Performing basic image editing tasks (e.g., image-to-image using a prompt, inpainting with a mask).
  • Using ControlNet pipelines with conditioning inputs (e.g., Canny edges, pose).
• (Optional) Basic GAN usage (loading pre-trained StyleGAN from the official PyTorch repo or other libraries). Generating samples, exploring latent space interpolation.

Text-to-Image Generation & Control (Lab 10)

• Objective: Use Stable Diffusion (PyTorch-based via diffusers) to generate images from text prompts and achieve control over generation using ControlNet.
• Tasks:
  1. Install the Hugging Face diffusers library.
  2. Load a Stable Diffusion pipeline (e.g., SD 1.5, SD 2.1, SDXL).
  3. Generate images using various text prompts. Experiment with prompt engineering (details, style keywords).
  4. Explore the impact of guidance_scale and num_inference_steps. Use negative prompts.
  5. Select an image and a corresponding conditioning map (e.g., Canny edge map, human pose skeleton).
  6. Load a ControlNet pipeline (PyTorch weights) compatible with the selected Stable Diffusion base model and conditioning type.
  7. Using the ControlNet pipeline, generate images by providing the text prompt and conditioning map. Observe how the output adheres to the control signal.
  8. (Bonus) Try generating images with multiple ControlNets active.
• Deliverables: Python script/notebook, generated images (from T2I and ControlNet), comparison of outputs with/without control, analysis of parameter effects.
• Tools: Python, Hugging Face diffusers, transformers, accelerate, OpenCV (for creating conditioning maps), PyTorch.
• Dataset: None (generative task). Conditioning maps can be generated from sample images.

Understand the motivation and mechanisms behind Self-Supervised Learning (SSL), Contrastive Learning, Multimodal Learning (Vision-Language Models), and Few/Zero-Shot Learning, focusing on PyTorch tools and libraries.

Re-evaluating Self-Supervised Learning (SSL):

• Motivation: Reducing label dependency, leveraging unlabeled data, learning general representations.
• Taxonomy Review: Contrastive, Masked Modeling, Non-Contrastive, Clustering.
• Impact on Pre-training of SOTA Models.

Deep Dive into Contrastive Learning:

• Core Idea: Pulling positives closer together and pushing negatives further apart in the embedding space.
• Components: Anchor/Positive/Negative samples, Encoder, Projection Head, Contrastive Loss (InfoNCE), Data Augmentation.
• Pioneering Methods: SimCLR, MoCo (Momentum Encoder, Queue), BYOL (No negatives, asymmetric architecture), SwAV (Online Clustering). (Many PyTorch implementations)
• Relationship with Metric Learning.

Multimodal Learning (Vision-Language Focus):

• Motivation: Richer understanding by combining modalities.
• Key Approaches:
  • Contrastive Pre-training: CLIP, ALIGN (Shared embedding space, Zero-shot capabilities). (OpenCLIP is in PyTorch)
  • Fusion Encoders (Brief mention).
  • Vision Encoders + LLMs (Dominant Trend): LLaVA, MiniGPT-4, BLIP-2, Flamingo, GPT-4V, Gemini. Architecture (Vision Encoder -> Adapter -> LLM). Training (Alignment Pre-training + Instruction Tuning). (Most open-source VLMs are in PyTorch)
• Applications: VQA, Captioning, Retrieval, Reasoning, Embodied AI (VLA).
• Challenges: Alignment, Hallucination, Evaluation, Data.

Few-Shot Learning (FSL):

• Motivation: Learning from very few labeled examples (K-shot, N-way).
• Episodic Training Framework (Support set, Query set).
• Approaches: Metric Learning (Prototypical Networks), Optimization-Based (MAML), Generative Approaches. (PyTorch libraries available for FSL).

Zero-Shot Learning (ZSL):

• Motivation: Recognizing classes with no labeled examples.
• Classical Approaches (Attribute-based, Word embeddings).
• VLM-Powered ZSL: Using CLIP/ALIGN embeddings for image and text class descriptions. Prompt Engineering/Tuning (TPT). Reasoning over VLM features (CHBR).
• Applications: Open-Vocabulary Detection/Segmentation.
• Using CLIP (Hugging Face transformers for PyTorch or OpenCLIP): Loading models, extracting image and text embeddings, computing similarity for zero-shot classification.
• Using a pre-trained VLM (e.g., LLaVA, BLIP via HF transformers for PyTorch or dedicated PyTorch libraries): Performing Visual Question Answering (VQA) and Image Captioning.
• (Conceptual) How Prototypical Networks work for Few-Shot Learning using PyTorch.
• (Conceptual) Setting up a contrastive learning pipeline using PyTorch libraries like lightly or PyTorch Lightning Bolts (focusing on components).

Zero-Shot Classification & VQA with VLMs (Lab 11)

• Objective: Apply CLIP (PyTorch version) for zero-shot image classification and use a Large Vision-Language Model (LVLM) (PyTorch-based) for VQA.
• Tasks:
  1. Select a classification dataset where classes can be described textually (e.g., CIFAR-10/100, Caltech-101).
  2. Load a pre-trained CLIP model (PyTorch version via HF or OpenCLIP).
  3. For a subset of images from the test set:
     • Extract the image embedding.
     • Create text prompts for all possible classes (e.g., "a photo of a [class name]").
     • Extract text embeddings for all class prompts.
     • Compute cosine similarity between the image embedding and all text embeddings.
     • Predict the class with the highest similarity.
  4. Calculate zero-shot classification accuracy. Compare with random chance.
  5. Load a pre-trained LVLM capable of VQA (e.g., BLIP-2, LLaVA via HF for PyTorch).
  6. Select several diverse images.
  7. Ask 3-5 different questions for each image (e.g., "What object is in the center?", "How many people are there?", "What color is the car?").
  8. Feed the image and question to the LVLM and record the generated answer.
  9. Qualitatively evaluate VQA performance: Are the answers relevant, accurate, and detailed? Note any hallucinations.
• Deliverables: Python script/notebook, zero-shot accuracy results, VQA image/question/answer pairs, qualitative analysis of VQA performance.
• Tools: Python, PyTorch, Hugging Face transformers, datasets, OpenCLIP (optional).
• Dataset: Standard classification dataset (CIFAR, Caltech-101), diverse image set for VQA.

Mini Project 2: Creative or Analytical Task (Week 12-13)

• Objective: Apply skills from generative models or multimodal learning using PyTorch tools.
• Sample Tasks:
  • Creative: Build a Gradio app using Stable Diffusion (diffusers - PyTorch) + ControlNet for guided image generation (e.g., turning sketches into photos).
  • Analytical: Use CLIP embeddings (PyTorch) to perform image retrieval based on text queries or find visually similar images within a dataset.
  • VLM: Build a simple tool that takes an image, automatically generates a detailed caption, and answers predefined questions about it using a PyTorch VLM.

Learn techniques for making trained PyTorch models smaller, faster, and suitable for real-world deployment across various platforms.

The Need for Efficiency: Latency, Throughput, Memory constraints (Edge/Mobile), Power Consumption, Cost.

Model Compression Techniques:

• Quantization: Reducing numerical precision (FP32 -> FP16/INT8/INT4). PyTorch built-in tools (PTQ, QAT), ONNX Runtime quantization. Hardware support (CPU, GPU, TPUs, NPUs).
• Pruning: Removing redundant weights/neurons/channels/layers. PyTorch torch.nn.utils.prune. Unstructured vs. Structured Pruning. Magnitude Pruning, Lottery Ticket Hypothesis concept.
• Knowledge Distillation: Training a smaller "student" model to mimic the outputs (logits/features) of a larger "teacher" model within PyTorch.
• Neural Architecture Search (NAS): Automating efficient architecture design (Brief: MobileNetV3, EfficientNet). Low-Rank Factorization.

Optimized Inference:

• Inference Engines/Runtimes: ONNX Runtime, TensorRT (NVIDIA), OpenVINO (Intel), PyTorch Mobile, Core ML (Apple).
• Graph Optimizations: Layer Fusion, Constant Folding (typically handled by runtimes/export).
• Batching & Caching.

Deployment Strategies:

• Cloud Deployment (Serverless functions, GPU VMs, Managed ML Services - SageMaker, Vertex AI). API design (REST).
• Edge Deployment (Mobile devices, Embedded systems - Raspberry Pi, Jetson Nano). Considerations: Power, connectivity, model size. Use ONNX Runtime or PyTorch Mobile.
• Web Deployment (using ONNX.js, WebAssembly).
• Containerization (Docker): Packaging models and dependencies.
• Monitoring and Maintenance in Production.
• Converting a PyTorch model to ONNX format (torch.onnx.export).
• Running inference using ONNX Runtime in Python.
• Applying basic Post-Training Quantization using PyTorch's built-in tools or ONNX Runtime. Comparing speed/size.
• (Conceptual) Overview of the TensorRT/OpenVINO optimization process (typically starting from ONNX).
• Using the Gradio library to quickly build a web demo interface for a trained CV model (can load PyTorch or ONNX).
• Building a simple Docker container for a CV model API (using Flask/FastAPI to serve a PyTorch or ONNX model).

Optimizing and Deploying a Model (Lab 12)

• Objective: Optimize a trained PyTorch CV model (e.g., from Lab 5 or 6) using quantization and deploy it as a simple web service or demo.
• Tasks:
  1. Select a trained PyTorch classification or detection model from a previous lab.
  2. Convert the PyTorch model to ONNX format. Verify the conversion.
  3. Benchmark the inference speed and measure the model size of the original PyTorch model (perhaps using torch.profiler) and the ONNX model using ONNX Runtime.
  4. Apply Post-Training Dynamic Quantization (INT8) to the ONNX model using ONNX Runtime tools (or try PyTorch PTQ and re-export).
  5. Benchmark the inference speed and measure the model size of the quantized ONNX model. Compare accuracy before and after quantization (on a small sample).
  6. (Option A) Build a simple web API using Flask/FastAPI that takes an image (upload or URL) and returns the model's prediction using the optimized ONNX model. Dockerize the application.
  7. (Option B) Build an interactive web demo using Gradio (using the optimized ONNX model, or loading the PyTorch model directly if sufficiently efficient) that allows users to upload an image and view the model's predictions.
• Deliverables: Optimized ONNX model (original and quantized), benchmarking script/results, Dockerfile and associated files (Option A) OR Gradio application script (Option B).
• Tools: Python, PyTorch, ONNX, ONNX Runtime, Docker (optional), Flask/FastAPI (optional), Gradio (optional).

Capstone Project - Proposal & Planning

Define a challenging and meaningful computer vision project applying SOTA techniques (using PyTorch), outlining the plan, dataset, methodology, and evaluation strategy.

• Brainstorming project ideas (aligned with participant interests and bootcamp scope).
• Instructor guidance and feedback on feasibility and scope.
• Project Proposal Document:
  • Problem Definition & Motivation
  • Dataset Identification & Acquisition Plan
  • Proposed Methodology (Models - PyTorch, Techniques)
  • Evaluation Metrics & Plan
  • Potential Challenges & Mitigation Strategies
  • Timeline & Milestones
• Peer review of proposals.

Capstone Project - Implementation & Development

Execute the capstone project plan, implement the solution using PyTorch, train/fine-tune models, iterate based on results, and document progress.

• Data collection and preprocessing (torchvision, custom Datasets).
• Model implementation and training/fine-tuning in PyTorch.
• Experiment tracking (W&B/TensorBoard).
• Debugging and troubleshooting PyTorch code.
• Regular check-in meetings with instructors/mentors.
• Interim progress presentations/demos.

Capstone Project - Presentation & Wrap-up

Present the capstone project findings, demonstrate the working PyTorch-based solution, discuss challenges and learnings, and review key bootcamp takeaways.

• Completing project implementation and documentation.
• Preparing the final presentation (Problem, Data, Method (PyTorch details), Results, Demo, Conclusion, Future Work).
• Project demos.
• Q&A and feedback from peers and instructors.
• Bootcamp evaluation and summary of key learnings.
• Career guidance in CV and next steps.

Evaluation:

• Lab Submissions (Accuracy, Code Quality (PyTorch specifics), Analysis)
• Mini Project Performance
• Quizzes/Assessments on Theoretical Concepts
• Capstone Project (Proposal, Implementation (PyTorch), Presentation, Results)
• Participation and Engagement

Environment Setup:

• Detailed instructions on Python, Conda environments, required IDEs (VS Code), PyTorch installation (CPU/GPU versions), and accessing GPU resources (Google Colab setup guide) will be provided prior to the bootcamp.
• Standardized environment files (environment.yml or requirements.txt) will be provided for consistency.

Advanced Topics (Can be added/modified based on time/interest):

• Reinforcement Learning for Computer Vision Tasks
• Graph Neural Networks in Computer Vision (PyTorch Geometric)
• Advanced Video Understanding (Dense Captioning, Complex Event Recognition)
• Explainable AI (XAI) for Computer Vision (Captum (PyTorch XAI library), Grad-CAM)
• Federated Learning for Computer Vision (Flower framework supports PyTorch)
• Ethical Considerations in CV (Bias, Fairness, Privacy)