Assignment 1: Perception Pipeline Implementation

Instructions

This assignment requires you to implement a complete perception pipeline that combines multiple AI models to process sensory data from a robot equipped with NVIDIA Isaac platform. You will design, implement, and evaluate a system that performs object detection, depth estimation, and environment mapping.

Learning Objectives

Upon completion of this assignment, you will be able to:

• Integrate multiple NVIDIA Isaac perception components into a unified pipeline
• Optimize AI models for real-time execution on edge computing platforms
• Implement sensor fusion for enhanced environmental understanding
• Deploy a complete perception system on NVIDIA hardware
• Evaluate the performance of your perception pipeline

Assignment Requirements

Part 1: System Design (20 points)

Design an architecture for a perception pipeline that:

1. Processes RGB camera data using optimized deep learning models
2. Integrates LiDAR data for depth estimation and object detection
3. Performs visual SLAM for localization and mapping
4. Uses reinforcement learning to adaptively focus attention based on task

Create documentation for your design that includes:

• Block diagram showing data flow and component interactions
• Justification for your choice of models and algorithms
• Analysis of computational requirements for edge deployment
• Error handling and fallback strategies

Part 2: Implementation (40 points)

Implement your perception pipeline with the following components:

1. Multi-modal Processing (10 pts)

   • Create nodes for processing RGB and depth data simultaneously
   • Implement CUDA-accelerated algorithms for feature extraction
   • Design efficient data structures for multi-modal fusion

2. Deep Learning Integration (10 pts)

   • Integrate at least 2 pre-trained models from the Isaac ROS package set (e.g., detection, classification, segmentation)
   • Optimize models for real-time performance using TensorRT
   • Implement dynamic batching for consistent performance

3. VSLAM System (10 pts)

   • Implement visual SLAM using Isaac tools or similar approach
   • Include loop closure detection and pose graph optimization
   • Create occupancy grid mapping for navigation

4. Perception-Action Coupling (10 pts)

   • Integrate perception outputs with a control or navigation system
   • Implement adaptive attention mechanisms that focus computational resources
   • Create feedback loops that improve perception based on actions

Part 3: Evaluation and Optimization (30 points)

1. Performance Evaluation (10 pts)

   • Measure inference times for each component
   • Evaluate accuracy of perception outputs against ground truth
   • Assess real-time capability (30 FPS minimum for perception pipeline)

2. Resource Optimization (10 pts)

   • Profile memory usage and optimize for edge deployment
   • Demonstrate model quantization without significant accuracy loss
   • Tune hyperparameters for your specific robot platform

3. Robustness Testing (10 pts)

   • Evaluate pipeline performance under varying lighting conditions
   • Test with partial sensor data (e.g., occluded sensors)
   • Assess recovery from tracking failures

Part 4: Documentation and Presentation (10 points)

1. Provide clear documentation for your implementation
2. Include instructions for building and running your system
3. Create a presentation (slides or written report) demonstrating your approach and results

Implementation Guidelines

1. Use ROS 2 and Isaac ROS packages where applicable
2. Optimize for real-time performance on NVIDIA Jetson or RTX platform
3. Implement proper error handling and graceful degradation
4. Include visualization tools for debugging and demonstration
5. Follow software engineering best practices (modular design, testing, etc.)

Example Structure

perception_project/
├── src/
│   ├── perception_pipeline/
│   │   ├── camera_processor.py
│   │   ├── lidar_processor.py
│   │   ├── fusion_module.py
│   │   ├── visual_slam.py
│   │   └── rl_attention.py
│   ├── models/
│   │   └── optimized_models.plan  # TensorRT optimized models
│   └── utils/
│       ├── visualization.py
│       └── evaluation_metrics.py
├── launch/
│   └── perception_pipeline.launch.py
├── config/
│   ├── perception_pipeline_params.yaml
│   └── models.yaml
├── test/
│   └── test_perception.py
└── docs/
    └── implementation_guide.md

Assessment Criteria

Part 1: System Design (20 points total)

• Architecture diagram showing component interactions (5 points)
• Justification of model/algorithm choices (5 points)
• Computational requirements analysis (5 points)
• Error handling strategy (5 points)

Part 2: Implementation (40 points total)

• Multi-modal processing working (10 points)
• Deep learning models integrated and optimized (10 points)
• VSLAM system functional (10 points)
• Perception-action coupling implemented (10 points)

Part 3: Evaluation and Optimization (30 points total)

• Performance measurements (10 points)
• Resource optimization demonstrated (10 points)
• Robustness testing completed (10 points)

Part 4: Documentation and Presentation (10 points total)

• Clear implementation documentation (5 points)
• Working instructions and demonstration (5 points)

Best Practices

• Use Isaac ROS packages where available and appropriate
• Implement proper logging and debugging capabilities
• Design modular components that can be individually tested
• Optimize for your target hardware early in the process
• Validate your perception outputs against realistic ground truth data

Submission Requirements

1. Complete source code with proper documentation
2. Configuration files for your perception pipeline
3. Launch files for easy execution
4. Evaluation results with metrics
5. Project report detailing your approach, challenges, and results

Additional Resources

• NVIDIA Isaac ROS documentation
• Isaac Lab reinforcement learning tutorials
• CUDA and TensorRT optimization guides
• ROS 2 perception tutorials
• Computer vision and SLAM literature

Rubric

• Total points: 100
• Passing score: 70/100 (70%)
• Late submission penalty: 5% per day