Skip to main content

Assignment 1: Implementing a Vision-Language-Action Pipeline

Objective​

Implement a complete Vision-Language-Action (VLA) pipeline that can interpret natural language commands, perceive the environment through vision systems, and execute appropriate robotic actions. This assignment integrates all aspects of the VLA module with previous course modules.

Prerequisites​

  • Completion of VLA module content (Weeks 10-13)
  • Understanding of ROS 2 concepts (Module 1)
  • Experience with simulation environments (Module 2)
  • Knowledge of NVIDIA Isaac and perception (Module 3)

Assignment Requirements​

Your VLA pipeline must include:

  1. Language Understanding Module: Process natural language commands and extract actionable information
  2. Vision Processing Module: Analyze visual input to understand the environment and objects
  3. Action Planning Module: Generate executable actions based on language commands and visual input
  4. Execution Module: Interface with robotic systems to execute planned actions
  5. Integration: All modules must work together in a cohesive system

Technical Specifications​

System Architecture​

Your system should follow a modular architecture with clear interfaces between components:

[Language Input] -> [Language Understanding] -> [Action Planner]
                     |                             |
                     v                             v
[Visual Input] -> [Vision Processing] -> [Action Executor] -> [Robot]

Core Components to Implement​

  1. Language Parser

    • Accept natural language commands
    • Identify task objectives, target objects, and spatial relationships
    • Generate structured action requests

  2. Perception System

    • Process camera input for object detection and localization
    • Integrate Isaac ROS perception packages
    • Maintain spatial map of objects in the environment

  3. Action Generator

    • Convert high-level objectives into sequences of robot actions
    • Integrate navigation and manipulation primitives
    • Handle error recovery and replanning

  4. Execution Coordinator

    • Manage the execution of action sequences
    • Monitor progress and adjust plans as needed
    • Provide feedback on task completion status

Implementation Steps​

Phase 1: System Design (20 points)​

Create a detailed system architecture document that includes:

  • Block diagram showing component interactions
  • Data flow between modules
  • ROS 2 message types and topics
  • Error handling mechanisms
  • Performance requirements and constraints

Submit a 2-3 page design document with architectural diagrams.

Phase 2: Component Implementation (50 points)​

Implement each of the core components:

Language Understanding Module (15 points)

  • Create a ROS 2 node that processes natural language commands
  • Use appropriate NLP techniques to extract intent and entities
  • Generate structured action requests based on the input

Vision Processing Module (15 points)

  • Integrate Isaac ROS perception components
  • Detect and localize objects relevant to tasks
  • Maintain spatial relationships between objects

Action Planning and Execution Module (20 points)

  • Plan sequences of actions based on command and perception inputs
  • Integrate with robot control systems (simulated or real)
  • Implement feedback and error recovery mechanisms

Phase 3: System Integration (20 points)​

Connect all components into a unified system:

  • Ensure proper data flow between modules
  • Handle asynchronous operations appropriately
  • Implement system-level error handling
  • Test integration in simulation environment

Phase 4: Evaluation (10 points)​

Evaluate your system's performance:

  • Test with at least 5 different command types
  • Measure success rate and execution time
  • Document limitations and potential improvements

Code Requirements​

Your implementation must:

  • Follow ROS 2 best practices and conventions
  • Include comprehensive documentation
  • Have appropriate error handling
  • Be modular and maintainable
  • Include unit tests for critical components
  • Use Isaac ROS packages for perception where applicable

Example ROS 2 Node Structure​

import rclpy
from rclpy.node import Node
from std_msgs.msg import String
from sensor_msgs.msg import Image
from geometry_msgs.msg import Pose
from vla_msgs.msg import ActionRequest, ActionResult  # You'll need to define this message type


class VLAPipelineNode(Node):
    """
    Main node for the Vision-Language-Action pipeline.
    Coordinates the interaction between language, vision, and action components.
    """
    def __init__(self):
        super().__init__('vla_pipeline')
        
        # Subscriptions
        self.command_subscriber = self.create_subscription(
            String,
            'natural_language_command',
            self.command_callback,
            10
        )
        
        self.image_subscriber = self.create_subscription(
            Image,
            'camera/image_raw',
            self.image_callback,
            10
        )
        
        # Publishers
        self.action_publisher = self.create_publisher(
            ActionRequest,
            'action_requests',
            10
        )
        
        self.status_publisher = self.create_publisher(
            String,
            'vla_pipeline/status',
            10
        )
        
        # Internal state
        self.latest_image = None
        self.perception_results = {}
        self.language_understanding = None
        
        self.get_logger().info("VLA Pipeline initialized")

    def command_callback(self, msg):
        """Process natural language command and initiate action planning."""
        command = msg.data
        self.get_logger().info(f"Received command: {command}")
        
        # Step 1: Use language understanding component
        action_request = self.process_language_command(command)
        
        if action_request:
            # Step 2: Incorporate perception data
            enhanced_request = self.enhance_with_perception(action_request)
            
            # Step 3: Publish action request
            self.action_publisher.publish(enhanced_request)
            
            # Step 4: Monitor execution and provide feedback
            self.monitor_execution(enhanced_request.id)
        else:
            self.get_logger().error(f"Could not process command: {command}")

    def image_callback(self, msg):
        """Process incoming image for environmental perception."""
        # Process image with Isaac perception components
        # Store relevant information for action planning
        pass

    def process_language_command(self, command: str) -> ActionRequest:
        """
        Process natural language command and generate initial action request.
        
        Args:
            command: Natural language command string
            
        Returns:
            ActionRequest message with initial plan
        """
        # In a real implementation, this would use NLP/VLA models
        # For this assignment, implement a rule-based parser or simple NLP approach
        pass

    def enhance_with_perception(self, action_request: ActionRequest) -> ActionRequest:
        """
        Enhance action request with perception data.
        
        Args:
            action_request: Initial action request from language processing
            
        Returns:
            ActionRequest enhanced with perceptual information
        """
        # Use perception data to refine action plan
        # Add object locations, spatial relationships, etc.
        pass

    def monitor_execution(self, request_id: str):
        """Monitor the execution of a specific action request."""
        # Monitor action execution and provide feedback
        pass


def main(args=None):
    rclpy.init(args=args)
    
    vla_pipeline = VLAPipelineNode()
    
    try:
        rclpy.spin(vla_pipeline)
    except KeyboardInterrupt:
        pass
    finally:
        vla_pipeline.destroy_node()
        rclpy.shutdown()


if __name__ == '__main__':
    main()

Testing Requirements​

Create test cases that demonstrate your system's capabilities:

  1. Simple Navigation: "Go to the red cube"
  2. Object Manipulation: "Pick up the blue cup"
  3. Spatial Relations: "Go to the left of the chair"
  4. Sequential Tasks: "Go to the table and pick up the book"
  5. Conditional Execution: "If you see a green ball, go to it"

Deliverables​

  1. Source Code: Complete, documented implementation
  2. System Design Document: 2-3 pages with architecture diagrams
  3. Test Results: Documentation of system performance on test cases
  4. Video Demonstration: Short video showing system in action (simulation or real robot)
  5. Reflection Report: 1-2 pages discussing challenges, solutions, and lessons learned

Grading Rubric​

  • System Design Document (20 points)
  • Component Implementation (50 points)
    • Language Understanding (15 points)
    • Vision Processing (15 points)
    • Action Planning/Execution (20 points)
  • System Integration (20 points)
  • Evaluation and Testing (10 points)
  • Total: 100 points

Performance Benchmarks​

For full credit on implementation:

  • Language Understanding: 80% accuracy on simple commands
  • Vision Processing: Successfully detect and localize target objects in 80% of attempts
  • Action Execution: Complete 70% of attempted tasks successfully
  • System Integration: All components communicate effectively without crashes

Extra Credit Opportunities (Up to 10 bonus points)​

  1. Implement a learning component that improves performance based on feedback
  2. Extend the system to handle multi-modal inputs (e.g., pointing + language)
  3. Add uncertainty awareness to the action planning component
  4. Implement sim-to-real transfer capabilities

Resources​

Submission Guidelines​

Submit your assignment as a ZIP file containing:

  1. A src/ directory with all source code
  2. A doc/ directory with your design document and reflection report
  3. A launch/ directory with launch files for your system
  4. A test/ directory with test cases and results
  5. A README.md file with setup and usage instructions

Name your submission as vla_assignment_lastname_firstname.zip.

Due Date: [Specify date per course schedule] Late submissions will incur a 5% penalty per day.