AI Human Activity Recognition for On-board BAS Experiments

1. Executive Snapshot

This dossier presents an end-to-end technical blueprint for SIH26174: AI Human Activity Recognition for On-board BAS Experiments, sponsored by the Indian Space Research Organisation (ISRO). In microgravity environments on board the Bharatiya Antariksh Station (BAS) and future deep-space crewed missions, high-latency and restricted bandwidth to Earth prohibit streaming continuous high-definition video to ground control [PS]. The proposed solution is AstraCheck-HAR, an edge-native, offline AI assistance system that continuously monitors fixed-payload video feeds, tracks experimental protocols, validates step sequences via hand-object interaction (HOI) and orientation-agnostic 3D Human Mesh Recovery (HMR), provides real-time offline voice guidance and anomaly alerts, logs structured execution telemetry, streams local IP feeds, and renders a real-time monitoring dashboard—all operating locally on space-constrained edge compute hardware [PS].

Problem Metadata

Fundamental System Objective: To deliver an air-gapped, edge-deployed AI activity recognition and protocol validation assistant that guarantees error-free execution of payload experiments on board the Bharatiya Antariksh Station without relying on ground mission control connectivity [PS].

2. Problem Deconstruction

The problem statement has been broken down into atomic, explicit requirements [PS]. Every explicit item from the official description and dataset notes is preserved without weakening or silent alteration [PS].

• Requirement [a] - Offline Standalone Edge Operation: System must process video feeds locally on edge hardware with zero dependency on cloud or ground control connectivity [PS].
• Requirement [b] - Fixed-Payload Camera Input: Ingestion must accept real-time video feeds from fixed cameras positioned around experimental payload racks [PS].
• Requirement [c] - Custom Synthetic & Focused Dataset Generation: The solution requires creating a focused local synthetic/replicated dataset (e.g., using webcams and physical prop boxes: outer container, red box, yellow box) for object detection, pose estimation, and hand-object interaction tracking [PS].
• Requirement [d] - Real-Time Sequence Recognition & Tracking: The AI model must recognize active human-object interaction steps and continuously track the progress of a pre-defined multi-step experiment [PS].
• Requirement [e] - Predictive Next-Step Guidance: At the start of the experiment and immediately after each completed step, the system must suggest the next step to be performed [PS].
• Requirement [f] - Voice-Based Out-of-Sequence & Skip Alerts: The system must trigger immediate voice alerts when an experiment step is skipped or performed out of sequence [PS].
• Requirement [g] - Timestamped Structured Telemetry Logging: System must continuously generate a lightweight, timestamped, structured text file (JSON/CSV) capturing executed steps, status, durations, and step outcomes [PS].
• Requirement [h] - Local Storage & Dual-IP Streaming: Video streams must be recorded to local edge storage while simultaneously streaming live feeds over a specific local IP address [PS].
• Requirement [i] - Real-Time Graphical User Interface (GUI): A local desktop/web GUI must display active step status, live video overlay, camera status, next-step recommendations, and log streams [PS].
• Requirement [j] - Deliverable Model & Executable: The final deliverable must be a trained offline AI model packaged into an offline standalone application [PS].
• Requirement [k] - (Optional / Advanced) Orientation-Agnostic 3D Human Mesh Recovery (HMR): The model should perform body tracking relative to the payload rack rather than a terrestrial ground/floor plane, accommodating microgravity 0G orientations where "up" and "down" do not exist [PS].

3. Stakeholders & User Needs

4. Requirements Traceability

5. Constraints & Environment Realities

• Spatial & Orientation Realities (Microgravity): In microgravity (0G), astronauts float at arbitrary angles (upside down, sideways) relative to cameras [PS, KNOWN]. Terrestrial pose estimators relying on vertical gravity vectors fail [PS]. Body posture tracking must be mapped relative to fixed payload rack reference frames [PS].
• Network & Data Limitations: LEO and space station communications suffer from bandwidth bottlenecks, intermittent line-of-sight contacts, and high latency [PS, KNOWN]. All AI processing, logging, and voice synthesis must occur completely air-gapped on the edge [PS].
• Compute & Power Budgets: On-board payload computers (e.g., NVIDIA Jetson Orin Industrial or space-qualified embedded systems) operate under tight thermal and power caps (15W–40W) [KNOWN]. Models must run efficiently at 15–30 FPS on TensorRT/ONNX Runtime without consuming excessive VRAM [INFERENCE].
• Optical & Environmental Variations: Payload rack lighting can change due to orbital solar angles, cabin illumination adjustments, or physical self-shadowing by astronauts [INFERENCE]. Astronauts wear gloves that alter hand geometry, requiring robust hand-object contact detection [RESEARCHED].

6. Research Questions

1. How can 3D Human Mesh Recovery (HMR) be rendered rotationally invariant when an astronaut is inverted relative to the camera?
   • Investigation Strategy: Project predicted 3D SMPL keypoints into a coordinate system anchored to detected payload rack fiducials [INFERENCE].
2. What lightweight temporal action model or state machine prevents false positives during hand occlusion?
   • Investigation Strategy: Combine frame-level YOLO detections with a hysteresis-backed Finite State Automaton (FSA) to require minimum time-in-state thresholds [INFERENCE].
3. How to synthesize realistic training data for microgravity hand-object interactions on a student budget?
   • Investigation Strategy: Capture multi-angle webcam videos while rotating camera angles or physically tilting props to simulate 0G orientation shifts [ASSUMPTION].

7. Data Requirements & Sources

Data Classification

• A. Explicitly Mentioned Data: Replicated experiment video with outer box, red box, and yellow box props [PS].
• B. Inferred Data: ArUco/fiducial marker coordinate reference frames for rack alignment [INFERENCE].
• C. Synthetic / Collected Data: Lab webcam footage recorded from multiple roll/pitch/yaw camera angles [ASSUMPTION].
• D. Data Requiring Validation: Exact physical dimensions and optical reflective properties of official ISRO payload racks [NEEDS VALIDATION].

8. Existing Solutions & Gap Analysis

9. Candidate Solution Approaches

Approach A: Pure Rule-Based Heuristic with 2D Bounding Box Overlap

• Mechanism: YOLOv11 detects hands and boxes. If bounding boxes overlap for $N$ consecutive frames, a step completion is registered.
• Pros: Extremely fast (<10 ms latency), zero GPU requirement, trivial implementation effort.
• Cons: Fails under occlusion; cannot verify hand posture or grasp state; highly sensitive to camera angle changes; zero orientation awareness.
• Evaluation: High hackathon feasibility, but fails the core requirement for robust protocol validation.

Approach B: Hybrid Architecture – Edge YOLOv11 + Rack-Relative 3D Mesh Normalization + Hysteresis Finite State Automaton (RECOMMENDED)

• Mechanism: YOLOv11 detects payload objects and hands; a lightweight 3D HMR (e.g., HMR2.0-S / CLIFF) extracts 3D SMPL body joint coordinates; 3D coordinates are transformed relative to detected payload rack corner fiducials; a deterministic Finite State Automaton (FSA) validates step sequence transitions and triggers local offline voice synthesis.
• Pros: Highly robust to 0G rotation; low edge compute footprint (<30ms end-to-end latency); zero state hallucination due to deterministic FSA; fully offline executable.
• Cons: Requires initial synthetic dataset generation and ArUco/fiducial marker setup.
• Evaluation: Perfect fit for hackathon MVP and production deployment requirements.

Approach C: End-to-End Multimodal Vision-Language Action Model (VLA / Edge VLM)

• Mechanism: A quantized local VLM (e.g., Qwen2-VL 2B or PaliGemma) ingests video clips and generates step prediction text directly.
• Pros: High flexibility; no manual state machine coding needed.
• Cons: High VRAM usage (>6GB); slow frame rate (<3 FPS on edge hardware); non-deterministic output risks false positive/negative protocol validations; high hallucination risk.
• Evaluation: Unsuitable for mission-critical space payload operations.

10. Recommended Approach & Why

Selected Strategy: Approach B (Hybrid Architecture) [ASSUMPTION].

Why Alternatives Were Rejected

• Approach A Rejected: Too fragile. Simple 2D bounding box overlap triggers massive false positives whenever an astronaut's hand passes over a box without actually picking it up or interacting with it [INFERENCE].
• Approach C Rejected: Edge VLMs are non-deterministic [KNOWN]. In space payload monitoring, hallucinating a step completion when an astronaut skipped a vital safety procedure can destroy an experiment or endanger personnel [INFERENCE]. Furthermore, VLMs cannot maintain 30 FPS processing on low-power edge platforms [KNOWN].

Victory Factors of Approach B

1. Deterministic Safety: Combining perception (AI) with a rule-based State Machine (Deterministic Engine) guarantees zero hallucinated state transitions [INFERENCE].
2. 0G Rotation Invariance: Transform matrix converts camera-space 3D coordinates to Rack-Space coordinates using 4 corner fiducial markers on the payload rack [INFERENCE].
3. Edge Optimization: Runs smoothly at 30 FPS on an NVIDIA Jetson AGX Orin or standard laptop GPU using ONNX Runtime / TensorRT [INFERENCE].

11. Technical Architecture

System Architecture Diagram

+-----------------------------------------------------------------------------------+
|                            INGESTION LAYER                                         |
|  +-----------------------------------+     +-----------------------------------+  |
|  | Fixed Payload Camera Feed (USB/IP)|     | Rack Fiducial Markers (ArUco/QR)  |  |
|  +-----------------+-----------------+     +-----------------+-----------------+  |
+--------------------|-----------------------------------------|--------------------+
                     |                                         |
+--------------------v-----------------------------------------v--------------------+
|                            PERCEPTION LAYER (EDGE AI)                             |
|  +----------------------------------+   +--------------------------------------+  |
|  | YOLOv11 Object & Hand Detector   |   | Lightweight 3D HMR / Pose Extractor  |  |
|  | (Outer Box, Red Box, Yellow Box) |   | (3D SMPL Skeleton Joint Keypoints)   |  |
|  +-----------------+----------------+   +--------------------+-----------------+  |
+--------------------|-----------------------------------------|--------------------+
                     |                                         |
+--------------------v-----------------------------------------v--------------------+
|                         SPATIAL & TEMPORAL ENGINE                                 |
|  +-----------------------------------------------------------------------------+  |
|  | Coordinate Transform: Camera-Space 3D Mesh --> Payload-Rack Frame (0G Norm) |  |
|  +-------------------------------------+---------------------------------------+  |
|                                        |                                          |
|  +-------------------------------------v---------------------------------------+  |
|  | Deterministic Finite State Automaton (FSA) Step Sequence Validator          |  |
|  | [Step 1: Open] -> [Step 2: Red Box Slot A] -> [Step 3: Yellow Box Slot B]...|  |
|  +-------------------------------------+---------------------------------------+  |
+----------------------------------------|------------------------------------------+
                                         |
+----------------------------------------v------------------------------------------+
|                       OUTPUT & INTERACTION LAYER                                  |
|  +------------------+   +--------------------+   +-------------------+  +----------+  |
|  | Offline TTS Voice|   | Timestamped Log    |   | Local MP4 Storage |  | RTSP IP  |  |
|  | Alert Engine     |   | Generator (JSON)   |   | H.264 Encoder     |  | Streamer |  |
|  +--------+---------+   +---------+----------+   +---------+---------+  +----+-----+  |
|           |                       |                        |               |      |
|  +--------v-----------------------v------------------------v---------------v---+  |
|  |                Local GUI Application (PyQt6 Dashboard)                     |  |
|  +----------------------------------------------------------------------------+  |
+-----------------------------------------------------------------------------------+

Step-by-Step Data Flow Narrative

1. Ingestion: Fixed payload cameras capture 1080p video frames at 30 FPS and feed them into the Edge Perception pipeline [INFERENCE].
2. Object & Hand Detection: YOLOv11-Nano extracts 2D bounding boxes and confidence scores for outer_box, lid_open, lid_closed, red_box, yellow_box, slot_A, slot_B, and astronaut_hand [ASSUMPTION].
3. 0G Spatial Transformation: ArUco markers placed on the payload rack frame define the 3D local coordinate system $(X_{rack}, Y_{rack}, Z_{rack})$. The predicted hand and body 3D keypoints are transformed into rack-relative space, eliminating camera/astronaut tilt orientation errors [INFERENCE].
4. Interaction Distance & Contact Analysis: Hand-object 3D spatial distance and IoU contact thresholds determine physical interaction states (HAND_GRASP_RED_BOX, RED_BOX_IN_SLOT_A, etc.) [INFERENCE].
5. State Machine Validation: The FSA compares detected interaction events against the pre-defined JSON experiment workflow [INFERENCE]:
   • If event matches expected active step: Advance state, set status COMPLETED, prompt next step via voice and GUI [INFERENCE].
   • If event violates sequence or skips a step: Retain active state, trigger instant offline voice alert (e.g., "Warning: Step 2 skipped! Please place Red Box in Slot A before proceeding.") [INFERENCE].
6. Logging & Streaming: The logging module writes structured telemetry to a local JSON file, while GStreamer encodes video to disk (/data/video_archive.mp4) and broadcasts an RTSP feed to rtsp://192.168.1.100:8554/live [ASSUMPTION].

12. Technology Stack

13. AI/ML Justification

AI Component 1: YOLOv11 Object & Hand Detection Model

• Purpose: Detect astronaut hands, outer experiment box, box lid status, red box, yellow box, slot A, slot B [PS].
• Model Type: YOLOv11-Nano fine-tuned on custom experiment dataset [ASSUMPTION].
• Input: 1920x1080 / 640x640 RGB video frame [ASSUMPTION].
• Output: Bounding box coordinates $[x_1, y_1, x_2, y_2]$, class label, confidence score [KNOWN].
• Data Required: 1,500 annotated synthetic/replicated experiment images with varied lighting and roll/pitch rotations [ASSUMPTION].
• Why AI is Necessary: Rule-based color thresholding or fixed background subtraction fails under changing lighting, shadows, and floating hands [INFERENCE].
• Evaluation Metric: [email protected]:0.95 > 0.88 [ASSUMPTION].
• Failure Mode & Mitigation: False detection due to glare/shadows $\rightarrow$ Hysteresis filter requires bounding box persistence across 5 consecutive frames before triggering state transition [INFERENCE].

AI Component 2: Rack-Relative 3D Pose / Mesh Estimator

• Purpose: Track 3D body keypoints and hand vectors relative to payload rack fiducials, overcoming 0G orientation shifts [PS].
• Model Type: Lightweight 3D Pose Estimator (MediaPipe 3D World Landmarks / HMR2.0-Small) [RESEARCHED].
• Input: Cropped astronaut body/hand bounding box RGB frame [RESEARCHED].
• Output: 3D coordinates $(x, y, z)$ for body joints and hand fingertips [KNOWN].
• Data Required: Pre-trained 3D body datasets + ArUco calibration matrices [INFERENCE].
• Why AI is Necessary: Analytical 2D projection cannot recover 3D depth or spatial orientation relative to arbitrary rack coordinates [INFERENCE].
• Evaluation Metric: Mean Per Joint Position Error (MPJPE) < 45mm [RESEARCHED].
• Failure Mode & Mitigation: Full occlusion by astronaut back $\rightarrow$ System falls back to object state transition tracking until pose is re-established [INFERENCE].

Where AI Should NOT Be Used

• Sequence Logic & Error Verification MUST NOT use AI [INFERENCE]. Step sequence verification, skip detection, out-of-order detection, and next-step recommendations are governed strictly by a deterministic Finite State Automaton (FSA) [INFERENCE]. Relying on an LLM or AI model for rule verification risks hallucinating step completion and causing mission failure [INFERENCE].

14. Data Architecture & Offline/Low-Network Design

Structured Telemetry Logging Schema (experiment_log.json)

{
  "mission_id": "BAS-EXP-2026-01",
  "experiment_name": "Sample Two-Box Allocation Experiment",
  "start_timestamp_utc": "2026-09-30T10:15:00.000Z",
  "system_status": "ACTIVE",
  "execution_summary": {
    "total_steps": 6,
    "completed_steps": 2,
    "current_step_id": 3,
    "anomalies_detected": 1
  },
  "step_telemetry": [
    {
      "step_id": 1,
      "step_name": "Open Outer Experiment Box",
      "status": "SUCCESS",
      "timestamp_start": "2026-09-30T10:15:05.120Z",
      "timestamp_end": "2026-09-30T10:15:08.450Z",
      "duration_seconds": 3.33,
      "confidence": 0.94
    },
    {
      "step_id": 2,
      "step_name": "Extract Red Box and Place in Slot A",
      "status": "SUCCESS",
      "timestamp_start": "2026-09-30T10:15:09.100Z",
      "timestamp_end": "2026-09-30T10:15:14.200Z",
      "duration_seconds": 5.10,
      "confidence": 0.91
    }
  ],
  "anomaly_events": [
    {
      "event_id": "ERR_01",
      "timestamp": "2026-09-30T10:15:18.000Z",
      "type": "OUT_OF_SEQUENCE_ATTEMPT",
      "description": "Astronaut attempted to Close Outer Box before placing Yellow Box in Slot B.",
      "voice_alert_triggered": "Warning: Out of sequence step. Please place Yellow Box in Slot B.",
      "resolved": true
    }
  ]
}

Offline & Zero-Internet Resilience Strategy

• Local Append-Only Database: Logs are continuously flushed to local SQLite / JSON files to prevent data loss in the event of power resets [INFERENCE].
• Local Video Ring-Buffer: Video streams are encoded into 5-minute H.264 MP4 chunks locally. Storage auto-prunes oldest non-annotated files when disk usage exceeds 90% [ASSUMPTION].
• Air-Gapped RTSP Streamer: The live IP feed streams over local isolated Ethernet/Wi-Fi (udp:// or rtsp://) on the payload rack local area network without requiring internet routing [INFERENCE].

15. Security, Privacy & Governance

• Personal & Astronaut Data Protection: Video footage contains biometrics of Vyomanauts [KNOWN]. In accordance with ISRO spaceflight governance, all processed video remains strictly on local space station storage; ground transmission is limited to lightweight encrypted JSON logs [INFERENCE].
• Local Encryption: Local video storage and SQLite telemetry files are encrypted using AES-256 with key storage bound to hardware TPM modules [ASSUMPTION].
• Air-Gapped Security & Integrity: System executable has zero listening network sockets exposed to public interfaces; local RTSP streaming is restricted to authenticated mission control IP subnets [INFERENCE].
• System Misuse & False Alert Safety: Voice alerts are advisory; astronauts retain manual override capability via physical rack switches or GUI controls to suppress alerts if an unscripted emergency step is required [INFERENCE].

16. MVP Scope (Hackathon Build)

+-----------------------------------------------------------------------------------+
|                                 MVP BUILD SCOPE                                   |
+-----------------------------------------------------------------------------------+
|  MUST BUILD (Core SIH Deliverable):                                               |
|  [x] Fine-tuned YOLOv11 for outer box, lid status, red box, yellow box, slots     |
|  [x] Local Webcam Ingestion & Experiment Sequence State Machine Engine            |
|  [x] Real-time Offline Voice Alerting Engine (eSpeak / Pyttsx3)                   |
|  [x] Lightweight PyQt6 Monitoring GUI showing live feed, bounding boxes, state    |
|  [x] Structured Timestamped Log File Generation (JSON / CSV)                      |
|  [x] Local H.264 Video Recording & Local IP Network RTSP Streamer                 |
|  [x] Single Executable Offline Application Package                                |
+-----------------------------------------------------------------------------------+
|  SHOULD BUILD (High-Value Differentiators):                                       |
|  [x] ArUco-based Payload Rack 3D Coordinate Normalization Frame                   |
|  [x] Orientation-Agnostic Synthetic Data Augmentation (0G Tilt Simulation)        |
+-----------------------------------------------------------------------------------+
|  NICE TO HAVE (Time Permitting):                                                  |
|  [ ] Full 3D SMPL Body Mesh Visualization in GUI                                  |
+-----------------------------------------------------------------------------------+
|  DO NOT BUILD DURING HACKATHON (Out of Scope / Production Only):                  |
|  [ ] Flight-grade radiation-hardened hardware drivers                            |
|  [ ] Direct satellite downlink telemetry integration                              |
+-----------------------------------------------------------------------------------+

17. Live Demo Simulation Plan

To present a compelling and transparent demonstration to SIH judges without pretending to have real spaceflight hardware [PS]:

1. Physical Mockup Setup: The team will construct a physical experiment box setup using an outer container, a hinged lid, a red sub-box, a yellow sub-box, and two designated slots labeled Slot A and Slot B [PS].
2. Camera Positioning: Dual webcams (one overhead fixed payload camera, one side angle camera) will capture live interactions [ASSUMPTION].
3. Microgravity 0G Simulation:
   • To demonstrate orientation-agnostic tracking, the webcam will be mounted on a rotating arm or physically flipped upside down / 90 degrees sideways during live testing [ASSUMPTION].
   • ArUco corner markers on the experiment prop box will dynamically normalize the 3D frame, demonstrating that sequence validation remains 100% accurate regardless of camera tilt [INFERENCE].
4. Live Sequence & Anomaly Simulation:
   • Phase 1 (Normal Execution): Open box $\rightarrow$ Pick Red Box $\rightarrow$ Place Slot A $\rightarrow$ Pick Yellow Box $\rightarrow$ Place Slot B $\rightarrow$ Close Box. GUI shows green status, voice prompts next step [PS].
   • Phase 2 (Skipped Step Anomaly): Open box $\rightarrow$ Directly grab Yellow Box (Skipping Red Box). System instantly triggers voice alert: "Warning! Step 2 skipped. Please place Red Box in Slot A first." [PS].
   • Phase 3 (Out-of-Sequence Anomaly): Attempt to close box while Yellow Box is outside. System alerts: "Warning! Experiment incomplete. Yellow Box not placed in Slot B." [PS].
5. Network & Storage Verification: Demonstrate live RTSP video playback on a second laptop connected over local Wi-Fi/LAN, and show real-time writing of experiment_log.json [ASSUMPTION].

18. Implementation Roadmap

Targeting the official deadline of 30 September 2026 [PS]:

Phase 1: Dataset & Pipeline Setup (Sep 05 - Sep 12, 2026)
  ├── Build physical prop experiment box (Outer box, Red box, Yellow box, Slots A & B)
  ├── Record 1,000 multi-angle video frames (Normal + 0G tilted camera views)
  └── Annotate bounding boxes using CVAT / Roboflow

Phase 2: Core Perception & State Machine (Sep 13 - Sep 19, 2026)
  ├── Fine-tune YOLOv11-Nano on experiment dataset
  ├── Implement ArUco payload rack coordinate normalization matrix
  └── Build Python Finite State Automaton (FSA) step tracking engine

Phase 3: Interaction, Voice & Logging (Sep 20 - Sep 24, 2026)
  ├── Integrate local offline voice alert engine (Piper / Pyttsx3)
  ├── Develop structured JSON timestamped log writer
  └── Set up GStreamer/OpenCV local video recorder & RTSP IP streamer

Phase 4: GUI Dashboard & Executable Packaging (Sep 25 - Sep 28, 2026)
  ├── Develop PyQt6 monitoring dashboard (Live feed, bounding boxes, step history)
  ├── Freeze python environment into standalone offline executable using PyInstaller
  └── Conduct end-to-end air-gapped field tests (Zero internet connection)

Phase 5: Final Demo Validation & Submission (Sep 29 - Sep 30, 2026)
  ├── Run false-positive and false-negative anomaly stress tests
  └── Complete project dossier, codebase push, and SIH submission

19. Testing & Validation Plan

• Functional Testing: Verify that all 6 steps of the sample experiment (Box Open $\rightarrow$ Red to Slot A $\rightarrow$ Yellow to Slot B $\rightarrow$ Box Close) trigger correct state transitions [PS].
• Anomaly & False-Positive Testing: Test 50 deliberate error scenarios (e.g., hovering hand without grasping, skipping red box, placing yellow box in Slot A) to ensure state machine triggers voice alerts with 0% false step completions [INFERENCE].
• Orientation Invariance Testing: Rotate the ingest camera by 45°, 90°, and 180° relative to the experiment box. Validate that ArUco rack-relative coordinate normalization maintains >95% state detection accuracy [INFERENCE].
• Performance Testing: Benchmark pipeline latency on target edge laptop/Jetson. Requirement: Processing latency < 35 ms per frame (>= 28 FPS) at 1080p resolution [ASSUMPTION].
• Air-Gapped Offline Testing: Disable all Wi-Fi, Ethernet, and cloud connections. Ensure system executes, logs, alerts, and displays GUI with zero network calls [PS].

20. Risk Register

21. Cost & Scalability

Hackathon Prototype Build Cost

• Hardware / Props: Standard Laptop with GTX 1650/RTX 3050 or Jetson Nano ($0 - $150), USB Webcams ($20), Cardboard/Acrylic Prop Box with Red/Yellow paint ($10). Total: ~$30 - $180 [ASSUMPTION].
• Software & APIs: 100% Open Source (Python, PyTorch, YOLOv11, OpenCV, PyQt6, eSpeak). Total Software Cost: $0.00 [KNOWN].

Spaceflight Production Deployment Considerations

• Flight Hardware: Space-qualified, radiation-hardened edge compute board (e.g., Vorago ARM or SpaceMicro/BAE RAD750 / Spaceborne Computer node) [KNOWN].
• Optical Hardware: Flight-certified fixed payload cameras with radiation-tolerant CMOS sensors [KNOWN].
• Compliance: Rigorous software verification under NASA/ISRO IEEE 12207 space software standard [KNOWN].

22. Differentiation

23. Future Scope

1. Multi-Camera Payload Rack Fusion: Expanding perception layer to synthesize feeds from 3 fixed camera angles around the payload rack to eliminate single-view occlusions [INFERENCE].
2. Glove & Tool Fine-Grained Force Tracking: Integrating soft tactile glove sensor telemetry with computer vision to track delicate micro-manipulations in space biology experiments [RESEARCHED].
3. Automated Protocol JSON Generator: Building a drag-and-drop web tool for Earth-based Principal Investigators to convert scientific paper protocols into execution JSON state machines automatically [INFERENCE].

24. Assumptions & Open Items Log

Consolidated Assumptions [ASSUMPTION]

• [ASSUMPTION] ArUco fiducial markers placed on payload rack corners provide an ideal coordinate reference frame for 0G spatial normalization.
• [ASSUMPTION] YOLOv11-Nano fine-tuned on 1,500 local synthetic images provides sufficient detection confidence for prop boxes and hands.
• [ASSUMPTION] eSpeak-NG / Piper TTS provides intelligible offline voice synthesis without requiring heavy cloud models.
• [ASSUMPTION] PyQt6 desktop dashboard delivers low-CPU local rendering suitable for payload monitoring computers.

Consolidated Items Needing Validation [NEEDS VALIDATION]

• [NEEDS VALIDATION] Exact physical dimensions, optical reflectivity, and mounting camera specifications of official ISRO BAS payload racks.
• [NEEDS VALIDATION] Specific payload rack compute module specifications (ARM vs x86, available CUDA memory, thermal dissipation limit).

Final Recommendation

Build AstraCheck-HAR using the Hybrid Edge Architecture (Approach B). The student team should immediately construct a physical prop box, record multi-angle webcam training data, fine-tune YOLOv11-Nano, implement the deterministic Finite State Automaton sequence validator, and package the complete system into a PyQt6 offline desktop application with voice alerting and RTSP streaming [ASSUMPTION].

Recommended Stack

• Perception Engine: YOLOv11-Nano (PyTorch / TensorRT) + ArUco 3D Transform Matrix [ASSUMPTION].
• State Machine & Logic: Python Deterministic Finite State Automaton (FSA) [INFERENCE].
• Voice Subsystem: Piper TTS / eSpeak-NG (Offline C++ / Python) [ASSUMPTION].
• GUI Dashboard: PyQt6 + OpenCV Canvas Overlay [ASSUMPTION].
• Video & Streaming: GStreamer + OpenCV H.264 Local Writer & RTSP Server [ASSUMPTION].
• Executable Packaging: PyInstaller / ONNX Runtime C++ [ASSUMPTION].

Biggest Dependency

Generating an adequately annotated local synthetic dataset covering varied camera angles, hand occlusion states, and lighting conditions to train the YOLOv11 detector [PS].

Biggest Risk

False-positive step completions triggered by brief hand occlusions over prop boxes during multi-step experiments [INFERENCE].

Biggest Technical Challenge

Achieving true 0G orientation invariance by transforming 3D hand/body coordinates into payload rack-relative reference space in real time [PS].

First 5 Actions

1. Assemble the physical prop experiment box (outer container, lid, red sub-box, yellow sub-box, slots A & B) [PS].
2. Record 1,000 multi-angle, multi-orientation webcam frames replicating normal and out-of-sequence experiment steps [PS].
3. Annotate bounding boxes using CVAT/Roboflow and export to YOLOv11 PyTorch format [ASSUMPTION].
4. Implement the Python Finite State Automaton (FSA) state machine for the 6-step sample workflow [INFERENCE].
5. Integrate Piper/eSpeak offline TTS voice synthesis and test immediate alerting on step skipping [ASSUMPTION].

Validation Before Coding

Verify that camera-to-rack transformation using 4 corner ArUco fiducials accurately recalculates 3D hand distances when the webcam is physically inverted 180 degrees [INFERENCE].

Buildability Score /100

92 / 100 (Extremely feasible for a 4-6 student team within the available timeframe using standard webcams, open-source computer vision frameworks, and local edge hardware) [INFERENCE].

Confidence

HIGH (The solution explicitly satisfies all required SIH parameters, addresses zero-gravity orientation constraints, and operates entirely air-gapped without cloud dependencies) [INFERENCE].

Research Sources

1. Bharatiya Antariksh Station (BAS) Program Overview

   • URL: https://en.wikipedia.org/wiki/Bharatiya_Antariksh_Station [1.1]
   • Source Type: Public Knowledge / Official ISRO Mission Roadmap [1.1]
   • Supported Claim: Confirms BAS operational timeline, microgravity science research objectives, and Gaganyaan relationship [1.1, 1.5].
   • Reliability: High [1.1].

2. Understanding Human Actions in Microgravity Environments

   • URL: https://arxiv.org/html/2506.02845v1 [2.1]
   • Source Type: Academic Research Paper [2.1]
   • Supported Claim: Analyzes astronaut activity recognition in ISS and Tiangong space station cabins, highlighting microgravity pose variability [2.1, 2.2].
   • Reliability: High [2.1].

3. SAM 3D Body & Lightweight Human Mesh Recovery (HMR2.0)

   • URL: https://ai.meta.com/research/publications/sam-3d-body-robust-full-body-human-mesh-recovery/ [3.5]
   • Source Type: Computer Vision Research Publication [3.5]
   • Supported Claim: Demonstrates 3D human pose and mesh estimation techniques decoupled from camera ground planes [3.4, 3.5].
   • Reliability: High [3.5].

4. Procedural Task Guidance and Sequence Verification on Edge

   • URL: https://arxiv.org/html/2607.24770v1 [2.1]
   • Source Type: Computer Vision & Edge AI Paper [2.1]
   • Supported Claim: Proves that Finite State Machines (FSM) combined with temporal perception provide reliable procedural step error detection on edge devices [2.1, 2.3].
   • Reliability: High [2.1].

5. Real-Time Egocentric Hand-Object Interaction Detection

   • URL: https://arxiv.org/html/2507.13326v1 [4.8]
   • Source Type: Computer Vision Research Paper [4.8]
   • Supported Claim: Validates real-time detection of hand-object contacts and active object states on low-power edge platforms [4.8, 3.1].
   • Reliability: High [4.8].