AI x IoT Project 3 : Real-Time Sign Language to Text Display using ESP32 and LCD

Welcome, Innovators! In this project, we'll bridge the gap between the physical and digital worlds by creating a system that recognizes simple sign language gestures using your computer's webcam and displays the translated text on an LCD screen powered by an ESP32.

This project is a fantastic introduction to computer vision with Python, microcontroller communication, and hardware interfacing. Let's get started!

How It Works: A High-Level View

The project has two main parts that talk to each other over a simple USB connection:

1. The Brain (Python on your PC): A Python script uses your webcam to "see." With the help of Google's MediaPipe library, it detects your hand and its landmarks (the joints of your fingers). It then runs some logic to recognize a few basic gestures like "Hello," "Peace," and "Thumbs Up."
2. The Display (ESP32 & LCD): The ESP32 microcontroller is constantly listening for messages from your computer. When the Python script recognizes a gesture, it sends the corresponding word (e.g., "Hello") over the USB serial port. The ESP32 receives this word and immediately displays it on the 16x2 LCD screen.

Required Components (from your Innovator Box)

• ESP32 Development Board
• 16x2 I2C LCD Display Module
• 4 x Male-to-Female or Female-to-Female Jumper Wires
• Micro-USB Cable

Part 1: Hardware Assembly & Wiring

The I2C LCD module is great because it only requires four wires to connect, making the setup clean and simple.

Wiring Instructions:

Connect the ESP32 to the I2C LCD module as follows. The default I2C pins on most ESP32 boards are GPIO 22 (SCL) and GPIO 21 (SDA).

Visual Guide:

Ensure your connections are firm. Once wired, you're ready to set up the software.

Part 2: Setting up the ESP32

We'll use the Arduino IDE to program the ESP32.

Step 1: Configure Arduino IDE for ESP32

If you haven't already, you need to teach the Arduino IDE how to work with ESP32 boards.

1. Open Arduino IDE.
2. Go to File > Preferences.
3. In the "Additional Boards Manager URLs" field, paste this URL:

4. https://raw.githubusercontent.com/espressif/arduino-esp32/gh-pages/package_esp32_index.json

5. Go to Tools > Board > Boards Manager....
6. Search for "esp32" and install the package by Espressif Systems.

Step 2: Install Required Libraries

This project needs two libraries to function: one to control the LCD and another to handle JSON data.

1. Go to Sketch > Include Library > Manage Libraries....
2. Search for and install LiquidCrystal_I2C by Frank de Brabander.
3. Search for and install ArduinoJson by Benoit Blanchon.

Step 3: Upload the Code to the ESP32

--------------------------------------------------------------------------------------

// ESP32 LCD Display Code for Sign Language Translation (Serial Version)
// Hardware: ESP32 + 16x2 LCD with I2C (PCF8574)
// Communication: USB Serial (no WiFi needed)

#include <LiquidCrystal_I2C.h>
#include <Wire.h>
#include <ArduinoJson.h>

// LCD configuration
LiquidCrystal_I2C lcd(0x27, 16, 2); // I2C address, columns, rows

// Display variables
String currentText = "";
unsigned long lastUpdateTime = 0;
const unsigned long displayDuration = 3000; // Display text for 3 seconds

void setup() {
  Serial.begin(115200);
  delay(1000);
  
  Serial.println("Starting ESP32 LCD Display...");
  
  // Initialize I2C
  Wire.begin(21, 22); // SDA=21, SCL=22 for ESP32
  
  // Initialize LCD
  lcd.init();
  lcd.backlight();
  lcd.setCursor(0, 0);
  lcd.print("Sign Language");
  lcd.setCursor(0, 1);
  lcd.print("Translator");
  delay(2000);
  lcd.clear();
  lcd.setCursor(0, 0);
  lcd.print("Ready for Serial");
  lcd.setCursor(0, 1);
  lcd.print("Commands");
  
  Serial.println("LCD initialized");
  Serial.println("Ready to receive commands via Serial");
  Serial.println("Send JSON: {\"text\":\"Your Message\"}");
  Serial.println("Or simple text commands:");
  Serial.println("  DISPLAY:Hello World");
  Serial.println("  CLEAR");
  Serial.println("  STATUS");
  Serial.println("=====================================");
}

void loop() {
  // Check for serial input
  if (Serial.available()) {
    String command = Serial.readStringUntil('\n');
    command.trim(); // Remove whitespace
    
    if (command.length() > 0) {
      processCommand(command);
    }
  }
  
  // Clear display after display duration
  if (currentText != "" && millis() - lastUpdateTime > displayDuration) {
    lcd.clear();
    lcd.setCursor(0, 0);
    lcd.print("Ready for Serial");
    lcd.setCursor(0, 1);
    lcd.print("Commands");
    currentText = "";
  }
}

void processCommand(String command) {
  Serial.println("Received: " + command);
  
  // Check if it's JSON format
  if (command.startsWith("{") && command.endsWith("}")) {
    // Parse JSON
    StaticJsonDocument<200> doc;
    DeserializationError error = deserializeJson(doc, command);
    
    if (!error) {
      String text = doc["text"];
      if (text.length() > 0) {
        displayText(text);
        Serial.println("JSON Command - Displayed: " + text);
      } else {
        Serial.println("Error: No 'text' field in JSON");
      }
    } else {
      Serial.println("Error: Invalid JSON format");
    }
  }
  // Check for simple commands
  else if (command.startsWith("DISPLAY:")) {
    String text = command.substring(8); // Remove "DISPLAY:" prefix
    displayText(text);
    Serial.println("Simple Command - Displayed: " + text);
  }
  else if (command == "CLEAR") {
    lcd.clear();
    lcd.setCursor(0, 0);
    lcd.print("Ready for Serial");
    lcd.setCursor(0, 1);
    lcd.print("Commands");
    currentText = "";
    Serial.println("Display cleared");
  }
  else if (command == "STATUS") {
    Serial.println("Status: Online");
    Serial.println("Current Text: " + currentText);
    Serial.println("Free Memory: " + String(ESP.getFreeHeap()));
  }
  else {
    // Treat any other text as display command
    displayText(command);
    Serial.println("Direct Text - Displayed: " + command);
  }
}

void displayText(String text) {
  lcd.clear();
  currentText = text;
  lastUpdateTime = millis();
  
  Serial.println("Displaying on LCD: " + text);
  
  // Handle text longer than 16 characters
  if (text.length() <= 16) {
    // Single line display
    lcd.setCursor(0, 0);
    lcd.print(text);
  } else {
    // Split text across two lines
    String line1 = text.substring(0, 16);
    String line2 = text.substring(16);
    
    // If line2 is longer than 16 characters, truncate it
    if (line2.length() > 16) {
      line2 = line2.substring(0, 13) + "...";
    }
    
    lcd.setCursor(0, 0);
    lcd.print(line1);
    lcd.setCursor(0, 1);
    lcd.print(line2);
  }
  
  // Show on second line how long text will be displayed
  if (text.length() <= 16) {
    lcd.setCursor(0, 1);
    lcd.print("(3sec)");
  }
}

--------------------------------------------------------------------------------------

1. Connect your ESP32 to your computer using the Micro-USB cable.
2. Open the Arduino IDE and paste the ESP32 code provided.
3. Go to Tools > Board and select a generic "ESP32 Dev Module" or the specific model you have.
4. Go to Tools > Port and select the COM port your ESP32 is connected to (e.g., COM3 on Windows, /dev/cu.usbserial-XXXX on Mac).
5. Click the Upload button (the right-pointing arrow). You might need to hold the "BOOT" button on your ESP32 when the IDE says "Connecting...".

After uploading, the LCD should light up and display a welcome message, then "Ready for Serial Commands".

Part 3: Setting up the Python Environment

Now, let's prepare your computer to run the computer vision part of the project.

Step 1: Install Python

If you don't have Python, download and install it from python.org. Make sure to check the box that says "Add Python to PATH" during installation.

Step 2: Install Required Python Libraries

Open your computer's terminal or command prompt and install the necessary libraries using pip:

pip install opencv-python mediapipe numpy pyserial

Step 3: Prepare the Python Script

--------------------------------------------------------------------------------------

# Sign Language Detection and Translation System (Serial Version)
# Computer Vision Part (Python) - Communicates via Serial

import cv2
import mediapipe as mp
import numpy as np
import serial
import json
import time
import serial.tools.list_ports

class SignLanguageDetector:
    def __init__(self, serial_port=None, baud_rate=115200):
        self.mp_hands = mp.solutions.hands
        self.hands = self.mp_hands.Hands(
            static_image_mode=False,
            max_num_hands=1,
            min_detection_confidence=0.7,
            min_tracking_confidence=0.5
        )
        self.mp_draw = mp.solutions.drawing_utils
        
        # Serial connection
        self.serial_port = serial_port
        self.baud_rate = baud_rate
        self.ser = None
        
        # Simple gesture recognition (you can expand this)
        self.gesture_buffer = []
        self.last_gesture_time = time.time()
        
        # Define some basic gestures based on landmarks
        self.gestures = {
            'hello': 'Hello',
            'thanks': 'Thank You',
            'yes': 'Yes',
            'no': 'No',
            'peace': 'Peace',
            'ok': 'OK',
            'one': 'One',
            'fist': 'Closed Fist',
            'bye': 'Bye',
            'thumbs_up': 'Thumbs Up'
        }
        
    def list_serial_ports(self):
        """List available serial ports"""
        ports = serial.tools.list_ports.comports()
        print("\nAvailable Serial Ports:")
        for i, port in enumerate(ports):
            print(f"{i}: {port.device} - {port.description}")
        return ports
    
    def connect_serial(self):
        """Connect to ESP32 via serial"""
        if self.serial_port is None:
            # Auto-detect or let user choose
            ports = self.list_serial_ports()
            if not ports:
                print("No serial ports found!")
                return False
            
            if len(ports) == 1:
                self.serial_port = ports[0].device
                print(f"Auto-selected: {self.serial_port}")
            else:
                try:
                    choice = int(input(f"Select port (0-{len(ports)-1}): "))
                    self.serial_port = ports[choice].device
                except (ValueError, IndexError):
                    print("Invalid selection!")
                    return False
        
        try:
            self.ser = serial.Serial(self.serial_port, self.baud_rate, timeout=1)
            time.sleep(2)  # Wait for Arduino to reset
            print(f"Connected to {self.serial_port} at {self.baud_rate} baud")
            
            # Send test command
            self.send_to_esp32("System Connected!")
            return True
        except serial.SerialException as e:
            print(f"Error connecting to serial port: {e}")
            return False
    
    def calculate_angle(self, p1, p2, p3):
        """Calculate angle between three points"""
        v1 = np.array([p1[0] - p2[0], p1[1] - p2[1]])
        v2 = np.array([p3[0] - p2[0], p3[1] - p2[1]])
        
        cos_angle = np.dot(v1, v2) / (np.linalg.norm(v1) * np.linalg.norm(v2))
        angle = np.arccos(np.clip(cos_angle, -1.0, 1.0))
        return np.degrees(angle)
    
    def recognize_gesture(self, landmarks):
        """Basic gesture recognition based on hand landmarks"""
        if not landmarks:
            return None
            
        # Get key landmark positions
        thumb_tip = landmarks[4]
        thumb_ip = landmarks[3]
        index_tip = landmarks[8]
        index_pip = landmarks[6]
        middle_tip = landmarks[12]
        middle_pip = landmarks[10]
        ring_tip = landmarks[16]
        ring_pip = landmarks[14]
        pinky_tip = landmarks[20]
        pinky_pip = landmarks[18]
        
        # Simple gesture detection logic
        fingers_up = []
        
        # Thumb (check if extended)
        if thumb_tip[0] > thumb_ip[0]:  # Right hand - thumb extended to the right
            fingers_up.append(1)
        else:
            fingers_up.append(0)
            
        # Other fingers (check if tips are above PIP joints)
        for tip, pip in [(index_tip, index_pip), (middle_tip, middle_pip), 
                        (ring_tip, ring_pip), (pinky_tip, pinky_pip)]:
            if tip[1] < pip[1]:  # Tip is above PIP (finger extended)
                fingers_up.append(1)
            else:
                fingers_up.append(0)
        
        # Gesture classification based on finger count
        total_fingers = sum(fingers_up)
        
        # Thumbs up: Only thumb extended, other fingers closed
        if fingers_up[0] == 1 and sum(fingers_up[1:]) == 0:
            return 'thumbs_up'
        elif total_fingers == 5:  # All fingers extended
            return 'hello'
        elif total_fingers == 4:  # Four fingers extended (any 4)
            return 'bye'
        elif total_fingers == 2 and fingers_up[1] == 1 and fingers_up[2] == 1:  # Peace sign
            return 'peace'
        elif total_fingers == 1 and fingers_up[1] == 1:  # Index finger only
            return 'one'
        elif total_fingers == 0:  # All fingers closed
            return 'fist'
        elif total_fingers == 3 and fingers_up[0] == 1 and fingers_up[1] == 1 and fingers_up[4] == 1:  # OK sign
            return 'ok'
        
        return None
    
    def send_to_esp32(self, text):
        """Send text to ESP32 via serial"""
        if self.ser and self.ser.is_open:
            try:
                # Send as JSON format
                message = json.dumps({"text": text}) + "\n"
                self.ser.write(message.encode())
                self.ser.flush()
                print(f"Sent to ESP32: {text}")
                
                # Read response (optional)
                time.sleep(0.1)  # Small delay for ESP32 to process
                if self.ser.in_waiting:
                    response = self.ser.readline().decode().strip()
                    if response:
                        print(f"ESP32 Response: {response}")
                        
            except serial.SerialException as e:
                print(f"Error sending to ESP32: {e}")
        else:
            print("Serial connection not available")
    
    def send_command(self, command):
        """Send a command to ESP32"""
        if self.ser and self.ser.is_open:
            try:
                self.ser.write((command + "\n").encode())
                self.ser.flush()
                print(f"Sent command: {command}")
                
                # Read response
                time.sleep(0.1)
                if self.ser.in_waiting:
                    response = self.ser.readline().decode().strip()
                    if response:
                        print(f"ESP32: {response}")
                        
            except serial.SerialException as e:
                print(f"Error sending command: {e}")
    
    def run(self):
        """Main detection loop"""
        # Connect to serial first
        if not self.connect_serial():
            print("Failed to connect to ESP32. Exiting.")
            return
        
        cap = cv2.VideoCapture(0)
        cap.set(cv2.CAP_PROP_FRAME_WIDTH, 640)
        cap.set(cv2.CAP_PROP_FRAME_HEIGHT, 480)
        
        print("Starting sign language detection...")
        print("Press 'q' to quit")
        print("Press 'c' to clear display")
        print("Press 's' for status")
        
        last_sent_gesture = None
        gesture_hold_time = 0
        
        while True:
            ret, frame = cap.read()
            if not ret:
                break
                
            # Flip frame horizontally for mirror effect
            frame = cv2.flip(frame, 1)
            rgb_frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
            
            # Process the frame
            results = self.hands.process(rgb_frame)
            
            if results.multi_hand_landmarks:
                for hand_landmarks in results.multi_hand_landmarks:
                    # Draw landmarks
                    self.mp_draw.draw_landmarks(
                        frame, hand_landmarks, self.mp_hands.HAND_CONNECTIONS)
                    
                    # Extract landmark coordinates
                    landmarks = []
                    for lm in hand_landmarks.landmark:
                        landmarks.append([lm.x, lm.y])
                    
                    # Recognize gesture
                    gesture = self.recognize_gesture(landmarks)
                    
                    if gesture:
                        cv2.putText(frame, f"Gesture: {gesture}", (10, 30), 
                                  cv2.FONT_HERSHEY_SIMPLEX, 1, (0, 255, 0), 2)
                        
                        # Send to ESP32 if gesture is stable
                        if gesture == last_sent_gesture:
                            gesture_hold_time += 1
                        else:
                            gesture_hold_time = 0
                            last_sent_gesture = gesture
                        
                        # Send after holding gesture for ~1 second (30 frames at 30fps)
                        if gesture_hold_time > 30 and gesture in self.gestures:
                            text_to_send = self.gestures[gesture]
                            self.send_to_esp32(text_to_send)
                            gesture_hold_time = -60  # Prevent spam
            
            # Display frame
            cv2.imshow('Sign Language Detection', frame)
            
            # Handle keyboard input
            key = cv2.waitKey(1) & 0xFF
            if key == ord('q'):
                break
            elif key == ord('c'):
                self.send_command("CLEAR")
            elif key == ord('s'):
                self.send_command("STATUS")
        
        # Cleanup
        cap.release()
        cv2.destroyAllWindows()
        if self.ser and self.ser.is_open:
            self.ser.close()
            print("Serial connection closed")

if __name__ == "__main__":
    # Initialize detector
    detector = SignLanguageDetector(serial_port="/dev/cu.usbserial-0001")  # Mac/Linux
    # detector = SignLanguageDetector(serial_port="COM3")  # Windows
    # detector = SignLanguageDetector()  # Auto-detect port
    detector.run()

--------------------------------------------------------------------------------------

1. Save the provided Python code as a file named sign_detector.py.
2. Crucially, you must tell the script which serial port your ESP32 is on. Find the correct port name from the Arduino IDE (Tools > Port).
3. Open sign_detector.py and find the line at the very bottom:
   Python

   # detector = SignLanguageDetector(serial_port="/dev/cu.usbserial-0001")  # Mac/Linux
   # detector = SignLanguageDetector(serial_port="COM3")  # Windows
   detector = SignLanguageDetector()  # Auto-detect port
   

4. For best results, specify your port directly. Uncomment the line for your operating system and replace the example port with your actual port. For example, if your port is COM5 on Windows, change it to:
   Python

   # detector = SignLanguageDetector(serial_port="/dev/cu.usbserial-0001")  # Mac/Linux
   detector = SignLanguageDetector(serial_port="COM5")  # Windows
   # detector = SignLanguageDetector()  # Auto-detect port
   

   
   Part 4: Run the Project!

• You're all set! It's time to bring your creation to life.
• Make sure your ESP32 is plugged into your computer and the LCD is on.
• Open a terminal or command prompt, navigate to the folder where you saved sign_detector.py.
• Run the script with the command:
• python sign_detector.py
• A window should pop up showing your webcam feed. Your hand will have a skeleton drawn on it when detected.
• Now, try making one of the pre-programmed signs in front of the camera. Hold the sign steady for about a second.
• Watch the magic happen! The recognized word should appear on your 16x2 LCD display.

Supported Gestures:

• Open Hand (5 fingers up): "Hello"
• Peace Sign (index & middle finger): "Peace"
• Thumbs Up: "Thumbs Up"
• Closed Fist: "Closed Fist"
• "One" (index finger up): "One"

Understanding the Code

ESP32 Code (.ino)

• setup(): Initializes the Serial communication, I2C, and the LCD screen. It prints a startup message to both the LCD and the Serial Monitor.
• loop(): This is the main cycle. It constantly checks if any data has arrived from the computer via the USB port (Serial.available()). It also has a timer that clears the message on the LCD after 3 seconds.
• processCommand(String command): This function is the command interpreter. It checks if the received text is in JSON format ({"text":"Hello"}), a simple command (DISPLAY:Hello), or just plain text. It then calls displayText to show it on the screen.
• displayText(String text): This function handles the actual printing to the LCD. It clears the screen, prints the new text, and can even split text longer than 16 characters onto two lines.

Python Code (.py)

• class SignLanguageDetector: This class bundles all the functionality.
• connect_serial(): Tries to connect to the ESP32. If you don't specify a port, it cleverly lists the available ones and asks you to choose.
• recognize_gesture(landmarks): This is the core of the gesture logic. It gets the coordinates of all hand landmarks from MediaPipe. It then uses a simple set of rules (e.g., "is the thumb tip to the right of the thumb knuckle?") to determine how many fingers are up and classifies the gesture.
• send_to_esp32(text): This function takes a string, wraps it in a JSON format like {"text": "Hello"}, and sends it over the serial connection to the ESP32.
• run(): The main loop that captures video from the webcam, processes each frame to find hands, calls the recognition logic, and sends the result to the ESP32. It includes a small delay to prevent spamming the LCD with the same gesture repeatedly.
•  

Troubleshooting

• LCD is blank or shows garbled characters:
  • Check your wiring! SDA and SCL might be swapped.
  • The I2C address might be different. The code uses 0x27. Some modules use 0x3F. You can run an "I2C Scanner" sketch in Arduino to find the correct address.
• Python script fails with a "Port not found" error:
  • Make sure you selected the correct COM port in the Python script.
  • Ensure no other program (like the Arduino Serial Monitor) is using the port at the same time. Close the Serial Monitor before running the Python script.
• Gestures are not being detected:
  • Make sure your hand is well-lit and fully visible in the camera frame.
  • The current gesture recognizer is very basic. It works best with a plain background.

1. Next Steps: Innovate Further!
   
2. This is just the beginning. Why not try to:
3. Add More Gestures: Modify the recognize_gesture function in Python to recognize more signs. You could try recognizing the numbers 2, 3, and 4.
4. Build Full Words: Create a system where you can spell out words letter by letter.
5. Go Wireless: Use the ESP32's WiFi capabilities to send the text from Python over your network instead of a USB cable.
6. Train a Real AI Model: For higher accuracy, you could train a custom machine learning model using TensorFlow or PyTorch to recognize a much wider vocabulary of signs.
   
   Happy building!