DIY 3D Hand Controller Using A Webcam And Scripting

October 25, 2024 by Heidi Ulrich 3 Comments

Are you ready to elevate your interactive possibilities without breaking the bank? If so, explore [Caio Bassetti]’s tutorial on creating a full 3D hand controller using only a webcam, MediaPipe Hands, and Three.js. This hack lets you transform a 2D screen into a fully interactive 3D scene—all with your hand movements. If you’re passionate about low-cost, accessible tech, try this yourself – not much else is needed but a webcam and a browser!

The magic of the project lies in using MediaPipe Hands to track key points on your hand, such as the middle finger and wrist, to calculate depth and positioning. Using clever Three.js tricks, the elements can be controlled on a 3D axis. This setup creates a responsive virtual controller, interpreting hand gestures for intuitive movement in the 3D space. The hack also implements a closed-fist gesture to grab and drag objects and detects collisions to add interactivity. It’s a simple, practical build and it performs reliably in most browsers.

For more on this innovation or other exciting DIY hand-tracking projects, browse our archive on gesture control projects, or check out the full article on Codrops. With tools such as MediaPipe and Three.js, turning ideas into reality gets more accessible than ever.

Gesture-Controlled Robot Arm Is A Nifty Educational Build

November 29, 2023 by Lewin Day 2 Comments

Traditionally, robot arms have been controlled either by joysticks, buttons, or very carefully programmed routines. However, for [Narongporn Laosrisin’s] homebrew build, they decided to go with gesture control instead.

The MeArm robotic arm is built using laser cut acrylic parts, and can be had in a kit if so desired. It features four servo motors, charged with rotating the arm’s base, pushing the arm forwards and backwards, up and down, and actuating its gripper. The servos are under the command of a micro:bit microcontroller board, which itself receives signals from a second micro:bit which is strapped to the human wishing to control the arm. The second micro:bit detects gestures with its accelerometer, and then sends the relevant commands to the robotic arm’s micro:bit over its built-in radio link. The arm controller then commands the servos to execute the maneuver.

It may be a small robotic arm that doesn’t have the capacity to lift much, but that’s not the point. This project is a great way to teach students how to program microcontrollers, work with sensor inputs, and just generally how to solve engineering puzzles. To that end, it looks like [Narongporn] has a great project on hand for teaching their students. Video after the break.

Continue reading “Gesture-Controlled Robot Arm Is A Nifty Educational Build” →

Watch Hides Gesture Controls In Wristband

June 25, 2023 by Bryan Cockfield No comments

Over the last five to ten years, smart watches have become fairly ubiquitous, with the Apple Watch being among the most prominent of them. Not everyone wants or needs all of the capabilities of these devices, though; plenty are still opting for simpler devices which only have a few functions built into them. [Josh] has been working on one of these devices but takes a major design cue from their smart counterparts with the addition of gesture controls for the watch built into the wristband instead of relying on a more traditional button interface.

The watch hosts most of the functionality of a non-smart digital watch, with a timer, alarms, and a stop watch built-in. To change the time or access any of these functions, a combination of resistive and capacitive touch sensors are built into the wristband near the watch face. The combination of sensors aims to use the benefits of either type of sensor, with the capacitive sensors being used for precision and gesture recognition and the resistive sensors being used for pressure sensitivity. Placing these sensors in the band instead of the watch face improves visibility as well, since the screen won’t be obscured by the user touching the screen.

[Josh] originally intended this type of watch to be used for those with prosthetics or other disabilities which would limit the ability to use standard watch buttons or interact with a touch screen on the watch face itself. The device is working quite well as can be seen in the video linked below, but is still in the prototyping phase and under active development. For finishing up the final versions, we’d recommend taking a look at the design of these open-source smart watches for their high quality fit and finish.

Continue reading “Watch Hides Gesture Controls In Wristband” →

Low-Cost Computer Gesture Control With An I2C Sensor

November 12, 2021 by Tom Nardi 6 Comments

Controlling your computer with a wave of the hand seems like something from science fiction, and for good reason. From Minority Report to Iron Man, we’ve seen plenty of famous actors controlling their high-tech computer systems by wildly gesticulating in the air. Meanwhile, we’re all stuck using keyboards and mice like a bunch of chumps.

But it doesn’t have to be that way. As [Norbert Zare] demonstrates in his latest project, you can actually achieve some fairly impressive gesture control on your computer using a $10 USD PAJ7620U2 sensor. Well not just the sensor, of course. You need some way to convert the output from the I2C-enabled sensor into something your computer will understand, which is where the microcontroller comes in.

Looking through the provided source code, you can see just how easy it is to talk to the PAJ7620U2. With nothing more exotic than a switch case statement, [Norbert] is able to pick up on the gesture flags coming from the sensor. From there, it’s just a matter of using the Arduino Keyboard library to fire off the appropriate keycodes. If you’re looking to recreate this we’d go with a microcontroller that supports native USB, but technically this could be done on pretty much any Arduino. In fact, in this case he’s actually using the ATtiny85-based Digispark.

This actually isn’t the first time we’ve seen somebody use a similar sensor to pull off low-cost gesture control, but so far, none of these projects have really taken off. It seems like it works well enough in the video after the break, but looks can be deceiving. Have any Hackaday readers actually tried to use one of these modules for their day-to-day futuristic computing?

Continue reading “Low-Cost Computer Gesture Control With An I2C Sensor” →

Groovin’ With A Gesture-Controlled MP3 Player

November 6, 2021 by Adam Zeloof 1 Comment

Touchscreens are great, but they’re not always the perfect solution. Trying to operate one with gloves on (even alleged “touchscreen-friendly” ones) can be cumbersome at best, and if the screen is on a publicly-shared device, such as a checkout kiosk it can easily become a home for bacteria, viruses and all sorts of other nasty stuff.

That’s what [Norbert Zare] was thinking when he built his gesture-controlled MP3 player. It uses a PAJ7620U2 gesture sensor to register a few intuitive hand motions including finger twirls to control the volume, hand swipes to skip forward and backwards, and a flat hand to play and pause the song. It even has a motorized knob and cute cutout music notes that move to provide some visual feedback for the gestures, which you can see in-action in the video below. If this seems familiar, it’s because on Tuesday we took a look at the camera-based, glance-to-skip-tracks controller he built.

To actually play some music, he gutted an old MP3 player and hooked the solder pads from the control buttons up to an Arduino, which reads gesture information from the sensor and emulates the MP3 player’s buttons by setting the appropriate pins to HIGH and LOW. Finally, he topped the whole thing off with an LCD screen and a case.

The great thing about [Norbert]’s approach is that it isn’t just limited to an MP3 player — it can be extended to replace the buttons on pretty much any device. Because the Arduino only needs to be connected to the button inputs of the device, it should be relatively easy to adapt most existing tactile interfaces to be touch-free. Paired with this gesture-tracking macro keyboard we saw earlier in the year, the days of actually having to touch our tech may soon be behind us.

Continue reading “Groovin’ With A Gesture-Controlled MP3 Player” →

Hedgehog Gesture Sensor Built With Cheap Time-of-Flight Modules

November 2, 2021 by Lewin Day 6 Comments

Time-of-flight sensors used to be expensive obscurities, capable of measuring the travel time of photons themselves and often used for tracking purposes. However, the technology is cheaper now, such that [jean.perardel] has used TOF sensors to build a useful and affordable gesture-tracking system.

The system relies on four VL53L1X time of flight sensors, which have a 16×16 scanning array and communicate over the I2C bus. Controlling the show is an Arduino MKR1010, though the project should be achievable with a range of other microcontrollers, too.

The device is built into a cute hedgehog-like form factor, with an LCD screen acting as the face. It displays facial expressions which show how the system is interpreting and responding to gestures. It gives the project lots of personality, which makes using the system more fun. Gestures from the system can be used to send keystrokes over USB, control relays or servos, or even fire IR signals to control TVs and other hardware.

It actually seems like a useful gesture control interface, one that could become a useful part of a workstation setup. We’ve seen gesture controls put to other uses too, like controlling robot arms. Video after the break.

Continue reading “Hedgehog Gesture Sensor Built With Cheap Time-of-Flight Modules” →

Gesture-Detecting Macro Keyboard Knows What You Want

June 13, 2021 by Kristina Panos 7 Comments

[jakkra] bought a couple of capacitive touchpads from a Kickstarter a few years ago and recently got around to using them in a project. And what a project it is: this super macro pad combines two touchpads with a 6-pack of regular switches for a deluxe gesture-sensing input device.

Inside is an ESP32 running TensorFlow Lite to read in the gestures from the two touchpads. The pad at the top is a volume slider, and the square touchpad is the main input and is used in conjunction with the buttons to run AutoHotKey scripts within certain programs. [jakkra] can easily run git commands and more with a handful of simple gestures. The gestures all seem like natural choices to us: > for next media track, ∧ to push the current branch and ∨ to fetch and pull the current branch, s for git status, l for git log, and the one that sounds really useful to us — draw a C to get a notification that lists all the COM ports. One of the switches is dedicated to Bluetooth pairing and navigating menus on the OLED screen.

We love the combination of inputs here and think this looks great, especially with the double touchpad design. Be sure to check out the gesture demo gif after the break.

Gesture input seems well-suited to those who compute on the go, and a gesture glove feels like the perfect fit.

Continue reading “Gesture-Detecting Macro Keyboard Knows What You Want” →