(WARNING: TECHNICALLY HEAVY CONTENT)
It has been almost a month since I last blogged. Now that our Hu Shu Special School CIP has come to an end, we have been working day and night on our projects. This post shall focus on our project updates rather than our day to day life in Hangzhou, which has become pretty mundane at this point of time!
So, I’m working on a pet gadget, which we have named the HeartCam. What does it do?
It’s a device that continuously monitors a dog’s heart rate and captures a photo from the front facing camera found on the device whenever it senses that the dog get’s excited. The photo then gets automatically uploaded to the cloud, where owners can view and share the contents.
Sounds simple enough right?
Boy were we wrong.
The following illustrates some of the obstacles we faced thus far.
1) Choosing the right hardware platform
There’s just a large pool of possible platforms for us to work on, each with its own pros and cons.
Should we work on the Arduino platform? If so, there’s like a gazillion Arduino boards to choose from.
How about Intel Edison boards?
FPGA (field programmable gate array) boards instead?
Raspberry Pi? Certainly better at doing CPU heavy task, but not the best when it comes to power management.
Arduino boards are good for interfacing with multiple sensors and low power management.
Linux based boards are best for fast image processing and handling. It takes up to 30 seconds for an Arduino Uno to save a low res picture to an SD card!
After spending a huge portion of our research phase deciding what’s best, we eventually went for the Arduino Yún, for it offers the best of both worlds. (fun fact, Yun means cloud in Chinese). I can program a python script on the Linux board and bridge it over to the Arduino processor, where all my sensors are interfaced. What more, its small and compact! How cool is that? =)
The Arduino Yún is a microcontroller board based on the ATmega32u4 and the Atheros AR9331. The Atheros processor supports a Linux distribution based on OpenWrt named OpenWrt-Yun. The board has built-in Ethernet and WiFi support, a USB-A port, micro-SD card slot, 20 digital input/output pins (of which 7 can be used as PWM outputs and 12 as analog inputs), a 16 MHz crystal oscillator, a micro USB connection, an ICSP header, and a 3 reset buttons.
The Yún distinguishes itself from other Arduino boards in that it can communicate with the Linux distribution onboard, offering a powerful networked computer with the ease of Arduino. In addition to Linux commands like cURL, you can write your own shell and python scripts for robust interactions. The Yún is similar to the Leonardo in that the ATmega32u4 has built-in USB communication, eliminating the need for a secondary processor. This allows the Yún to appear to a connected computer as a mouse and keyboard, in addition to a virtual (CDC) serial / COM port.
The Bridge library facilitates communication between the two processors, giving Arduino sketches the ability to run shell scripts, communicate with network interfaces, and receive information from the AR9331 processor. The USB host, network interfaces and SD card are not connected to the 32U4, but the AR9331, and the Bridge library also enables the Arduino to interface with those peripherals.
2) Choosing the right sensors
Key components needed:
1) Arduino Yún board
2) Camera module
3) Heart Rate Sensor
4) SD card
So we’ve narrowed on the Arduino board. Next step, finding a good camera module. Most of the common camera modules in the market were CMOS based, such as the OV7670, which only had image resolutions up to 728 x 488. That’s like going back in time to 2001, where the first camera phones came out. The horror. Even then, a quick google search, and it was evident that there was alot of problems interfacing these “sub par modules” to work properly.
Hence, we decided to tackle the problem from another angle. How about using an USB webcam (even a HD one is pretty cheap in the market), and interfacing it with the Linux board for communication using the UVC protocol? We explored the idea and it proved to be feasible. Hence, we settled for a cheap webcam from taobao.com decent 720p webcam, the Logitech C270 (refer to point 3 to find out more).
We bought the pulse rate sensor from pulsesensor.com to start with, considering the affordable price point. The Pulse Sensor measures subtle changes in light from expansion of the capillary blood vessels to sense our heartbeats. However, after playing with the sensor for a couple of days, we realized that it was not reliable enough as the placement of the sensor had to be reaaaaaaallly precise! A little off and the readings go wayward. Furthermore, it would be impractical and challenging to incorporate this on a dog when we weren’t even getting good readings from a human.
The chest strap wirelessly transmits data to the heart rate monitor receiver in the 5Ghz frequency range, which then converts the ECG signals into easy-to-use heart rate data. It then implements a sophisticated algorithm for computing an average heart rate even with noisy or intermittent data from the transmitter.
3) Working with the hardwares
Working with the hardwares proved to be more challenging than I expected it to be. Firstly, I had to learn all about the terminal console and nano GUI as I never had any Linux experience prior to this project. I broke down the project into smaller stages for easier troubleshooting.
The first obstacle I ran into was having insufficient onboard memory to install the Linux packages from Github. The solution? Moving the onboard storage to an external SD card and running the Linux processes from the SD card instead. (think of it as something similar to running your desktop computer off an external hard disk)
After the board was all set up and packages installed, the next step was to test the webcam, from the terminal console. Note: At this point of time, we were still using the cheap taobao-ed webcam. Guess what? The images taken were all corrupted. After troubleshooting the problem, we started playing around with different camera settings such as the frame-rate, shutter speed and image resolution. The results were better provided they were of a lower image quality. When taken in high res, the corruption was still prevalent. It was then found that the cheaper webcams in the market have “poorer usb buffering” along the cables, which might cause images to be corrupted.
Not wanting to take any chances, we then ordered a more decent and reputable webcam, the Logitech C270. The images were still corrupted by default, but after some fine tuning and tweaking, we were able to get high resolution photos without any corruption! =)
The next logical step was to link the Dropbox account to the Arduino for automatic image uploading. After registering for a developer account with Dropbox, we proceeded to create a webapp and obtain the unique developer token id needed to authorize our automated python script. After setting it all up, it was time for the moment of truth. Testing it out! As expected, it was not a fairy tale ending and we were faced with a SSL certificate error when connecting to the server, resulting in a failed upload. The next 2 days was spent in frustration as we tried fixing the problem. As this was a networking domain based problem, we had to read up more on networking as a whole and understand how proxies work. At the end of the 2 days and many workarounds, we still couldn’t fix the issue.
NOTE: China’s firewall blocks almost every single saint internet page the outside world uses on a day-to-day basis. This includes Facebook, Twitter, Dropbox, Google and many others.
Just as we were about to give up, we decided to try using another VPN client to bypass China’s firewall instead. No harm trying right? annnnnnd BOOM! It worked! I suspect the existing VPN client messed around with some proxies settings that affected communication with the dropbox server.
Here’s a video of the working prototype, with Mark, being the model. As we do not have access to a dog on which we can test it on at this point of time, we made Mark exercise vigorously to trigger the spike in heart rate =)
With less than a month to go for the product exhibition, there’s still alot of improvements and optimizations to be done before we focus our efforts on the product packaging and aesthetics. But the journey thus far has been sweet bittersweet! =)