Pixy Stuck <\/span><\/p>\n Attempts to Find an Augmented Reality Development Solution for Unity3d<\/span><\/p>\n Project Team: Matt Cotter (Grad CSC), Sam Henry (Grad CSC)<\/span>, Jim Martin (A+D) and Daniel Lunk ( Grad, A+D) An interdisciplinary group of faculty and students from the North Carolina State University Colleges of Engineering and Design began collaborating with the goal of finding a practical method to develop augmented reality applications. In the first meeting it was decided that the method needed to work with the Unity game engine in an effort to make augmented reality development more accessible. Such a method would would pave the way for future augmented reality development by artists and designers in the Advanced Media Lab at the NCSU College of Design, and elsewhere. <\/span><\/p>\n We came into the project planning to work with the Microsoft Kinect, which has had a non\u00adcommercial open\u00adsource SDK available since 2011. In the initial meeting the idea of using Kinect was shelved, in favor of Pixy (CMUcam5), which is marketed as a fast, easy\u00adto\u00aduse<\/em> vision sensor. The reasons for this decision were that Kinect\u2019s strengths lie in detecting and tracking human bodies through depth perception, while the Pixy\u2019s hardware accelerated color and object detection seemed like a perfect fit for the type of interactive games that the group was most interested in exploring. <\/span><\/p>\n<\/div>\n<\/div>\n Experience with the Pixy <\/span><\/p>\n In an effort to allay any concerns over the uncontrolled environmental variables on the porch we also tested inside in a clear box, again under ideal circumstances for color detection, using bright sticky notes and a white background with the Pixy\u2019s camera placed in a fixed position above the box. While over the box, the Pixy was consistently detecting objects and displaying this information through the PixyMon application, which is useful for debugging but isn\u2019t particularly useful for our project. The libraries that were provided with Pixy weren\u2019t working the way they were supposed to. <\/span><\/p>\n <\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n Connecting Pixy to Unity\u00a0 There are currently no libraries available to connect a Pixy directly to Unity, so we were forced to try to route the Pixy\u2019s data through another platform. Pixy was designed and came with libraries to connect to the Arduino platform, so that was the obvious choice to try next. Unfortunately, when the Pixy was connected to the Arduino via serial port the data transmitted to the computer was incomplete and we could not detect objects through Pixy\u2019s provided libraries with consistency and accuracy. We identified buffering issues as the cause of the missing data, but were unable to ameliorate the buffering issues with delays due to our lack of knowledge of the Pixy\u2019s architecture. <\/span><\/p>\n Still searching for a way to connect the Pixy to Unity, we discovered <\/span>Uniduino<\/span>, which \u201cgives you the power to connect the Unity game engine to Arduino\u201d according to its website. When we found this plugin we planned to connect Pixy\u00ad>Arduino\u00ad>Uniduino\u00ad>Unity. Alas, the Uniduino didn\u2019t support SPI connections, rendering it useless for this application; however, it did successfully connect Unity to the Arduino through an analog port, which could be useful in future projects. <\/span><\/p>\n OpenCV as an Alternative to Pixy <\/span><\/p>\n Feeling that we had exhausted our options on the Pixy front, we went back to the drawing board and decided to attempt to integrate OpenCV with Unity. We found some <\/span>videos <\/span>of <\/span>Unity3d projects <\/span>featuring OpenCV but were unable to replicate this integration. It didn\u2019t help that the only <\/span>guides <\/span>we could find on the subject were inconsistent, badly written, and\/or outdated. <\/span><\/p>\n 2D Demo Application in Unity <\/span><\/p>\n We did create a demo in Unity that we planned to use to demonstrate our (theoretical) augmented reality solution. The demo features multiple objects falling randomly from the top of the screen under the effects of gravity. When the user clicks, a new block is placed on screen, at the point of the mouse, that will disrupt the falling objects\u2019 descent, as it would in real life. The plan was for users to be able to place blocks in the real world and immediately see the effects they have on the virtual falling objects, projected onto the surface. <\/span><\/p>\n Lessons and Conclusion <\/span><\/p>\n Any time a project involves new technologies it runs a risk of failure. New technology doesn\u2019t have years of field testing and bug\u00adfixing, or the public documentation that accumulates over time. We didn\u2019t have the requisite hardware knowledge to compensate for this, leading to us wasting time on the Uniduino when someone with a different background could have identified the mismatched connectors right away. <\/span><\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n Going forward, OpenCV looks like the most viable hope for this project\u2019s original goal of enabling augmented reality development, in Unity3d or another environment. Kinect is also an option, as it could potentially differentiate objects by their depth in the right augmented reality situation. <\/span><\/p>\n CMUcam<\/em><\/strong><\/p>\n The goal of the CMUcam project is to provide simple vision capabilities to small embedded systems in the form of an intelligent sensor. The CMUcam open source programmable embedded color vision sensors are low-cost, low-power sensors for mobile robots. You can use the CMUcam vision systems to do many different kinds of on-board, real-time vision processing tasks. http:\/\/www.cmucam.org\/<\/a><\/p><\/blockquote>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n <\/p>\n","protected":false},"excerpt":{"rendered":" Pixy Stuck Attempts to Find an Augmented Reality Development Solution for Unity3d Project Team: Matt Cotter (Grad CSC), Sam Henry (Grad CSC), Jim Martin (A+D) and Daniel Lunk ( Grad, A+D) Faculty Advising: Ben Watson (CSC), Pat FitzGerald (A+D) […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[79,89,75],"tags":[126,131,134,133,132,136,135],"class_list":["post-1560","post","type-post","status-publish","format-standard","hentry","category-reasearch","category-tutorial","category-tutorials","tag-arduino","tag-kinect","tag-opencv","tag-pixy","tag-processing","tag-uniduino","tag-unity"],"_links":{"self":[{"href":"https:\/\/onebitpixel.com\/aml\/index.php?rest_route=\/wp\/v2\/posts\/1560","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/onebitpixel.com\/aml\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/onebitpixel.com\/aml\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/onebitpixel.com\/aml\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/onebitpixel.com\/aml\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1560"}],"version-history":[{"count":7,"href":"https:\/\/onebitpixel.com\/aml\/index.php?rest_route=\/wp\/v2\/posts\/1560\/revisions"}],"predecessor-version":[{"id":1569,"href":"https:\/\/onebitpixel.com\/aml\/index.php?rest_route=\/wp\/v2\/posts\/1560\/revisions\/1569"}],"wp:attachment":[{"href":"https:\/\/onebitpixel.com\/aml\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1560"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/onebitpixel.com\/aml\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1560"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/onebitpixel.com\/aml\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1560"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
\nFaculty Advising: Ben Watson (CSC), Pat FitzGerald (A+D) and Lee Cherry<\/p>\n![\"pixy01\"](\"https:\/\/onebitpixel.com\/aml\/wp-content\/uploads\/2014\/07\/pixy01.png\")
Once we received the Pixy we began to explore its capabilities, testing its color and object detection abilities by placing brightly colored sticky notes on a white ceiling approximately four feet away from the Pixy\u2019s camera. During this test we began to discover some of the Pixy\u2019s limitations, as it correctly identified both full and half sticky notes, but couldn\u2019t detect quarters of sticky notes. We also threw a large red ball off of the porch with the camera aimed at the ball\u2019s descent path, only to have the ball leave the Pixy\u2019s sensor range about 10 feet down. <\/span><\/p>\n<\/div>\n<\/div>\n![\"pixy02\"](\"https:\/\/onebitpixel.com\/aml\/wp-content\/uploads\/2014\/07\/pixy02.png\" "\\"padding:")
<\/span><\/p>\n