~ computer vision fun ~ with photogrammetry

Catie Buhler - dfab research

A couple months ago we became interested in computer vision, a broad subject of teaching computers how to see.  We began by feeding images into Grasshopper, a visual scripting plug-in for Rhino, and generating printable 3D models. Photogrammetry opened up the possibility to generate models from a variety of sources, from things directly in front of you to videos found online.  There are a number of exciting practical and creative applications for photogrammetry.  While we played around with a variety of proprietary and open source programs, it seemed important to develop an accessible workflow of all open or shared source software for anyone to explore the multitude of possibilities.  While the software we used was amazing in many ways, they definitely have their flaws and limitations when compared to their proprietary competitors.

–Anna Brancaccio + Catie Buhler

Here are links to tutorials that we found most helpful:

VisualSFM –> Meshlab: http://wedidstuff.heavyimage.com/index.php/2013/07/12/open-source-photogrammetry-workflow/

View original post 2,164 more words

Soft Robotics: Sandal Actuator

This semester our class assembled the control board from the Soft Robotics tool Kit, a system for testing pnuematic actuators.  After printing out their sample file for a mold of an actuator on the Object printer, casting it in rubber and testing it successfully, we began designing one of our own.

Soft Robotics toolkit website

Our project is to make a sandal with actuating straps that start flat and inflate to grip your feet.  We are close, but still working out some issues to make the sandals move properly.

For our first test, we modeled and printed the mold on the object printer.  The the sizing and spacing of the chambers and walls of the actuator were based off of the measurements of this Soft Robotic Gripper.  While one strap curled as we injected air into it, the other side didn’t, and with the side that worked, the connecting chamber seemed to be over inflating, and the individual mini chambers weren’t inflating enough.

We decided to switch from 3d printing the molds, to milling them out of HDPE for cheaper, faster molds.  We also designed three separate test straps with slight variations on the actuator shape and size as well as the chamber size and spacing.

Our first round of tests all failed because the paper (which was probably cut too big) prevented the two halves of the molds from bonding, and each test blow out almost instantly upon the air being applied.

For our second round, we used a form of synthetic silk in place of the paper.  We wanted something that was rigid enough to prevent the actuator from bending on one side and that the rubber could more easily bond to.  Above is a photo of the recently poured dragon skin rubber in one of our test molds with the piece of silk layed on top of the shallow side of the mold.  By restricting the bend on one side, it encourages the other to curl in a controlled manner.

This second round still had a few problems like chamber clogging and air holes around the seem.

We made some more modifications to the design and decided to cast the tubing into the molds, since inserting the tubing at the end proved to be problematic.

An image of our sandal mold being milled on the CNC router in a sheet of HDPE.

Setting up the tubing.  We’ve been using small amounts of water based clay in our molds to secure the tubing.  The tubing is slightly too thick for the entrance points of the actuators.

Us mixing the rubber.

We used a degassing chamber to remove many of the bubbles within the rubber which could affect the air flow.  We experimented with dragon skin rubber with different curing times.  The faster curing rubber seemed to work the best.

We added valves to the shoes to regulate the air flow so that the actuator could fill up and then be plugged to stay inflated in position.  We milled the sole of the sole to be bonded with the top piece that contains the actuators.

Catie Buhler - dfab research

Guidelines for Dfab Research ~ Computer Vision ~ Catie Buhler + Anna Brancaccio

Parameters for Grading

• create an efficient and accessible workflow
• become proficient in programs used in final workflows (VisualSFM, Meshlab, Blender)
• a final collection of models created using photogrammetry that addresses the functional uses of photogrammetry

General Plan

• a finished workflow documented as a written tutorial (to potentially also include the more experimental input/outputs we have used in Grasshopper, or with impractical collections of images)
• create a navigable collection of meshes made from various image sources and softwares (collective virtual psychogeography ? — like a shared instagram map for translating real space into a virtual one)

4/14 – 4/21

• separately watch Blender tutorials
• using VisualSFM>>Meshlab>>Blender create a finished textured mesh
• determine final group of photo series to translate to models

4/21 – 4/28

• download Blender, Unity, Cloud Compare
• determine final workflow
• tutorials visualSFM, VLC media player finished, and…

View original post 40 more words

Point Clouds in Grasshopper

We quickly experimented with taking the point clouds generated in VisualSFM into Grassopper and manipulating there shape with the help of Ryan Hoover.  Here we just gave the points a shape (sphere) and enlarged them.  This gave us a weird blobby version of our gazebo.                                                                             

Creating 3D Meshes from Videos

As we’ve been trying out the different software we’ve also been testing out the construction of point clouds and meshes from different types of source imagery.  I think what makes photogrammetry so exciting is the ability to create at least partially 3d forms or spaces from things not physically available to you as opposed to 3d laser scanning.  So much of the world is now understood through images so the translation of images or video to a virtual space explorable in three dimensions seems natural.  Catie took some stills from Nelly Fertado’s music video I’m Like A Bird and imported them first into Agisoft and then into visualSFM to Meshlab to Blender.  While definitely flawed and piece-y both Agisoft and VisualSFM were able to match parts of the images together to form a distorted Nelly Fertado laying in a field as in part of the music video.  We also tested a series of Facebook pictures taken at a girl’s pool.  We couldn’t get a good point cloud from this yet.

VisualSFM=>Meshlab=>Blender

Moving from VisualSFM to Meshlab and finally to Blender seems to be the best workflow for our project.  All three programs are open source and for the most part have been producing pretty good results.  We tried out Agisoft, the proprietary alternative to visualsfm which is generally agreed to work better, and it worked pretty well for one of our smaller image sets, but crashed on a larger set of images where VisualSFM did not.  The problem may have been that we used an older, slower computer to run Agisoft on.  Major advantages of Agisoft are that it combines the work done in Visualsfm and Meshlab into a few simple bottoms in the same program as well as having a nice user-friendly interface.  We decided free software is worth the few extra steps.  We’re still working through a couple problems with texturing some of our meshes.  As we get further into compiling our meshes into a single Blender scene we’ll see if we’re going to have to tweak our meshes/textures so the file doesn’t become huge and slow.  We’re thinking of trying out an additional open source program CloudCompare which is a point cloud editing program which would allow us to merge point clouds meshes and reduce the size of a point cloud if we wanted to generate a more simple/lighter weight mesh in mesh lab.  I think there are ways of doing these things in just visualsfm and mesh lab too, but I’ve read that some people find CloudCompare to work better.  It might be good to limit the number of programs in our workflow as much as possible since constantly importing and exporting files from one program to another can become really tedious.

Starting with VisualSFM

Catie and I began looking at VisualSFM.  The first time we tried it our dense point reconstruction was extremely light and the image was barely recognizable.  This was with about 50 images.  We then took photos of a gazebo in Druid Hill Park photographing it in a grid and overlapping parts of each consecutive photo.  This time we used 142 photos and the quality was much better.  Here is the resulting Dense and sparse point clouds:

I’m not sure why, but the photos from one half of the gazebo were placed with the other half, building only one side of the gazebo.  I tried the F2 command to try and get rid of some of the misplaced cameras, but that seemed to get rid if a lot of rightly placed cameras causing a worse quality point cloud (shown in the first image on the left).

123D Catch

Here’s a screenshot from a mini animation I made by using 123D Catch to scan my mushrooms I was growing in a pair of shoes and then imported the mesh into Maya to orbit around the scan.  It works really well, I only fed it 11 images.  The down side as Jesse describes here http://wedidstuff.heavyimage.com/index.php/2013/07/12/open-source-photogrammetry-workflow/ is that the process is completely hidden, therefore very limiting and bad for any sort of experimentation.

Photogrammetry and Mapping Space

Catie and I have begun more specific research into constructing or reconstructing environments using computer vision.  Right now we’re interested in using photogrammetry to create our virtual space, which is the process of using photos to generate 3d models.   We’re interested in the possible inclusion of GPS coordinates from the photos we take. This would open up the possibility of creating a participatory or communal 3D scan of an environment in which people could contribute their photo from a specific location to help construct the space, thinking about the function of GPS in apps like Instagram or Google Earth. We recently found the Photosynth project https://photosynth.net, which overlaps a lot of our interests.

We have been hearing about the project Aerialscape done last year by MICA students Sophie Stoerkel and Clara Hickman in which they rigged an iPhone up to a small balloon device to take continuous aerial photos of landscapes to create accurate 3d models using the software Agisoft which specializes in photogrammetry.  Sophia’s blog explains it in more detail: https://sophiestoerkel.wordpress.com.  We met with Sophie to have her run through their process and show us the basics of Agisoft. Most agree that Agisoft is the top photogrammetry software program out right now and is used my most major 3D scanning companies like Direct Dimensions. We’ll definitely be trying this out.

I found a blog post the other day that is extremely relevant to our research:

http://wedidstuff.heavyimage.com/index.php/2013/07/12/open-source-photogrammetry-workflow/

While most agree that Agisoft produces better resulting point clouds than VisualSFM (used by Jesse in the above post), we became very interested in VisualSFM for a couple reasons discussed by Jesse: free and open source as well as the fact that each of the steps are exposed and it allows you to participate in the process.

A second Idea we’re exploring is constructing 3D spaces from videos. In the spirit of Akihiko Taniguchi we expect the resulting spaces to be filled with holes and other imperfections (if we get it to work at all). For this we’re also looking into Agisoft, VisualSFM and Grasshopper.

We’ve done some quick room scans with the Kinect using Skanect software and while the scans came out pretty good, they were really heavy and were crashing my computer. Also, neither of our ideas could be done with the Kinect; we would have to use photogrammetry.

Nicholas Burrus-artist, researcher, teacher

Here’s a link to his website.
The website is a really good resource for information on working with the Kinect and computer vision where he logs all his research along with his projects.