July 2007 Archives

Robert Richards was on the Space Show yesterday and touched on several topics including the upcoming Phoenix launch to Mars, the International Space University which he co-founded and the company he works for Optech which is participating in Phoenix as well as Canadian space industry topics.

I have to mention that one of the great products that Optech produces is a LIDAR (Light Detection and Ranging). They just don't get enough credit for producing this great technology. LIDAR is a is an optical remote sensing technology that measures properties of scattered light to find range and/or other information of a distant target. The Phoenix mission incorporates a LIDAR into the Meteorological Station (MET) science instrument. An Optech LIDAR is currently being used by the NASA Ames K-10 rover team at the Haughton-Mars Project Research Station on Devon Island. As well a LIDAR is included in the PRIME (Phobos Reconnaissance and International Mars Exploration) mission instrument package that is in the concept study phase being undertaken by Optech, the Mars Institute (my institution), MDA for the Canadian Space Agency.

From the Space Show on the interview:

On Friday at the Open Source conference, Jimmy Wales, founder of Wikipedia and Wikia, the open source search engine project, announced the release of an open-source Web crawling site called Grub. Grub crawls the web indexing pages for the Wikia search engine.

It's a clever idea building upon other distributed projects such as SETI @ Home. Crawling the web is costly so if you have thousands of clients doing it for you that will save you money and could make crawling cost effective. However I have to wonder what percentage of an actual crawl will be performed by Grub distributed clients. Also if my computer is contributing to this project, which although is open source, is still a for profit venture, shouldn't I profit from it as well?

Grub aims to compete with Google. If they can get enough computing power behind them they might be able to get an index as large as Google's and maybe even bigger but the key to getting and keeping market share will be the results returned. And that is all in the algorithms of the search engine.

A distributed web crawling client for Project Phoenix is something to consider and providing people a portion of the revenue stream could make it attractive to users.

The 2007 Space Elevator Games will be held in Salt Lake City this year. The qualifying rounds will start on October 15, and the event will be open to the public between October 19 and October 21.

Hosted by the SpaceWard Foundation the games feature over 20 teams competing $1,000,000 in cash prizes provided by NASA’s Centennial Challenges program.

New this year is the "Light Racers" -- that allows kids and young adults to take part in a realistic lunar exploration scenario and win cash prizes for their performance.

• Learn more about the 2007 Space Elevator Games
• What you will see
• Attractions
• Tickets
• Directions
• Sponsorship
• Exhibitors

When dealing with vast amount of data you need a scalable distributed storage system. All of my database driven web sites use MySQL and for smaller databases you can mount a MySQL search to the web site. But you soon find out that to deliver fast searching capabilities to a site, or if as in my case you intend to offer a search service of hundreds of millions of crawled niche data you need a scalable distributed storage system.

Recently Google hosted a Conference on Scalability in Seattle where they talked about MapReduce, BigTable, and other distributed systems for large datasets. Listed here are the talks which are now available on Google video:

• Keynote I: MapReduce, BigTable, and Other Distributed System Abstractions for Handling Large Datasets by Jeff Dean, Google, Inc.
• Keynote II: Scaling Google for Every User Marissa Mayer, Vice President, Search Products & User Experience, Google, Inc.
• Lustre File System by Peter Braam, Founder and President, Cluster File Systems, Inc.
• SCTP's Reliability and Fault Tolerance by Brad Penoff, Mike Tsai, and Alan Wagner, The University of British Columbia Department of Computer Science.
• Scalable Test Selection Using Source Code Deltas by Ryan Gerard, Symantec Corporation.
• VeriSign's Global DNS Infrastructure, Patrick Quaid, Technical Director and Scott Courtney, Principal Architect, VeriSign.
• Using MapReduce on Large Geographic Datasets, Barry Brummit, Software Engineer, Google, Inc.
• YouTube Scalability, Cuong Do, Engineering Manager, YouTube.
• Building a Scalable Resource Management, Khalid Ahmed, Platform Computing Corp.
• Lessons In Building Scalable Systems, Reza Behforooz, Google Inc.

(Kudo's to Greg Linden for compiling the list of videos.)

The video's provide some technical detail while Marissa Mayer's provides some insight into Google's big picture plans.

Google's technology however is closed so if you're interested in a solution that you can use then turning to open source projects is the way to go. And this is where Hadoop with HBase come in.

Posted the other day on the Google Lat Long Blog is mention of a new NASA layer being added to Google Earth. It incorporates Astronaut Photography of Earth, Satellite Imagery, and Earth City Lights. The photography layer shows the best images going back to the Mercury missions and is from the online Astronaut Photography collection. The satellite imagery is also a compilation of the best NASA images taken over the years.

Just over a week I was at the NASA Ames Participatory Summit, something I'll write about later, and met Mark Jannot the editor of Popular Science Magazine. He had just finished giving a talk on a new service they had just launched a couple of weeks ago called the PopSci Predictions Exchange (PPX). Here's how they describe the service;

"Welcome to the PPX, the first place to bet on the future of science and technology. It's easy and free: Log on, and we'll give you POP$250,000 in our virtual PopSci Dollars. Use that money to buy propositions you think are likely to happen. If other traders also want to buy, that proposition's price will go up, and you'll make PopSci bucks. Expand your portfolio with bets on energy, space, consumer technology and extreme science, and compete against other players for prizes and bragging rights."

What a cool interactive service that encourages learning about so many different things.

If you haven't tried the social networking service called Twitter then I suggest you do. It can be addictive in a useless way or it can be productive depending on who you talk to. Twitter is a web based service that allows you to stay in touch and keep up with your friends, colleagues, business contacts etc. It's not just limited to the web, you can use your mobile phone to access your Twitter and to Twitter with everyone. As well, there is Twitterrific a free Mac desktop application from the Iconfactory that let's you Twitter. And if that's not enough for you there's Twittervision which allows you to view global perspective in normal or 3D view what people are Twittering about.

I've just started Twittering a few months ago and I have to admit I had no idea of what use it would have. But after trying it out a little I can see some value to Twittering beyond just keeping in touch with your friends. I've been using it mostly for fun but I may try setting up a Twitter channel for a project and see how it goes.

Recently I pinged my LinkedIn network from some thoughts on Twitter. One response from Aimee Evans was particularly interesting from the business perspective.