Whats new in Spark this week #1

Whats new in Spark will look the activity in the Spark commit logs every week and attempt to summarize what new features and bug fixes have occurred. This not intended to summarize everything, mostly things that might be useful to application developers. Without further ado lets get started: 

• zipPartions was added to the Scala & Java APIs. zipPartions joins an up to 4 RDDs using a user supplied function. It requires that all of the RDDs have the same number of partitions with no such requirement on size within each partition. https://github.com/mesos/spark/commit/c9c4954d994c5ba824e71c1c5cd8d5de531caf78 https://github.com/mesos/spark/commit/c9c4954d994c5ba824e71c1c5cd8d5de531caf78 
• unPersist() was added to the Scala & Java APIs allow the removal of an RDD from persistence once it is no longer needed. https://github.com/mesos/spark/commit/93091f6936262a4006d875bf69b3f8c31c291617
•  Bugfix was added to validate that local directories can be created when being added. https://github.com/mesos/spark/commit/c9c4954d994c5ba824e71c1c5cd8d5de531caf78
• Spark’s block UI manager had a bug with Spark Streaming blocks which was fixed https://spark-project.atlassian.net/browse/SPARK-740 https://github.com/mesos/spark/commit/538ee755b41585c638935a93ec838b635149f659
• The shuffle writer now looks at spark.shuffle.file.buffer.kb to determine the buffer to use. Previously the default buffer was 8kb which could cause a lot of unnecessary disk seeks, and the new default is 100kb. https://github.com/mesos/spark/commit/1055785a836ab2361239f0937a1a22fee953e029 

That is all that I found interesting in skimming this weeks commit logs, if I missed something important feel free to let me know :)

Automatic spelling corrections on Github

English has never been one of my strong points (as is fairly obvious by reading my blog), so my latest side project might surprise you a bit. Inspired by the results of tarsnap’s bug bounty and the first pull request received for a new project(slashem - a type safe rogue like DSL for querying solr in scala) I decided to write a bot for github to fix spelling mistakes.

The code its self is very simple (albeit not very good, it was written after I got back from clubbing @ JWZ’s club [DNA lounge]). There is something about a lack of sleep which makes perl code and regexs seem like a good idea. If despite the previous warnings you still want to look at the code https://github.com/holdenk/holdensmagicalunicorn is the place to go. It works by doing a github search for all the README files in markdown format and then running a limited spell checker on them. Documents with a known misspelled word are flagged and output to a file. Thanks to the wonderful github api the next steps is are easy. It forks the repo and clones it locally, performs the spelling correction, commits, pushes and submits a pull request.

The spelling correction is based on Pod::Spell::CommonMistakes, it works using a very restricted set of misspelled words to corrections.

Writing a “future directions” sections always seems like such a cliche, but here it is anyways. The code as it stands is really simple. For example it only handles one repo of a given name, and the dictionary is small, etc. The next version should probably also try and only submit corrections against the conical repo. Some future plans extending the dictionary. In the longer term I think it would be awesome to attempt detect really simple bugs in actual code (things like memcpy(dest,0,0)).

You can follow the bot on twitter holdensunicorn .

Comments, suggestions, and patches always appreciated. - holdenkarau (although I’m going to be AFK at burning man for awhile, you can find me @ 6:30 & D)

I <3 Topatoco

I just had a wonderful customer service experience with them, and I feel the need to share and let everyone know how awesome Topatoco is.

Topatoco sent me what has to be one of the best packages I've received in a long time. After some initial problems with my first bear-monster hoodie, they mailed a replacement, which (thanks to USPS) got mixed up, but no worries, they re-sent it. With buttons! and stickers :) And most of all a hand written note.

Translation of Pigs Can Fly Site Monitor

Pigs Can Fly Site Monitor is now available in Spanish and French. The French translation is a bit more shifty than the Spanish one. If you find a translation error, drop me an e-mail , holden@pigscanfly.ca .

Pigs Can Fly Site Monitor Notification

Pigs Can Fly Site Monitor Notification
Originally uploaded by dmcopernicus

I'm pleased to announce the launch of Pigs Can Fly Site Monitor for the Google Android platform, a free application. Pigs Can Fly Site Monitor (pcfsm) behaves like other regular site monitoring software, polling your website at customizable intervals to ensure it is online. Since the site monitor runs directly on your phone, you don't have to worry about haveing a second computer to act as the monitoring station, or any difficulties with slow SMS delivery.

Pigs Can Fly Site monitor is available from the Google Market for free. Users with the Google Market on their Android can download it by click here or looking under the Tools section in the Google Market on their Android phone. For users without access to the Google Market place (like NeoFreeRunner uses such as my self) you can install it from pcfsm.com.

In addition to the traditional polling, PCFSM also handles basic regular expression matching, and can optionally check if your site is linked from slashdot or reddit (as being linked to from there may cause massive spikes in traffic).

If you don't have an unlimited data plan, I'd recommend setting the polling interval to a very high number, on the other hand if you do have an unlimited data plan go wild :)

PCFSM is still pre-1.0, so there may be some bugs. If you find any please e-mail me at holden@pigscanfly.ca , make sure to include PCFSM in the subject so that I notice it.

DeviceScape now available on the OpenMoko

I'm pleased to be able to post the DeviceScape ipkg's of DeviceScape for download. The binaries consist of two packages devicescape (also mirrored on the csc) and a different wpa version(or on the also mirrored on the csc). It has been tested on the ASU software image of the OpenMoko.

While I will respond to bug reports ( holden@pigscanfly.ca ), this is likely the end of the line for this software package.

It seems like the OpenMoko software stack doesn't have a lot of life left in it, and I've got an application (involving site monitoring) I'd like to write for the Android. As such I'm going to be putting Android on my FreeRunner and hopefully crank out that application that this Saturday/Sunday.

xkcd404 - the xkcd that wasn't

xkcd404 - the xkcd that wasn't
Originally uploaded by dmcopernicus

I was reading the xkcd archives to switch my mind away from math (temporarily) and I was reminded of xkcd #404 . xkcd comics are of the from http://xkcd.com/[comicnumber] and #404 just gives back a 404 error.

update to web2.0collage

my web2.0collage
Originally uploaded by dmcopernicus

I've updated web2.0collage, it now uses stateless web servelets (thanks to a patch in plt scheme :)), which gives the load balancer a much easier time, sine session are no longer sticky. It is covered in a few other places, besides slashdot, now (like gigaom/nyt) and being the hobo that I am, I'm trying to see if I can get the story up on digg ftw :)

If you run into any bugs with it (which is likely) or have suggestions I'd love to hear them :)

Your browser history is showing (an open source web application in scheme)

my web2.0collage
Originally uploaded by dmcopernicus

Over the course of last weekend I wrote web2.0collage, a browser history sniffing collage generator in scheme. Web2.0collage is designed to graphically illustrate just how easy it is for sites to determine what your browser history is. When you visit the site it sniffs your browser history, and creates a collage of the (safe for work) sites that you visit. It is an interesting application of potentially scary technology (imagine a job application site using this to screen candidates). Ideally, given some time in my schedule, I'd like to make it a bit more user friendly and robust so that I could perhaps show it to the general public to increase awareness of privacy issues on the web.

The code, while not good since I was learning how the plt-webserver & imagemagick bindings worked at the time, is available under the agpl. Today it hit the front page of slashdot, causing some less than fortunate scaling issues to be discovered. Hatguy & myself managed to fix them (sort of) without too many interruptions.

Devicescape, OpenMoko, StarBucks & Boingo mobile

I finally got a replacement battery for my FreeRunner allowing me to perform a rather important test, namely Starbucks support. Unfortunately the Canadian Starbucks use a different Wi-Fi provider than the American Starbucks, so the free wifi login support with Devicescape doesn't currently work. However, Boingo has a free 30 day trial for boingo mobile, which is a roaming partner with Bell (one of the Canadian Starbucks wireless providers) and Devicescape does support boingo hotspots.

Much to my pleasant surprise, the existing code worked with only a few minor modifications. I came across and fixed a minor bug involving not being able to stop the connection process, so you can take back over manual control if you so desire. Once again, if you are interested in testing this release give me a shout ( holden@pigscanfly.ca ), make sure to include openmoko in the subject somewhere so it gets through.

Now that my FreeRunner is working again I'm hoping to get a UI prototype up at the end of next weekend or two.

First glance at su.pr

After reading about su.pr I was interested in giving it a spin (especially since I need to be concentrating on abstract algebra, I needed to take it for a spin right away). They posted beta invite code to the stumbleupon twitter feed. Of the features mentioned in su.pr's initial press release, all seem to be functional with the notable exception of "seo friendly" links (aka 301 redirects) so that search engines count the links as going to your site and shorten on your domain. I haven't had a chance to try out any of the other features, like suggested posting times, as it seems like they are tailored to each user so it requires a bit of data first. The settings panel seems a bit buggy (I haven't managed to get it to add more than one site that I'm "promoting"), but that seems like an easy fix. Overall I'm not entirely sure what all the buzz was about, it seems kind of cool but lacks sufficient compelling features to convince me that its not a bad thing to use a url shortener.

Less than fun server times

So it looks like my host pulled the plug on my server on the morning of the 27nth. There appears to be some mix up with finding my account, which could be anything from a mix-up to "oops we wiped that server". Since its been awhile I've decided that the server probably isn't coming back online anytime soon, so I've got a VPS set up. Most of the stuff is backed up, sadly back in Waterloo. Fortunately I learnt from my more recent laptop failures and the most important bits (namely my delicious code) is in a variety of locations (three cheers for git :) ).
On the plus side I've lost a lot of cruft of configuration that had built up over the years, but on the downside I've got a lot of configuration and sys admin work to do for the next couple of days.

Almost done with interviews

Interviewing for full-time is quite different than what my Co-Op interviews have prepared me for. For one thing, companies are much more interested in having you come on site, which is pretty cool in that I've gotten to see a lot of different work environments, but also has the downside of keeping me busy flying all over. Fortunately, I've managed to get the remaining 2 west coast companies I'm interviewing with to co-ordinate so I don't have to make separate trips out :) I was a little worried with job hunting during this economic slump, but it seems like most technology companies are still hiring (albeit maybe not as many people as before). Having the Amazon offer has made the whole process much less stress-full in some ways, but in other ways its made my schedule a lot more packed since the deadline is the end of this month.

Random beer

Oddly enough a lot of people from Ottawa end up going to the University of Waterloo (or at least they seem to, in the Math/CS segment). Apparently, I am so far out of touch with Ottawa that I didn't know about the creation of a new brewery (called beaus) (complete with blog). Kevin was kind enough to bring down a big (~2L) jug of "Lug Tread" which was surprisingly good. So that this isn't a total non-sequenter with the rest of what I write, I wonder what sort of challenges they faced doing a startup and how those compare to tech startups? And now back to that free beer....

Update Yahoo! Zimbra Desktop vulneraible to Man in the Middle

Once again, Yahoo! has made a slight mis-step with protecting their users' information. In my attempt to enable interoperability between pcfspam & Yahoo! Mail, I uncovered another problem with the most recent Yahoo! Zimbra Desktop. The new Zimbra Desktop (build 1344) uses the same login methodology as the web login, which is already known to be replayable. Unfortunately, unlike the web login, it doesn't notify the user in the event of an SSL certificate mismatch. This makes Yahoo! Zimbra vulnerable to a man-in-the-middle attack, exposing both usernames and passwords.

To reproduce this bug, simply download Zimbra desktop & set your host file (/etc/hosts) for login.yahoo.com to point to your local machine (127.0.0.1) by adding:

127.0.0.1 login.yahoo.com

Alternatively, you can configure bind and add the Yahoo! zone:

;
; BIND data file for the fake yahoo zone
;
$TTL 604800
yahoo.com. IN SOA localhost. root.localhost. (
;@ IN SOA localhost. root.localhost. (
         2  ; Serial
    604800  ; Refresh
     86400  ; Retry
   2419200  ; Expire
    604800 ) ; Negative Cache TTL
;
yahoo.com. IN NS ns1.yahoo.yahoo.com.
login.yahoo.com. IN A 127.0.0.1
login.yahoo.yahoo.com. IN A 127.0.0.1
ns1.yahoo.yahoo.com. IN A 127.0.0.1

. Then start an SSL webserver (I used apache) on port 443 and take a look at the access log to see the request:

127.0.0.1 - - [21/Nov/2008:00:27:39 -0500] "GET /WSLogin/V1/get_auth_token?appid=0YbgbonAkY2iNypMZQOONB8mNDSJkrfBlr3wgxc-&login=albertsanchezo&passwd=kingof HTTP/1.1" 404 401 "-" "Jakarta Commons-HttpClient/3.0"

You can clearly see the variables login & passwd contain the username and password. It should be noted that no warning message was shown to user and this was done with a self-signed cert for a localhost.localdomain. A malicious attacker would have to exploit only one of the many DNS poisoning vulnerabilities and pass the authentication information through to be able to capture the usernames & passwords of a large number of Yahoo! users. While you can see that I didn't bother passing the information through, you could also get a similar effect with squid (or another proxy) and still allow authentication to complete.

The impact of this is much lower than the previous vulnerability with Yahoo! Zimbra desktop, but is still serious as it exposes usernames & passwords with only a trivial amount of effort.

At the time of the writing Yahoo! security has been notified.
p.s.
For anyone from Yahoo! reading this, I'm still waiting for the shirt I was promised from the first time I reported a vulnerability, but its all good :)