Posts

Showing posts with the label life

Co-Op term at Amazon

This semester I'm at Amazon, its quite a lot of fun. Seattle is a really nice city, especially in the summer. At Microsoft some people referred to the summer as "demo weather" because thats when they brought down the interns to try and recruit them :) Most of my summer is being spent working on the problem of detecting miss-classified items (so for example if a fridge was listed as a shoe that would be bad news). Its the kind of work which I really enjoy because its fairly open-ended in terms of the solution and hard. A lot of traditional stock solutions proved to be unworkable since we have such a large data set. Oddly enough searching for information about scaling the approaches I dumped into some posts by a third-party trying to accomplish something similar with a subset of the Amazon data :). Since the general rule of software projects seem to be if you create it you get to name it, I decided on naming my project "BrowseEpicFail" (I later renamed it to be mo...

Google Summer of Code

My proposal to create scheme bindings for subversion for Google Summer of Code has been accepted, which is awesome. I'm looking forward to a summer of largely functional programming (scheme & ml). If time permits I'm going to try and pick up haskell and make haskell bindings as well. The awesome thing about this is that one of the things I need to do for All The Code is create a subversion interface, so the work is related to what I was going to be doing anyways :)

Downtime, presentatons, etc.

So All The Code recently suffered some downtime and I figured I should probably talk about it. The problem came from the reverse proxy software that I'm using to act as a load balancer and (ironically) avoid nodes that are down. Even then, the reverse proxy going down shouldn't be the end of the world since there are two reverse proxy boxes and ZoneEdit is supposed to failover to the second server in the event the first one goes down. During the brief stint on slashdot, I disabled the second reverse proxy and started using it as just another node to keep up with the load. Sadly, I forgot to switch it back to its old configuration after the load went back to normal, so the failover system didn't work out as planned :) I've got a full schedule for the next little while, sadly not enough of it involves fun coding, but such is life. This Friday I'm giving a presentation to a small group (3 people) looking at the business aspects of All The Code. I'm also going to b...

Features & Bugs oh my!

Being on slashdot, reddit and other news sites has its ups & downs. On the positive side, you get a lot of viewers, and for the most part people seemed fairly positive. I also got a lot of useful bug reports about such wonderful things as cross sight scripting vulnerability, places where things should have been escaped, and a few feature requests. Since All The Code has a fairly small database, since it only indexes .java files and does not look inside cvs/svn/git or even zip files for the time being, a lot of searches didn't return many results. The solution to this is to allow substring matching IE (you search for "btree" it finds "tree" and gives you tree). This has the wonderful effect of giving you lots of results, but depending on what you are searching for this stemming can lead to much sadness, giving not so good results at time. My present solution involves some magic determining when to use stemming and when not to use stemming. After being on slas...

so much stuff, so little time

So All The Code recently made it onto slashdot for the win. Theres a lot of stuff that I need to do and write about but life is way too busy.

Taste the alpha

So, after All The Code was up for a few hours I made one small change to the production system, breaking codeviewing for the better part of half an hour. Anyways, I came up with a way to improve results that shouldn't take more than a hundred lines or so to implement, so as soon as I get some free time I'm going to give that a shot (not on the production version though, I've learnt my lesson there). On a less than totally random note, newsforge put up a small blurb on it for awesome.

more progress

So, the database built again. There seems to be a bug in the code where by certain cases involving weird class inheritance isn't being properly handled [whereas it use to be handled properly], so I'm going to have to look into that and probably do another database build. I have a meeting on Wednesday with the person at UW responsible for students wishing to do business on their Co-Op term. From what I recall, his background is much more manufacturing / physical oriented, so I might have a bit of difficulty explaining exactly what I do. I may also try talking to the people I talked to about eight months ago, but I'm not sure if thats a good idea or not. Once the final alpha 0.1 database is ready, I will probably give a demo to the csclub , which will hopefully go better than my previous demo :). The present pre-alpha todo list is: 1) fix class bug 2) rebuild DB 3) fix up HTML so it works on most popular browsers (presently breaks in IE6,Konquerer, and probably a few more). ...

the spirit of "why not?"

The University of Waterloo is celebrating its 50th anniversary, and they have chosen the slogan to be "the spirit of why not?". Right now I'm in my 3A academic term, and I'm trying to figure out what I should do with the Co-Op term between 3A and 3B. I've been going back and forth between applying only to awesome jobs, and not applying to any jobs and working on my own projects (like AllTheCode ). I figure, why not?

Last minute fun

As a lot of people have noticed, you don't really notice a lot of bugs until you actually try and ship something. One of the more frustrating ones is that my AMI image (for Amazon EC2) I had constructed didn't want to boot, which is somewhat inconvenient to say the least. I hadn't noticed the problem, since all of the earlier versions were run on my one dedicated server [with a limited data set]. Fortunately, after a few days of butting my head against the wall I managed to get a working AMI image, so now its just down to getting it to run through my data in time [which is a bit questionable], in part because I did my bench marking on my laptop and assumed that the EC2 images would be faster; but my laptop would appear to be about 4 times faster. While I was doing some random poking around online, waiting for a batch job to run, I found snap.com which has a really cool preview feature. So I randomly added it to AllTheCode, but then I thought back to how I originally discove...

Last Day

Friday was my last day @ Xandros (conveniently also the day of the company Christmas party). Looking back on the past four months, its been more fun than my previous Co-Op jobs. I've had a chance to play around with the wonderful world of device drivers, file systems, and boot loaders for increased stylishness. Part of the reason why I think it was so fun, is that I was learning a lot of it as I went along, whereas I was pretty well qualified for most of my previous jobs. The plan for the next week is to split my time 50/50 between All The Code and another project that I'm working on. I've fixed the concurrency bug with All The Code, but I've also noticed a few not so good thing, and a pretty awesome thing. The not so good thing is that about ~10% of the records in the DB weren't java programs (which I've fixed). The awesome thing is that I've figured out a way to get ~10 languages supported really quickly, so I'm going to try and get that done over the ...

$2.72

That is how much money I've spent [in machine time] trying to figure out why my new and improved parser dies horrible on awsp. I thought maybe there was a problem with the framework I was using, perhaps I wasn't flushing the output to the file properly in my ocaml app, any number of possibilities. Turns out I was having 7 machines all write to the same file [where I thought they were all writing to different files in there local cache]. The reason why it never showed up in my ,albeit limited, testing is I only have on machine, and only on the platform does it run on multiple nodes at the same time. I hope I don't have too many more gotchas like this hanging around since Monday is fast approaching :)

Optimism

One thing which I find universal among people trying to make something is optimism, or at the very least localized optimism. For me, this means thinking that I can do tasks in 30 minutes when they really take an evening, the database will build on the first try, and all sorts of wonderful things like that. While this can be really useful at times (namely when starting or keeping on going), it can also make things un-necessarily difficult, by setting release dates too early and other fun things like that. It still looks like I'll be able to hit December 11nth, just 30 minutes to fix this one last bug, rebuild the DB, another half hour to put together a new front end, and few other "minor" things ..... :)

Going Live Dec 11nth

In the spirit of releasing early, All The Code will be going live on December 11nth for the win. Now, everyone seems to be releasing there products in "Beta", but I'm not sure it will be exactly beta quality by then, so I'm going to call it a public alpha. Take that you fancy Web 2.0 "beta" people :) The planned features for the public alpha is basic search functionality and only Java supported. I will probably set up a poll for what language I should support next. I'm hoping to get searching inside archives and popular source control systems done over Christmas. After that I will probably add about one additional language a month (perhaps a little more depending on how much free time I end up with).

DemoCamp Ottawa 2, A look back

DemoCamp Ottawa was quite interesting, although not exactly what I was expecting. My spam filter managed to eat the confirmation e-mail from the organizer, so I didn't appear on the schedule but fortunately I was able to demo anyways. I'm certainly going to have to look into my spam filter, its eaten a few other decidedly non-spammy e-mails that I much rather have. All of the other presentations had actual UIs [most of them quite nice] and the technology was way beyond the "I built the DB yesterday/this morning I hope it works", but hey whatever :-) One of the demos (Context Discovery) looked really cool to me, essentially it attempts to create a summary of a text document based on an enhanced version of an algorithm that came out of the nrc . They seem to be getting ready to do a beta, and I'm certainly going to take a look at it once its released [although the demo was on a WIN32 platform, so there is a reasonable chance they won't release a linux version ...

Final Stage

The Final Stage is now building, albeit with a whole bunch of limits placed on it. It would appear that I definitely need to figure out how to paralyze the build process in the future, I've got a few ideas, but I'll also check with the people at AWSP to see if they have any recommendations on how to do it [since they have much more experience with that sort of stuff than I do]. I'm not really sure if the final stage is going to be built in time for my presentation tomorrow, but in a worst case scenario I can just go with the phase two stuff I have from before, it doesn't really demonstrate the unique parts of ATC so well, but I suppose that is life.

Top 5 Signs that your IDE is bloated

I've been playing around with a lot of new languages recently as part of getting All The Code ready, and I've tried numerous IDEs (camelia, radrails/eclipse, etc.) for different languages (ocaml , ruby , c, etc.) and they all seem to suffer from extreme bloat. #5 It uses more ram memory FireFox with 30 tabs #4 You can see a refresh rate as you type #3 It takes longer to load than the rest of your operating system #2 If someone replaced it with a web-based application you wouldn't notice the delays #1 The developers claim it does more than emacs. Whats the lesson of this story? There isn't one, except that writing blog posts while waiting for batch jobs to run is good way to pass the time.

Phase 0 Done, Phase 1 Started

So Phase 0 of the ATC build process is complete. What is phase 0 you may ask? Its a cool sounding name for I finally have the data on my computer ready to be processed. Since I'm only indexing java programs for monday (I haven't written parsers for anything else yet) I wasn't expecting there to be too much data I'd be able to pull from AWSP, maybe 700mb or something in that range. However, without doing any of the fancy things that I want to do later (like looking inside archives, csv/svn/git) I now have 2.4Gb of data to shift through for monday. Sadly, the code I wrote for the Phase 1 build code has a few bugs, which I need to get fixed.

Amazon WebServices, or why I might get a chance to sleep after all

I wasn't expecting such a fast response from amazon with regards to my problems using AWS , but I must say I'm pleasantry surprised. Monday afternoon my time (Monday morning there time) I got a conference call, and we went over how I was trying to use the Alexa Websearch service, and they explained the reason for all my frustration. Namely, that it times out with very large start values, which is why I couldn't get more than a few hundred records. They didn't place a hard limit on the start values, since when the problems will start to occur is apparently dependant on a large number of factors. On the upside, it looks like I won't have to write my own crawler since they also enabled my Amazon Web Search Platform account, which should not have these problems [AWSP is designed for people trying to build search engines, and has a different model than the regular AWS system]. The next seven days are going to killer finish my demo, but it should be possible :-)