I've written a book "Fast Data Processing with Spark" which covers Python, Scala, and Java

I recently finished writting "Fast Data Processing with Spark". Apache Spark is a framework for writing fast, distributed programs.


Fast Data Processing with Spark covers how to write distributed map reduce style programs with Spark. The book guides you through every step required to write effective distributed programs from setting up your cluster and interactively exploring the API, to deploying your job to the cluster.


Personally, while the fast nature of Spark is not to be understated, I really enjoy its functional style APIs and find it a lovely environment to code in.

Comments

Post a Comment

Popular posts from this blog

New year new job, same projects

 I’m stoked to announce that I’ve joined Snowflake to continue working on OSS Apache Spark :) I’ve got a post on the Snowflake blog talking about the work we’re doing  https://careers.snowflake.com/us/en/blogarticle/building-apache-spark-in-the-open-at-snowflake  — 
Read more

Making Hibernate work on Ubuntu 22.04 (jammy) on the Framework Laptop w/full disk encryption

  Making Hibernate work on Ubuntu 22.04 (jammy) on the Framework Laptop w/full disk encryption The 0th step is disabling secure boot in the bios (reboot, press F2 to get into the BIOS, disable secure boot). The first step is to make a swapfile large enough for your device to suspend to. sudo swapoff /swapfile sudo dd if = /dev/zero of = /swapfile bs = $( cat /proc/meminfo | awk '/MemTotal/ {print $2}' ) count =1024 conv = notrunc sudo mkswap /swapfile sudo swapon /swapfile Then make sure it's enabled on reboot, e.g: echo '/swapfile swap swap defaults 0 0' | sudo tee -a /etc/fstab If there is another swap partition or file in /etc/fstab you may need to take it out. Find the device your swapfile is located on (e.g. findmnt  -no source -T /swapfile ), note here some guides recommend using the UUID, in my experience, this resulted in some errors. To find the resume offset of your file run filefrag -v /swapfile and then look at the first value under physical_offset...
Read more

Unexpected gmail account name disclosure on calendars

Image
I came across a an interesting blog post showing how to get the name of a Gmail account. Since the bug was visible through Google Calendars I hoped that it was maybe limited to users who had signed up for Google Calendar. This is not the case. The steps that I followed: 1)Register brand new gmail account with default settings register new Gmail account 2)Go to google calendars (as a different user) select a calender invite the user to share one of my calendars (click on the down arrow next to your calendar) choose a calendar to invite the user to share 3)Invite the user Invite the user to share your calendar 4)Save the changes (note that it indicates the user isn't registered for Google Calendars) & go back Invite user to Google Calendars 5)See the users name see the username Normally I'm all for notifying the company of the problem before public disclosures, but the cat is well out of the bag and having a walk around. It apparently works regardless of whether the user is ...
Read more