Using Spark ML on Spark Errors - What Do the Clusters Tell Us? @ Spark Summit EU 2018

Thanks for joining me on 2018-10-04 for Using Spark ML on Spark Errors - What Do the Clusters Tell Us?.The talk covered:

If youre subscribed to user@spark.apache.org, or work in a large company, you may see some common Spark error messages. Even attending Spark Summit over the past few years you have seen talks like the "Top K Mistakes in Spark." While cool non-machine learning based tools do exist to examine Sparks logs they dont use machine learning and therefore are not as cool but also limited in by the amount of effort humans can put into writing rules for them. This talk will look what happens when we train "regular" clustering models on stack traces, and explore DL models for classifying user message to the Spark list. Come for the reassurance that the robots are not yet able to fix themselves, and stay to learn how to work better with the help of our robot friends. The tl;dr of this talk is Spark ML on Spark output, plus a little bit of Tensorflow is fun for the whole family, but probably shouldnt automatically respond to user list posts just yet. Session hashtag: #SAISML10

.

The slides are at http://bit.ly/2NoIFxr.

Comment bellow to join in the discussion :).Talk feedback is appreciated at http://bit.ly/holdenTalkFeedback

Comments

Post a Comment

Popular posts from this blog

New year new job, same projects

 I’m stoked to announce that I’ve joined Snowflake to continue working on OSS Apache Spark :) I’ve got a post on the Snowflake blog talking about the work we’re doing  https://careers.snowflake.com/us/en/blogarticle/building-apache-spark-in-the-open-at-snowflake  — 
Read more

Making Hibernate work on Ubuntu 22.04 (jammy) on the Framework Laptop w/full disk encryption

  Making Hibernate work on Ubuntu 22.04 (jammy) on the Framework Laptop w/full disk encryption The 0th step is disabling secure boot in the bios (reboot, press F2 to get into the BIOS, disable secure boot). The first step is to make a swapfile large enough for your device to suspend to. sudo swapoff /swapfile sudo dd if = /dev/zero of = /swapfile bs = $( cat /proc/meminfo | awk '/MemTotal/ {print $2}' ) count =1024 conv = notrunc sudo mkswap /swapfile sudo swapon /swapfile Then make sure it's enabled on reboot, e.g: echo '/swapfile swap swap defaults 0 0' | sudo tee -a /etc/fstab If there is another swap partition or file in /etc/fstab you may need to take it out. Find the device your swapfile is located on (e.g. findmnt  -no source -T /swapfile ), note here some guides recommend using the UUID, in my experience, this resulted in some errors. To find the resume offset of your file run filefrag -v /swapfile and then look at the first value under physical_offset...
Read more

Unexpected gmail account name disclosure on calendars

Image
I came across a an interesting blog post showing how to get the name of a Gmail account. Since the bug was visible through Google Calendars I hoped that it was maybe limited to users who had signed up for Google Calendar. This is not the case. The steps that I followed: 1)Register brand new gmail account with default settings register new Gmail account 2)Go to google calendars (as a different user) select a calender invite the user to share one of my calendars (click on the down arrow next to your calendar) choose a calendar to invite the user to share 3)Invite the user Invite the user to share your calendar 4)Save the changes (note that it indicates the user isn't registered for Google Calendars) & go back Invite user to Google Calendars 5)See the users name see the username Normally I'm all for notifying the company of the problem before public disclosures, but the cat is well out of the bag and having a walk around. It apparently works regardless of whether the user is ...
Read more