It’s been quite a while since I’ve ran the updates for the Hacker News Hiring Trends. Here are some updated stats for May 2014.
Number of Posts
Looks like May was just a little slower than April considering the number of posts (323 vs 343).
Looks like AngularJS is already kicking butt for the month of June. It’ll be interesting to see how things are at the end of the month.
I was wondering what months get the most comments on Hacker News “Who’s Hiring” posts. January seems to be the slowest month, as I expected. I also expected December to be slow, but it turns out that’s not the case.
A by-product of scraping and analyzing the posts from Hacker News, I get Document-Term Matrix of all the words mentioned each month. This gives a bit of insight into trends about what programming languages, frameworks and technologies people are looking to hire for. I was expecting a little more clear cut trends for technologies to be dropping of rising over the two and a half years for which I have data. The popularity of a term is determined by the number of mentions in total per month, I could have broken things down a bit differently; maybe count the number of posts that have at least one mention of a term.
Using an IPython Notebook to scrape Hacker News with Requests, parse HTML with BeautifulSoup, analyze with Pandas and draw graphs with matplotlib feels a lot more natural to me than working in an environment like R. I’m looking forward to more little projects of analyzing data similar to this in the future.
I’ve been playing around with creating a small Cocoa App for the last week or so. Finding good examples and documentation has been a little problematic, so I’ve put together a tiny application as a reference point to get going when starting fresh. It’s built entirely in Python using Cocoa through the PyObjC wrappers. It includes a XIB created with Interface Builder in Xcode. Bundling the App for distribution is a snap using the py2app command in setuptools.
Source on GitHub: simple_pyobjc_cocoa_xib
All the required Python:
The most useful resource for me has been the source available in the PyObjC examples. One thing that took a while to figure out was how the initial NIB and window are loaded when some of the examples start. The Info.plist file generated by py2app includes the NSMainNibFile element which instructs OS X to load the MainMenu NIB when launching by default.
The py2app tutorial has been useful for getting the initial project set up.
Below are the steps I took to create the application with a 5 minute video of the process at the bottom.
The tool used to create a setup.py file is py2applet. It is not in the PATH by default so the full path can be used to run it.
$ /System/Library/Frameworks/Python.framework/Versions/Current/Extras/bin/py2applet --make-setup SimpleXibDemo.py
By default, the setup.py created by py2applet has argv_emulation set to True. This flag enables dragging of files onto our application in OS X. It’s not supported with the latest libraries, and this application that functionality. So it can be removed from the OPTIONS dict.
To include the XIB that will be created later in the application, it’s added to the DATA_FILES list.
DATA_FILES = ['SimpleXibDemo.xib']
The Python Source
The Python source for the application is all in one file, SimpleXibDemo.py. It consists of one class (SimpleXibDemoController) and a __main__ block to get the application up and running.
Inheriting from the NSWindowController class gives the SimpleXibDemoController class the responsibility of managing a window. It handles events like windowDidLoad to get hooks into the windows lifecycle.
The local variable counterTextField is initialized to objc.IBOutlet(). This allows in Interface builder to assign a reference to a Label to it for providing output to the user.
The @objc.IBAction decorators on a couple functions expose them to Interface Builder as so the functions can be called when a Push Button is activated.
Creating a XIB in Interface Builder
In Xcode, click File… New… File… Create and OS X User Interface Window. Save it with the file name that was referenced above in the setup.py; SimpleXibDemo.xib.
Drag three Push Button elements and one Label element into the new window. The buttons text can be changed by double clicking on them.
Add the SimpleXibDemo.py file to Xcode in File… Add Files… Interface Builder will recognize the Python file and find the objc.IBOutlet() and @objc.IBAction elements.
Click on File’s Owner in the left pane with the projects objects and choose the identity inspector tab in the top right. Change the Class value to our Python class SimpleXibDemoController.
Attaching the IBOutlet and IBActions in Interface Builder:
Hold down CTRL and drag from File’s Owner to the Label. Select the counterTextField outlet.
Hold down CTRL and drag from the decrement button to the File’s Owner. Select the decrement: action. Do the same for the increment button, but select the increment: action.
Hold down CTRL and drag fro the Quit button to the Application object and chose the terminate: action.
Run in Alias Mode for Development
While working on the App, it’s convenient to build it in Alias mode. Resources in the build are aliased to source code, so changes can be previewed without going through a whole rebuild.
$ python setup.py py2app -A
Run the application in Aliased mode:
Build for Distribution
The application is built for distribution using the same py2app command, but without the Alias flag.
$ python setup.py py2app
The distributable Application can be found in the dist folder.
Video of the workflow
In the book On Intelligence, Jeff Hawkins (founder of Palm Computing) makes the argument that the roughly about 30 billion neurons and 30 trillion synapses (junctions between neurons) in our brain are much like general purpose computers and can be trained to process information that they were not even initially supposed to.
Most of sight doesn’t happen in the eyes. The eyes are just sensors that feed data that is processed in the brain. As shown in the video below, Erik Weihenmayer has been able to regain some sight by using a Brainport device that takes video and sends signals to his tongue. From this, he’s about to “see” some objects around him, even read simple text, and play tic-tac-toe with his daughter.
This makes me wonder how the devices around us could be interacted with in more ways than just visually and audibly. For instance, Google Glass vibrates at the side of your head. Could it lead you around a city giving you directions not using the display at all? I can imagine different pulses in the vibrations let you know how soon you have to turn, and even different pulses would tell you to turn either left or right.
Brainport Vision Device helps a blind man “see”
A fellow attendee at the Open Data Day Toronto event hosted by Urban+Digital Toronto was wondering how to display Toronto neighbourhood data on a map that she was working on. I couldn’t find any GeoJSON of the neighbourhoods out in the open, so I grabbed a shapefile from the City of Toronto Open Data Portal – Toronto Neighborhood Planning Areas.
To export it to GeoJSON I used the ogr2ogr tool from GDAL (Geospatial Data Abstraction Library).
$ ogr2ogr -t_srs EPSG:4269 -f geoJSON Neighbourhoods.json Neighbourhoods.shp
The resulting GeoJSON file was 1.4M in size. That’s a bit big to be sending down to the browser, so I used the Simplify geometries tool in Quantum GIS (QGIS) and then re-exported to GeoJSON. The resulting file is a manageable 308K. It still looks good, so it could be shrunk down even further if needed.
I keep a lot of information in my Gmail account; in about 72,488 email messages. All of my email accounts forward to my Gmail Inbox. The convenience of having all of my email in one place is great, but I am risking things by putting all of my eggs in one basket. It’s unlikely, but Google could lose those emails, or for some reason revoke my access to the free service that I’ve been using for so long.
My current backup is still running on the cheapest instance available over at DigitalOcean:
$ fab docean status ... Status: backup running : yes number of emails : 52697 size on disk : 1.2G
A few requirements I had for my Gmail backup:
- Free (as in beer and speech)
- Runs remotely – to not heat up my laptop, and not shutdown when I close the lid
- Filesystem based for later statistical analysis
- Easy to backup
In searching for a tool to host my own backup of Gmail, I cam across gmailbackup. I’ve forked it and made a small change to it’s parsing of date fields in emails. One culprit message that caused the parse error is the introductory message sent by Google entitled “Gmail is different. Here’s what you need to know.” sent when if first signed up on June 23, 2004. When presented with the “Show Original” option, it lacks the required orig-date filed from RFC2822 (Internet Message Format).
Backing up 72,000 emails takes a long time, so I’d rather not run it on my laptop. I’ve put together a fabric script to handle the major functionality of running the backup on a remote server. I originally got this working inside a Vagrant managed virtual machine, but have moved to running it remotely on a DigitalOcean VPS. Setting up a new instance and ripping it down for testing is incredibly quick and simple with them, and they’re not currently charging for backups, so it’s as close to free as can be. I’ll probably bring back the Vagrant configs shortly.
You can check out the source and instructions on how to get it running over at GitHub: https://github.com/adamw523/gmailarchive
Happy Lunar New Year! I’m looking forward this year to exploring a bit more around the world of Python. Reading the excellent book Programming Collective Intelligence has inspired me to explore the language a bit more.
IPython has become on of my indispensable tools when working in Python. I’ve been enjoying using it’s shell for regular development, and also in the browser using the Notebook feature – A web-based user interface for authoring Python code. It’s great at displaying images and other artifacts right in the web browser.
I’ve started a bit of my own Python development. I switched my remote server provisioning and deployment procedures to fabric.
I’ve also put together a quick project to play around with creating, building and sharing a PyPI hosted package. My first such project is a library and command line tool for interfacing with the RESTful API of the cloud hosting provider DIgitalOcean.
A Google Street View car drives into Central Park in New York and then magically dissappears. Interesting. It almost seems as if the car had an out of body experience. See for yourself. Follow the cars path into Central Park for a surprise.