More Web Scraping with Phantomjs & CoffeeScript

Thu, 20 Mar 2014 03:33:00 +0000

I’ve been playing with Phantomjs and CoffeeScript a bit more. In this example I load the first page of google news, extract all the links and link text, and write them to stdout in markdown format.

# headlines: generate markdown-formatted output with just the headlines
# and links from the main google news page (no sections, nothing fancy)
page_url = 'http://news.google.com/'
jquery_url = 'http://ajax.googleapis.com/ajax/libs/jquery/2.1.0/jquery.min.js'
page = require("webpage").create()
	
# this is a handler that relays console messages from inside our page to
# the phantomjs console. I've made it ignore everything that isn't
# prefixed with the word 'headline:'
page.onConsoleMessage = (msg) ->
    if msg.substring(0, 9) is "headline:"
        console.log(" - " + msg.substring(9))
	
	
page.open page_url, (status) ->
    if status isnt "success"
        console.log "# Error: couldn't load the page"
        phantom.exit()
    console.log "# The News\n"
    page.includeJs jquery_url, ->
        page.evaluate ->
            $('a.article').has('span.titletext').each (indx, elem) ->
                title = $(elem).text()
                href = $(elem).attr 'href'
                console.log "headline:[#{title}](#{href})"
        phantom.exit()  # I need this here to let the evaluate finish

Getting to know CoffeeScript

The combination of CoffeeScript an jQuery feels natural. But the overall program feels pretty brittle, and the phantomjs error messages that were printed while I was writing this little script weren’t very helpful. To be fair, part of this relates to CoffeeScript. I’m still getting used to it, and I’m in the phase where overlooked unbalanced parens or unintentionally ambiguous expressions cost me lots of time to debug.

Phantomjs on galvanist

More Web Scraping with Phantomjs & CoffeeScript

Getting to know CoffeeScript