Visualizing Defiance, Ohio Shows

I won a free O’Reilly ebook from a hackathon I participated in a while ago.  I chose Visualizing Data and I’ve been working through the book.  I always find that its more helpful to work on a project of my own that approximates the examples in a technical book instead of just reading or copying and pasting the example code.  It forces me to learn new things, gets me more excited about the project and makes me re-read portions of the book in detail.

For a useful time series, I chose one that was close to my life, the list of past Defiance, Ohio shows.  It was also useful because I was trying to set expectations for a new job and wanted to let them know how much time I had spent away from home in the past.  The original version of this list was pretty messy, so I had to clean it up a lot with Google Refine before visualizing it using processing.

<br /> No Java 2 SDK, Standard Edition v 1.4.1 support for APPLET!!<br />

Most striking, you can see how our activity has become more sparse in the last few years. If you’re interested in the code, you can download the processing code.

Creating a video CD (VCD) from a YouTube video

As a Christmas gift for my father, I transferred a YouTube video of a public service announcement that he remembered fondly from his childhood to optical media so he could watch it on the television instead of on the computer.

There were some unique sets of constraints that made this an interesting project. First, I came to my parents’ house with only a Dell Mini netbook (with no optical drive) running Ubuntu Linux.  None of the computers had DVD writeable optical drives, but they could burn CDs.  My parents’ DVD player can play video CDs (VCDs) so I decided this would be a good option.

Capturing the video

I downloaded the YouTube video using the UnPlug extension for Firefox.  I chose to save the MP4 version of the video.

Transcoding the video

Since I would be burning the video on my mom’s notebook, I read the help documentation for the CD writing software to determine which video format the software needed to create a VCD.

VLC is usually my go-to tool for working with video, but I got errors about not having the correct codecs.  Rather than messing around, I wanted to just get the project done.  So, my next choice was FFmpeg.   It was a good choice because the software, though a command-line utility, has a preset for creating an mpeg file suitable for burning a VCD.

ffmpeg -i High\ Flight\ \(John\ Gillespie\ Magee\ Jr\ Poem\).mp4 -target ntsc-vcd high_flight.mpg

I was able to do this with the version of ffmpeg included with Ubuntu 10.04, but as ffmpeg is cross-platform it should be straightforward to do this on other platforms.

Burning the VCD

I copied the file to my Dropbox and downloaded it to my Mom’s computer using the services web interface.  Her burning software allowed me to just drag-and-drop the file into the burning program’s window.

MoveSmart

Home_MoveSmart_St._Louis_(Dev)_-_2014-01-15_22.41.50

MoveSmart helped users discover new, affordable, neighborhoods based on neighborhood assets like quality public schools, walkability and farmers markets.

Role

Sustaining developer

Description

I didn’t build the original MoveSmart, but I learned a ton maintaining Bec White‘s code. Building on a clever API that leveraged Drupal Views, I created custom  modules to add new datasets to the system and to support neighborhoods in the St. Louis metropolitan area.

In addition to continued development of the system, I performed data cleaning and merging of the datasets used in the neighborhood search.

More information

Draw the Line Midwest

Open source mapping software allowed users to view state and federal legislative districts and draw their own district plans during the 2010 redistricting process.

Role

Integrating developer

Details

I deployed and customized instances of the open source District Builder software for a number of Midwestern states.  Users could view existing districts or create new district maps based on block and tract census geographies.  Some instances provided a platform for competitions to see if users could create more balanced district plans based on political, racial and population demographics.

In order to minimize hosting costs, the instances were deployed on a single server with resources provisioned to virtual hosts using VMWare.  In addition to the provisioning and hosting, I provided data preprocessing and cleaning in order to add additional demographic data to use to evaluate district plans.

Lakeview Wal-Mart

I’m pretty interested in reports that Wal-Mart may be trying to open a location in my neighborhood.

I read a report in the Red Eye that said the retailer had signed a letter of intent on a location.

This is Ald. Tom Tunney’s (44th) response to this and similar reports:

On Thursday, December 9, I heard reports in the media that Wal-Mart may be interested in renting a location here in the 44th Ward.  This was surprising as I had not seen any conceptual plans for a proposed store in Lakeview.

After follow-up on my part, Wal-Mart issued the following statement on Monday, December 13:

“Contrary to media reports, Walmart has not executed a lease or a letter of intent with the developer to locate a store on the property known as ‘Broadway at Surf,’ at 2840 North Broadway in the Lakeview community of Chicago.  The company is evaluating a number of potential opportunities across the city of Chicago, and will continue to work with elected officials, business groups, community associations and key stakeholders to ensure that sites and formats are compatible with the communities we seek to serve.”

Maggie Sans – VP Public Affairs and Government Relations
Wal-Mart
702 Southwest 8th Street
Bentonville, AR 72716-0350

If any plans come forward in the future, it will be vetted through a rigorous community process.  As a small business owner, I understand the impact any big-box retailer would have on our neighborhood.  We will work together, residents and businesses, to continue to make our community a better place to live, raise a family, shop and own a business.

Dealing with Drupal comment spam

I got word that the Pages to Prisoners website was drowning in comment spam.  I locked down permissions, installed the AntiSpam and reCAPTCHA modules and wrote up some quick docs about my approach to solve it before stumbling on this really helpful how-to post, How to: Take control of your Drupal site back from the comment spammers, which outlines most of the steps that I took.

In the final step of this how-to, the author recommends bulk deleting spam comments in SQL.  The example SQL query in the how-to is pretty intense, deleting all comments.  I found that by looking at the comments, I could identify frequently used phrases in the subject that clearly had nothing to do with content on the site.  I could then compose a SQL query that would blow away these comments in a way that felt safer than deleting everything.  This is the query that I used which took me about 15 minutes to blow away over 3000 comments:

DELETE FROM `comments` WHERE `subject` LIKE "%boot%" OR `subject` LIKE "%jewelry%" OR
`subject` LIKE "%ugg%" OR `subject` LIKE "%watches%"  OR `subject` LIKE "%handbag%" OR 
`subject` LIKE "%cartier%"  OR `subject` LIKE "%jordan%" OR `subject` LIKE "%vuitton%" OR 
`subject` LIKE "%replica%" OR `subject` LIKE "%Louboutin%" OR `subject` LIKE "%MOD%"  OR 
`subject` LIKE "%OEM%" OR `subject` LIKE "%viagra%" OR `subject` LIKE "%coach%" OR 
`subject` LIKE "%chanel%"  OR `subject` LIKE "%tiffany%"  OR `subject` LIKE "%armani%"  OR 
`subject` LIKE "%windows 7%"  OR `subject` LIKE "%bvlgari%" OR `subject` LIKE "%chanel%" OR 
`subject` LIKE  "%The whirlwinds Mens%"  OR `subject` LIKE "%footwear%"  OR 
`subject` LIKE "%gucci%" OR `subject` LIKE "%BALENCIAGA%" OR `subject` LIKE "%jersey%" OR
 `subject` LIKE "%shoe%" OR `subject` LIKE "%jacket%" OR `subject` LIKE "%vibram%" OR 
`subject` LIKE "%mobile phone%" OR `subject` LIKE "%cell phone%" OR `subject` LIKE "%burberry%" OR
`subject` LIKE "%blu ray%" OR `subject` LIKE "%blu-ray%" OR `subject` LIKE "%phone%" OR 
`subject` LIKE "%nike%" OR `subject` LIKE "%abercrombie%" OR `subject` LIKE "%dress%" OR 
`subject` LIKE "%charm%" OR `subject` LIKE "%jewellery%"  OR `subject` LIKE "%london%" OR 
`subject` LIKE "%www.%"  OR `subject` LIKE "%video game%" OR `subject` LIKE "%bag%" OR 
`subject` LIKE "%wedding%" OR `subject` LIKE "%rosetta stone%" 

Really, It’s Worth It

This was originally posted on the Local Fourth blog as part of my participation in a community media innovation project at the Medill School of Journalism.

You’re in the middle of a big project with tight deadlines. Parts of your infrastructure are a little, well, jenky. Do you take the time to make things cleaner and more coherent, or do you focus on coding, hoping that your stack holds together until you have time to clean it up? Will a new tool pay off or is it just a distraction from perhaps more tedious, but more crucial work that needs to be done.

This was my situation early this week when I started looking at our deployment procedure when moving developing on our workstations to making our work public on our webhost. I was working on a bug where things that worked on our development machines weren’t working on the webhost. Instead of just setting up a new instance for testing on our webhost, I decided to invest the time in exploring a new-to-me tool called Fabric.

Fabric is a Python library and tool for scripting commands to be run on a remote server over SSH. You can store it in the root of your Django or other Python project and run tasks on the remote server. For instance, to build up a new staging instance on our webserver, I type ‘fab staging mkinstance’ on my shell.  In this example, staging and mkinstance are just tasks that you’ve defined as Python functions. staging is a task that sets context variables about this particular instance, such as the home directory where the other tasks will execute on the remote server. mkinstance just calls other tasks to create a new virtualenv, install required Python packages, download the most recent version of the source from git and more. I may go into the details of our Fabric configuration in the future, but, more importantly, I want to explain why I think exploring this tool to improve our deployment was a good way to spend my time early this week.

Avoid rm -rf . “Oh #$%!(*&!!”

I would say that even savvy system administrators who know better, who will lecture their junior admins about what not to do at great length, sometimes hop in a shell to run a quick command only to carelessly wreak disaster. I recently caught myself typing commands into shells on the wrong host. There had to be a better way. Fabric modules compel me to explicitly type the action that I intend to take and the instance I want to take it on. fab staging destroyeverything has a different resonance to ssh somehost.com && cd /home/ghing/webapps/ && rm -rf myproject.

A deployment reasoning tool

Automating your deployment process in a script helps you break down the deployment process. What are the steps that need to be taken? What order should they be run in? What are the dependencies between the steps? Writing these things out in code helps identify redundancies or potential problems in the process.

Code as documentation

An artist who taught game development to high school students told me that she introduced students to the idea of programming by having them work in pairs and take turns writing pseudocode to instruct their partner about how to move around in a space. I’d imagine one of the big takeaways is the difference between code and natural language. The latter can make for some beautiful journalism, but it can make describing a process convoluted. After I finished our Fabric module, I wondered how I would have written free form paragraphs to describe our deploy process, or explain them out loud. Even with few comments, its clear what’s happening in the deploy process. In around the same time it would take to write a step-by-step description of the process, I have one in code that can also do all the steps for me.

Further Reading

Refactoring fabfile.py for fast, robust Django deployment by Christopher Groskopf got me stoked about using Fabric, and it’s a good place to look for good practices for deploying Django in the cloud. It was, however, a little daunting as an entry point to the different tools.

Tools of the Modern Python Hacker: Virtualenv, Fabric and Pip does a good job of describing the connection between these tools and how they can be successfully used together for deployment.

Deploying Django with Fabric helped get me started in writing my Fabric file and understanding the conventions of the tool.  Some of the information on this page may be a little different with more recent versions of Fabric.

Deploying with Django’s Sites Framework on Webfaction (virtualenv, mod_wsgi, git) helped me figure out some of the non-Fabric stuff to make deployment consistent on our webhost.

A Heads-Up About Head-Down Coding

This was originally posted on the Local Fourth blog as part of my participation in a community media innovation project at the Medill School of Journalism.

I first became acquainted with the term hackathon in college. A computer science student group organized an all-night event in one of the computer labs and ordered in pizza and a seemingly limitless supply of caffeinated soda. I remember the event as fun, but I mostly worked on classwork. I didn’t yet have a backlog of personal hacking projects that could really benefit from hours of uninterrupted coding.

At the start of this project, Shane, Steve and I participated in a mobile hackathon sponsored by The Media Consortium. It was a fun way to familiarize ourselves with our chosen software tools, get used to coding as a team and anticipate some of the hurdles we’d encounter during the innovation project. Though I spend a lot of most days in front of the computer, there’s something pleasurable about having an entire day, or weekend, of frenetic, uninterrupted programming. This might sound like hell to a lot of people, but it’s fundamentally satisfying to building something from scratch and to be able to be fully immersed in a project, really feeling every aspect of the process and design.

Most of my coding time doesn’t feel this way. Within the innovation project, there are countless meetings, e-mail threads to read, documents to share. While there is a certain amount of coordination required between developers working on the same code base, the time overhead required to connect with other teams feels like it increases exponentially. While communicating about the project often feels like an interruption, finding uninterrupted time to program is even more difficult when I have to factor in childcare responsibilities.

On a typical Wednesday, I try to leave the workspace before 3:30 to pick the kids up from their after school program. Then we head home where I have to help with homework, cook dinner, do the dishes, go to the park if we have time and then remind them when it’s time for bed. It’s around 9 before I can even think clearly about code again. Thursdays, I leave even earlier to pick them up from school, drop Florence at home and drive Oona to the West Loop for her dance class (though this has made me really good at doing long division in my head while navigating traffic). If the traffic’s bad, I’m in the car for around two hours total. I sometimes try to squeeze in a little work while they’re doing homework, or reading after dinner, but it usually feels like a kind of attention purgatory where I’m neither able to focus on the infuriating bug that I’m trying to fix or the needs of the kids. On the days when I have to leave early, it feels like I never end my work day at a coherent stopping point, where I’ve at least discovered the cause of a bug and have a plan for a fix or where I’ve sketched out enough of the implementation of a feature where I can come back to it and everything will make sense.

I’ve lived with and helped take care of my friend’s twin daughters, now 10 years old, for a few years.  But, my responsibilities, and their impact on my time, really ramped up since we all moved to Chicago. I’m somewhere between a babysitter and a parental figure and this ambiguity mirrors the range of likely responses to talking about trying to work on a software project and share in childcare responsibilities. For a lot of my peers, who are childless, the response is “I can’t imagine what that’s like.” For older folks who are parents, it’s more like “duh, that’s how it is.” It’s a tough topic to talk about, made tougher because, from week to week, I either feel overwhelmed by all the juggling or like the responsibilities that I have are a cakewalk compared with those of a full-time single parent.

As a recent thread about parenting and start-ups on the excellent Geek Feminism Blog points out, its not a very productive conversation to generalize about how parenting or any other real-life experience affects technology projects and the people behind them. Still, the experience of someone struggling to balance the needs of a project and the needs of others is different than those who aren’t struggling with such ambivalence. I’ve read accounts of developers who are fathers feeling alienated when they leave at the end of the workday, just as their childless colleagues are ordering pizza for a late-night coding session. During this past year’s Ada Lovelace Day, a campaign to forefront women in technology, a blogger voiced criticism that some men used it as an opportunity to acknowledge how their female partners gave them time to work on technical projects rather than celebrating the direct technical contributions of women. In the context of childcare responsibilities, it’s easy to see how this underscores a trade-off between one person’s ability to participate in demanding technology projects and another’s. But I want to go beyond pointing out that there might be some disparities and prejudices around different people’s availability for technology work or projects. Instead, I try to question whether the amount of time someone spends hacking on a project, or in the office necessarily represents the only value of their labor.

I yearn for days where I can work, uninterrupted and on my own schedule, on a juicy programming problem. But I, and hacker culture in general, may also overly romanticize this kind of heads-down coding. Interruptions are frustrating, but they can also mean that our work and lives as programmers are grounded in the world and not just in code. Good software exists to meet needs, not for its own sake, and a life without significant demands outside of a software project can make it much more difficult for a developer to design software that is both elegant and useful. Furthermore, more code, or time coding, doesn’t necessarily mean better code. I’ve definitely spent hours of intense coding, only to find that I’ve hacked my way into a corner, leaving behind a series of commits more convoluted than when I started.

Now I just need to remember this when I’m stuck in traffic rushing to make that after-school pickup.

Paying It Forward

This was originally posted on the Local Fourth blog as part of my participation in a community media innovation project at the Medill School of Journalism.

I’m finding the word community increasingly confusing, especially when navigating the world of hyperlocal publishing.  When someone says community, do they mean community like the city of Evanston, or the city’s West Side neighborhood, or a block club or church. Or, do they mean the community of users of a particular site? When do these groups intersect, when are they too disparate?  The 2010 Knight News Challenge goes as far as defining a specific Community category for entries:

Community: Seeks groundbreaking technologies that support news and
information specifically within defined geographic areas. This is designed to
jump-start work on technologies and approaches that haven’t arrived yet.
Unlike the first three categories, sub-
missions in this area must be tested in a geographically designated community.

But, in a Sept. 20 post announcing the 2010 challenge, the poster wrote “I think of this as our io9 category,” referring to a Gawker Media-run science-fiction and popular culture site.  Perhaps the poster was referring to the future-focused voice of the site, but it also surfaces the possibility that people may increasingly identify with communities and person-to-person interactions that aren’t geographically bound.

In looking at strong, geographically disparate online communities, groups of people engaging around free/libre/opensource software, or FLOSS projects are one of the most compelling. While they can exhibit the same segregation or bickering of physical communities, they can also be a model for people coming together to build something that serves a clear need. The way in which many projects are firmly grounded in utility and the way in which similar projects seem to sustain themselves not by competing but by understanding how their project does a job that’s different than other software is a lesson that media organizations, particularly in the hyperlocal space, would do well to learn.

FLOSS projects also complicate traditional notions of sustainability. While many projects have found ways to sustain themselves financially, either through donations, sponsorship or by incredible use of volunteer time coding, documenting and providing help and training for the project, FLOSS projects tend to put utility ahead of commercial viability. Making technology that serves a need and remains relevant and responsive to changing needs and to feedback from users is as important to the sustainability of the project as the dollars and cents.

We make use of a lot of FLOSS for implementing the technological part of the innovation project. While there are lots of ways that we could give back for this technology that is so useful us as developers (Palintir, a Chicago-based web development shop that specializes in sites based on the Drupal content management system, for instance, contributes code that they use to develop new features for their clients back to the larger Drupal community), the tight time constraints of graduate school and a rapid project mean that dollars are the best way that I can give back to these projects.

Even though most of the tools that I use to make technology are available free of cost, paying something for them helps me think of how I value the tools for this project. I’ve decided to donate the amount of money that I spend each week on a common indulgence during this project, going out for lunch with other team members, to some of the FLOSS tools that I’ve used the most in the last few weeks.

Python – most of the code in this project is written in this language. It’s flexible, easy to learn, has a large number of useful contributed libraries and is very readable making it easy to understand someone else’s code. Donate to the Python Software Foundation.

jQuery – If the back end of the project is written in Python, the front end is highly dependent on the jQuery javascript framework. JQuery makes it easier to implement some of the rich user interactions that people have come to expect on the web. Donate to the jQuery project.

Django – Django is a Python web framework that has its roots in the newsroom. . The first time I used the framework, I was amazed at how it streamlined the most tedious aspects of web development. When I’m curious about how to do something in the framework, I often discover that there’s an elegant approach provided in the framework along with clear documentation. Donate to the Django Software Foundation.

Vim – I was compelled to learn to use this editor when I started at my first tech job at a regional Internet service provider. The network administrator said that it was important to learn vi (Vim, an enhanced version of the classic UNIX editor, stands for vi improved) because you could be assured that it would be available on any UNIX system that you found yourself poking around. While the navigation of the program, which is keystroke heavy, seemed unintuitive at first, once I got used to it, the lightweight but highly customizable and extensible editor felt like it was designed just for me. Rather than asking for donations to sustain the project, Vim’s lead developer solicits donations for a charity that supports children in Uganda.

Firebug – I don’t know how I wrote programs for the web before Firebug. This Firefox extension helps me understand and tweak the HTML and CSS of a design and also see what is going on behind the scenes with Javascript errors and AJAX requests. Donate to the Firebug project.