Programmatically reshaping data

Yesterday at work, my coworker needed to reshape a data table, from CSV. The data was a time series, with one row per date. He wanted to reshape the data so there was one row per day of the year with a column for the desired variable for each year. He observed that there were 93 distinct days of the year for each year and used some clever Excel functions and auto-fills to do this in about 5 minutes. I spend considerably more time figuring out how to do this in PostgreSQL, Python and R.


CPS School Budgets

cps-budgets-og

On Monday, July 13, 2015, Chicago Public Schools (CPS) released preliminary spending plans for the 2015-2016 school year. Many schools face significant budget cups. We made a simple [news app to let users explore how different school budgets changed](http://apps.chicagotribune.com/news/local/cps_school_budgets/).

Additional coverage:

* [CPS Budget: Deep Cuts For Neighborhood Schools, Cash Infusion For Charters](http://www.dnainfo.com/chicago/20150713/loop/cps-student-budgeting-holds-steady-but-schools-will-feel-pinch) (DNAinfo Chicago)
* [Chicago Public Schools’ budgets spend $500 million district doesn’t have](http://www.chicagotribune.com/news/local/breaking/ct-chicago-public-schools-budgets-20150713-story.html)


Chicago Police Department accountability coverage

Update July 27, 2015

Update July 24, 2015, allegations by ex-IPRA investigators of pressure to change findings

Update July 19, 2015, more stories

A few weeks ago, the Trib published a story about the review of complaints against Chicago Police Department officers to which I contributed data analysis. This week, a couple of stories about Chicago’s Independent Police Review Authority (IPRA) came across my radar, so I wanted to share them here:


Become a Better Programmer Through Mundane Programming

“Become a Better Programmer Through Mundane Programming” is a session that Dan Nguyen and I did at SRCCON this year. Here’s the description from the session page

One of the most difficult challenges for learning programmers is being able to go beyond basic lessons to building “the next Facebook”, or even just applying programming to your work projects. The secret is that programming can be practiced — and used — on everyday, somewhat “mundane” computational tasks. When you learn how to program to do things that are helpful to you, you not only become a better programmer, but you gain insight on how programming fits into the bigger projects you hope to build.

This session will involve both discussion of everyday programming and demonstrations of creating and refining “mundane” programming scripts, with the goal of brainstorming ideas and use cases to expand the scope of problems we can use programming to solve.

The most validating experience was one of the participants showing me some shell script she had written using the t Twitter client to auto-follow SRCCON tweeters.

Resources:


Tribune analysis: Cops who pile up complaints routinely escape discipline

Activists bearing posters of Mayor Rahm Emanuel and police Superintendent Garry McCarthy protest police treatment of residents during a demonstration in front of the mayor’s City Hall office in February 2015. (Nancy Stone, Chicago Tribune)

This is a story that Jeremy Gorner and I wrote from police complaint data that Gorner got from an FOI request. Read the story, Tribune analysis: Cops who pile up complaints routinely escape discipline


Introducing Clarify

state_summary_page__KY

Source has a blog post that Derek Willis and I wrote about Clarify, a small Python library for parsing data from Clarity Election Night Reporting (ENR) systems.

This was developed during OpenNews’ Elections Code Convening, which was really fun, productive and a compelling alternative to hackathons as a model for collaborative hacking.


Cleaner Data

11899178333_3a4781efb9_h

Along with David Eads, I gave a talk on cleaning data based on my experience working with convictions and elections data at the Chicago School of Data Conference. The session detail page is here and my slides are here. You might find my speaker notes helpful. They’re in the GitHub repo for the slide deck.

Image: Mud Volcanos creative commons licensed (BY-ND) flickr photo by “Caveman Chuck” Coker


New music for back to school

Prince (the other one)

I work up this morning to the din of the first day of school. Even being out of school for a while, I feel like the fall still has this sense of beginning to me, even as the summer comes to the close.

Out of curiosity looked up the current band of a friend I’ve fallen out of touch with and was instantly pulled in by what I heard. Have you ever been haunted by a song and found yourself playing and replaying the same song a dozen times to see if the lyrics match up with the music’s powerful first impression? That’s what happened to me when I heard “How ya been feelin'”, a track from a forthcoming 7″ from Austin’s Prince. It has such a great combination of energy and sadness. The lyrics are direct, but use imagery that evokes something more than the words. The other recordings on the band’s Bandcamp page are pretty great as well.

Una Bèstia Incontrolable and Iron Lung

Last week I saw Una Bèstia Incontrolable and Iron Lung play and I enjoyed hearing two bands play interesting heavy music that’s still grounded in punk and hardcore idioms. It’s hard because seeing heavy bands live sometimes feels like the way we engage with the music is so predictable compared with the music.

I really enjoyed this Noisey interview with Iron Lung because it revealed some surprising influences that I hadn’t listened to very intently including Flipper’s “Generic Flipper” and Rudimentary Peni’s “Death Church”.

Dystopian Society

Some dear friends just moved to Florence, Italy and I was curious what was going on in local music. This death rock band is the first thing that came up when I google “Florence Italy DIY punk”.


Tools for visualizing network graphs

Tools for extracting structured data from a PDF file

These are tools that have been suggested to me to extract structured data from a PDF files: