Floodlight

Floodlight is a web-based platform for telling community stories from people, places and organizations around the Denver metropolitan area.  The project was originally supported by The Piton Foundation and funded in part by a grant received as part of the Knight Foundation Community Information Challenge.

Role

Lead developer

Details

Since many of the stories told on Floodlight are about work being done in particular Denver, Colorado neighborhoods, authors needed to be able to tag their stories with boundary geographies (cities, neighborhoods, zip codes) or provide addresses of specific places, which are automatically geocoded and associated with boundary geographies.

For this project, I relied on the excellent spatial data support of the PostGIS database and abstractions offered by the Django Frameworks ORM to implement a data model and create scripts to easily load new boundary geographies from shapefiles and define their relationships with other boundaries.

Users searching for stories in their community can use a faceted browsing interface that not only filters based on address and boundary geography, but also other taxonomies.  In order to integrate these different types of filters, I used the Solr search server.  On the front end, the Backbone framework provides an interface to the faceted browse and the Leaflet mapping library is used to display boundary geographies and story markers based on the user’s filters.

The application is powered by a RESTful API that is consumed by a Backbone-based story builder.

In addition to the web platform, the project also included a great capacity-building component.  Team members conducted “story raising” events that trained community members in different digital storytelling skills and a “story navigator” worked with community groups to discover and tell stories using the platform.

More Information

Screenshots

Home_Floodlight_-_2014-01-15_22.07.30

Explore_Floodlight_Floodlight_-_2014-01-15_22.11.14

Floodlight_Why_Stories_Matter_A_Reflection_on_the_Denver_Camping_Ban_and_the_Conversation_around_Homelessness_-_2014-01-15_22.14.54

Testament

Testament is a Django-based prison book project management application originally written by Andrew Badr. It lets volunteers track sent and returned packages and inmate requests in the hopes of minimizing delays for inmates and helping the book projects evaluate their services for themselves and funders.  It is used by the Midwest Pages to Prisoners Project and other prison book projects.  I am a contributor to this application’s code and migrated from a pencil and paper on index card database to Testament while coordinating the Midwest Pages to Prisoners Project.

Draw the Line Midwest

Open source mapping software allowed users to view state and federal legislative districts and draw their own district plans during the 2010 redistricting process.

Role

Integrating developer

Details

I deployed and customized instances of the open source District Builder software for a number of Midwestern states.  Users could view existing districts or create new district maps based on block and tract census geographies.  Some instances provided a platform for competitions to see if users could create more balanced district plans based on political, racial and population demographics.

In order to minimize hosting costs, the instances were deployed on a single server with resources provisioned to virtual hosts using VMWare.  In addition to the provisioning and hosting, I provided data preprocessing and cleaning in order to add additional demographic data to use to evaluate district plans.

Paying It Forward

This was originally posted on the Local Fourth blog as part of my participation in a community media innovation project at the Medill School of Journalism.

I’m finding the word community increasingly confusing, especially when navigating the world of hyperlocal publishing.  When someone says community, do they mean community like the city of Evanston, or the city’s West Side neighborhood, or a block club or church. Or, do they mean the community of users of a particular site? When do these groups intersect, when are they too disparate?  The 2010 Knight News Challenge goes as far as defining a specific Community category for entries:

Community: Seeks groundbreaking technologies that support news and
information specifically within defined geographic areas. This is designed to
jump-start work on technologies and approaches that haven’t arrived yet.
Unlike the first three categories, sub-
missions in this area must be tested in a geographically designated community.

But, in a Sept. 20 post announcing the 2010 challenge, the poster wrote “I think of this as our io9 category,” referring to a Gawker Media-run science-fiction and popular culture site.  Perhaps the poster was referring to the future-focused voice of the site, but it also surfaces the possibility that people may increasingly identify with communities and person-to-person interactions that aren’t geographically bound.

In looking at strong, geographically disparate online communities, groups of people engaging around free/libre/opensource software, or FLOSS projects are one of the most compelling. While they can exhibit the same segregation or bickering of physical communities, they can also be a model for people coming together to build something that serves a clear need. The way in which many projects are firmly grounded in utility and the way in which similar projects seem to sustain themselves not by competing but by understanding how their project does a job that’s different than other software is a lesson that media organizations, particularly in the hyperlocal space, would do well to learn.

FLOSS projects also complicate traditional notions of sustainability. While many projects have found ways to sustain themselves financially, either through donations, sponsorship or by incredible use of volunteer time coding, documenting and providing help and training for the project, FLOSS projects tend to put utility ahead of commercial viability. Making technology that serves a need and remains relevant and responsive to changing needs and to feedback from users is as important to the sustainability of the project as the dollars and cents.

We make use of a lot of FLOSS for implementing the technological part of the innovation project. While there are lots of ways that we could give back for this technology that is so useful us as developers (Palintir, a Chicago-based web development shop that specializes in sites based on the Drupal content management system, for instance, contributes code that they use to develop new features for their clients back to the larger Drupal community), the tight time constraints of graduate school and a rapid project mean that dollars are the best way that I can give back to these projects.

Even though most of the tools that I use to make technology are available free of cost, paying something for them helps me think of how I value the tools for this project. I’ve decided to donate the amount of money that I spend each week on a common indulgence during this project, going out for lunch with other team members, to some of the FLOSS tools that I’ve used the most in the last few weeks.

Python – most of the code in this project is written in this language. It’s flexible, easy to learn, has a large number of useful contributed libraries and is very readable making it easy to understand someone else’s code. Donate to the Python Software Foundation.

jQuery – If the back end of the project is written in Python, the front end is highly dependent on the jQuery javascript framework. JQuery makes it easier to implement some of the rich user interactions that people have come to expect on the web. Donate to the jQuery project.

Django – Django is a Python web framework that has its roots in the newsroom. . The first time I used the framework, I was amazed at how it streamlined the most tedious aspects of web development. When I’m curious about how to do something in the framework, I often discover that there’s an elegant approach provided in the framework along with clear documentation. Donate to the Django Software Foundation.

Vim – I was compelled to learn to use this editor when I started at my first tech job at a regional Internet service provider. The network administrator said that it was important to learn vi (Vim, an enhanced version of the classic UNIX editor, stands for vi improved) because you could be assured that it would be available on any UNIX system that you found yourself poking around. While the navigation of the program, which is keystroke heavy, seemed unintuitive at first, once I got used to it, the lightweight but highly customizable and extensible editor felt like it was designed just for me. Rather than asking for donations to sustain the project, Vim’s lead developer solicits donations for a charity that supports children in Uganda.

Firebug – I don’t know how I wrote programs for the web before Firebug. This Firefox extension helps me understand and tweak the HTML and CSS of a design and also see what is going on behind the scenes with Javascript errors and AJAX requests. Donate to the Firebug project.

Finding duplicate records in a books to prisoners database application

High on my list of neglected tech. projects is the Testament books to prisoners database web application.  This is the database program that projects like the Midwest Pages to Prisoners Project use to track packages sent and returned and books requested in the hopes avoiding delays in delivering books to incarcerated people and to provide metrics that grant providers like.

One of the design challenges has to do with duplicate records.  Recipients of books are identified by their state/federal department of correction (DOC) number (if they’re in a state or federal prison – most jails don’t use ID numbers), their state of incarceration and their name.  I assume that the database was designed originally to minimize barriers for the book project volunteers so both the name and DOC# are free text fields.  Javascript is used to match existing records based on the DOC#, but there is still a large possibility for duplicate records.

The reason for duplicate records is that both the person writing to request books and the volunteer may list their name and/or DOC# inconsistently.  For instance, the state may store the DOC# in their database as A-123456 but the incarcerated person may write it as A123456 A-123-456 or just 123456.  Volunteers who don’t know about this and aren’t careful may not check beforehand for an existing record.

This is probably preventable through more sophisticated validation, but we still need a way to find duplicates in the existing records.  As this application is written in the Django framework, I want to try to use the Django API to find matches.

At first thought, it seems like I will have to iterate through each inmate record and check if there is a duplicate record.  This seems pretty slow, but I can’t think of a better way to do this.  At this point, there aren’t so many records that this approach will fail, but it would be nice to do something slicker.

The other problem is how to match a duplicate.  One approach might be to build a regexp for the DOC# (for instance, match either the first character or omit it, allow dashes or spaces between all characters, …) and then use the iregexp field lookup to try to find matches. One challenge with this is that the current Testament codebase is using Django 0.97 (I think) and iregexp is only available starting in 1.0.  Maybe it’s time we updated our code anyway.

There is also the Python difflib module that can compute deltas between strings.  However, it seems like this would slow things down even further because you would have to load each inmate object and then use difflib to compare the DOC#s.  I assume that the previous approach would be faster because the regexp matching happens at the database level.

Django: Querying data from the Python shell

I needed to get some stats for some research that we’re doing and was happy to see that you can use Django and the python shell to query testament data in a way that’s database independent.  It’s a little unintuitive if you’re thinking in SQL mode, but it is usable and super-helpful.  I wanted to share it with ya’ll in case you needed to quickly pull stats or examine info.

Helpful reference Django docs:

  • http://docs.djangoproject.com/en/dev/topics/db/queries/#making-queries
  • http://docs.djangoproject.com/en/dev/ref/models/querysets/#queryset-api-reference

Print the prison name, city, and state of all prisons that received a package sent by the Midwest Pages to Prisones Project from 2009-01-01 to 2009-03-22

geoff@btp:/var/www/testament/testament_trunk/btp$ python manage.py shell
>>> import datetime
>>> from core.models import Prison, Package
>>> start_date = datetime.date(2009, 1, 1)
>>> end_date = datetime.date(2009, 3, 22)
>>> prisons = Prison.objects.filter(package__sent_on__range=(start_date, end_date), package__group__username__exact='mwpp').distinct()
>>> for prison in prisons:
>>>    print "%s %s, %s" % (prison.name, prison.city, prison.state)