The document was turned in, as scheduled, on Sept. 1. I have some worthwhile feedback, and I am editing now. Expect to have edits complete by October 1 and if all goes well, the book goes to the binder and the final copy goes on the shelf on Oct 15.

In the meantime, I have been showing the application to several people in the Massachusetts government, with interesting feedback.

There may be a real possibility with redistricting at the state level. I have met twice with our local Representative as well as the staff of the Committee on Elections. I am meeting on Oct. 8 with the state Senator who has the most institutional experience with redistricting. Then they want me to meet with the Chairman of the state Committee on Elections. At a minimum, they recognize that citizens don’t understand redistricting and want some way to get them involved.

The state uses precincts instead of tracts or blocks. There 40 state Senators and 160 Representatives. Note that one Senator District does not contain 4 Representative’s Districts. I can revive the voter registration, election history and financial demographic information I collected for Massachusetts and include that. My original plan had been to do that for CA, FL, NY and TX as well, but the data just wasn’t available.

I didn’t intend it, but I believe the application has an architecture that will accommodate this approach. Geography files describing nested regions that divide a state. Statistical characteristics associated with each region. Visualization to explore those characteristics at several levels. Display information on the map. Interactions to move regions around and see the results. Import and export plans. See the results on Google Earth / Google Maps.

We’ll see where this goes.

For this update, I have put the loaded plans in a table, each showing a bar representing relative scores for population, moi and city splitting. The table is also sortable on any of those score values.

I also added two new coloring schemes. I have been doing a lot of plan-refinement, and I could see that these features would be worthwhile. First – color each district by the percent over or under the target. There is already a small thumbnail image of the state which gives an overall picture, but this new approach gives more detail and helps with complex situations like the greater metropolitan NYC area.

When I import  a plan – for instance Brian Olson’s work – there may be tracts or blocks that don’t work – they are not assigned – or they are disconnected. They have to be resolved before proceeding. This coloring schemes shows each tract by the district it is assigned to. You can quickly see where the disconnected regions are.

Finally, when refining, I sometime want more information about the Tract/CCD involved. Right-clicking now gives a dialog with information on the name, population and racial composition of the associated Tract and CCD.

Here is a screen-cast showing the new features.

I think there is reasonable contrast between the value and density ranges. I may have to do some more work on the labeling, and probably state the specific item being measured where it currently says “Demographic Characteristic” but good enough for now.

http://www.nautilusone.biz/redistricting/olsonMaps/caDistricts.kmz

http://www.nautilusone.biz/redistricting/olsonMaps/txDistricts.kmz

1. In the list of plans to choose – show a glyph that indicates the relative merit of that plan in each of the three dimensions: population equivalence, moment of inertia and city-splitting. Sort the list by any of those dimensions.

2. When showing the demographic measurements of any region – like district, city or tract, encode the population density as well as the metric (like percent Caucasian).

There are some other details that would be nice – but these are the major ones, and I want to complete them so I can get on with the writing. However, they have some twists and turns. To compare plans – I need more plans. I have imported Brian Olson’s plans and resolved some issues along the way. They are only at the “tract” level, but you can actually see them on Google Maps. That way, you can take advantage of the rich trove of information that those maps offer. CA and TX are too large, but for the others, paste any of these URLs in the search box of Google Maps:

http://www.nautilusone.biz/redistricting/olsonMaps/olsonFL.kml

http://www.nautilusone.biz/redistricting/olsonMaps/olsonMA.kml

http://www.nautilusone.biz/redistricting/olsonMaps/olsonNY.kml

In each case, the districts will show in solid colors. If you “uncheck” a district in the control panel on the left, the underlying geographical details – like roads, city names and aerial photographs will be available. Later I may put in a transparency slider, like Google Earth has. Another way of exploring the district characteristics.

I have tried splitting CA and TX into two pieces, but they still seem to be too big. I may try some other approaches later.

Another potential source for plans is the approach outlined by Span, Gulotta and Kane, in their paper about their Moment of Inertia method. I was able to meet with Dan Kane, and I understand their work and it may be worth a day or two to try implementing their algorithm.

I have the current plans for the 111th COngress, the results of my Cricnione work, recursive splitting on cities and tracts and Brian Olson’s work. I am till hoping to get Georg Clark’s plan for CA. That should be enough for a reasonable range of comparisons.

The other piece to include density when I display the demographic characteristics of a region. Otherwise, given two regions of equal population, a small region will lose significance compared to a large region – even though they are equivalent. I currently use a ColorBrewer scale for the low to high range of any characteristic – with no indication of population density. Below the state average is green, above is purple.

It is defined with RGB values. I have tried translating it to the HSL color-space, and varying the lightness to encode the relative density. I could not get enough contrast between different values. I have been exploring the Lab color space, and that seems more promising. The image below shows a circle of color – with an *a value indicating the relative blue-green/magenta component, and *b indicating the relative blue/yellow component. Both of these axes come close to matching the eye’s channels for visual perception. Then the vertical dimension is lightness.

I found this applet tool for generating related values of Lab colors, and also translating them to the RGB space – Graphic CIE L*a*b* Calculator.

You can choose a and b values with the bottom and right sliders, and then vary the lightness with the left slider. After some experimenting, I decided to keep b fixed at 20 and vary a from -80 to +79. I would use that to represent how much a measure differed from the state average. Then I varied the lightness from 25% to 75%. Tha would encode the relative density of that region, relative to the state average. Results below seem OK for now. I may refine them later.  Initially, I varied the colors manually, but found some good formulas here – first translate L*ab vlaues to the XYZ space, and then to the RGB space.

Here is my current color scheme.

Other possibilities include different starting ending colors – using a scheme based on varying the b factor – or one moving diagonally across the color circle, simultaneously varying both a and b. I need to improve my color generating function and then try it on my real range of demographic values.

You can see a screen-cast (~3 minutes) here

This is one of the key concepts in my application – that citizens need to relate to a plan based on the city they live in – not just the tract, since I believe few people know or understand what tract they live in.  It turns out that this requires a lot of wrangling and some compromises to be realistic. The census doesn’t really use “cities” or “towns” that we relate to. They have CCDs – a “Census County Division” which closely approximates something we are familiar with. Although most CCDs are large and contain multiple tracts – some tracts span across more than one city, and some CCDs are actually smaller than a tract. Anyway – part of the challenge.

I believe the final “major function” to implement is the plan table – showing the relative merit of plans by the three measurements, and sortable by those values.

There are several external refinements, and quite a number of internal issues, but I am confident I’ll finish the function by early July and be ready to move on to the writing.

Biggest refinement right now is changing the color scale used to show the demographic values on the map. At Hanspeter Pfister’s suggestion, I want to apply a luminance factor based on the population density of each tract or CCD as the demographics are illustrated. Otherwise, dense areas, which are significant, don’t stand out from large sparsely populated areas. Making the dense areas more intense will improve the perception of the demographics.

Below I show an image of the three CCDs, with the centers marked. Then another image of an aerial photographic image of the underlying terrain. You can see that there actually is a Cedar Creek Lake – one CCD is in the north-west corner, another is in the south -west corner, and another is off to the east a bit. No other CCDs around. There are some other instances where there are two places with the same name, but this one stands out with three.

For now, I will leave the ambiguity alone. There will be more than one entry for CCDs like this. Each one will take you to the matching CCD on the map.

A different problem occurs in New York. There are several places – maybe 25, where a “town” and a “city” share the same name. Most are relatively small and upstate, although Elmira and Poughkeepsie show up. The field int the KML file where I extracted the name  provides the name, without the city/town qualifier. If I have the time, there is another field that has the fully qualified name For now, I am going to leave that alone as well.

California is always the toughest example. When I am moving tracts there by dragging from district to district, I see a slow response. I am not ready to solve it yet, waiting until I get all the functions done. However, I wanted a better handle on what might be causing it. At least for assurance that my direction wasn’t digging a bigger hole. On the mouse down I have to recognize the tract and the district it is coming from. On the mouse released, I recognize which district is the destination. Then I change the district id for that tract and update house-keeping information and then repaint the map and the supporting displays. An early thought was that connecting the mouse location to the tract or district was slow, since I only iterate through the possible choices and check whether the polygon contains the point. I could improve that with something like a KD-Tree. I am intrigued with that sort of algorithm, but I wanted some measurements first. This wasn’t yet a full-fledged profiling exercise, because I was just doing one move at a time. So, I just put in print statements and watched when the came out. As a next step, I could capture the system time for each event in a  sequential list and then analyze it at the end, to avoid any overhead or interference from the printing.

The results showed that recognizing the tract and district happened quickly. Updating the district information to display the population and MOI  measures followed. The bottleneck turned out to be evaluating the city splitting. I have a somewhat inefficient method that is correct, but it updates the entire state. For the tract move, I was re-invoking that entire method. At a minimum, I only need to update the origin and destination districts, and really only the specific CCD associated with the tract being moved. I can address that later. Get on with the rest of the application for now.