Downloading and Extracting US Census Bureau TIGER/Line Shapefiles

The United States Census Bureau (census.gov) site is a great resource for free spatial data such as state, county, census tract, etc. boundary files. These files can be used as is or used as a building block for creating custom sales territories. They can be loaded into SQL Server for use in spatial queries or for displaying boundaries on a map in an SSRS report. They can also be used to build map tiles for use with the Bing Maps API. This post, however, is focused on downloading and extracting these files.

These TIGER (Topologically Integrated Geographic Encoding and Referencing)/Line Shapefiles are available in many vintages, including definitions from the 1990, 2000 and 2010 censuses, with annual updates through 2013 (2014 should be available soon). The Census Bureau site provides several ways to access and download these files, but the easiest way is to use their FTP site (ftp://ftp2.census.gov/).

You can use an FTP client or, as shown here, you can simply use Windows Explorer.

Start by opening Windows Explorer and pasting ftp://ftp2.census.gov/ directly into the address bar. 

Navigate to the geo/tiger/. From this folder you will see the different vintages of TIGER files available. To illustrate the next step, lets say we're interested in the 2013 Census Tract boundaries.

Navigate to the TIGER2013/TRACT folder. In the right hand window (i.e. file list), you'll see a list of compressed (.zip) files with names based on the State FIPS Code, which is a two character numeric code assigned to each state, the District of Columbia and outlying areas of the U.S. (Puerto Rico, Guam, etc.).

Select the ZIP files you're interested in. I'm only interested in the U.S. states and DC, so I'm selecting files tl_2013_01.zip (AL) through tl_2013_56.zip (WY).

Right-click one of the selected files and select Copy from the popup menu.

Navigate to a folder on your local network, right-click the folder and select Paste. This may take a few minutes. I'll wait.

Now that you have the files on your local network, we need to extract (i.e. unzip) them. Unfortunately Windows Explorer falls short when needing to unzip multiple files. If you already have WinZIP or WINRAR installed, you could use either of those to unzip multiple files. I don't, so I'm going to use the following PowerShell script that I found on Steve Schofield's Blog: 

function UnZipMe($zipfilename,$destination) 
{ 
    $shellApplication = new-object -com shell.application 
    $zipPackage = $shellApplication.NameSpace($zipfilename) 
    $destinationFolder = $shellApplication.NameSpace($destination) 

    # CopyHere vOptions Flag # 4 - Do not display a progress dialog box. 
    # 16 - Respond with "Yes to All" for any dialog box that is displayed. 

    $destinationFolder.CopyHere($zipPackage.Items(),20) 
} 

# replace the following with the folder that contains your zipped files
$a = gci -Path C:\Data\USCBCensusTracts2013 -Filter *.zip 

foreach($file in $a) 
{ 
    Write-Host "Processing - $file" 
    UnZipMe –zipfilename $file.FullName -destination $file.DirectoryName 
}

Once this script finishes running, you will have all of your shapefiles ready to work with.

Creating Map Tiles (Part 3) - Generating Tiles from a TileMill Project Using MBUtil and PowerShell

This post is a continuation of Creating Map Tiles (Part 2) - Creating a Sales Territory Boundary Set Using TileMill, where I prepared my transparent boundary set in TileMill. In this post I use PowerShell to call TielMill and MBUtil to generate the map tiles.

As I mentioned in Part 1, for much of the process of creating map tiles I relied on an excellent series of posts by Pedro Sousa on his blog, Pedro's Tech Mumblings. In Part 2 of his series, he walks you through how to export your TileMill project to a MBTiles SQLite file and then use MBUtils (a command line MapBox tool) to export the map tiles. Below, I've taken a the same approach but created a PowerShell script to automate the process.

In order to perform this process manually or using the PowerShell script below, you need to install Python with SQLite and MBUtils. Pedro walks you through this, however I performed the following steps outlined in the accepted answer to Is there any way to use mbutil on windows? posted on StackExchange.

• Download and install ActiveState Active Python 
• Install MBUtil by downloading and unzipping to C:\mbutil-0.2.0.
• From a command-line navigate to C:\mbutil-0.2.0 and run "setup.py install" to install mbutils

At this point the pre-requisites for the following PowerShell script exist. 

function CreateTiles() {
    param(
        [Parameter(mandatory=$true)] [string]$project,
        [Parameter(mandatory=$true)] [string]$inputPath,
        [Parameter(mandatory=$true)] [string]$outputPath
    )
    # delete .mbtiles, .export and .export-failed files
    remove-item  "$($inputPath)$($project).*"
    
    # delete directory
    $pngPath = $outputPath + $project
    if (test-path $pngPath){
       write-host "Deleting $($pngPath)... this may take awhile." 
       #remove-item $pngPath -recurse  # very slow
       $fso = New-Object -ComObject scripting.filesystemobject
       $fso.DeleteFolder($pngPath, $true)
    }
            
    # Create mbtiles file. Requires TileMill.
    # changed the following so that the executable's path doesn't 
    # contain spaces (i.e. using the call (&) operator is optional)
    cd "C:\Program Files (x86)\TileMill-v0.10.1\tilemill\"
    $mbtilesFile = "$($inputPath)$($project).mbtiles"
    & ".\node.exe" ".\index.js" "export" $project $mbtilesFile "--format=mbtiles"
    
    # Create PNGs from mbtile file. Requires:
    # Python with SQL Lite (install ActivePython);
    # MB Util (download mbutil zip from https://github.com/mapbox/mbutil/tree/v0.2.0).
    python "C:\mbutil-0.2.0\mb-util" --scheme=xyz $mbtilesFile $pngPath
}

CreateTiles "State_And_County_Boundaries" "C:\SHPs\" "C:\MapTiles\"

The first step in the script removes the existing MBTiles files, if they exist, which will block the creation of the new file. It also deletes the existing path of previously created tiles. This is necessary because I don't want to have any old tiles mixed with the new (remember not all tiles are generated within the bounds set in the TileMill project because my map has a transparent background). Depending on the number of tiles generated in a previous run, this can take awhile.

The second step is what you can do manually from TileMill by clicking the Export button in the main toolbar and selecting MBTiles. It calls TileMills index.js file and is documented here on the MapBox site.

The final step calls the MBUtils executable to produce the map tiles in a set of folders based on the XYZ tiling scheme.

The files now just have to be copied to a web server where they can accessed by the Bing Maps API. The following, modified from a function taken from another of Pedro's blog posts (taken in turn from a blog post by Alastair Aitchison), shows how the tiles can be called from the Bing Maps API using JavaScript:

var tileSource = new MM.TileSource({ 
    uriConstructor:  function getTilePath(tile) {
        // only created custom tiles for zoom 0-12
        if (tile.levelOfDetail <= 12) {
            var x = tile.x;
            var z = tile.levelOfDetail;
            var yMax = 1 << z;
            var y = yMax - tile.y - 1;

            return "Images/Tiles/BaseMap4/" + z + "/" + x + "/" + y + ".png";
        }
    }
});

var tileLayer = new MM.TileLayer({ mercator: tileSource, opacity: 1 });

// Push the tile layer to the map
map.entities.push(tileLayer);

One of the modifications that I made to the above prevents the call to retrieve tiles that were not generated for certain zoom levels. This will prevent the server from having to generate a multitude of 404 error response codes as you zoom below the level for which tiles were generated. Another modification that I made to my own code (not included above) was to add a version argument to the end of the URL which is retrieved from a web service call to the database when the page initially loads. I use this argument to force the fetching of new tiles when the tiles have been replaced on the server.

The Tile Layers Used with the Bing Maps API

Using the PowerShell script above, I have been able to generate three different, fairly static boundary sets, consisting of 3 to 5 layers each (resulting in around 100,000 tiles for each set). Along with the script from Part 1, Creating Map Tiles (Part 1) - Exporting MapInfo TAB Files to ESRI SHP Files Using PowerShell and MapInfo, I have been able to automate the creation of an additional two boundary map tile layers that have to be generated monthly, due to changes in the boundary definitions.

Creating Map Tiles (Part 2) - Creating a Sales Territory Boundary Set Using TileMill

This post is a continuation of Creating Map Tiles (Part 1) - Exporting MapInfo TAB Files to ESRI SHP Files Using PowerShell and MapInfo, where I prepared my MapInfo boundary files for use in TileMill. In this post I focus on creating a boundary map in TileMill that can eventually produce tiles which will be overlaid onto a Bing Maps tile layer using the Bing Maps API. To keep it very simple, my map will only consist of state and county boundaries.

The first step in creating a transparent boundary set is to specify "png (24-bit)" as the Image Format and to uncheck the "Default data" checkbox when creating the new project (like MapInfo, in TileMill you have a project, or workspace, which will contain your layers). 

When you open the project, you will notice that TileMill automatically created a stylesheet (style.mss) with a default Map style consisting of a solid blue background (unlike MapInfo which use the Layer Control, TileMill uses a CSS type stylesheet to control the appearance of the layers). The second step in creating a transparent boundary set is to remove or comment out  (i.e. using comment blocks /* */) the Map selector.

The next step is to open the Layers List using the bottom button in the Editing tools (located at the bottom left of the map). Next, add a layer by clicking the Add Layers button. Add the bottom, or most detailed, layer first (counties, in my example). Browse for the ESRI Shape file and click "Save & Style" when done (entering a Carto ID and Class is optional). The layer will be added to the map with a default style.

The third step in creating a transparent boundary set is to comment out or remove the polygon-opacity and polygon-fill attributes of polygon layers. You can also change the line-color and line-width to your preference. At this point, you can add more polygon layers (I'm adding states) and perform the same edits to the attributes for each selector. My result looks like:

Right now the map is pretty cluttered with all of the county boundary lines. This brings up the other requirement for my transparent boundary set which was to have the more detailed layers only become visible as you zoom in. To accomplish that, we can apply a zoom selector which can specify at what zoom levels the layer is visible. For my simple state and county set, the stylesheet now looks like following:

#countyregion {
  [zoom >= 6]
    {
      line-color:#00ffff;
      line-width:1;
    }
}

#stateregion {
  line-color:#0000ff;
  line-width:1;
}

Because this map will eventually overlay a Bing Maps road or satellite layer, I also want to make the boundaries stand out a little more. This can be done using additional styles and is basically taken straight from the TileMill manual. The "outline" CSS style pseudo class is used to identify the purpose of the style, but is not required. It is important to put this style setting before the existing, as it will be rendered first.

#countyregion::outline {
  [zoom >= 6]
    {
      line-color:#00ffff;
      line-width:7;
      line-opacity:0.1;
      line-join: round;
      line-cap: round;
    }
}

#countyregion {
  [zoom >= 6]
    {
      line-color:#00ffff;
      line-width:1;
    }
}

#stateregion::outline {
  line-color:#0000ff;
  line-width:7;
  line-opacity:0.1;
  line-join: round;
  line-cap: round;
}

#stateregion {
  line-color:#0000ff;
  line-width:1;
}

Additionally, the line-join and line-cap attributes were used to smooth out the ends of the line segments. The result looks like this (note that I'm zoomed in to 6 to show the county boundaries):

The final step in setting up the TileMill project is to create the settings that will be used when generating the tiles. You do this from the Project Settings panel, accessed using the wrench button in the main toolbar (i.e. the upper right of the application). By defining the Bounds (-179.7253,18.1928,-66.3464,71.4848), Center (-97.6355,40.6181,10) and the Zoom (0-12), you can significantly reduce the number of tiles that will be generated. Additionally, because of the transparent background, tiles will not be generated if there is no visible boundary. This results in a much smaller number of tiles than if there were a visible background to the map.

At this point I have created a simple transparent boundary set with a detailed (i.e. county) layer that becomes visible as one zooms in. The next blog post, Creating Map Tiles (Part 3) - Generating Tiles from a TileMill Project Using MBUtil and PowerShell, will present a PowerShell script that will generate the layers based on this TileMill project.

Creating Map Tiles (Part 1) - Exporting MapInfo TAB Files to ESRI SHP Files Using PowerShell and MapInfo

I recently had the requirement to display sales territory boundaries within a Bing Maps web application that I had written. The boundaries were actually a set of custom sales territories (e.g. region, district, markets) using counties as a building block. After some research, I determined the best way to incorporate these boundaries into my Bing Maps application was to generate a set of map tiles which would progressively show the boundaries (e.g. regions then districts then markets and finally counties) as you zoomed in.

For much of the process of creating map tiles, I relied on an excellent series of posts by Pedro Sousa on his blog, Pedro's Tech Mumblings. Starting with Part 2 of his series, he introduces you to TileMill and shows how it can be used to design a map.

One of the first steps after installing TileMill is to create a new project and add map layers to it, and one of the options for the map layers is to use ESRI Shape (.shp) files. I had a bit of a head start in the process as I had been using MapInfo and MapBasic to create MapInfo TAB files for many years for use in a desktop application that I had written back in the late 1990's. Because of my familiarity with MapInfo, I knew that it included a MapBasic tool called the Universal Translator which is a data conversion tool built on Safe Software's FME (Feature Manipulation Engine) technology. This could be used to translate my MapInfo TAB files into ESRI Shape files. However, I had several different boundary sets, consisting of several different boundaries (a dozen total), that I needed to create, and using an interactive tool would be tedious. I also had two boundaries that needed to be generated monthly, so automating the process was a must.

Because I could use MapBasic to write something to call the Universal Translator within MapInfo I knew I could automate the process. However, I got lucky when I unexpectedly stumbled upon the Universal Translator User Guide, which included information on how the MapInfo Universal Translator could be called from the command line. Armed with this information, I was able to quickly put together the following PowerShell script to convert all of my files.

function ExportTabToShp() {
    param(
        [Parameter(mandatory=$true)] [string]$TAB_PATH,
        [Parameter(mandatory=$true)] [string]$outputPath
    )

    $baseName = [System.IO.Path]::GetFileNameWithoutExtension($TAB_PATH)
    
    # Application name of the MapInfo Universal Translator
    $CMD = "C:\Program Files (x86)\MapInfo\Professional\UT\Imut.exe"
    
    # Command that generates a semantic control file
    $IMUT_COMMAND = "CFGenerate"
    
    # Input format is MapInfo .TAB
    $INPUT_FORMAT = "MAPINFO"
    
    # Output format is the ESRI Shapefile format
    $OUTPUT_FORMAT = "SHAPE"
    
    # Path and filename of the semantic control file
    $FME_PATH = $outputPath + $baseName + ".fme"
    
    #Path and filename of the logfile to be generated
    $LOG_FILENAME = $outputPath + "mut.log"

    # Generate a mapping file:
    write-host "Note the RemoteException that is thrown is simply a message stating that the 'Mapping File Generation' was successful:" -ForegroundColor Green
    & $CMD $IMUT_COMMAND $INPUT_FORMAT $OUTPUT_FORMAT $TAB_PATH $FME_PATH LOG_STANDARDOUT "YES" LOG_FILENAME $LOG_FILENAME LOG_APPEND "YES"

    # Run the mapping file:
    write-host "Note the RemoteException that is thrown is simply a message stating that the 'Translation' was successful:" -ForegroundColor Green
    & $CMD $FME_PATH --HOMOGENOUS_GEOMETRY "YES" --_SHAPE_FILE_TYPE "2d" --SourceDataset $TAB_PATH --DestDataset $outputPath --_EXTENSION "TAB" --_BASENAME $baseName --_FULLBASENAME $baseName
}

ExportTabToShp "C:\TABs\county.tab" "C:\SHPs\"

The first call in the ExportTabToShp function generates an FME file, which is a "mapping file" that controls the translation. The second call runs the translation using the generated FME file. Refer to the User Guide link above for more information required to expand this function to support other translations.

Note the use of the Call operator (i.e. &) in the above PowerShell script. This allows one to call an executable formatted as a string and is useful when there is a space in the path to the executable.

In the next part of this series, Creating Map Tiles (Part 2) - Creating a Sales Territory Boundary Set Using TileMill,I'll walk through a couple of the steps that I used in TileMill to create the sales territory boundary sets.