Saturday, January 17, 2009

Geo-coding Your Constituents

Everyone likes maps & map mash-ups. There seems to be something very appealing about being able to see things visually compared to textually. Since we launched our Interactive Class Map about 8 months ago it's become one of the longest viewed pages on our website. The average constituent spends almost 4 minutes and 30 seconds doing nothing but panning and zooming around the map looking at their classmates. We have had a simple NetCommunity Directory Part that displayed the same information just in list form for years. It's still widely used but users don't spend anywhere near the amount of time viewing it as they do the Interactive Class Map. 

In this post I'm going to cover what is really the most important part of working with maps on your NetCommunity website, geo-coding. So, what is geo-coding? It's the process of taking an address and finding its latitude and longitude. Once you have done that it becomes much easier to display that location on a map. Just a warning, this is going to be a bit technical. 

I hacked out a quick and dirty webservice that I'll make available at the end of this post for download. It uses the REAPI to get a constituent’s address from RE and then uses Google to find the lat/lng. I'll take you through the code step by step. 

CAUTION: This method is NOT the best approach if you are planning on processing a lot of records. Google imposes a 15K limit on the number of requests per day from a single IP address. They also frown upon making requests too quickly and may block you for that. If you are planning on using this method to process your entire database you really should batch it up in small chunks over the course of a few days. Also be sure your timer makes each request to Google a few seconds apart. The code I'm going to show you does not do that, but it wouldn't be too hard to add. Since you have the source code you should be able to add that functionality.

So let’s get started.

The first step is to create a new Web Service project in Visual Studio 2008. The idea behind this project is that other NetCommunity custom parts could use it. If you are planning to bulk geo-code your constituents you could also use it in something like a Windows Service that runs over a few days to process your whole database.

Now that the webserivce project has been created we want to go ahead and start setting up the two functions we're going to use.

We're going to need to import these references;
Imports System.Web.Services
Imports System.Web.Services.Protocols
Imports System.ComponentModel
Imports Blackbaud.PIA.RE7.BBREAPI
Imports System.Xml
Imports System.Net
Imports System.IO

You'll probably need to add Blackbaud.PIA.RE7.BBREAPI to your project. If you don't have the REAPI you'll need it. Otherwise you'll have to modify this code to use the NetCommunity API which is free. I know the REAPI better for this stuff which is why I used it. 

'''
    ''' this method will take a REID and get the address
    ''' then it will use google maps to lookup that address address
    ''' note: the value of the returned array with the lat/lng is nothing if
    ''' either the address coudln't be geocoded or it wasnt' specific enough to get back a single result
    '''
    '''
    '''
    '''
     _
    Public Function getConstitLatLng(ByVal sREID As String) As ArrayList
        Dim oConstit As New CRecord 'this is the constit object
        Dim aLatLng As New ArrayList(2) 'we'll be returning this.
        Dim RE7 As REAPI = initAPI() 'this is located in the source. be sure to change up your serial, username, pass and db number. 
        Dim oAddress As CConstitAddress 'address object
        Try
            With oConstit
                .Init(RE7.SessionContext) 'init
                .Load(sREID) 'load the passed in REID
                For Each oAddress In .Addresses 'loop over all the addresses
                    If oAddress.Fields(ECONSTIT_ADDRESSFields.CONSTIT_ADDRESS_fld_PREFERRED) = -1 Then 'grab the prefered one, yes I know it's -1 trust me on this -1 = true for some strange reason in the BBREAPI...I don't ask questions if it works. 
                        'pass the retrieved criteria to the seekCoords method
                        aLatLng = getCoords(oAddress.Fields(ECONSTIT_ADDRESSFields.CONSTIT_ADDRESS_fld_ADDRESS_BLOCK) & " " & oAddress.Fields(ECONSTIT_ADDRESSFields.CONSTIT_ADDRESS_fld_CITY) & ", " & oAddress.Fields(ECONSTIT_ADDRESSFields.CONSTIT_ADDRESS_fld_STATE) & " " & oAddress.Fields(ECONSTIT_ADDRESSFields.CONSTIT_ADDRESS_fld_POST_CODE))
                        Exit For
                    End If
                Next
            End With
        Finally 'close down your stuff...always
            oConstit.CloseDown()
            System.Runtime.InteropServices.Marshal.ReleaseComObject(oConstit)
            System.Runtime.InteropServices.Marshal.ReleaseComObject(RE7)
        End Try
        Return aLatLng 'return the value
    End Function


Now lets look at the getCoords function. This one just takes the address we retrieved from the API and uses a Google web service to return the lat/lng. One important thing to note is that it will return Nothing if the address can't be geo-coded OR if the address isn't specific enough and Google returns more than one possible result. I did this for simplicity but you can modify the project so things are handled how you want them to be. 

Private Function getCoords(ByVal sAddress As String) As ArrayList
        Dim stream As IO.Stream = Nothing
        Dim doc As XmlDocument
        Dim req As HttpWebRequest
        Dim aLatLng As New ArrayList
        If sAddress <> "" Then 'we check to make sure that we have an address
            req = DirectCast(WebRequest.Create("http://maps.google.com/maps?output=kml&q=" + sAddress), HttpWebRequest) 'pass the address to googole and assign the results to req
            'now we parse up the results 
            'generally speaking kml = xml so we can use the std handlers in .net for this
            'why google doesn't just call it xml is beyond me
            Dim response As HttpWebResponse = DirectCast(req.GetResponse(), HttpWebResponse)
            stream = response.GetResponseStream()
            Dim sr As New StreamReader(stream)
            Dim text As String = ""
            text = sr.ReadToEnd()
            sr.Close()
            doc = New XmlDocument()
            doc.LoadXml(text)
            longitudes = doc.GetElementsByTagName("longitude")
            latitudes = doc.GetElementsByTagName("latitude")
            'for simplicty we only want to worry about results that have 1 possible result
            'none, means the address wasn't geocodable
            'multiple means the address wasn't specific enough
            If longitudes.Count = 1 Then
                aLatLng.Add(latitudes(0).InnerText)
                aLatLng.Add(longitudes(0).InnerText)
            End If
        End If
        Return aLatLng
    End Function

So that's pretty much it. You can download the full project below, enjoy. 

Don't forget to Join the LinkedIn and/or Facebook group and be sure to follow me on Twitter.

5 comments:

Paul Morriss said...

Does that violate Google's terms of service or some other agreement. I thought of doing some screenscraping to get journey driving times and came across a comment to that effect.

Garrett said...

I wouldn't imagine it violates any TOS. It's using a piece of the google maps API and they put mechanisms in place to prevent abuse to I think it's ok.
This isn't really screen-scraping. That I would think would violate their TOS.

graham said...

Wow, that's impressive! I don't have the technical chops to delve into the API, but here is a free, easy low-cost way to get RE data on Google Maps (the link is a how-to video that anyone should be able to follow). Here goes: Low Tech Way to Put RE Data on

Google Maps

Garrett said...

I like the concept. I'd be REALLY concerned that this might expose constituent data. I'm not sure what batchgeocode.com is but using this method its feasible that they could get access to whatever data you paste up there.
Neat idea but I'd be cautious.

Yellow Blade said...

Công ty vận chuyển hàng hóa nội địa chúng tôi xin giới thiệu các dịch vụ vận chuyển, dịch vụ ship hàng uy tín để phục vụ nhu cầu Tết của quý khách hàng. Cụ thể chúng tôi sẽ cung cấp dịch vụ chuyển quà tết. Chúng tôi sẽ giúp bạn vận chuyển hàng hóa đến tay người thân, bạn bè ở xa một cách nhanh chóng nhất. Đảm bảo giá cả hợp lý chất lượng dịch vụ tuyệt vời. Ngoài ra chúng tôi còn cung cấp nhiều dịch vụ khác như dịch vụ ship hàng cod, giao hàng cho shop, dịch vụ chuyển phát nhanh trong nước,... Nếu cần chuyển hàng hãy nhớ liên hệ với chúng tôi nhé.