Building a simple JSON API with GoLang

I want to write more Golang; In order to do this I need a reason to write Golang. And the truth is I could use the performance gains of Go to resolve some of the performance issues I have been experiencing with a large Flask restful application. So today just to force myself into writing a little bit of Golang I wrote a small RESTful API for using both Google and Bing.

Now to be fair, Bing lets me do this; but Google does not and for this example you will see Bing provides the expected results where as Google does not. But since this was a learning exercise it is ok and I really didn’t feel like going down the rabbit hole of reverse engineering Google Search or setting up a Custom Search so were going to keep it how it is now.

Binging it

Only at programmatic searching does Microsoft do anything right…. man I just have so much hate… Hugs @Microsoft Hugs

[[email protected] Searchy]$ curl localhost:8080/bing/mattharris.org
["http://mattharris.org/","http://mattharris.org/contact","http://mattharris.org/resume","http://mattharris.org/linux/","http://mattharris.org/html/","http://mattharris.org/system-administration/","http://mattharris.org/mysql/","http://mattharris.org/ruby/","http://mattharris.org/python/","http://mattharrismusic.com/","https://twitter.com/mattcharris","https://twitter.com/mattcharris","http://mattharris.org/php/","http://www.matt-harris.com/","https://www.facebook.com/public/Matt-Harris","https://www.linkedin.com/pub/dir/Matt/Harris","http://www.csun.edu/mike-curb-arts-media-communication/music/Matt-Harris","http://go.microsoft.com/fwlink/?LinkId=521839\u0026CLCID=0409","http://go.microsoft.com/?linkid=9844325","http://go.microsoft.com/fwlink/?LinkID=617297","http://go.microsoft.com/fwlink/?LinkID=246338\u0026CLCID=0409","http://go.microsoft.com/fwlink/?LinkID=286759\u0026CLCID=409"]

Googling it

Like I said, I didn’t feel like fighting with google today…

[[email protected] Searchy]$ curl localhost:8080/google/mattharris.org
["http://www.google.com/search?q=mattharris.org\u0026um=1\u0026ie=UTF-8\u0026hl=en\u0026tbm=isch\u0026source=og\u0026sa=N\u0026tab=wi","http://maps.google.com/maps?q=mattharris.org\u0026um=1\u0026ie=UTF-8\u0026hl=en\u0026sa=N\u0026tab=wl","https://play.google.com/?q=mattharris.org\u0026um=1\u0026ie=UTF-8\u0026hl=en\u0026sa=N\u0026tab=w8","http://www.youtube.com/results?q=mattharris.org\u0026um=1\u0026ie=UTF-8\u0026sa=N\u0026tab=w1","http://news.google.com/nwshp?hl=en\u0026tab=wn","https://mail.google.com/mail/?tab=wm","https://drive.google.com/?tab=wo","https://www.google.com/intl/en/options/","http://www.google.com/history/optout?hl=en","https://accounts.google.com/ServiceLogin?hl=en\u0026passive=true\u0026continue=http://www.google.com/search%3Fq%3Dmattharris.org","http://maps.google.com/maps?q=mattharris.org\u0026um=1\u0026ie=UTF-8\u0026sa=X\u0026ved=0ahUKEwjssNX_3LfNAhVExYMKHWzsBSoQ_AUICQ"]

The source code

This should be pretty simple to follow but if not main() handles the routing to the functions based on the route provided by the server. And then we make a GET request to get the webpage and then finally parse out the links on the page.

// A sample RESTful API for search engines
// Used for tutorial on mattharris.org
package main

import (
    "encoding/json"
    "log"
    "net/http"
    "regexp"

    "github.com/gorilla/mux"
    "golang.org/x/net/html"
)

// Handle RESTful Routing
func main() {
    router := mux.NewRouter().StrictSlash(true)
    router.HandleFunc("/google/{search_term}", GoogleIt)
    router.HandleFunc("/bing/{search_term}", BingIt)
    log.Fatal(http.ListenAndServe(":8080", router))
}

// Handle Google Searching
func GoogleIt(w http.ResponseWriter, r *http.Request) {
    vars := mux.Vars(r)
    response, err := http.Get("http://google.com/search?q=" + vars["search_term"])
    if err != nil {
        // Oh no! Google isn't working!!!
        log.Fatal(err)
    } else {
        defer response.Body.Close()
        links := GetLinks(response)
        json.NewEncoder(w).Encode(links)
    }
}

// Handle Bing Searching
func BingIt(w http.ResponseWriter, r *http.Request) {
    vars := mux.Vars(r)
    response, err := http.Get("http://www.bing.com/search?q=" + vars["search_term"])
    if err != nil {
        // Oh no! Bing isn't working!!!
        log.Fatal(err)
    } else {
        defer response.Body.Close()
        links := GetLinks(response)
        json.NewEncoder(w).Encode(links)
    }
}

// Parse out Links
func GetLinks(response *http.Response) (links []string) {
    z := html.NewTokenizer(response.Body)
    for {
        tt := z.Next()

        switch {
        case tt == html.ErrorToken:
            // EOD
            return
        case tt == html.StartTagToken:
            t := z.Token()

            isAnchor := t.Data == "a"

            if !isAnchor {
                continue
            }

            ok, url := GetHref(t)
            if !ok {
                continue
            }

            links = append(links, url)
        }
    }

    return
}

// Get the link
func GetHref(t html.Token) (ok bool, href string) {
    for _, a := range t.Attr {
        if a.Key == "href" {
            matched, err := regexp.MatchString("^http", a.Val)
            if err != nil {
                ok = false
            } else {
                if matched {
                    href = a.Val
                    ok = true
                }
            }
        }
    }

    return
}

Write a Comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.