How good is knowledge if you can't share it?

Previously, I have shown how to visualise the data gathered by the crawler part of i-must-go. To pass the data to the graphing library - sigma.js in this case, it has to be stored in a text file, which then is used to recreate the object. The process of translating the object to a format that can be stored more easily is called serialising or marshalling the object.

Out of many different available formats, JSON (JavaScript Object Notation) happens to be the most common nowadays, being used very often to transport data to and from RESTful APIs. Its clear structure and readability has made it one of the people's favourites. Of course, golang has its library for dealing with this, and I'm going to show you how I use it.

Code revision at the time of writing: 60a319a21, still.

JSON, the Red Ranger

Golang's library for JSON is, unexpectedly, named json. JavaScript Object Notation format consists of two basic data structures (objects and arrays) and 7 values (string, number, true, false, null, object and array).

Intuitively, one can understand that golang structs would map to objects and slices would map to arrays. Additionally, maps in the form of map[string]T where T is a serializable type, can also be mapped to JSON objects. Obviously, string, bool and "number-like" types work just fine.

Serialisation

In my case, I have settled on using types defined especially for representing specific elements of a graph as follows:

// {"id": "", "label": ""}
type JsonNode struct {
    Id    string `json:"id"`    // The last part is called a tag
    Label string `json:"label"`
}

// {"id": "", "source": "", "target": ""}
type JsonEdge struct {
    Id     string `json:"id"`
    Source string `json:"source"`
    Target string `json:"target"`
}

// {nodes: [{"id": "", "label": ""}, ...], 
//  edges: [{"id": "", "source": "", "target": ""}, ...]
// }
type JsonOutput struct {
    Nodes []JsonNode `json:"nodes"`
    Edges []JsonEdge `json:"edges"`
}

Important thing about storing objects: Only exported fields are serialised, and json fields' names are by default the same as the names in the struct. That means that all serialised fields spelled with a capital letter by default due to golang's exporting rules.

This behaviour is fine for most of the use cases, but can also be customised using tags. They allow you to define additional json-specific information, such as how to map struct's field to a JSON field. If the field's name is specified as "-", it will always be skipped in the marshalled form.

I have created a function for converting the NetworkGraph structure to a byte array containing the JSON-encoded data. It drops some of the data because all I want to achieve is to define nodes and edges for the graphing library.

func Jsonify(ng *NetworkGraph) []byte {
    var out_nodes []JsonNode
    var out_edges []JsonEdge

    for _, node := range ng.GetNodes() {
        out_nodes = append(out_nodes, JsonNode{
            Id:    string(node.GetIP()),
            Label: string(node.GetIP()),
        })
    }

    for index, edge := range ng.GetEdges() {
        out_edges = append(out_edges, JsonEdge{
            Id:     strconv.Itoa(index),
            Source: string(edge.LocalAddress),
            Target: string(edge.RemoteAddress),
        })
    }

    output := JsonOutput{out_nodes, out_edges}

    b, _ := json.MarshalIndent(output, "", "  ")
    return b
}

Deserialisation

For the sake of completeness, here's how to read the data out of a byte array and into the object. Unmarshal function is used for unmarshalling the JSON. It accepts two arguments: a byte array and an output variable implementing null interface interface{}.

In my case, unmarshalling would have to look like this:

var data JsonOutput

// variable myJSON contains the output of Jsonify() function above.
err := json.Unmarshal(myJSON, &data)

// variable data contains the data from myJSON now!

As of now, I'm not even using JSON unmarshalling in the application since there is no use for it. This might be useful once I want to persist the structure of the network between re-runs.

But wait, there's more!

This post covers only the basic usage that I have needed to get my work going. If you want some more information, I highly recommend an article on golang blog called JSON and Go.

Stay tuned for the next posts!