Last week I set out to create a somewhat better log viewer for the Google App Engine development server.

As an App Engine developer I'm intimately familiar with this screen:

Boring monochrome log

It's useful and I love App Engine's logging, but I quickly become overwhelmed when there's a lot of requests coming in (for example, a page with a lot of static assests or a series of task queue tasks). It becomes hard to search and comprehend the logs.

So the first thing I did was add some color:

Exciting color log

This helped a lot, but I was curious as to how far I could take this. I had an idea about creating a web-based log viewer that would make it much easier to digest and search logs. I ended up creating this:

Even more exciting web log

The rest of the post talks about some of the implementation details behind this. If you just want to check it out you can install it via pip:

$ pip install gae-flightdeck

Then just run it:

$ cd some-app-engine-project
$ gae-flightdeck

It's on github if you want to dig through the source.

Behind The Scenes

I decided to try to build this in Python. I chose to use gevent and gevent-socketio to implement the server and AngularJS for the front-end. In retrospect it probably would've been easier to just write this using node.js, but I was curious about how to pull it off with Python.

Server

The server part of this needs to read the log output from the App Engine dev server and then push it through socket.io to the front-end. We can use piping to send the stdout and stderr to our program:

$ dev_appserver.py . 2>&1 | ourprogram.py

The trick is that we need to read from stdin in a non-blocking fashion so that gevent-socketio can communicate to our front-end.

Non-blocking stdin

The first experiment was to make a simple echo program with gevent that would read from stdin in a way that plays nice with gevent. After a bit of research I came up with this:

The program flow here is that producer blocks and yields on sys.stdin.readline while consumer blocks and yields on q.get(). Once a line comes into stdin the producer adds it to the queue and yields while waiting for the next line. The consumer then kicks in an prints it out and then yields back to the producer while waiting for the next item to come in the queue.

In order to make stdin work with gevent without blocking in an "un-friendly" way we have to use gevent.monkey:

monkey.patch_sys(stdin=True, stdout=False, stderr=False)  

Note that we're not patching stdout and stderr. I found that if you do that while having greenlets that essentially run forever then keyboard interrupts will not work to kill the program. I don't need to do non-blocking I/O on stdout and stderr anyway so it's not a problem.

Handing it off to socket.io

Before you can use socket.io with gevent you have to setup a simple WSGI server to serve up the static files as well as hold the socket.io "namespaces". For the sake of brevity I'm just going to talk about the namespace implementation here. You can checkout the implemenation of make_server here.

In socket.io a namespace is basically an "endpoint" or a "reqeust handler" in WSGI parlance. They're a way of separating different socket channels, but in this app we only need one. Our namespace is going to take the place of the consumer in the above example and just emit the log to the socket.io client.

class LogNamespace(BaseNamespace, BroadcastMixin):  
    def recv_connect(self):
        def sendlogs():
            while True:
                val = q.get(True)
                self.emit('log', {'line': val})

        self.spawn(sendlogs)

It spawns a greenlet when the client connects. This greenlet very similar to the consumer we had before. It waits for a log line to come into the queue and then just emits it to the client. Presently, this only works for one client at a time, however, using the broadcast mixin it's possible to make it support multiple ones.

The complete server program (minus the web_server.py file mentioned above) looks like this:

The Client

For the client we're going to use the socket.io client to recieve logs from our server and AnguarJS to display them.

We first need to wire up socket.io to an AnguarJS controller:

angular.module('app', [])  
.controller('logs', function($scope){
  var socket = io.connect('/logs');

  $scope.logs = [];

  socket.on('log', function(data) {
    $scope.$apply(function(){
      $scope.logs.append(data);
    });
  });
});

Socket.io is dead simple. The only trick here is that the socket.on event isn't aware of angular's dirty checking so we have to use $scope.$apply to make everything work together. There are various libraries out there that wrap socket.io and do this for you automatically, but I felt it was overkill for this really simple application.

All that's left is to display the logs. Here's a complete front-end for our log viewer:

Making the client more useful

So far we've just managed to make the browser display exactly what's in the terminal. In order to make this more useful a few more features were implemented. Instead of embedding the code here I'll link to the relevent sections on github.

Log formatting

I wanted to do more than just display the console logs on my browser. I wanted to format them to better match what's on the production App Engine log viewer. For the most part this is straightforward if not tedious: break the string up in to relevent sections (here) and just display them in a table-like view (here).

However, there are multiple logs that can come in for a single request. In the production log viewer these logs are grouped by request. This turned out to be somewhat easy to implement: just accumlate logs until the request itself is logged then associate the logs with that request (here). Then it's just a matter of displaying the logs along with the request (here).

Filtering by regex

I implemented this via a angularjs filter. The application watches the search box and tries to compile a regex (here). If the regex is valid it'll use it inside the filter (here).

Auto scroll

I wanted the browser window to stay at the bottom of the page when new items were added. The implementation is pretty straightforward and somewhat hacky.

Content is © 2017. All Rights Reserved.

All code is licensed under the Apache License, Version 2.0 and is © Google Inc unless otherwise noted.

All opinions here are my own, and do not necessarily reflect the opinions of my employer.

Any code does not constitute an official Google product (experimental or otherwise), it just happens to be owned by Google.

Ghostium Theme by @oswaldoacauan

Proudly published with Ghost