This is the fifteenth article in the series in which I document my experience writing web applications in Python using the Flask microframework.

The goal of the tutorial series is to develop a decently featured microblogging application that demonstrating total lack of originality I have decided to call microblog.

Here is an index of all the articles in the series that have been published to date:

• Part I: Hello, World!
• Part II: Templates
• Part III: Web Forms
• Part IV: Database
• Part V: User Logins
• Part VI: Profile Page And Avatars
• Part VII: Unit Testing
• Part VIII: Followers, Contacts And Friends
• Part IX: Pagination
• Part X: Full Text Search
• Part XI: Email Support
• Part XII: Facelift
• Part XIII: Dates and Times
• Part XIV: I18n and L10n
• Part XV: Ajax (this article)
• Part XVI: Debugging, Testing and Profiling

Table of Contents:

[TOC]

This will be the last of the I18n and L10n themed articles, as we complete our effort to make our microblog application accessible and friendly to non-English speakers.

In this article we will take a departure from the "safe zone" of server-side development we've come to love and will work on a feature that has equally important server and client-side components. Have you seen the "Translate" links that some sites show next to user generated content? These are links that trigger a real time automated translation of content that is not in the user's native language. The translated content is typically inserted below the original version. Google shows it for search results in foreign languages. Facebook shows it for posts. Today we are adding that very same feature to microblog!

Server-side vs. Client-side

In the traditional server-side model that we've followed so far we have a client (the user's web browser) making requests to our server. A request can simply ask for a web page, like when you click the "Your Profile" link, or it can make us perform an action, like when the user edits his or her profile information and clicks the Submit button. In both types of requests the server completes the request by sending a new web page to the client, either directly or by issuing a redirect. The client then replaces the current page with the new one. This cycle repeats for as long as the user stays on our web site. We call this model server-side because the server does all the work while the client just displays the web pages as it receives them.

In the client-side model we have a web browser that again, issues a request to the server. The server responds with a web page like in server-side, but not all the page data is HTML, there is also code, typically written in Javascript. Once the client receives the page it will display it and then execute the code that came with it. From then on you have an active client that can do work on its own without reaching out to the server. In a strict client-side application the entire application is downloaded to the client with the initial page request, and then the application runs on the client without ever refreshing the page, only contacting the server to retrieve or store data. This type of applications are called Single Page Applications or SPAs.

Most applications are a hybrid between the two models and combine techniques of both worlds. Our microblog application is clearly a server-side application, but today we will be adding a little bit of client-side action to the mix. To do real time translations of user posts the client browser will send requests to the server, but the server will respond with translated texts without causing a page refresh. The client will then insert the translations into the current page dynamically. This technique is known as Ajax, which is short for Asynchronous Javascript and XML (even though these days XML is often replaced with JSON).

Translating user generated content

We now have pretty good support for foreign languages thanks to Flask-Babel. Assuming we can find translators willing to help us, we could publish our application in as many languages as we want.

But there is one element missing. We made the application very friendly and inviting to people of all places, so now we are going to get content in a variety of languages, so our users are likely to come across blog posts that are written in languages they don't understand. Wouldn't it be nice if we could offer an automated translation service? The quality of automated translations isn't great, but in most cases it is good enough to get an idea of what someone said in a language we don't speak, so all our users (even ourselves!) can benefit from such a feature.

This is an ideal feature to implement as an Ajax service. Consider that our index page could be showing several posts, some of which might be in several different foreign languages. If we implemented the translation using traditional server-side techniques a request for a translation would cause the original page to get replaced with a new page showing just the translation for the selected post. After reading the translation the user would have to hit the back button to go back to the post listing. The fact is that requesting a translation isn't a big enough action to require a full page update, this feature works much better if the translated text is dynamically inserted below the original text while leaving the rest of the page untouched. So today we are implementing our first Ajax service!

Implementing live automated translations requires a few steps. First we need to identify the source language for the text we want to translate. Once we know the language we also know if a translation is needed for a given user, because we also know what language the user has chosen. When a translation is offered and the user wishes to see it we invoke the Ajax translation service, which will run on our server. In the final step the client-side javascript code will dynamically insert the translated text into the page.

Identifying the language of a post

Our first problem is identifying what language a post was written in. This isn't an exact science, it will not always be possible to detect a language, so we will consider it a "best effort" kind of thing. We are going to use the guess-language Python module for this. So go ahead and install it. For Linux and Mac OS X users:

flask/bin/pip install guess-language

For Windows users:

flask\Scripts\pip install guess-language

With this module we will scan the text of each blog post to try to guess its language. Since we don't want to scan the same posts over and over again we will do this once for each post, at the time the post is submitted by the user. We will then store the detected language along with the post in our database.

So let's begin by adding a language field to our Posts table:

class Post(db.Model):
    __searchable__ = ['body']

    id = db.Column(db.Integer, primary_key = True)
    body = db.Column(db.String(140))
    timestamp = db.Column(db.DateTime)
    user_id = db.Column(db.Integer, db.ForeignKey('user.id'))
    language = db.Column(db.String(5))

Each time we modify the database we also need to do a migration:

$ ./db_migrate.py
New migration saved as microblog/db_repository/versions/005_migration.py
Current database version: 5

Now we have a place to store the language of each post, so let's detect the language of each added blog post:

from guess_language import guessLanguage

@app.route('/', methods = ['GET', 'POST'])
@app.route('/index', methods = ['GET', 'POST'])
@app.route('/index/<int:page>', methods = ['GET', 'POST'])
@login_required
def index(page = 1):
    form = PostForm()
    if form.validate_on_submit():
        language = guessLanguage(form.post.data)
        if language == 'UNKNOWN' or len(language) > 5:
            language = ''
        post = Post(body = form.post.data,
            timestamp = datetime.utcnow(),
            author = g.user,
            language = language)
        db.session.add(post)
        db.session.commit()
        flash(gettext('Your post is now live!'))
        return redirect(url_for('index'))
    posts = g.user.followed_posts().paginate(page, POSTS_PER_PAGE, False)
    return render_template('index.html',
        title = 'Home',
        form = form,
        posts = posts)

If the language guessing doesn't work or returns an unexpectedly long result we play it safe and save an empty string to the database, a convention that we will use to signal that a post has an unknown language.

Displaying the "Translate" link

The next step is to show a "Translate" link next to any posts that are not in the language used by the user (file app/templates/posts.html):

{% if post.language != None and post.language != '' and post.language != g.locale %}
<div><a href="#">{{ _('Translate') }}</a></div>
{% endif %}

We are doing this in the post.html sub-template, so any page that displays blog posts will get this functionality. The Translate link will only be visible if we could detect the language of the blog post and the language we detected is different than the language selected by the localeselector function from Flask-Babel, which based on the work we did in the previous article we can now access at g.locale.

This link required us to add a new text, the word "Translate" which needs to be included in our translation files, so we entered it with the translation wrapper to make it visible to Flask-Babel. To get it translated we have to refresh our language catalog (tr_update.py), translate in poedit and recompile (tr_compile.py), as seen in the I18n article.

We don't really know what we will be doing to trigger a translation yet, so for now the link does nothing.

Translation services

Before we can proceed with our application we need to find a translation service that we can use.

There are many translation services available, but unfortunately most are either paid services or free with big limitations.

The two major translation services are Google Translate and Microsoft Translator. Both are paid services, but the Microsoft offering has an entry level option for low volume of translations that is free. Google offered a free translation service in the past but that isn't available anymore. That makes our choice of translation service very easy.

Using the Microsoft Translator service

To use the Microsoft Translator there are a couple of requirements that need to be met:

• The application developer needs to register with the Microsoft Translator app at the Azure Marketplace. Here the level of service can be selected (the free option is at the bottom).
• Then the developer needs to register an application. The registered application will get Client ID and Client Secret codes to be used as part of the requests sent to this service.

Once the registration part is complete the process to request a translation is as follows:

• Get an access token, passing in the client id and secret.
• Call the desired translation method, either Ajax, HTTP or SOAP, providing the access token and the text to translate.

This really sounds more complicated than it really is, so without going into details here is a function that does all the dirty work and translates a text to another language (file app/translate.py):

import urllib, httplib
import json
from flask.ext.babel import gettext
from config import MS_TRANSLATOR_CLIENT_ID, MS_TRANSLATOR_CLIENT_SECRET

def microsoft_translate(text, sourceLang, destLang):
    if MS_TRANSLATOR_CLIENT_ID == "" or MS_TRANSLATOR_CLIENT_SECRET == "":
        return gettext('Error: translation service not configured.')
    try:
        # get access token
        params = urllib.urlencode({
            'client_id': MS_TRANSLATOR_CLIENT_ID,
            'client_secret': MS_TRANSLATOR_CLIENT_SECRET,
            'scope': 'http://api.microsofttranslator.com',
            'grant_type': 'client_credentials'
        })
        conn = httplib.HTTPSConnection("datamarket.accesscontrol.windows.net")
        conn.request("POST", "/v2/OAuth2-13", params)
        response = json.loads (conn.getresponse().read())
        token = response[u'access_token']

        # translate
        conn = httplib.HTTPConnection('api.microsofttranslator.com')
        params = {
            'appId': 'Bearer ' + token,
            'from': sourceLang,
            'to': destLang,
            'text': text.encode("utf-8")
        }
        conn.request("GET", '/V2/Ajax.svc/Translate?' + urllib.urlencode(params))
        response = json.loads("{\"response\":" + conn.getresponse().read().decode('utf-8-sig') + "}")
        return response["response"]
    except:
        return gettext('Error: Unexpected error.')

This function imports two new items from our configuration file, the id and secret codes assigned by Microsoft (file config.py):

# microsoft translation service
MS_TRANSLATOR_CLIENT_ID = '' # enter your MS translator app id here
MS_TRANSLATOR_CLIENT_SECRET = '' # enter your MS translator app secret here

To use the service you will need to register yourself and the application to receive the codes that go in the above configuration variables. Even if you just intend to test this application you can register with the service for free.

We have added more texts, this time a few error messages. These need translations, so we have to re-run tr_update.py, poedit and tr_compile.py to update our translation files.

Let's translate some text!

So how do we use the translator service? It's actually very simple. Here is an example:

$ flask/bin/python
Python 2.6.8 (unknown, Jun  9 2012, 11:30:32)
>>> from app import translate
>>> translate.microsoft_translate('Hi, how are you today?', 'en', 'es')
u'?Hola, cómo estás hoy?'

Ajax in the server

Now we can translate texts between languages, so we are ready to integrate this functionality into our application.

When the user clicks the Translate link in a post there will be an Ajax call issued to our server. We'll see how this call is made in a bit, for now let's concentrate on implementing the server side of the Ajax call.

An Ajax service in the server is like a regular view function with the only difference that instead of returning an HTML page or a redirect it returns data, typically formatted as either XML or JSON. Since JSON is much more friendly to Javascript we will go with that format (file app/views.py):

from flask import jsonify
from translate import microsoft_translate

@app.route('/translate', methods = ['POST'])
@login_required
def translate():
    return jsonify({
        'text': microsoft_translate(
            request.form['text'],
            request.form['sourceLang'],
            request.form['destLang']) })

There is not much new here. This route handles a POST request that should come with the text to translate and the source and destination language codes. Since this is a POST request we access these items as if this was data entered in an HTML form, using the request.form dictionary. We call one of our translation functions with this data and once we get the translated text we just convert it to JSON using Flask's jsonify function. The data that the client will see as a response to this request will have this format:

{ "text": "<translated text goes here>" }

Ajax in the client

Now we need to call the Ajax view function from the web browser, so we need to go back to the post.html sub-template to complete the work we started earlier.

We start by wrapping the post text in a span element with a unique id, so that we can later find it in the DOM and replace it with the translated text (file app/templates/post.html):

<p><strong><span id="post{{post.id}}">{{post.body}}</span></strong></p>

Note how we construct the unique id using the post's id number. If a given post has an id = 3 then the id of the <span> element that wraps it will be post3.

We are also going to wrap the Translate link in a span with a unique id, in this case so that we can hide it once the translation is shown:

<div><span id="translation{{post.id}}"><a href="#">{{ _('Translate') }}</a></span></div>

Continuing with the example above, the translation link would get the id translation3.

And to make it a nice and user friendly feature we are also going to throw in a spinning wheel animation that starts hidden and will only appears while the translation service is running on the server, also with a unique id:

<img id="loading{{post.id}}" style="display: none" src="/static/img/loading.gif">

So now we have an element called post<id> that contains the text to translate, an element called translation<id> that contains the Translate link but will later be replaced with the translated text and an image with id loading<id> that will be shown while the translation service is working. The <id> suffix is what makes these elements unique, we can have many posts in a page and each will get its own set of three elements.

We now need to trigger the Ajax call when the Translate link is clicked. Instead of triggering the call directly from the link we'll create a Javascript function that does all the work, since we have a few things to do there and we don't want to duplicate code in every post. Let's start by adding a call to this function when the translate link is clicked:

<a href="javascript:translate('{{post.language}}', '{{g.locale}}', '#post{{post.id}}', '#translation{{post.id}}', '#loading{{post.id}}');">{{ _('Translate') }}</a>

The template variables obscure this code a bit, but really the function that we are calling is simple. Assuming a post with id = 23 that is written in Spanish and is viewed by a user that uses English then this function will be called as follows:

translate('es', 'en', '#post23', '#translation23', '#loading23')

So this function will receive the source and destination languages and the three DOM elements associated with the post to translate.

We can't write this function in the post.html sub-template because its contents are repeated for each post. We'll implement the function in our base template, so that there is a single copy of it that is accessible in all pages (file app/templates/base.html):

<script>
function translate(sourceLang, destLang, sourceId, destId, loadingId) {
    $(destId).hide();
    $(loadingId).show();
    $.post('/translate', {
        text: $(sourceId).text(),
        sourceLang: sourceLang,
        destLang: destLang
    }).done(function(translated) {
        $(destId).text(translated['text'])
        $(loadingId).hide();
        $(destId).show();
    }).fail(function() {
        $(destId).text("{{ _('Error: Could not contact server.') }}");
        $(loadingId).hide();
        $(destId).show();
    });
}
</script>

We rely on jQuery for this functionality. Recall that we included jQuery back when we styled the application using Twitter's Bootstrap framework.

We start by hiding the element that contains the translate link and showing the spinning wheel image.

Then we use jQuery's $.post() function to send the Ajax request to our server. The $.post function issues a POST request that to the server will be identical to a request issued by the browser when a form is submitted. The difference is that in the client this request happens in the background without causing a page reload. When a response is received from the server the function given as an argument to the done() call will execute and will have a chance to insert the data received into the page. This function receives the response as an argument, so we just overwrite the translate link in the DOM with the translated text, then hide the spinning wheel and finally display the translated text for the user to see. Neat!

If there is any error that prevents the client from getting a response from the server then the function inside the fail() call will be invoked. In this case we just show a generic error message, which also needs to be translated into all languages and thus requires a refresh of our translation database.

Unit testing

Remember our unit test framework? It is always a good idea to evaluate if it makes sense to write tests every time we add a new feature. Invoking the text translation service could be something we could inadvertently break while we work on our code, or it could break due to updates in the service. Let's write a quick test that makes sure we can contact the service and obtain translations:

from app.translate import microsoft_translate

class TestCase(unittest.TestCase):
    #...
    def test_translation(self):
        assert microsoft_translate(u'English', 'en', 'es') == u'Inglés'
        assert microsoft_translate(u'Espa?ol', 'es', 'en') == u'Spanish'

We do not have a client-side testing framework at this time, so we will not test the end-to-end Ajax procedure.

Running our testing framework should give us no errors:

$ ./tests.py
.....
----------------------------------------------------------------------
Ran 5 tests in 5.932s

OK

But if you run the test framework without having done the proper configuration for the Microsoft Translator you would get:

$ ./tests.py
....F
======================================================================
FAIL: test_translation (__main__.TestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./tests.py", line 120, in test_translation
    assert microsoft_translate(u'English', 'en', 'es') == u'Inglés'
AssertionError

----------------------------------------------------------------------
Ran 5 tests in 3.877s

FAILED (failures=1)

Final words

And with this we conclude this article. I hope it was as fun to read for you as it was for me to write!

I have recently been alerted of some issues with the database when Flask-WhooshAlchemy is used for the full text search. In the next chapter I will use this problem as an excuse to go over some of my debugging techniques for Flask applications. Stay tuned for episode XVI!

The code for the updated application is available on github, here. For those that do not use github below is a friendly download link:

Download microblog-0.15.zip.

I look forward to see you in the next article!

Miguel

Origin: http://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-xv-ajax