Code Walkthrough: Libretto

In February, I released the first prototype of the Android App, Libretto, a multimodal e-book platform designed to allow readers to compare different versions of a musical as it develops over time, and to listen to the music associated with the text. I hope to release an update soon, but in the meantime, I wanted to take a minute to walk through the code (which is available at GitHub) for those interested in working with the code.

The first thing to know about Libretto is that it is a "native" app. That is, it runs directly on the Android operating system rather than in a web browser. Today, many app developers prefer to develop web apps (mobile friendly versions of their interactive web pages) because it allows for almost instant compatibility across devices. Native apps, on the other hand, must be developed for iOS, Android, Windows Phone, etc. separately.

I developed Libretto as a native app, because it needs to do two things not easily done in a web app:

1. Interact with content on file system of the device (Libretto allows the user to associate songs they have previously purchased with moments in the script). This sort of file interaction is prohibited (for security reasons) on most browsers.
2. Unzip a file (this requires more processing power and memory than is usually available in a web app).

It's possible that as mobile web technologies develop, it will be possible to migrate Libretto to a web app, but for now, a "native" app seemed like the best approach. It's not fantastic code. It's probably even pretty bad in a lot of places. But Oscar Hammerstein II reportedly once gave Stephen Sondheim an assignment to take a bad play and make it into a good musical. Maybe this blog can enable the same kind of assignment.

Prerequisites

Android apps are written and compiled in Java, and knowing Java first would be really helpful, but I will assume many web programmers do not know it well. In this blog, I will, however, assume a basic knowledge of common programming vocabulary. If you don't know about variables, for loops, if/then conditionals, functions, and data types like arrays and strings, this probably isn't the best place to start. There are several really good introductions to coding out there, and it may be best if you go check them out first and come back in little while.

I'm going to aim this blog at web programmers who have written some of their own JavaScript, PHP, or Ruby code, but who have may not have programmed in Java before. My goal is to help web programmers gain an understanding of native apps for those cases, like Libretto, where it is necessary.

Getting started

You can write Android apps in any text editor (Notepad, TextWrangler, Sublime, whatever), but I don't recommend it. Compiling and deploying your app can be done using a set of command line apps that you could, theoretically, string together in a shell script of some kind. Don't.

It's fairly clear that Google means for you to develop in Eclipse. Eclipse is a very sophisticated open source code editor (or integrated development environment [IDE]) designed for Java developers. It gets a bad rap for being an unstable memory hog, and some of that reputation is deserved, but the Android-branded version is surprisingly free of most of the problems that have caused headaches in the past. Unless you have very good reasons for doing otherwise, I recommend using the out-of-the-box installation rather than attempting to add Android as a plug-in to your existing Eclipse environment.

Once you've installed the Android Development Kit version of Eclipse, you'll be asked to create a "workspace." This is a folder that contains all of your projects made in the program as well as the configuration files for your development environment. I like to create a folder in my Dropbox folder so that I can have access to my environment from multiple machines. I assume this would work equally well with Google Drive or other synched cloud drives.

Google has a good tutorial for starting an entirely new project, so I won't describe the process further here. Instead, minimize the welcome screen if it's open using the dash in the upper right hand corner of the panel. Do not close the tab. It's not a big deal if you do, but it may make thing a little confusing later on.

Since we're walking through the Libretto code, we're going to import a new project. Make sure you've downloaded the Libretto code from GitHub and unzip it. Go to "File->Import…" and from the "Android" folder select "Existing code into workspace." You'll be asked to specify where the code for the existing project is.

Click the Browse button and navigate to the folder where you unzipped the Libretto code. Once it's imported, the folder will appear in the right panel with a red x on the folder.

If you open this folder, you'll see that the "src" and "res" folders seem to have problems. If you hover over the "src" folder errors, you'll find a mention of some thing to do with the variable or library "R" (such as "org.nypl.libretto.R"). The R library is an Android generated tool that keeps track of objects you've named in your code. It's generated automatically. When Eclipse reports that it's having trouble finding "R", it almost never originates with the R library itself. It just means something has gone wrong at a level much deeper than Eclipse is prepared to deal with.

Often, you can get more useful tips about what the problem is by looking in the "res" folder. The "res" folder contains files that are something like templates in a language like Ruby on Rails or Backbone.js. They are little XML files that specify the structure of parts of your graphical interface. If you look at the errors in the "Res" folder, you'll probably find some reference to "ActionBarSherlock."

"ActionBarSherlock" is a "theme" library, sort of like Twitter's Bootstrap: a set of common interface structures and visual elements that would be a pain to reinvent for each app. It was written by a guy named Jake Wharton and can be downloaded from his website, but I've also hosted the version used for testing in my GitHub account. Unfortunately, as far as I can tell, you have to import the ActionBarSherlock library into Eclipse independently. So let's do that now...

(This part feels unnecessarily complicated to me, but I don't know another way, let me know if you do…)

Download the ActionBarSherlock library and put it somewhere where you can find it again. Import the library the same way you imported the Libretto code: Go to "File->Import" and from the "Android" folder select "Existing code into workspace." Point the software to the "actionbarsherlock" folder. The library will then be added to your workspace.

Once "actionbarsherlock" is imported, open the "libs" folder, select "android-support-v4.jar", right click, and select "Add to Build Path."

Now, open the "Project" menu and select "Properties." First, select "Android" from the menu on the left and in the lower section, click "Add" and select "actionbarsherlock." It will appear in box at the bottom.

Finally, select "Java Build Path" on the left and click the "Libraries" tabs. Make sure "android-support-v4.jar" and "jsoup-1.7.2.jar" appear in the list. If they don't, click "Add JARS…" and select "jsoup-1.7.2.jar" from the "Libretto/libs" folder and "android-support-v4.jar" from the "actionbarsherlock/libs" folder. Then click the "Order and Export" tab and make sure "jsoup-1.7.2.jar" and "android-support-v4.jar" are checked.

Click "OK"

Now click the "Libretto" folder in the "Package Explorer" list and open the Project drop down one more time and click "Clean."

Whew! Ideally, all the errors will now go away. If not, close and restart Eclipse. If that doesn't work, post a note in the comments.

Code overview

Ok, at this point, the code should be working. You could connect your Android device to your computer, click the green "play" button at the top of the interface, and deploy the app to your device. Google provides a great tutorial for testing apps both in the Android emulator and on actual devices, though, so let's get right to the code.

Let's begin by a quick look at the files and folders at the top level of the Libretto folder.

• src: Contains all of the Java source code that runs the app
• gen: Contains auto-generated files (like that R thing I mentioned early)
• assets: Contains files that might be used by the program (like images, HTML, or fonts)
• bin: Contains files generated after the code is compiled (including the "apk" file that can be installed on a device manually if you like)
• libs: Contains the jsoup file I've packaged with the app
• res: Contains the templates used for the interface (described above)

And AndroidManifest.xml is the file that gives the app form and structure and will serve as the beginning of this walkthrough.

You can view and edit the manifest file with the user-friendly interface built into Eclipse, or directly using the built in XML editor (to get to the XML, click the "AndroidManifest.xml" tab at the bottom). Using either view, you can see that the manifest consists of:

1. basic metadata about the app (displayed in the "Application" pane or at the top of the XML),
2. the list of access permissions the app requests from the user on installation (viewed in the "Permissions" pane),
3. specifications for testing and deployment (none in this app, but available in the "Instrumentation" pane),
4. and, finally, a list of functions (or "activities") the app can perform (listed at the bottom of the Application pane in the "Application Nodes" list or as children of the "Application" XML tag).

Most of the activities in the Libretto manifest simply have a name, a theme (in this case a reference to an ActionBarSherlock template), and a specification for whether the screen should be allowed to rotate.

Two of these activity nodes, though, have additional information. The first, SplashActivity, has an "intent", which is a kind of generic Android term for starting an activity. Here, SplashActivity is given an intent with a "MAIN" action and a "LAUNCHER" category, which means that the activity is called when the app opens.

We'll come back to SplashActivity in just a moment. The other important thing that happens in the manifest is the declaration of the data provider (in this case "org.nypl.libretto.LibrettoContentProvider") which describes a database for the app.

As I mentioned above, the Java source code mostly lives in the src folder. If you open the folder, you'll see a bunch of little brown squares with crosses over them. These are Java packages ("brown paper packages tied up with string"). Essentially, the packages allow similar types of code to live together and share some information.

The packages in the src folder perform, roughly, the following functions:

• org.nypl.libretto: All of the "activities" of the app. These are the basic functions that run the show (making use of the user interface templates and the data stored elsewhere).
• org.nypl.libretto.adapater: These "adapters" connect the activities to data coming from the user interface or the database.
• org.nypl.libretto.database: As might be expected, this package contains code for setting up and accessing the tables of the internal database.
• org.nypl.libretto.holder: This package contains a set of little objects, named Beans, that are used to pass data from the database around.
• org.nypl.libretto.parsing: This code pulls apart an Libretto Variroum ePUB file and stores in the data in the app.
• org.nypl.libretto.ui: As the name suggests, this is code that listens to the interface and invokes functions in response to user actions.
• org.nypl.libretto.utils: These are miscellaneous tools needed by the rest of the app (e.g. an unzip function)
• org.nypl.libretto.widget: This package contains a single Java class needed for the user interface used to present the list of plays on the first screen.

The Libretto code is, alas, a bit messy (an artifact of its complicated development history which I will describe elsewhere). The short version of the story is that there are several eras of code written by at least two or three different hands, and I, the last coder working feverishly to finish it up, abandoned some niceties of style and elegance in favor of functionality.

*IMPORTANT: Eclipse gives you some nice tools to try to trace your way through the code. If you encounter a function or a variable you don't recognize, you can highlight it (that is, select it), and click F3 (or right click and select "Open Declaration…"). This will open the file that contains the first use of that function or variable. If you want to find all the occurrences of a variable (or any bit of text) in your project, you can open the Search menu and select "File Search." Make sure the "File Search" tab (NOT the Java Search tab which is often selected by default). *

The datastore

Android apps can use a built-in SQLITE database to store data that should be preserved from session to session. In Libretto, the creation of this database happens in the LibrettoContentProvider.java file in the org.nypl.libretto package. This file (which describes a section of reusable code Java developers call a "class") builds on and "extends" the functionality of a built-in Java class called "ContentProvider." The basic ContentProvider class has a function called "onCreate", which is redefined (or "Overridden"), in LibrettoContentProvider (on line 68 or so). When Android checks the manifest and determines that the "ContentProvider" for this app is "LibrettoContentProvider" it runs this function, which, you will see, sets up a new SqliteDBHelper (a class that is defined in the org.nypl.libretto.database package).

In SqliteDBHelper there is an "onCreate" function that first invokes the DatabaseTable class (located in the org.nypl.libretto.database package) that sets up all of the tables and columns in the database. The numbers of these tables are non-sequential because I removed some non-working annotation and bookmarking functions late in the development process. The tables here are:

• PLAY: A single epub (e.g. The Black Crook)
• VERSION: A "witness" or version of the text (e.g. Harvard Theater Collection Promptbook)
• ANCHOR: A point that is synchronized across versions of the text.
• AUDIO: The location on the file system of an audio file
• CHAPTERS: Divisions within an text (scenes, acts, chapters, whatever)
• SHEETMUSIC: The location of Sheet music SVG files

After the tables are created, the "onCreate" function calls the CsvToSqliteImport class and it's "readFromCsvForPlayTable" function. This function looks for a file called "playjsonformat.json" in the assets folder of the app (a folder that sits outside of the "src" folder). Initially, this file only contains a reference to the "About" document that describes the app. While the database was being created, the SplashActivity was running, and it has by this point attempted to update this file.

Getting a list of scripts

SplashActivity also has an "onCreate" function which runs when it is invoked. This function checks to see whether it has a wireless signal (either from a cell network or a wifi router) and then tries to download a library file hosted on Google Drive. If it succeeds, it will add all scripts linked from this Google Drive file to the playsjsonformat.json file. In any case, once the downloading has completed or failed, a function called startUp is called which invokes a new activity.

Activities in Android are started through "Intents." This is the way Android can pass control of the program from one activity to another (along with relevant data), while still keeping the first activity alive in the background if desired. On around line 130 of SplashActivity, a new Intent is created and invoked to pass control over to PlaylistActivity.

The Home Screen (PlaylistActivity)

The PlaylistActivity displays the list of available plays from the playsjsonformat.json file. In the "onCreate" function, it identifies the template from the res folder it will use with the "setContentView" function (around line 75). This activity uses the aforementioned R library to identify plays_activity.xml which is in the res/layout folder as the base template and then populates the tag values through a series of calls to the XML using commands like "findViewById."

Downloading/installing a libretto

When a title not currently on the device but listed on the PlayListActivity screen is clicked, the function defined in setOnChildClickListener in PlayListActivity runs (starting at around line 198). This function first checks to see whether the "About" file has been clicked (which would invoke a different set of actions). Otherwise, it tries to download and unzip to new play. The unzipped folder is stored in a content directory reserved for the app, and, starting at line 446 the app reads the nav.xhtml file that defines the content of an ePub book and parses all of the versions and audio files into the database (using the CsvToSqliteImport class from the org.nypl.libretto.database package and the functions it calls from the org.nypl.libretto.parsing package). I won't go into detail here about how the ePUB itself is parsed, but hopefully the code itself will be relatively clear. If there is any interest, I will describe this part of the code in a later blog.

Reading a libretto

Once the libretto is installed, a new Intent is invoked at around line 238 of PlayListActivity. This intent includes a lot of variables (called "extras") pushed into the intent with the "i.putExtra" commands that follow the declaration. These extras comprise a data object that gets packaged up and sent to the new intent, PlaysDetailsActivity.

PlaysDetailActivity sets up the reading environment with the ability to browse through scenes. It also establishes the ViewPagerAdapter—an extension of a built-in Android interface called a PagerAdapter that allow users to page through a set of screens with a horizontal swipe. In Libretto, this interface element is used to let users swipe through versions of an play (in ViewPagerAdapter) and through pages of sheet music (in SheetMusicPagerAdapter).

Within ViewPagerAdapter (and, for that matter, SheetMusicPagerAdapter), a mini-web browser (called a WebView) is established. This built-in Android object allows us to load HTML files in and pass data from JavaScript to the native code and back again. In Libretto, for instance, we want to make sure the interface scrolls to approximately the same position in a scene as the user flips between versions. This is done by tracking a series of anchor points (built into the HTML files of the scenes that are packaged in the ePUB) using the JavaScript code in "anchor.js" (which is added to the HTML files during the parsing of the ePUB). In ViewPagerAdapter, a JavaScript interface called appScrollManager is created at around line 246 or so and attached to the object "jsScrollPosition" which is defined at line 140 as an instance if the custom ScrollPosition object created at the top of the class. The anchor.js code can talk to this object and cause things to happen in the native Android Java code. This is, of course, a little bit of a security vulnerability, so its wise to be sure you know exactly what JavaScript code might be run from the WebView.

It somewhat safer, though not always practical, to allow Java to capture user actions in the WebView by overriding a URL change. That is, when a user clicks a link, it is possible to prevent the usual behavior and cause something else to happen. This is what Libretto does when a user clicks either an audio icon or a sheet music link (see the shouldOverrideURLLoading function at around line 308 of ViewPagerAdapter).

Conclusion

Hopefully this will give interested developers some guidance for modifying Libretto or creating their own similar apps. There is much of the code I've left unexplored, but this blog entry is already getting a little long and confusing. If there's anything you like to know more about, leave a comment or email me at dougreside@nypl.org.