Julien Lecomte's Blog

I am pleased to announce that the YUI Compressor is now hosted on GitHub. Hopefully, this move will make the external contribution process more streamlined. Have fun coding!

If you haven’t done so already, head over to the YUI Blog for the official announcement. I am proud to have made a (very modest) recent contribution to the library by porting the browser history utility to YUI3 (take a look at the source code) By the way, this new version of the browser history utility supports Opera and IE8 (in both quirks mode and IE7 standards mode) Moreover, it has more features than the YUI2 version, should be easier to use, and weighs a few hundred bytes less (after minification with the YUI Compressor) I’ll try to find some time in the next few weeks/months to write a bit more about YUI3 in general, and the new browser history utility and the YUI Compressor. Stay tuned!

The YUI Compressor uses a slightly modified version of the parser used in the Rhino JavaScript engine. The reason for modifying the parser came from the need to support JScript conditional comments, unescaped forward slashes in regular expressions — which all browsers support and many people use — and a few micro optimizations. The problem is that many users had the original Rhino Jar file somewhere on their system, either in their classpath, or in their JRE extension directory (<JRE_HOME>/lib/ext) This caused many headaches because the wrong classes were being loaded, leading to many weird bugs.

Today, I finally decided to do something about it. This meant writing a custom class loader to load the modified classes directly from the YUI Compressor Jar file. You can download the source and binary package here:

Download version 2.4 of the YUI Compressor

The skeleton of the custom class loader is pretty straightforward:

package com.yahoo.platform.yui.compressor;

public class JarClassLoader extends ClassLoader
{
    public Class loadClass(String name) throws ClassNotFoundException
    {
        // First check if the class is already loaded
        Class c = findLoadedClass(name);
        if (c == null) {
            // Try to load the class ourselves
            c = findClass(name);
        }

        if (c == null) {
            // Fall back to the system class loader
            c = ClassLoader.getSystemClassLoader().loadClass(name);
        }

        return c;
    }

    protected Class findClass(String name)
    {
        // Most of the heavy lifting takes place here
    }
}

The role of the findClass method is to first locate the YUI Compressor Jar file. To do that, we look in the classpath for a Jar file that contains the com.yahoo.platform.yui.compressor.JarClassLoader class:

private static String jarPath;

private static String getJarPath()
{
    if (jarPath != null) {
        return jarPath;
    }

    String classname = JarClassLoader.class.getName().replace('.', '/') + ".class";
    String classpath = System.getProperty("java.class.path");
    String classpaths[] = classpath.split(System.getProperty("path.separator"));
    for (int i = 0; i < classpaths.length; i++) {
        String path = classpaths[i];
        JarFile jarFile = new JarFile(path);
        JarEntry jarEntry = findJarEntry(jarFile, classname);
        if (jarEntry != null) {
            jarPath = path;
            break;
        }
    }

    return jarPath;
}

private static JarEntry findJarEntry(JarFile jarFile, String entryName)
{
    Enumeration entries = jarFile.entries();
    while (entries.hasMoreElements()) {
        JarEntry entry = (JarEntry) entries.nextElement();
        if (entry.getName().equals(entryName)) {
            return entry;
        }
    }
    return null;
}

Once we know where the YUI Compressor Jar file is, we can load the appropriate class from that file. Note the need to define the package the class belongs to before calling defineClass!

protected Class findClass(String name)
{
    Class c = null;
    String jarPath = getJarPath();
    if (jarPath != null) {
        JarFile jarFile = new JarFile(jarPath);
        c = loadClassData(jarFile, name);
    }
    return c;
}

private Class loadClassData(JarFile jarFile, String className)
{
    String entryName = className.replace('.', '/') + ".class";
    JarEntry jarEntry = findJarEntry(jarFile, entryName);
    if (jarEntry == null) {
        return null;
    }

    // Create the necessary package if needed...
    int index = className.lastIndexOf('.');
    if (index >= 0) {
        String packageName = className.substring(0, index);
        if (getPackage(packageName) == null) {
            definePackage(packageName, "", "", "", "", "", "", null);
        }
    }

    // Read the Jar File entry and define the class...
    InputStream is = jarFile.getInputStream(jarEntry);
    ByteArrayOutputStream os = new ByteArrayOutputStream();
    copy(is, os);
    byte[] bytes = os.toByteArray();
    return defineClass(className, bytes, 0, bytes.length);
}

private void copy(InputStream in, OutputStream out)
{
    byte[] buf = new byte[1024];
    while (true) {
        int len = in.read(buf);
        if (len < 0) break;
        out.write(buf, 0, len);
    }
}

The last thing we need to do is bootstrap the application. In order to do that, we simply load the main class (YUICompressor) using our new custom class loader. All the classes that will be needed at runtime will use the same class loader:

package com.yahoo.platform.yui.compressor;

public class Bootstrap
{
    public static void main(String args[]) throws Exception
    {
        ClassLoader loader = new JarClassLoader();
        Thread.currentThread().setContextClassLoader(loader);
        Class c = loader.loadClass(YUICompressor.class.getName());
        Method main = c.getMethod("main", new Class[]{String[].class});
        main.invoke(null, new Object[]{args});
    }
}

As you can see, it's not terribly complicated to write a custom class loader. Note: I left out all the exception handling code and the import statements for clarity. The final code can be found in the downloadable archive. Cheers!

This new version of the compressor fixes a few bugs and implements a few additional micro optimizations. Please refer to the CHANGELOG file for a complete list of changes, and don’t hesitate to report any issue you may experience with this version of the YUI Compressor.

Download version 2.3 of the YUI Compressor

My name is Julien Lecomte, and this is my personal weblog. I write mainly about my passion, amateur astronomy, as well as my technical interests, including operating system development and web technologies.

I am currently a senior software engineer in the Yahoo! Search frontend team. Prior to working on web search, I was part of Yahoo!’s client evangelism team, a group that provides architectural assistance to Yahoo! developers on the design and implementation of rich interactions in the browser, working alongside Douglas Crockford, Bill Scott and Iain Lamb (co-founder of OddPost). Among other things, I worked extensively on Yahoo! Mail, wrote the YUI Browser History Manager and the YUI Compressor, and contributed to many different projects throughout the company.

Prior to working at Yahoo, I was a senior software engineer at Scalix (now part of Xandros), where I spent a couple of years developing their web client. Prior to joining Scalix, I was a software engineer at easyplanet where I worked on a consumer VoIP application based on the OpenH323 library.

During my spare time, I like to go hiking in the hills surrounding the Santa Clara valley, observe the night sky, build furniture in my woodshop, work on my hobby operating system — Simplix — enjoy some good wine and foods, and travel to exotic destinations with my beautiful wife.

In 2001, I obtained a Master’s degree in computer science from Supelec, a leading engineering institute in France.

My resume can be found at http://www.julienlecomte.net/resume.html.

Additionally, you may contact me at julien [dot] lecomte [at] gmail [dot] com

I implemented a few enhancement requests and fixed a bug in this new version of the YUI Compressor. Let me know if you encounter any issue with it.

Update (9/27/07): YUI Compressor version 2.2.2 is now available. It fixes a lot of bugs that have been reported recently. By the way, I really appreciate all the bug reports, so keep them coming!

Update (9/28/07): New bugs have been reported and fixed in version 2.2.3 now available for download (check out the CHANGELOG file in the download page) And keep these bug reports coming!

Update (10/1/07): A few more minor bugs have been fixed in version 2.2.4. Thanks for the bug reports!

Download version 2.2.4 of the YUI Compressor

This new version of the YUI Compressor supports stdin and stdout. This means that you can now call the YUI Compressor using the following command line:

java -jar yuicompressor-2.2.jar --type js < input.js > output.js

You can still use the following syntax as well:

java -jar yuicompressor-2.2.jar -o output.js input.js

This has three main consequences:

1. All informational and error messages are now printed to stderr.
2. If no input file is specified, the YUI Compressor defaults to stdin. In that case, you must specify the --type option because the YUI compressor has no way of knowing whether it should invoke the JavaScript or CSS compressor (there is no file extension to look at)
3. If no output file is specified, the YUI Compressor defaults to stdout (in prior versions, it used to create a file named after the input file, and appended the -min suffix)

The other main feature brought by this new version of the YUI Compressor is the support for JScript conditional comments:

/*@cc_on
   /*@if (@_win32)
      document.write("OS is 32-bit, browser is IE.");
   @else @*/
      document.write("Browser is not 32 bit IE.");
   /*@end
@*/

Note that the presence of a conditional comment inside a function (i.e. not in the global scope) will reduce the level of compression for the same reason the use of eval or with reduces the level of compression (conditional comments, which do not get parsed, may refer to local variables, which get obfuscated) In any case, the use of Internet Explorer’s conditional comments is to be avoided.

Finally, a few improvements have been made to the CSS compressor.

Download version 2.2 of the YUI Compressor

The importance of having a solid build process

Modern web applications are large and complicated pieces of engineering, using many different technologies, sometimes deployed on hundreds (or even thousands) of servers throughout the world, and used by people from dozens of locales. Such applications cannot efficiently be developed without relying on a solid build process to do all the dirty and repetitive work of reliably putting all the pieces together.

Apache Ant

Many tools (make, gnumake, nmake, jam, etc.) are available today to build applications. However, when it comes to building a web application, my personal favorite is definitely Apache Ant (here is a good explanation of Ant’s benefits over the other tools) This short tutorial will assume you already have some basic knowledge of Ant (if you don’t, you should flip through its user manual beforehand) This article will focus mainly on building front-end code using Ant.

Build types

It is often useful to build an application differently at different stages of its life cycle. For instance, a development build will have assertions and tons of logging code, while a production version will be stripped of all that. You may also need some intermediate build to hand off to QA, and still be able to debug things easily while running actual production code. With Ant, you can create one target per build type, and have a different dependency list for each target. Properties can also be used to customize specific build types. Here is an example:

<target name="dev" depends="load.properties.dev, js.preprocess, js.check.syntax, copy.jsp, copy.image.files, copy.css.files, copy.js.files, compile.jsp, copy.properties.files, compile.webapps, copy.libs"
    description="Development build">
</target>

<target name="prod" depends="load.properties.prod, js.preprocess, js.check.syntax, js.concatenate, js.minify, copy.jsp, copy.image.files, copy.css.files, copy.js.files, compile.jsp, copy.properties.files, compile.webapps, copy.libs"
    description="Full production build">
</target>

<target name="load.properties">
    <property file="ant.properties"/>
</target>

<target name="load.properties.dev" depends="load.properties">
    <property name="js.preprocess.switches" value="-P -DDEBUG_VERSION=1"/>
    <property name="js.compressor.switches" value="--nomunge --line-break"/>
</target>

<target name="load.properties.prod" depends="load.properties">
    <property name="js.preprocess.switches" value="-P -DDEBUG_VERSION=0"/>
    <property name="js.compressor.switches" value=""/>
</target>

Concatenate your JavaScript and CSS files

Concatenating JavaScript and CSS files contributes to making your site faster according to Yahoo!’s Exceptional Performance team. File concatenation using Ant is trivial using the concat task. However, it is often important to concatenate JavaScript and CSS files in a very specific order (If you are using YUI for example, you want yahoo.js to appear first, and then dom.js/event.js, and then animation.js, etc. in order to respect module dependencies) This can be accomplished using a filelist, or a combination of filesets. Here is an example:

<target name="js.concatenate">
    <concat destfile="${build.dir}/concatenated/foo.js">
        <filelist dir="${src.dir}/js"
            files="a.js, b.js"/>
        <fileset dir="${src.dir}/js"
            includes="*.js"
            excludes="a.js, b.js"/>
    </concat>
</target>

It is also possible to prepend some comments to the destination file, which is often used for license and copyright information, using a nested header element (see the Apache Ant manual) Just keep in mind that minifying your code will make these comments go away, so you may want to do this later in the build process.

Preprocess your JavaScript files using CPP

Preprocessing JavaScript code is very useful to facilitate the development process. It allows you to define your own macros, and conditionally compile blocks of code. One easy way to preprocess JavaScript code is to use the C preprocessor, which is usually installed by default on UNIX machines. It is available on Windows machines as well using cygwin. Here is a snippet of an Ant build.xml file illustrating the use of cpp to preprocess JavaScript files:

<target name="js.preprocess" depends="js.concatenate">
    <apply executable="cpp" dest="${build.dir}/preprocessed">
        <fileset dir="${build.dir}/concatenated"
            includes="foo.js"/>
        <arg line="${js.preprocess.switches}"/>
        <srcfile/>
        <targetfile/>
        <mapper type="identity"/>
    </apply>
</target>

You can now write JavaScript code that looks like the following:

include.js:

#if DEBUG_VERSION
function assert(condition, message) {
    ...
}
#define ASSERT(x, ...) assert(x, ## __VA_ARGS__)
#else
#define ASSERT(x, ...)
#endif

foobar.js:

#include "include.js"

function myFunction(arg) {
    ASSERT(YAHOO.lang.isString(argvar), "arg should be a string");
    ...
#if DEBUG_VERSION
    YAHOO.log("Log this in debug mode only");
#endif
    ...
}

Just a word of caution here: make sure you use the UNIX EOL character before preprocessing your files. Otherwise, you’ll have some issues with multi-line macros. You may use the fixcrlf Ant task to automatically fix that for you.

Minify your JavaScript and CSS files

Minifying your JavaScript and CSS files will help make your application faster according to Yahoo!’s Exceptional Performance team. I warmly recommend you use the YUI Compressor, available for download on this site. There are two ways to call the YUI Compressor from Ant. Here is the first one using the java task:

<target name="js.minify" depends="js.preprocess">
    <java jar="yuicompressor.jar" fork="true">
        <arg value="foo.js"/>
    </java>
</target>

Here is another way using the apply task that allows you to pass several files to the compressor using a fileset:

<target name="js.minify" depends="js.preprocess">
    <apply executable="java" parallel="false">
        <fileset dir="." includes="foo.js, bar.js"/>
        <arg line="-jar"/>
        <arg path="yuicompressor.jar"/>
        <mapper type="glob" from="*.js" to="*-min.js"/>
    </apply>
</target>

Update: Starting with version 2.2.x of the YUI Compressor, the ant target described above needs to be slightly modified:

<target name="js.minify" depends="js.preprocess">
    <apply executable="java" parallel="false">
        <fileset dir="." includes="foo.js, bar.js"/>
        <arg line="-jar"/>
        <arg path="yuicompressor.jar"/>
        <srcfile/>
        <arg line="-o"/>
        <mapper type="glob" from="*.js" to="*-min.js"/>
        <targetfile/>
    </apply>
</target>

Also consider the following ant target to minify CSS files:

<target name="js.minify" depends="js.preprocess">
    <apply executable="java" parallel="false">
        <fileset dir="." includes="*.css"/>
        <arg line="-jar"/>
        <arg path="yuicompressor.jar"/>
        <arg line="--line-break 0"/>
        <srcfile/>
        <arg line="-o"/>
        <mapper type="glob" from="*.css" to="*-min.css"/>
        <targetfile/>
    </apply>
</target>

Work around caching issues

Adequate cache control can really enhance your users’ experience by not having them re-download static content (usually JavaScript, CSS, HTML and images) when coming back to your site. This has a downside however: when rev’ing your application, you want to make sure your users get the latest static content. The only way to do that is to change the name of these files (for example by adding a time stamp to their file name, or using their checksum as their file name) You must also propagate that change to all the files that refer to them.

The copy and file name replacement will be handled by a custom Ant task named FileTransform (see FileTransform.java) The first step is to build the task from the source and define the custom task:

<target name="-setup.build.tools" depends="-load.properties">
    <mkdir dir="${build.dir}/tools/classes"/>
    <javac srcdir="tools/src"
           destdir="${build.dir}/tools/classes"
           includes="**/*.java">
        <classpath>
            <pathelement location="tools/lib/ant.jar"/>
        </classpath>
    </javac>
    <taskdef name="FileTransform"
        classname="com.yahoo.platform.build.ant.FileTransform"
        classpath="${build.dir}/tools/classes"/>
</target>

(Note the use of a “-” in front of the name of the target. This is used to make the target “private” i.e. not invokable directly) The second part is the copy of the static content that is going to be cached using the newly defined FileTransform task:

<target name="-copy.js.files" depends="-setup.build.tools">
    <mkdir dir="${build.dir}/js"/>
    <FileTransform todir="${build.dir}/js"
        changefilenames="true"
        propertiesfile="${build.dir}/js.files.mapping.properties">
        <fileset dir="site/js" includes="*.js"/>
    </FileTransform>
</target>

The mapping between the old names and the new names is stored in a properties file. The final step is the copy of the files that refer to files which name was changed in the previous step. For this, we use the copy task:

<target name="-copy.php.files" depends="-copy.js.files">
    <mkdir dir="${build.dir}/php"/>
    <copy todir="${build.dir}/php">
        <fileset dir="site/php" includes="*.php"/>
        <filterset>
            <filtersfile file="${build.dir}/js.files.mapping.properties"/>
        </filterset>
    </copy>
</target>

You can download an archive containing a very simple Ant project illustrating this advanced technique.

Deploy your application

Deploying a web application is usually done by copying a set of files over to the production servers and running a list of commands on those servers (stop the services, run a few shell commands, and finally restart the services)

You have many options to copy files to remote servers with Apache Ant. You could use the copy task to copy files to a locally mounted file system, or you could use the optional FTP task. My personal preference is to use scp and rsync (both utilities are available on all major platforms) Here is a very simple example demonstrating the use of scp with Apache Ant:

<apply executable="scp" failonerror="true" parallel="true">
    <fileset dir="${build.dir}" includes="**/*"/>
    <srcfile/>
    <arg line="${live.server}:/var/www/html/"/>
</apply>

And here is an example showing how you can run remote commands:

<exec executable="ssh" failonerror="true">
    <arg line="${live.server}"/>
    <arg line="sudo webctl restart"/>
</exec>

Conclusion

I hope this article has given you some ideas of what’s possible to automate with modern build tools such as Apache Ant (other cool things include automatic generation of documentation, source control integration, packaging using rpm and such, etc.) My professional experience has taught me that the build process cannot be an afterthought. Just like performance, it should be part of your project from the very beginning. The second lesson is that all web developers need to be active participants in creating and maintaining the build process (with maybe one build engineer to centralize all the information) instead of relying on somebody else to build their component. Finally, keep your build process alive and maintain it actively as your needs will change during your project life cycle.

This new version of the compressor fixes a few bugs and optimizes string concatenations. You can now keep your code nicely formatted by concatenating string literals, but not suffer any performance hit. For example, the following code:

var url = "http://www.flickr.com/services/rest/"
    + "?method=flickr.test.echo"
    + "&format=json"
    + "&api_key=02b6a73df2c4a33c282f3d02fe5724e3"
    + "&callback=jsonCallback";

will become (↵ marks a line continuation)

var url = "http://www.flickr.com/services/rest/↵
?method=flickr.test.echo&format=json↵
&api_key=02b6a73df2c4a33c282f3d02fe5724e3&callback=jsonCallback";

Also, note that, from now on, all YUI Compressor-related downloads will be done from a separate download page and not directly from this blog.

Download version 2.1 of the YUI Compressor

I am very pleased to announce the release of version 2.0 of the YUI Compressor. In addition to fixing several bugs and implementing a few enhancements suggested by the community, I also integrated Isaac Schlueter’s regular expression based CSS minifier. Therefore, the YUI Compressor is now able to compress both JavaScript and CSS files (see the CHANGELOG for a full list of changes) And as always, keep the feedback coming!

Download version 2.0 of the YUI Compressor