File transfers using SOAP (XFire)

This past week I’ve been spending some time working on an API for uploading your resources (applications, war files, database dumps, whatever) to HostedQA. While HostedQA can currently automate the entire test process, including setting up your database and app server, it didn’t yet have continuous integration support. Of course, that sort of defeats the purpose of automation, since that is where most automation begins.

So over the last week I spent some time adding an API that allowed for sending large files (100MB or more). At first I figured I’d just go with a simple HTTP POST (REST-style). In fact, at the time that’s what we were using to send large data between our client virtual machines and the main HostedQA server (for the rest of the API we were using Hessian). I decided to bit the bullet and try to offer a true SOAP API using XFire.

So, hopping on to #xfire on irc.codehaus.org, Dan Diephouse helped out immediately and ended up actually spending a lot of time making file attachments even better. In almost no time I had switched out Hessian support and created an Ant task that could upload files over SOAP via XFire. The only trick was enabling MTOM. Although not spelled out in that doc, if you’re using Spring for your XFire services, you can enable MTOM by simply adding this to your ServiceBean definition (_not_ your actual service bean):

<property name="properties">
<map>
<entry key="mtom-enabled" value="true"/>
</map>
</property>

After that, simply adding a javax.activation.DataSource parameter on the service interface was all you had to do to get file upload support working. Sort of.

It turns out that while it worked, large files would cause OutOfMemoryErrors because they were being buffered up in memory. No good, considering that with HostedQA we have to support database snapshots and war files that could be somewhat large. Even at just 10 or 20MB, I don’t want to take up all that memory. So back to #xfire and Dan promises to hack away support for it.

After a day, Dan had a pretty good solution in place that even worked with .NET. Not bad! The only trick was that HTTP chunking needed to be turned on by the client now, otherwise streaming wouldn’t even happen at the HTTP level. Since Jetty and Resin (our development and production app servers, respectively) both support chunking, the only thing that I had to do was update my findService method in RpcUtils.java:

public static <T> T findService(String url, Class<T> clazz) {
Service serviceModel = new ObjectServiceFactory().create(clazz);
T service = (T) new XFireProxyFactory().create(serviceModel, url);

Client client = ((XFireProxy) Proxy.getInvocationHandler(service)).getClient();
client.setProperty("mtom-enabled", "true");
client.setProperty(HttpTransport.CHUNKING_ENABLED, "true");

return service;
}

Now files are being streamed across the wire perfectly, all using XFire and open protocols. Hessian was immediately ditched and our continuous integration support is just about complete. There were a few hiccups, however:

• Currently, servers that are given DataSources (in a method parameter) clean up the temporary File automatically. However, clients that get DataSources returned to them do not have their temp files cleaned up. I’m working on a patch to fix this.
• On top of all of this, we were also adding SSL support to the whole site. SSL is generally a pain, but even more so when you have a wilcard domain (we make http://yourcompany.hostedqa.com to each of our customers) and many virtual machines all needing to communicate back to the main server. Not impossible, but lots of moving parts that can easily get goofed up.
• The STAX XML parser is a pain in the butt. Again, I hit more problems than most will due to the complex deployments we do (JVM launches a VM, which launches a JVM, which launches an app server on another JVM… don’t ask :P). The solution was to make sure I was using the stax-api-1.0.1.jar and also that -Djavax.xml.stream.XMLInputFactory= com.ctc.wstx.stax.WstxInputFactory was set. Otherwise you might run in to problems where you get a ClassNotFoundError about a BEA class (due to XBean looking for BEA’s implementation).
• Generally, XFire requires a ton of jars - more than I really want. I always liked how Hessian was just a single jar. Because of these jars, I did run in to a couple compatibility issues, especially around Spring parsing applicationContext.xml.
• Finally, as my first time really using SOAP, I had to change some of the objects I was passing over the wire. Using Hessian, which uses serialization, I could have circular references in my objects. SOAP doesn’t support that, so I ended up having to create some simpler versions of my model (TestSuite.getApi() returns TestSuiteAPI).

Overall though, XFire was actually a lot easier than we thought!