Tangled in the Threads

Jon Udell, February 22, 2002

WSDL Interop Adventures

I come neither to praise WSDL, nor to bury it, but only to better understand it. The Web Services Description Language, intended to make consumption of Web services more frictionless, has itself lately generated a fair amount of friction. The concept, which can (sometimes) dramatically be seen in action in Visual Studio.NET, is that WSDL makes Web services self-explanatory. You just walk up to a service (i.e., access its WSDL file), and it says: "Here are the services, methods, and types that I define." Consider, for example, a service called PlaceFinder, available at xmethods.org. Naming its WSDL file in VS.NET's Add Web Reference dialog enables you to write code like this: 

com.geographynetwork.www.PlaceFinder ws = new com.geographynetwork.www.PlaceFinder();
com.geographynetwork.www.LocationInfo l = ws.findPlace("New York");
Console.WriteLine(l.candidates[0].description1);
Console.WriteLine(l.candidates[0].x.ToString());
Console.WriteLine(l.candidates[0].y.ToString());

At each step along the way, VS.NET's IntelliSense (namespace completion) makes the API of this service quite magically discoverable. When you call ws.findPlace, for example, the UI shows you that a string argument is required, and a LocationInfo object will be returned. After assigning that object to a variable ("l"), typing its name plus a dot reveals that "candidates" is an available member, and that it is an array of Location objects. Accessing one of those location objects reveals that it has numeric members called "x" and "y", and a string called "description1."

Four SOAP toolkits

This is just plain wonderful when it works, but it often doesn't. I'd heard conflicting reports as to why not, so I sat down with a collection of SOAP toolkits to see what I could discover. The toolkits, all of which can both produce and consume SOAP services, were:

Rather than implement yet another stock-quote or temperature-conversion service, I decided to work on something that would help me build a real groupware application. There is data, in Radio UserLand, that it would be interesting and useful to share. One of its databases, in particular, keeps track of the inbound newsfeeds that you subscribe to. Enabling a team of Radio users working together as a team to amalgamate those feeds could be a worthwhile groupware feature.

A Radio UserLand Web service

The newsfeed data lives in a table in one of Radio UserLand's object databases. Answering a SOAP call with that data is, at one level, completely trivial. You just drop a Frontier script into the folder \radio\Web Services, and it's automatically available as a SOAP method. Here was my first attempt: 

on subs()
  {
  local (t);
  new (tableType, @t);
  for adr in (@aggregatorData.services)
   {
   local(title = adr^.compilation.channeltitle);
   t.[title] = nameOf(adr^);
   };
  return (t)
  }

This code reduces aggregatorData.services, a Frontier table with lots of information in it, to a simpler table whose keys are the URLs of RSS files, and values are channel titles. Then it simply returns the table, to be serialized as a SOAP response.

This didn't work. When serializing this structure, Radio tries to make URLs into the names of XML elements, but URLs contain characters not legal in XML names.

Here was my second attempt: 

on subs()
  {
  local (t);
  new (tableType, @t);
  for adr in (@aggregatorData.services)
   {
   local(title = adr^.compilation.channeltitle);
   t.[title] = nameOf(adr^);
   };
  return (table.tableToXML(@t));
  }

Here, the table is explicitly serialized as XML, and returned as a string. Although this works, it's not too useful. As a consumer of this service, you don't want an opaque bundle of XML that you then have to parse and map into your language's data structures. You want to receive something as one of your language's data structures.

Here was my third attempt: 

on subs()
  {
  local (l);
  new (listType, @l);
  for adr in (@aggregatorData.services)
   {
   new (listType, @t);
   t[0] = adr^.compilation.channeltitle;
   t[0] = nameOf(adr^);
   l[0] = @t;
   };
  return (l);
  }

This returns, in Perlish terms, a LoL (list-of-lists). But when I tested it locally -- the easiest way to do this is to execute the one-liner ["soap://localhost:5335/"].radio.subs() in the Quick Script window -- there was again the problem that Radio didn't want to serialize this structure as a SOAP response.

Here was my fourth attempt: 

on subs(urlsOrTitles)
    {
    local (l);
    new (listType, @l);
    for adr in (@aggregatorData.services)
        {
        case urlsOrTitles
            {
            "urls" 
                { l[0] = nameOf(adr^); };
            "titles" 
                { l[0] = adr^.compilation.channeltitle; }
            }
        };
    return ( l );
    };

Diving into WSDL

Here, I'm just returning a single array of strings -- either the URLs or the titles. This checked out fine locally. It's not ideal, since the natural form of the returned data is a pair of arrays, but it's a starting point. The next question was: how to write the WSDL file to describe this service to, say, VS.NET?

There's been a lot of discussion recently about whether, or how, scripting languages like Frontier's UserTalk, or Perl, or Python, can produce WSDL files that describe SOAP services they publish in terms that languages like Java and C# can understand. The bottom line, for these dynamic languages, is that you have to provide some type description. Here's how the ActiveState folks have done that for Perl: 

=for interface
  static ustring echostring(ustring something);
  <soap namespace="http://www.ActiveState.com/echo/"/> 
=cut 

sub echostring {
     my $echo = shift;
     return $echo;
 }

Clearly this requires double maintenance: the type description has to evolve in tandem with the method. I would summarize the arguments surrounding this issue like so:

Since Frontier doesn't yet offer WSDL-generating assistance like that shown here for Perl, I needed to write a WSDL file to describe my service. Asking VS.NET to generate one for me seemed like a bright idea. So I wrote a VS.NET Web service with this code at its core: 

string[] channels = new string[10];
return channels;

Here is the <types> element of the WSDL file that VS.NET wrote: 

<types>
<s:schema elementFormDefault="qualified" targetNamespace="http://tempuri.org/">
<s:element name="subs">
  <s:complexType /> 
</s:element>

<s:element name="subsResponse">
<s:complexType>
<s:sequence>
<s:element minOccurs="0" maxOccurs="1" name="subsResult" type="s0:ArrayOfString" /> 
</s:sequence>
</s:complexType>
</s:element>

<s:complexType name="ArrayOfString">
<s:sequence>
<s:element minOccurs="0" maxOccurs="unbounded" name="string" nillable="true" type="s:string" /> 
</s:sequence>
</s:complexType>
<s:element name="ArrayOfString" nillable="true" type="s0:ArrayOfString" /> 
</s:schema>
</types>

A working, but suboptimal, solution

Looked reasonable to me, but I had no luck consuming my Radio service from VS.NET by way of this WSDL. Eventually I arrived at a solution that did work. I am embarrassed to say how long it took me. Let's just say that when, in the wee hours, I found this mailing-list posting, I realized I wasn't the only one with a headache. Here is the solution I finally came to: 

<?xml version="1.0" encoding="utf-8"?>
<definitions xmlns:http="http://schemas.xmlsoap.org/wsdl/http/"
 xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/"
 xmlns:s="http://www.w3.org/2001/XMLSchema"
 xmlns:s0="uri:xsd.subs"
 xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/"
 xmlns:tns="uri:subs"
 targetNamespace="uri:subs"
 name="subs"
 xmlns="http://schemas.xmlsoap.org/wsdl/">

  <types>
    <s:schema targetNamespace="uri:xsd.subs">

    <s:complexType name="ArrayOfstring">
      <s:complexContent>
        <s:restriction base="soapenc:Array">
          <s:sequence>
            <s:element name="item" type="string" minOccurs="0"
               maxOccurs="unbounded" nillable="true"/>
          </s:sequence>
          <s:attribute ref="soapenc:arrayType" wsdl:arrayType="s:string[]" 
                xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/"/>
        </s:restriction>
      </s:complexContent>
    </s:complexType>
    </s:schema>
    </types>

  <message name="subsRequest">
    <part name="what" type="s:string"/>
  </message>

  <message name="subsResponse">
    <part name="titlesOrURLs" type="s0:ArrayOfstring"/>
  </message>

  <portType name="subsPort">
    <operation name="subs">
      <input message="tns:subsRequest"/>
      <output message="tns:subsResponse"/>
    </operation>
  </portType>


  <binding name="subsSoap" type="tns:subsPort">
    <soap:binding style="rpc"
      transport="http://schemas.xmlsoap.org/soap/http"/>
    <operation name="subs">
      <soap:operation soapAction="/radio" style="rpc"/>
      <input>
        <soap:body use="encoded" namespace="uri:subs" 
          encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"/>
      </input>
      <output>
        <soap:body use="encoded" namespace="uri:subs" 
          encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"/>
      </output>
    </operation>
  </binding>

<service name="subsService">
    <document>
        Enumerate Jon's RSS subscriptions, by title or URL.
    </document>
    <port name="subsPort" binding="tns:subsSoap">
      <soap:address location="http://192.168.1.101:5335/"/>
    </port>
  </service>

</definitions>

GLUE users reading this are probably chuckling by now. Because in this case, the WSDL type definitions generated by the equivalent service in Java: 

java.lang.String[] channels = new java.lang.String[10];
return channels;

exactly match the WSDL type definitions that I finally figured out how to write. Sigh.

Was nirvana now achieved? Not quite. It's true that I could now point VS.NET at my WSDL. I could use IntelliSense to discover the service ("subsService"), method ("subs"), return type (array of strings), and argument name and type ("titlesOrUrls", string). The C# program that I wrote to call this service did, indeed, receive the requested array of strings. But this was still kind of lame. The user of the service has to call it twice, then pair up the arrays. That's awkward,and what if things change in the meantime? So I went back to the drawing board.

Back to the drawing board

When I showed my work to Jake Savin, over at UserLand, he came up with a couple of alternative ideas. Here's the one I liked best: 

on subs() {
local (t);
new (tableType, @t);
for adr in (@aggregatorData.services) {
local (st); new (tableType, @st);
st.title = adr^.compilation.channeltitle;
st.url = nameOf (adr^);
table.uniqueName ("ch", @t, 5)^ = st};
return (t)}

This approach generates XML-legal names, like ch00001: 

<SOAP-ENV:Body>
  <subsResponse>
    <Result>
       <ch00001>
          <title xsi:type="xsd:string">Jon Udell</title>
          <url xsi:type="xsd:string">http://udell.roninhouse.com/udell.rdf</url>
       </ch00001>
       <ch00002>
          <title xsi:type="xsd:string">Privacy Digest Weblog</title>
          <url xsi:type="xsd:string">http://www.newsisfree.com/HPE/xml/feeds/29/1929.xml</url>
       </ch00002>

... etc ...

OK, now how to describe this in WSDL? Well, you can't. Simon Fell (http://www.pocketsoap.com/), whose tools map between SOAP and COM, explains:

  1. This is 100% valid section 5 SOAP.

  2. Any SOAP toolkit that either (i) likes WSDL or (ii) attempts to map it to a static type will throw its hands up in horror, as the structure is not describable at compile time, only at runtime. BTW, there are equivalent nasties in .NET [datasets], which the MS guys are all too happy to tell everyone to use. I don't think it's as simple as static vs dynamic typing, more of a compile time vs runtime type discovery thing.

  3. It looks like an array, and if it were encoded as an array, most toolkits would be much happier.

This got me to wondering how the WSDL-generating toolkits at my disposal, VS.NET and GLUE, would encode a hashtable. I started with VS.NET but ran into a wall. It simply refuses to serialize a hashtable:

Cannot serialize .NET Hashtable:

System.InvalidOperationException: Method subsService.subs can not be reflected. ---> System.InvalidOperationException: There was an error reflecting 'subsResult'. ---> System.NotSupportedException: The type System.Collections.Hashtable is not supported because it implements IDictionary.

Dang. I was sort of looking forward to be able to pass hashtables around. This was unexpected and disappointing.

Next I tried GLUE. Here's the WSDL it produces for a java.lang.Hashtable: 

<types>
<schema xmlns="http://www.w3.org/2001/XMLSchema" xmlns:tns="http://xml.apache.org/xml-soap" 
  targetNamespace="http://xml.apache.org/xml-soap">
<complexType name="Map">
<sequence>
<element name="item" minOccurs="0" maxOccurs="unbounded">
<complexType>
<sequence>
<element name="key" type="anyType" /> 
<element name="value" type="anyType" /> 
</sequence>
</complexType>
</element>
</sequence>
</complexType>
</schema>
</types>

This, I learned, is known as the Apache Map serialization, because it's supported in the Apache SOAP stack. It's also supported in two of the SOAP toolkits I was using: SOAP::Lite and GLUE. This means, I was able to verify, that you can pass a java.util.Hashtable from GLUE and receive it via SOAP::Lite as a Perl hashtable. However, VS.NET doesn't like this format, so I couldn't consume hashtable-returning services by way of Apache-Map-style WSDL.

Interop isn't "yes" or "no," it's "how well?"

I should clarify, though. In truth I was never completely unable to consume services from VS.NET by way of WSDL. Interoperability isn't always binary; often there are shades of gray. When the producer and consumer cannot agree, in WSDL terms, on some rich type definition, they can punt and change from: 

  <message name="subsResponse">
    <part name="titlesOrURLs" type="s0:ArrayOfChannelStructs"/>
  </message>

to: 

  <message name="subsResponse">
    <part name="titlesOrURLs" type="soap:anyType"/>
  </message>

This results in partial functionality. You can do an Add Web Reference in VS.NET, and can discover the name of the service, the names of its arguments, and their types (if simple). The result will simply be an object. If you poke at it, you'll find it to be an array of System.XML.XMLNode. This is only slightly better than our original tableToXML example. It's not useless, but it's not wonderful either.

Here's what I think this boils down to. In XML-RPC, the kinds of structures that scripting language like to work with are beautifully and simply described in the spec:

Here's an example of a two-element <struct>: 

<struct>
   <member>
      <name>lowerBound</name>
      <value><i4>18</i4></value>
      </member>
   <member>
      <name>upperBound</name>
      <value><i4>139</i4></value>
      </member>
   </struct>

<struct>s can be recursive, any <value> may contain a <struct> or any other type, including an <array>.

But if you can't count on passing around lists-of-lists and hashtables-of-lists-of-hashtables -- which at this point you cannot -- then there's a problem. De facto solutions like the Apache Map format can help, as we've seen. Both SOAP and WSDL are evolving, and we can hope they will nail down these conventions. If not, Simon Fell suggests, one of two things will happen: a move to XML-RPC, or a move to "the doc/literal style of SOAP, which promotes XML on the wire to a first class citizen, and ditches the concept of language-to-XML mapping." Given the range of business interests and programming models stirring the pot, I don't know which of these outcomes I'd bet on. But one way or the other, let's get it settled sooner rather than later. What we need is working applications, not cosmic architectures.

Acknowledgements: Shane Caraveo, Simon Fell, Tony Hong, Sam Ruby, Rich Salz, Jake Savin.

Jon Udell (http://udell.roninhouse.com/) was BYTE Magazine's executive editor for new media, the architect of the original www.byte.com, and author of BYTE's Web Project column. He is the author of Practical Internet Groupware, from O'Reilly and Associates. Jon now works as an independent Web/Internet consultant. His recent BYTE.com columns are archived at http://www.byte.com/tangled/.

Creative Commons License
This work is licensed under a Creative Commons License.