ColdFusion

Digg This!

For those not familiar with web services, they are a way for servers to exchange messages with each other using XML standards. Despite the extreme hype, web services do not enable you to do anything that was not previously possible. Since the earliest versions of ColdFusion, CF developers have used the CFHTTP tag to post an HTTP request to another server and then analyzed the response. Submitting (or receiving) a request via web services simply does the same thing, albeit using XML standards.

CFMX and Web Services
With ColdFusion MX, Macromedia made implementing web services very easy. (Although CFMX 7 was recently released, the web services implementation has no major changes or enhancements.) To receive a web services request, simply create a ColdFusion Component (CFC) and set the "access" value to "remote" for those functions which should be accessible as a web service.

One benefit of web services is that they are "self-describing," meaning the web service itself provides the documentation that tells others systems (or people) the inputs (name and field type) and output field type for that web services function. This documentation is Web Services Definition Language, or WSDL. To generate the WSDL file for each "access=remote" function in a ColdFusion Component, simply call that component via your browser followed by "?wsdl". For instance:

http://localhost/YourComponent.cfc?wsdl

The resulting file is a WSDL document, which is just an XML document.

To submit a web service via ColdFusion, you can use the CFINVOKE tag just as if you were submitting a request to a normal CFC function. Other methods of invoking functions will work as well, such as within CFSCRIPT. It doesn't matter whether the web service request is sent to a server running ColdFusion, Java, .NET, PHP, and so on.

How to Web Services-Enable Applications: A Case Study
So implementing web services functions in ColdFusion is simple. But that's very different from web services-enabling your application. This requires more than just creating some CFC functions set to "access=remote." It requires authenticating the user submitting the request, authorizing they have permission for that function, and validating that they own each target of that request.

For instance, if the function is to update a product, you first authenticate the user, i.e., whether they are logged in. If the user isn't logged in or their session has expired, they must first log in. Once you have determined their identity, you next authorize whether they have permission to update products. This is based on how you implement permissions, which can be via roles, individual permissions, and so on. Finally, you must verify whether the user owns the product he/she is trying to update, i.e., that the product belongs to that user and not another user (not to mention that the product itself exists).

Of course, you also need to validate the updated values just as you would when submitting via a normal browser-based form. And if there are any errors, just as you display error messages to the user, you need to tell the user submitting the web service that their update request was not successful.

What follows is a case study on web services-enabling Averum Billing. It will hopefully provide MXDJ readers with an overview of the various issues we encountered. Some of these problems are specific to ColdFusion while others are generic problems that apply regardless of your development platform.

There are 10 primary issues, which I list below and will discuss:

1. File size limit
2. Session management
3. Redundant CFCs
4. Application scoped CFCs
5. Custom IDs vs. internal IDs
6. All fields are required
7. Returning queries
8. Boolean fields
9. Error messages
10. Debugging

1.  File Size Limit
For developers using Fusebox or other methodologies where all pages are called via index.cfm or some other file, it's a natural preference to prefer that all web services requests use the same ColdFusion Component as well. Unfortunately, that just ain't gonna happen. Java has a 64k file size limit, which can quickly be reached with a few functions and their arguments.

You can try moving all logic outside of the component (which is recommended anyway) but you will still hit the limit eventually. Plus you will also find that the CFRETURN tag must be in the CFC itself and not an included file, so you will need a variable to store the returned value.

Bottom line: if you are truly web services-enabling your application, you must give up the dream of using the same CFC for all web services functions. You'll need to use multiple CFCs. So you may as well separate the functions into logical groupings ­ CFC1 and CFC2 are not exactly intuitive.

2.  Session Management
If your application requires users to log in, they'll need to do so before accessing any data via web services as well. Despite ColdFusion's documentation, managing sessions in web services is not simply a matter of using CFAPPLICATION or cookies. Normal browser-based sessions based on cookies do not work. A more creative solution is required.

It's possible that including CFID and CFTOKEN in the URL when calling the web service component would work, but we didn't try this. It's not a standard practice in web services for the URL to invoke a web service to vary. That's why you have input variables. Moreover, the default setting for a browser-based ColdFusion session to timeout is 20 minutes. However, our clients preferred a longer timeout period. For Averum Billing, we chose 2 hours, which would have been dangerous if storing those sessions in memory.

As usual, the best way to resolve new issues is to see how other sites have dealt with them. After reviewing the web services implementations of several other popular sites, we choose to use a UUID variable as the session ID, which can generally be assumed to be unique.

After logging in with a username and password (and in our case, a company name as well), Averum Billing's web services login function returns a UUID, which must then be submitted with each subsequent web services request. This authenticates the user so that the username and password do not need to be submitted in each request.

For additional security, Averum Billing tracks the IP address used when logging in. All subsequent web services requests for that UUID must originate from the same IP address. Otherwise the request is rejected because the user isn't logged in.

Like any session, there is a need to store "session" variables. However, because we cannot store normal session variables (between requests), we technically do not have a session and cannot store variables in the Session scope. Instead, to simulate session variables, we chose to add a WebServiceSession table to our database that tracks the necessary session information.

While this limits the number and type of session variables we can store, it was the only scalable solution. We initially implemented sessions using an application-scoped structure where the structure index was the UUID of the web services session. The resulting value would also be a structure to store the necessary session variables. For instance:

<CFSET Application.webServiceSession[theUUID].userID = "1">

However, this method has an inherent scalability problem. Adding a new session or updating an existing session required updating an application-scoped variable, which in turn requires using CFLOCK to ensure only one process is updating the variable at a time. If you expect more than a few customers to be accessing the web services interface at the same time, the constant locking of the application-scoped variable becomes a major bottleneck.

Finally, like any session, it must end at some point either by logging out or timing out. While we chose not to build a "logout" web services function, a web services session can time out due to inactivity, just like a browser session. However, whereas CFAPPLICATION does this for you automatically, we had to implement this capability ourselves.

To do this, we created a "session" variable that stores the last date/time the "user" submitted a web services request. This is used to determine whether the web services session should be "timed out" due to inactivity, which is checked with each request. We chose to time out sessions after 2 hours of inactivity. With each web services request, we first validate the UUID to ensure it is an active session and, if so, then compare the current date/time with the last request to determine whether the session has "timed out." To actually delete the timed-out sessions, we set up a scheduled script that deletes all web services "sessions", i.e., the rows in the database where the last request was more than 2 hours ago.

3.  Redundant CFCs
While some developers prefer to store all components in a single directory, we chose to store each component in the same directory with its associated functionality. But this would require that customers refer to the web services components in a whole bunch of different directories. Of course, it would be more convenient if all components accessed via web services were in the same directory.

While it initially seems ludicrous to have redundant components, it was actually necessary anyway. The web services components have the same name, but with a "WS" prefix. Most web services functions in Averum Billing have the same function (method) name as their associated non-web services function.

For many reasons, the normal CFC function and the web services version of that function accept a different list of arguments. We will go into more detail later about some of the reasons for this, including custom IDs, searching, updating, and boolean fields. Of course, the most obvious example is the UUID that must be passed in to validate the session.

Another simple reason is that Averum Billing contains fields that are used internally, but which are not specified directly by the client. In general, our client is not even aware the fields exist. These fields exist as arguments in the normal CFC function, but not in the web services CFC function. While Java does allow you to have multiple functions with the same name but with different arguments, ColdFusion does not.

4.  Application Scoped CFCs
Once we created our first web services component, a funny thing happened when we tested it. For simplicity, we used CFINVOKE to call the component just as we would any other. We assumed there was no difference when calling the function as a component versus calling it as a web service. In other words, we figured "<CFINVOKE Component=" worked the same as "<CFINVOKE WebsService=".

As you have probably guessed by now, this was not the case. Many of our components access functions in other components. In those instances, we simply used "CFINVOKE Component=" to access the other component. When calling a component that is not in the same directory, the "Component=" value must start from the root public directory of your site, which requires setting up a mapping in the ColdFusion Administrator. This is fairly annoying, and we would have preferred that CFMX be more intelligent in this respect rather than implementing this in a normal Java fashion.

This method of calling a function in another component works fine if the component is called via "<CFINVOKE Component=" but, for some strange reason, doesn't work when called as a web service. Yes, it's the exact same function that is being called, but CF is evidently a bit more cranky when called as a web service. So we found that the only way to access a function in another component when called via a web service is to store that component in the Application scope.

It's a good idea to store commonly-used components in the Application scope for performance reasons. So we were considering doing this anyway, at least for the more heavily-used components. But the web services requirement did not give us much choice in the matter.

The only downside of storing components in the application scope is that we then had to remember to "update" the component value in the application scope whenever that CFC file is updated. To do this, we added a simple link in our admin interface that can reset these values when necessary.

The components that are called as web services are not stored in the application scope though because they are never called by other components. They are only called directly by clients when accessing a web services function. So storing them in the application scope provides no benefit.

On a random note, you will also find that if you update a web services CFC, ColdFusion will not recognize the changes in the updated CFC until you restart ColdFusion. It is possible it may recognize the changes after a certain amount of time, but we have not been willing to wait around long enough to find out.

We also use several functions that are written in CFSCRIPT. Many of these functions are also stored in the application scope since ColdFusion gets cranky about accessing functions via cached components. Before including the ColdFusion file that defines the function, you must first check whether the function already exists. If it does exist and you try to define a function with the same name (even if it's the same function) ColdFusion returns an error. Doing a check in the file that defines the function is too late ­ ColdFusion returns an error even if the CFSCRIPT is within a CFIF statement that says to only execute the CFSCRIPT if the function does not exist.

A final note about storing components in the application scope (or any persistent scope actually). Any variables stored using the "Variables" scope in a persistent component are stored persistently as well. While the ColdFusion documentation does warn you about this, it did not occur to us. As you know, you must use the "var" command when creating a temporary variable in a component. The complication is when a web service mirrors the functionality of a non-component based browser feature, where a persistent Variables scope is not a problem. So while we initially thought it was good habit to use the Variables scope instead of leaving a variable un-scoped, it came back to haunt us as we had to de-scope many variables that were accessed via a web service. Plus we had to use "var" in our web services CFCs to create a temporary variable.

5.  Custom IDs vs. Internal IDs
Now that we have dealt with the basics of implementing web services, we can deal with the specifics of the actual web services functions and how your customers will use them. Averum Billing has a custom ID field for each item so that our clients can specify their own primary key values rather than using the primary key assigned automatically by Averum Billing.

For instance, when a new product is created, the productID is assigned automatically by the database. This number generally has no meaning to our client since they have their own productID already for that product. By using the custom ID, a field we creatively named productID_custom, they can refer to the product in Averum Billing using the same ID they use in their own system. Whereas the productID is an integer field, the productID_custom is a varchar (text string) field so that clients can enter any value they want.

When specifying the productID via web services, our client can specify either our internal productID or their productID_custom. But the web services function must know which ID they are specifying. There are several ways this can be achieved:
a.  Having both productID and productID_custom fields as arguments in the web services function, where productID is numeric and productID_custom is string. Then if the productID_custom is not blank and productID is 0, this means they are using the productID_custom value. However, this method is not explicit and the client could accidentally specify values for both fields, in which case it is unclear which field to use. Plus, in some functions, it may be possible to specify multiple productID's in a comma-delimited list, or multiple productID_custom's.
b.  To ensure it is explicit whether to use the productID or productID_custom, instead of having both productID and productID_custom fields, we can instead have a single productID field and then add an argument where the client can specify which version to use. We named this field "useCustomIDFieldList". The default is to use the productID. To specify that the productID_custom should be used instead, just enter "productID" for the "useCustomIDFieldList" value. If there are other arguments in the function for which the custom ID may be used, just enter that field name in useCustomIDFieldList, which is a comma-delimited list of fields for which the custom ID should be used. This method works great, except that it still is not explicit whether to use the productID or productID_custom field if there is only one. Plus if the productID argument is a string, that violates the web services concept of a self-defining function.
c.  The final method is to use the useCustomIDFieldList argument and also have both productID and productID_custom arguments. This is the most explicit method and maintains the field type integrity (except where the productID may be a comma-delimited list, in which case it must be of type string instead of numeric).

6.  All Fields Are Required
One of the annoying things about web services is that all arguments are required, whether you need to specify a value for that argument or not. Unlike accessing normal functions in ColdFusion, the Required clause in the CFARGUMENT tag is ignored for web services. All arguments are required. Period.

While this does not seem like a big deal, it is actually quite annoying, especially if the web services function is for searching or updating. When searching, there are dozens of search criteria you might use to filter the results. Because all arguments are required though, you need a way to specify which arguments should be used for searching. In Averum Billing, we have solved this problem with an argument named "searchFieldList". This is a comma-delimited list of the arguments to use for searching.

Similarly, when updating an item, for instance a product, you may want to update the product name but not the price. Again, you do not want to be forced to update all product fields if you only want to update the name. That would require you to know the current value of the product fields you do not want to update. But we certainly do not want to create a separate web services function for each product field so you can only update that field. Instead, we added an argument named "updateFieldList" which is a comma-delimited list of the fields to update.

A somewhat-related side effect of the required field problem is Averum Billing's support for custom fields. Averum Billing allows clients to create their own custom fields for users, companies, products, etc. But it is not possible to submit arguments to web services that are not listed in the function. Java may support such "overloading" but ColdFusion does not. And it certainly violates the self-describing spirit of web services standards.

For customers to specify custom fields via web services, we instead have an argument named "customField" where they can enter the custom field values in an XML string where <customField> is the main tag and the individual custom field tags are the custom fields names as defined when they created the custom field.

7.  Returning Queries
Like all functions in CFCs, web services functions specify the field type that is returned. When the return type is a query, you need to consider that queries are treated differently in ColdFusion than in other languages. They are generally treated as a dual-index array whereas ColdFusion essentially treats queries as a structure/array combination. For instance, to access a particular row in a ColdFusion query, you can use:

queryName.fieldName[row]

But when the query is returned via web services to another language, instead of "fieldName" being the index, the index is a number. But the number must correspond to the field name, and the client requesting the query must know which index number corresponds to the field name. Fortunately, ColdFusion seems to be consistent in how the field names are ordered ­ they match the order in which the fields are listed in the SELECT clause of the query.

In your documentation, simply list the returned fields in the order in which they are selected. This provides a minor issue if the query returns internal fields that are not listed in the web services documentation. You cannot hide this fact since it is easy enough to determine the array length. So you either have to specify that the query not return these internal fields when the query is requested for web services, or just have some fun and let your client guess what the unlisted field is.

8.  Boolean Fields
Boolean fields are generally specified as 0/1 or True/False. They presented a minor challenge in our web services interface. Internally, our components used numeric field types for Boolean fields that are stored as bit fields in the database. We could have specified a numeric field type for our web services as well, but in keeping with the self-describing nature of web services, we preferred the argument type to accurately reflect the data type.

This presented two issues though. First, since our internal components and code use the numeric value, we created a simple function to convert the Boolean value to a 0 or 1. Second, there are instances where the Boolean value may be null, e.g., a nullable bit field. Well, it seems that null is not a valid Boolean value and the web service will return an error if you submit a blank value for a Boolean field. This is very annoying. In these instances, the data type for the "Boolean" argument must instead be a string, which can be blank.

On a side note, the MySQL database does not seem to support bit fields. So we used a tinyint field instead. The alternative was to use a varchar field of size 1, but we preferred to stick with a numeric field.

On a complete tangent, we also learned the hard way that in ColdFusion MX, if your query returns a bit field, ColdFusion will display the value as a 0 or 1. However, if you use ColdFusion's ValueList function to return a list of all values in the query for a bit field, for some annoying reason, ColdFusion converts the 0's to False's and the 1's to True's.

9.  Error Messages
When a user submits a form, you validate the field values and, if there are any mistakes, you display the error messages and re-display the form. This is slightly more difficult in web services, which are generally submitted by a server and thus no one is there to read the error messages. More importantly, each web service function must specify the data type of the return value. If the function returns a numeric value or a query, clearly you cannot return an error message as a text string.

In .NET, I believe it is possible to create an exception where the returned value is not of the type specified. But trying to do this in ColdFusion will cause an exception error. (Unfortunately, ColdFusion also returns an error if you call a web services function without all of the arguments or if the argument value does not match the field type. This is incredibly annoying because there is no way to catch this error via CFTRY.)

There are two issues regarding how to handle error messages:
1) how to tell the client submitting the request that there was a problem with their request, but in a way that their server can understand in an automated manner;
2) when clients are initially setting up the web services on their end, how to let them access the error messages so they know why the request was unsuccessful.

For the server, our documentation lists the value returned if the request is not successful. For a numeric return type, we return ­1. For a string, we return a blank value. For a date field, we return January 1, 1970 at 12:00 AM. For a query, we return a blank query with a single field named "error". (If the user does not have permission, we do not want to return a blank query with the actual field names. Granted they can get this from our documentation, but we still need to differentiate between an error and a blank results set.)

Now that the server knows the request was rejected, the programmer might want to know why. In the "session" variables for each web services login session, Averum Billing stores the last error message generated for that session. We could have stored the last error message generated for each web services function or perhaps the last 10 error messages, but we determined that would be a waste of storage.

There are many types of error messages, including:

• The user's web services session has timed out (or never existed)
• The user does not have permission for the requested function
• The client does not have permission for the requested item, e.g., the product may belong to a different Averum Billing client.
• Form validation issues

10.  Debugging
Finally, on the topic of error messages, we should warn you that debugging your web services is more complicated than debugging normal ColdFusion code. The error messages generated by ColdFusion are often useless and misleading. Checking the ColdFusion error log may help, but it is generally the same non-descriptive error message.

There are several ways to debug your web services code. One way is to call the function as a normal component instead of a web service, but that's not always useful if the error is web services-specific.

Another method is to use CFTRY within the code where the CFCATCH clause includes a CFMAIL tag to email the error message to you or a CFFILE tag to write/append the error message to a text file. Of course, our experience seems to suggest that ColdFusion ignores the CFCATCH code unless it is within a file that is not included directly by the CFC itself, i.e., it must be at least 2 files down from the component.

Finally, like other code, you can walk through the code and comment out parts that may be causing errors, and then gradually un-comment the code to find the error. When doing this, just make sure there's still a return value specified for the function.

If you're truly stuck and are totally frustrated, we recommend inserting a bunch of CFFILE tags that append a number or other text to a file, where each successive CFFILE increments the number. This is the quickest way to get feedback about where the error is occuring. Unfortunately, this method does not work well if the error is in the CFC itself since ColdFusion evidently caches any functions with remote access and updates its cache only when ColdFusion is restarted. (This was mentioned in #4 above.)

One interesting fact we learned the hard way is about the function QuerySetCell. Even if setting the value to an integer, when listing the values in a query using ValueList, ColdFusion returns the value with a single decimal point. The only way to resolve this was to set the value using the ToString function around the actual integer value. ValueList also converts bit fields from 0/1 to True/False.

We also ran into a problem with a query that was looped thru via CFLOOP Query="". The variables within the query were properly-scoped using "queryName.fieldName", but the query was returning junk data. Using ValueList confirmed that the query was correct. Oddly, the solution was to remove the "queryName." in the CFLOOP.

Finally, for temporary values that will be used in your function, you should create the variable using "<CFSET var >". However, this does not apply to values that will be returned via a CFINVOKE tag when accessing another function. The "var" value will actually take precedence over the returned value, including for queries.

Summary and Disclaimer
This article was intended to help ColdFusion developers web-services enable their CFMX application. It recounts the issues encountered by Averum, Inc. while web services-enabling Averum Billing, Averum's billing system for Application Service Providers (ASPs).

We do not suggest the methods used by Averum are the only way or best way to implement web services throughout your application. However, the primary goal of this article was to make you aware of the various issues we encountered, share our solutions, and stimulate debate among the ColdFusion community about best practices for web services-enabling your application.