RESTfmSync makes heavy use of the new in v12 “Insert From URL” script step to send data to the FileMaker Server. This works very well, but has a few limitations. Seeing as we’ve been through all of the testing and trials to work out exactly what those limitations are, we thought it worth documenting them for others using this step in their solutions.
GET vs POST
First of all, the Insert From URL ( hereafter referred to as IFU ) step only supports a GET operation. It can’t do PUT, POST or DELETE. So some interactions with web servers are not possible. In fact most web APIs utilise much more than just a GET, and so for lots of things Web Service related, IFU just isn’t enough.
We have the advantage with RESTfmSync of controlling both the client ( FileMaker Pro on the desktop or FileMaker Go on iOS ) and the server ( RESTfm via Custom Web Publishing in FileMaker Server ). So we’ve built into RESTfm the ability to override the method. This means we can send a GET request but have RESTfm treat it like any a different request. This is done via an extra parameter
RFMmethod=POST
When you send this parameter in your URL you can do a POST via a GET from the Insert From URL step.
GET has length issues
GET also has limitations in terms of the length of the url string that it will accept. Unfortunately this varies with both the web server and web client, so there’s no real way of knowing what it will be in advance and you need to limit your data length to the worst case option. In RESTfmSync we’ve chosen a 1950 character limit as we found that some values of 2000 characters or more would fail in some setups.
Again we can work around this in RESTfm by appending data and sending multiple requests to the server. Using the
RFMappend
parameter, you can elect to have the data sent be appended onto the end of the fields instead of replacing them.
Insert From URL likes to mess with your data
However the most frustrating limitations in IFU are that it modifies some data in an effort to help you, but that only really hinders things. It has some encoding built into the step that modifies the data in your url before it’s sent. The idea seems to be that it auto encodes things for you, meaning you don’t have to encode it yourself. However it also means it encodes the wrong things.
When the WebViewer was introduced, it also had this same behaviour, and in FileMaker 11 and later, there was an optional “Automatically Encode URL” checkbox introduced that defaults to on.
The help describes this as :
Select Automatically encode URL to allow FileMaker Pro to apply encoding rules to the URL, if necessary, so that it complies with a browser’s required format. To keep the URL in the format in which it is entered, deselect this checkbox.
And it goes on to further explain :
- The following characters are never automatically encoded: ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz 0123456789-_.~!*'();:@=+$,/?[]
- An ampersand (&) is encoded only if a space follows it. For example, “& “is encoded, but “&x” is not.
- A pound sign (#) is encoded only if a number character (0 through 9) follows it.
- The backslash (\) and percent (%) characters are always encoded.
- All other characters are always encoded.
However, none of this mentions the IFU script step, just the WebViewer. But it’s clear there is some encoding being done to the your data in IFU as well.
For example, if you want to send to a url a new field value of “abc&def” for the field called “MyData” you would send the following :
MyData=abc&def
But you can’t send the & as part of the data, the & separates field values, so it needs to be encoded. The encoded value for & is %26. FileMaker Pro will even do this for you via GetAsURLEncoded ( “abc&def” ). So you actually need to send :
MyData=abc%26def
But if you’re paying attention to the encoding rules above, FileMaker decides that the % needs to be encoded as it’s URLEncoded value and so even though you’re sending %26 it converts this silently behind the scenes and sends:
MyData=abc%2526def
And your MyData field ends up with the value of “abc%26def”.
In most situations this is completely unhelpful as it makes it impossible to distinguish between text that is an encoded ampersand and text that is actually %26 and is meant to show that. And there’s no way to turn this off.
RESTfm to the rescue again
We’re fortunate in RESTfm to be able to work around this because of our ability to control client and server. Inside RESTfmSync, we use a Custom Function that does the following:
GetAsURLEncoded ( Substitute ( text ;
[ "%" ; "%25" ] ;
[ "!" ; "%21" ] ;
[ "#" ; "%23" ] ;
[ "$" ; "%24" ] ;
[ "&" ; "%26" ] ;
[ "'" ; "%27" ] ;
[ "(" ; "%28" ] ;
[ ")" ; "%29" ] ;
[ "*" ; "%2A" ] ;
[ "+" ; "%2B" ] ;
[ "," ; "%2C" ] ;
[ "/" ; "%2F" ] ;
[ ":" ; "%3A" ] ;
[ ";" ; "%3B" ] ;
[ "=" ; "%3D" ] ;
[ "?" ; "%3F" ] ;
[ "@" ; "%40" ] ;
[ "[" ; "%5B" ] ;
[ "]" ; "%5D" ] ) )
This way at the other end we can undo that extra encoding, and then url un-encode and get back the original data. We detect this whenever the parameter RFMfixFM01 is included. If it’s not included the it is assumed that it’s a normal encoding.
Issues
The obvious issue with this is in situations outside of a managed client/server environment, such as external public APIs. You have no control over the server and so some data just cannot be sent accurately.
For RESTfmSync, we can work around the issue, but it means that one character ( & ) is turned into five ( %2526 ) and so makes it much more likely that we’ll run into the URL length limitations.
Alternatives
The advantage of the IFU step is that it can happen in the background on a layout where the field being referenced is off layout and not seen by the user. Our alternative option in RESTfmSync is to use the WebViewer to do the POST instead. This works, and has none of the limitations of the length of the url, or strange encoding issues. It does however force us to have the WebViewer visible to the user, and also requires us to have a scripted callback mechanism that the Wv calls to complete the process.
All of this is not as pleasant in terms of user interface but in the case of very large data sets can be much faster overall.
Conclusions
It would be really nice to see FileMaker build in native functionality for the sorts of HTTP calls that we can do in the BaseElements plugin.
Our hope with the plugin is that it is made obsolete by advances in FileMaker itself. FileMaker has already obsoleted two of our functions, which were the SQL calls to grab Table and Field structure directly from the file. Being able to act as a full Web Services client would be a great next step on that path.
In the short term a “Automatically Encode URL” checkbox for the Insert From URL step would be a great help.