Yesterday I updated my list of new FileMaker features for version 11. The list I maintain doesn't include everything (like the new Quick Start Screen, or text highlighting), but includes all of the items that affect our ability to program in FileMaker. So things like new script steps or changes to calculations.

The reason for doing this yesterday was that I was building into BaseElements the ability to recognise which calculation functions and script steps came from which versions of FileMaker. Using this list you can then say that to open this file, you need to use at least a minimum of FMP version X. There is a little more to do in BaseElements though to make it comprehensive, like extra features within certain script steps or layout features like web viewers or filtered portals. However, the functions area is more technical and is built into the XSLT in BaseElements, so I wanted to get that coding done first.

But I stumbled across an interesting bug I'd never noticed before in the FileMaker DDR. To test the various functions, including the Get functions, I built a file with a single calculation that included every possible built in function. Then I ran the DDR for this file and had a look at the details. To understand how the DDR works, when you have a long calculation, FileMaker breaks it into chunks. There are chunks for Fields, Custom Functions, Functions and everything else is a "NoRef" chunk (in other words text).

All of the built in FileMaker functions were reported, correctly, as "FunctionRef" nodes. I had thought that the most likely answer for the collection of the "Get" functions, was that the Get part would be a FunctionRef and the rest would be NoRef. Because in reality, the parameter you pass to the Get function is just a reserved word and the actual function is called simply "Get". I was hoping that it might distinguish the reserved words in some way so I could figure out which ones it used, but I wasn't sure what it would do.

It turns out in FMP 10, and I presume earlier versions, FileMaker reports some of the parameters as "FunctionRef" chunks and ignores some of them, so they end up in the text along with the round brackets and any whitespace and also anything that follows the function. I wasn't sure what the logic for this was, there didn't seem to be any consistency to it. And even stranger it reported one Get reserved word as a Custom Function.

I was all set to report a bug to FMI when I realised I had generated the DDR with FMP 10 and realised I should run the same test with 11. Well in 11, every Get reserved word is reported as a FunctionRef, so I can accurately report on them in BaseElements. So no bug report is in order.

What this does mean is that there could also be other differences and inaccuracies with the DDR in older versions that are fixed in 11, and I'd suggest that if you're running any tool like BaseElements where you utilise the DDR that you should run it from version 11, just to be sure.

I do have a big feature request for the DDR though : That it broke the calculation into more discreet chunks. For example, plugin functions are reported as "FunctionRef" along side every built in function, it would be great to have a "PluginRef" type by itself. Comments and quoted text aren't put into anything separate, it would be handy to have them put into a "TextRef" or CommentRef section. And technically the Get function types aren't really functions, they're reserved words, so a "ReservedWordRef" would be nice. Maybe in v12 :)

I haven't seen a notice yet about this on FileMaker's site, but I found FileMaker Go for iPhone and iPad in the iTunes store this morning. And what else do I do but test BaseElements 3 on it. First impressions are that it looks good.

There are some font issues and some other formatting strangeness, but it does load and I can click around on things. I did also get some dialogs to display which looked neat.

More to the point though is what we can do with this for clients and remote access to solutions. There is a lot to like, even if this is just the beginning. There is going to be a bit to catch up on about the differences between a full FMP app and the iPhone/iPad app, I'm looking forward to that. I've already got staff saying they need an iPad for testing now :)

I've uploaded another beta version of BaseElements 3. I was hoping to have a windows build available as well this time, but it's not to be. There are a whole bunch of path related issues inside the xslt that are causing me grief so I've got to sort out those before it's ready for prime time. In the mean time for those on the mac :

BaseElements 3 Beta 2

Changes since b1

• Fixed display issues on the Variables and File layouts - thanks Steve Wright.
• Fixed an issues with File References used on the right side of Relationships.
• Fixed an issue where processing of items wasn't always happening.
• Fixed an issue where the right field in Join Predicates was missing.
• Fixed an issue where quotes in comments would cause problems - thanks Doug Rowe.
• Fixed a bunch of issues related to the processing of files and file paths.
• Sorted out some privilege related issues that stopped you doing reports and importing more than one DDR - thanks Gordon Cox and others :)
• Added some processing for TO Groups back in.

The links you need are :

BaseElements_Mac.dmg - fp7 files.
BaseElements_Mac_MU.dmg - Sharing enabled version.
BaseElements_Mac_Runtime.dmg - Runtime.

Also this has the xslt code built in for the plugins and variables although only variables is working at the moment. This means I'm not adding any more processing steps to the import steps. This is as detailed as it's going to be. I'd love to hear from you about any speed differences between this version and the same DDR under BaseElements 2.x as I don't expect it to change much at all between now and final.

As always send me any feedback or issues you find : or on twitter or our contact form.

Following on from part 1.

The second big change to the XSLT engine in BaseElements 3 is that we consolidated a lot of areas in the system. All of the old field, Custom Function, TO, ValueList, Variable and Plugin reference tables were all joined into a single "References" table. Although having one base table instead of 5 doesn't make things much faster inside BE, it meant we were able to do a single xslt run instead of five.

For example we're traversing the entire XML tree to find all the calculations. We do this again to find all of the references within the calculations. But if you imagine that a calculation can reference fields, CFs, TOs, variables and plugins, then repeating that traversal five times for each one is time consuming. So by putting the references into a single table we can do that just once.

But that's not all, we've brought into the xslt code some work that was previously being done as processing steps in BaseElements. If you've watched a BaseElements import, you will have seen some steps where it mentions "Processing Calculations". This was doing a few things : Digesting the calculation steps and producing plugin references and variable items, and also sorting out Indirection steps with function calls. It's a time consuming step in BaseElements 2, and can be up to a quarter of the total processing time. To give you an idea of the work being done, here is a screenshot of the code from BE2 :

In that there are lots of recursive custom functions ( great for simplifying code, not so great for massive data processing ), and lots of loops that run off and generate new records.

In BaseElements 3 this entire block of code is gone. In it's place we've built in the same functions into the References xslt file. I'm going to digress into some detailed xslt so if you're not interested, feel free to skim the technical steps or tune out :)

The first thing we needed to do was to be able to differentiate plugins from normal functions. FileMaker helpfully puts function calls in a separate part of the DDR, so we can track those, but it doesn't differentiate between plugins and other functions. To do this in xslt you build a lookup table reference :

<xsl:key name="plugin-lookup" match="PluginList/Plugin/Function" use="@name"/>

then you setup a variable to reference the external file :

<xsl:variable name="plugin-file" select="document('BE_Plugins.xml')" />

And finally when you've got a function call to lookup you look it up in your external table :


<xsl:for-each select="$plugin-file">
<xsl:if test="key('plugin-lookup', $FunctionName)">
<xsl:call-template name="RESULTSET"/>
</xsl:if>
</xsl:for-each>


We've used these key lookups in a few places other places as well and it makes a huge difference to the processing time.

The second part of that long script was the processing of variables used. FileMaker doesn't even distinguish variables from other parts of the text, and so they're just lumped in with text, math symbols and other whitespace as a "NoRef" type. As well as that, variables are now also in the text on layout objects, inside file references, perform find steps and set variable steps. Each one is slightly different, but essentially the process is the same for all of them : sort through the text and find the variables. This is actually harder than it seems. First, you don't want to find variables inside comments, so you need to remove those. Then you need to remove anything inside quote marks, but you need to first strip out any escaped quotes. Then you need to split the remaining text into dividers. Because a variable name can have spaces, the only way to tell when it ends is to just find the next non-variable name character after the $. This could be any of of the various math symbols or other characters that end a text block. So finally you have a list of the remaining text, and you convert those to nodes and loop through them to find any that start with a $.

This process is more complex to setup inside xsl than it is to code in FileMaker, but the end result is that the process runs much faster because you're not looping through records and generating multiple new variable reference records. The entire process fits inside the main reference xslt and is included in the single import that imports those.

I won't post the code here, but if anyone is interested in how it works, let me know and I'll document it in more detail.

It's these two changes, as well as the csv imports, that constitute the bulk of the reduced import time in BaseElements 3. It's funny, I thought I'd done a good job of making things faster in BE 2, and I wasn't sure where any extra savings would come from. It was only after the v11 update broke our old xsl that I went back to the drawing board and found other optimisations with the new xsl engine. So maybe if v11 still worked with the old code BE 3 might not have been as fast as it is...