I've uploaded another beta version of BaseElements 3. I was hoping to have a windows build available as well this time, but it's not to be. There are a whole bunch of path related issues inside the xslt that are causing me grief so I've got to sort out those before it's ready for prime time. In the mean time for those on the mac :

BaseElements 3 Beta 2

Changes since b1

• Fixed display issues on the Variables and File layouts - thanks Steve Wright.
• Fixed an issues with File References used on the right side of Relationships.
• Fixed an issue where processing of items wasn't always happening.
• Fixed an issue where the right field in Join Predicates was missing.
• Fixed an issue where quotes in comments would cause problems - thanks Doug Rowe.
• Fixed a bunch of issues related to the processing of files and file paths.
• Sorted out some privilege related issues that stopped you doing reports and importing more than one DDR - thanks Gordon Cox and others :)
• Added some processing for TO Groups back in.

The links you need are :

BaseElements_Mac.dmg - fp7 files.
BaseElements_Mac_MU.dmg - Sharing enabled version.
BaseElements_Mac_Runtime.dmg - Runtime.

Also this has the xslt code built in for the plugins and variables although only variables is working at the moment. This means I'm not adding any more processing steps to the import steps. This is as detailed as it's going to be. I'd love to hear from you about any speed differences between this version and the same DDR under BaseElements 2.x as I don't expect it to change much at all between now and final.

As always send me any feedback or issues you find : or on twitter or our contact form.

Following on from part 1.

The second big change to the XSLT engine in BaseElements 3 is that we consolidated a lot of areas in the system. All of the old field, Custom Function, TO, ValueList, Variable and Plugin reference tables were all joined into a single "References" table. Although having one base table instead of 5 doesn't make things much faster inside BE, it meant we were able to do a single xslt run instead of five.

For example we're traversing the entire XML tree to find all the calculations. We do this again to find all of the references within the calculations. But if you imagine that a calculation can reference fields, CFs, TOs, variables and plugins, then repeating that traversal five times for each one is time consuming. So by putting the references into a single table we can do that just once.

But that's not all, we've brought into the xslt code some work that was previously being done as processing steps in BaseElements. If you've watched a BaseElements import, you will have seen some steps where it mentions "Processing Calculations". This was doing a few things : Digesting the calculation steps and producing plugin references and variable items, and also sorting out Indirection steps with function calls. It's a time consuming step in BaseElements 2, and can be up to a quarter of the total processing time. To give you an idea of the work being done, here is a screenshot of the code from BE2 :

In that there are lots of recursive custom functions ( great for simplifying code, not so great for massive data processing ), and lots of loops that run off and generate new records.

In BaseElements 3 this entire block of code is gone. In it's place we've built in the same functions into the References xslt file. I'm going to digress into some detailed xslt so if you're not interested, feel free to skim the technical steps or tune out :)

The first thing we needed to do was to be able to differentiate plugins from normal functions. FileMaker helpfully puts function calls in a separate part of the DDR, so we can track those, but it doesn't differentiate between plugins and other functions. To do this in xslt you build a lookup table reference :

<xsl:key name="plugin-lookup" match="PluginList/Plugin/Function" use="@name"/>

then you setup a variable to reference the external file :

<xsl:variable name="plugin-file" select="document('BE_Plugins.xml')" />

And finally when you've got a function call to lookup you look it up in your external table :


<xsl:for-each select="$plugin-file">
<xsl:if test="key('plugin-lookup', $FunctionName)">
<xsl:call-template name="RESULTSET"/>
</xsl:if>
</xsl:for-each>


We've used these key lookups in a few places other places as well and it makes a huge difference to the processing time.

The second part of that long script was the processing of variables used. FileMaker doesn't even distinguish variables from other parts of the text, and so they're just lumped in with text, math symbols and other whitespace as a "NoRef" type. As well as that, variables are now also in the text on layout objects, inside file references, perform find steps and set variable steps. Each one is slightly different, but essentially the process is the same for all of them : sort through the text and find the variables. This is actually harder than it seems. First, you don't want to find variables inside comments, so you need to remove those. Then you need to remove anything inside quote marks, but you need to first strip out any escaped quotes. Then you need to split the remaining text into dividers. Because a variable name can have spaces, the only way to tell when it ends is to just find the next non-variable name character after the $. This could be any of of the various math symbols or other characters that end a text block. So finally you have a list of the remaining text, and you convert those to nodes and loop through them to find any that start with a $.

This process is more complex to setup inside xsl than it is to code in FileMaker, but the end result is that the process runs much faster because you're not looping through records and generating multiple new variable reference records. The entire process fits inside the main reference xslt and is included in the single import that imports those.

I won't post the code here, but if anyone is interested in how it works, let me know and I'll document it in more detail.

It's these two changes, as well as the csv imports, that constitute the bulk of the reduced import time in BaseElements 3. It's funny, I thought I'd done a good job of making things faster in BE 2, and I wasn't sure where any extra savings would come from. It was only after the v11 update broke our old xsl that I went back to the drawing board and found other optimisations with the new xsl engine. So maybe if v11 still worked with the old code BE 3 might not have been as fast as it is...

As a follow up to my previous post about MarsEdit, I was having an issue when trying to upload images. I was getting a really useless error :

It is not possible to upload the file, because it exceeded the maximum filesize of 0 bytes

So check the prefs inside drupal :

Right. Something's not quite talking there. It turns out the problem was in permissions. Although I'm logged in as user 1, which on the site has every privilege, the blog api uses the proper roles and won't allow you to upload if the permissions for your role doesn't have it explicitly set. When I turned on the correct privileges all was good again.

As of today we have a new payment processing engine on the website. We used to use a combination of Ubercart, Paypal and WorldPay before today. All of these have been fine and have worked well for their relative strengths.

Ubercart is evolving quickly and has a good following in drupal land, although what we're doing, selling two products, doesn't really warrant or justify a "shopping cart" type solution. It's not yet flexible enough for us to do more (like invoicing consultant work via ubercart), so in lots of ways is overkill.

Worldpay has been very good, and is quite reasonably priced. It may seem odd to have a company in Australia, selling products in US dollars via a subsidiary of Bank of Scotland, but there isn't any local bank who will let us sell in US dollars without charging huge fees. I'm not particularly fussed about buying in overseas currency myself, but 90% of our sales are to the US, so it makes sense to have a fixed rate for that currency. I think this is becoming less of an issue though as time goes on, people are becoming more attuned to buying on the web from overseas companies.

Paypal has been good for us, you do hear stories about people who have had paypal issues before and we've had none of that. However we did get a recent email with a warning about Instant Payment Notifications. This is part of paypal processing that sends the data back to our site after a transaction completes. I have no idea what went wrong, nothing changed on our end, so I've no idea what caused the warning. It worries me that included in the email is the phrase

"If this problem continues, IPNs may be disabled for your account."

. Seeing as I can't find a reason for the issue, nor does the email contain any details, I'd hate for something to just stop working. I think this is the sort of thing people get concerned about with paypal, as the issue isn't clear and there is no place in the email to follow up or find out why or what is happening, and no contact details either.

The big issue in all of this for me though, is that I have to manage all of this myself. Keeping all of the various parts updated (especially ubercart and the various modules) and making sure it's all configured properly to work how it needs to work is not something I need to be doing. I'd prefer to be spending my time coding on BaseElements or UpdateManager and turning those into better products. I seem to come to that conclusion more and more these days.

So far we've outsourced timesheets to WorkFlowMax (works well but doesn't 100% suit our workflows, we work around that), our accounting to Xero (love it), and now our payments to FastSpring.

I'm yet to see how well FastSpring will work for us. I took it on because I've seen other developers using it, so it already has some sort of a reputation. Plus it offers features that are simple to add that would be complex in ubercart. At this point we have a very plain vanilla store, although we'll add more in the future.

In the mean time we still have the paypal and worldpay accounts, and we even have an old Kagi one. Worldpay is the only one of those that costs us money to keep open, although we still use it for taking credit cards at DevCon and will probably do that again this year. After that though, I'm not sure if it's worth keeping.

If you have any issues with purchasing or questions when buying a product, please contact us.