Goya Blog

baseElements 3 Beta 8

I never thought I'd do 5 beta versions of this, let alone 8. But here it is :

BaseElements_Mac.dmg - fp7 files.
BaseElements_Mac_Runtime.dmg - Runtime.

BaseElements_Win.zip - fp7 files.
BaseElements_Win_Runtime.zip - Runtime.

The list of changes are :

  • Added the TO mismatch to the error count.
  • Fixed an issue where Summary fields weren’t being referenced.
  • Made some more optimisations in the references import.
  • Fixed an issue where valuelist references to external files wasn’t working.
  • Sorted out some reference issues with ValueLists.
  • Added a custom menu item for printing of reports and list layouts.
  • Fixed an issue in the Go report for SMTP steps.
  • Removed and, or, not and xor from the plugin list :)

Critical changes are to the way ValueLists were handling references, the layout unreferenced calc and also the FileMaker Go Compatiblity Report, especially it's print option.

This version has now had quite a bit of testing, so I'm looking to post the upgrade details on Monday.

New BaseElements 3 hidden feature - copy from BE to FMP

One thing I've been wanting to do for ages in BaseElements is the ability to copy ( and maybe paste ) information out of BaseElements and back into FileMaker. Imagine that you've got an old analysis of a solution that has an older version of a Custom Function or a Script, and you need to put it back into your solution. Just open up BE, select the CF and hit copy and paste into FMP.

Obviously this isn't a built in function in FileMaker Pro and although it's relatively simple on the mac using applescript, there is no easy way on Windows. There was a plugin ( since discontinued I think ) that had this feature, and the new ScriptMaster Advanced from 360Works has the ability to set and get the clipboard. We've built this functionality into our own BaseElements plugin which we'll be open sourcing soon so anyone can give it a try.

So the just released beta 7 version of BaseElements 3 includes the ability to copy Custom Functions directly out of BaseElements and paste them back into FileMaker. And it has the bonus feature of including any modifications you make to the name, parameters or calculation in BaseElements into the copied version.

Hidden Features

The reason I call this a hidden feature is that it's implemented with zero interface. If you're looking at a Custom Function and you don't have an active field selection, then command-c will copy that CF to the clipboard. And if you're looking at a list or table view of CFs, then it will copy the entire found set of CFs to the clipboard. If you've got some text selected in BE then it will copy the selection instead.

I really like that it's implemented this way. There are some downsides in that some people may never find out about the feature, but hopefully the more I post blogs or when I finally finish the manual, there will be lots more places to find out about these things. But it's the fact that a feature that is really helpful, but you won't necessarily use 100% of the time, isn't taking up valuable screen real estate, where it would have to appear 100% of the time.

The ability to do GTRR into a new window by holding down the option key, and the ability to do GTRR using found set by holding shift are in the same arena. These, to me, are the two best features in BaseElements, but they don't ever announce their presence. ( They announce their presence inside ScriptMaker by their 8500 lines of code though - my kingdom for a dynamic GTRR step ).

It's always tempting to shout out these features by including them on the layout somewhere, but I think you're making a mistake in doing that, they don't need to be on the layout, leave that to the actual data, and the feature itself hides away and is called into action only when needed.

More to come.

For BE 3.0.0 the copy function will be limited to Custom Functions, and to copying only not pasting. I'm going to work on Fields next, and after that Tables, and then hopefully steps and scripts. The reason for that is the XML for CFs is relatively simple. We're not storing in BaseElements the actual XML like you do in ClipManager ( great product by the way, use it a lot, saves heaps of time, will post more later ), so we need to recalculate the XML from the stored data. For CFs that's pretty simple, some heading details and then a calc.

For fields it's more complex, there are lots of different types with storage and validation and calc options, there is a lot to recreate. Tables obviously depends on fields, so that would be next, but it's a relatively simple step once fields are done.

Script steps are a whole other box of hurt. There are 150 odd script step types and each one has very different coding for what data it comes with and how it's laid out in the XML. Have a look at the XML DDR pdf document for v11 if you want to get an idea of what's involved. The problem though with scripts is that we import into BaseElements the script text ( which isn't the same as the XML ) and also any reference items. So lots of the settings and options aren't included. These would need to be included in order to regenerate the XML. The reasons for that are two-fold : keeping the import time down, fewer items is faster imports, and also 99% of the time you won't need all of the details in individual fields, but you can see all of the details in the step text.

So the options with steps are to try to turn the actual text back into XML, or import more data. I'm going to play with both over time and see how it goes. I may end up with some sort of hybrid approach.

Pasting into BaseElements

This one I haven't worked out a full use case for just yet. It seems logical that if you can copy out of BaseElements, you should be able to paste back into BE. But where and how? Obviously if you're looking at a CF, pasting a script doesn't make sense. Even if you're pasting a CF what should happen? Should it over write the current CF? What if there is more than one in the clipboard?

My opinion is that it shouldn't alter any of the current data in BE. I don't think that helps in maintaining a record of your solution as it was imported at the time.

So there would be need to be somewhere else that you could paste into. Maybe some sort of scratch pad or toolbox type option. I'm still considering how to go with this, including the option of tight integration with ClipManager, so that I don't have to implement my own toolbox at all, I'd just use the ClipManager one, and reference it.

I'd love to know other's opinions on how best to use this and if they see a use case for it. Send me email, fill in the form, or reply on twitter.

BaseElements 3 Beta 7

Once more for old times sake :

BaseElements_Mac.dmg - fp7 files.
BaseElements_Mac_Runtime.dmg - Runtime.

BaseElements_Win.zip - fp7 files.
BaseElements_Win_Runtime.zip - Runtime.

I've got all of the import functionality complete in this one and one really cool new feature so will post more soon.

Reducing the number of objects on a layout in BaseElements

One of my big priorities for BaseElements 3 was to reduce the number of objects on each layout to reduce complexity, and make it easier to work within BaseElements. There is a lot happening in BE, and trying to make things clearer will always help.

First, no more id fields.

One of the first things I thought about was the id fields. These aren't real fields or details in FileMaker, these are the internal ids that FileMaker generates for every object (layout, script, field, TO etc) that users never see until they run the DDR. We import these and use them for matching of related records.

In BE 1.0 the ids took centre stage, they were listed first, before the object name itself. I realised that this wasn't necessary, and in version 2, they were moved to the end of the item, as well as being removed from the portals altogether. You never needed to see the ids in a portal to understand what was going on, so removing them just meant I could show more detail in portals.

So in version 3 I was thinking about them again, and decided that they didn't need to be on any layout at all. I've almost never used them, but if they're required for one particular use case, then the fields are still there and whoever needs them can just add the field to the layout as required. ( There is full layout access in BaseElements )

I think this is wise, as 99% of the time, they're not used, and take up space and make everything else harder to work with. For the 1%, they're still available on demand whenever needed.

Second, the "Items that use this" count.

The other item I didn't like was the counts for "Items that use this xxx" or "Items used by this xxx". For example this one in the fields layout :

Screen shot 2010-08-05 at 1.30.00 PM.png

This takes up a lot of room, and doesn't provide a wealth of information, considering that in most cases more than half of the totals were empty. When I started doing conditional formatting on the tabs, I realised I could easily use that to display the "in use" or "not is use" status that these counts were taking up. There were two other functions that these fields were showing : the actual number, and the quick link that you had by clicking on the field label to go directly to that item.

My first attempt was to try to build the count into the tab name itself. Because it's a field not an actual tab name, that could work, but I couldn't come up with a format that looked good and displayed the right level of information. In the end I went back to think about it a little more. The critical thing is whether or not there are related records, the actual count is almost superfluous. If there are 3 or 30 it's not important as long as you know they are there. So the highlight works well enough, and I disposed of the counts.

The only other thing was the quick links. I haven't yet put these back into BE3. I'm not sure if there is a good way to go about it. You could have the ability to go directly to a list of items by option clicking on the tab or something similar. The only issue is that I'm alreading using option, shift and command clicking to do different things with GTRR so it may confuse things. And there isn't any sort of user interface convention that does this that I know of, so it would be a very non-obvious feature. I'm still considering options there...

BaseElements 3 : More Speed : Loops vs Import

The last thing I needed to do to finalise the feature set in BaseElements 3 was to generate the list of Variable references from each of the Calculations. This isn't a minor thing though, as there are no variable reference types in the DDR, so you need to pull a list of variables out of the calculation manually and then generate the list of related records.

First get the variables.

In order to find variables inside a calculation, you do a few things : First remove the comments, both /**/ and // types, then remove any text between quotes, making sure you ignore escaped quote characters. Then you can work through the text and get the $ values, making sure you've got the right break characters. I've done this code before in FileMaker, and it's not simple, and involved a few recursive custom functions. In other words it works, but it's slow.

In BaseElements 3 we've worked around this entire issue with a single plugin function call : BE_ExtractScriptVariables. This sort of thing is much faster in plugin code, so that is a big advantage.

Next create the related records.

The second thing you need to do then is turn that list of variables into a set of records. You can't do a simple import from one table to another as there will be multiple variables in some calculations and you need a separate reference record for each. The standard way to do this would be a loop through the records, and a loop inside that through the variable list, which generates a new record each time. The issue that makes this slow is the constant context switch. You're switching from the Calculation layout to the Reference layout every time you want to move to the next record.

Slow : Looping New Records.

For curiosities sake I built a quick version of this code and timed it. Total for the 2500 odd BaseElements calculations was over 2 and half minutes, to generate nearly 4000 variable references.

Fast : Write to disk and Import.

The actual code I'm using in the next version of BaseElements is a loop through the calculations record that stores the contents of a csv export file in a local variable. So you do a very basic loop and just fill up a variable. No context switch, no new records, no modifying or committing of data, just reading from fields and writing to a variable, so it's fast. Time for that section of the code is about 2 seconds for the entire 2500 records.

Then you write the file to disk ( less than one second ) and import the csv file ( about 8 seconds ). So total time : 11 seconds.

Obviously a huge difference between a loop with new records and a write to disk and import. For anyone doing any large moving of found sets, you might want to consider something similar.

BaseElements 3 Beta 6

Another quick update on last nights b5, which contains a couple of quick fixes for false errors, some adjustments to the FM Go compatibility report and an updated plugin for good measure. Download away :

BaseElements_Mac.dmg - fp7 files.
BaseElements_Mac_Runtime.dmg - Runtime.

BaseElements_Win.zip - fp7 files.
BaseElements_Win_Runtime.zip - Runtime.

Update : There is an issue with the plugin version on windows, so stick to b5 if you're testing on windows :

BaseElements_Win.zip - fp7 files.
BaseElements_Win_Runtime.zip - Runtime.

The next release will also have a fix for layouts that were incorrectly reporting Unreferenced. I should have this new version released in the next day or two.

BaseElements 3 Beta 5

I've posted what is hopefully the final beta of BaseElements 3. The download links are :

BaseElements_Mac.dmg - fp7 files.
BaseElements_Mac_Runtime.dmg - Runtime.

BaseElements_Win.zip - fp7 files.
BaseElements_Win_Runtime.zip - Runtime.

New Licence Required.

This version requires a new v3 licence code, an existing v2 licence code won't work. You can use a demo code in the mean time until I put version 3 purchase details on the web.

The total list of changes are :

  • fixed some display issues in the MenuSet layout - thanks Doug Rowe.
  • fixed an issue with dialogs appearing when no layouts were present - thanks Adam Dempsey.
  • fixed some display issues with Inbound Authorisations - thanks Steve Wright.
  • fixed a bunch of issues with the display of portal errors and resolved some other TO error reference display issues.
  • cleaned up the Items Used portals and made them display errors in TOs.
  • Finished work on the FileMaker Go Compatibility Report
  • Extended the beta period to 10th August
  • More plugin updates and url fixes.
  • Fixed the issue of plugins not being reported.
  • Sorted out some processing issues with plugins and variables.
  • Added plugins and variables to the privilege set for Smart Find - thanks Adam Dempsey.
  • Fixed the issue of saving of the reg details.
  • Increased the requirement for a version 3 licence.
  • Added a warning dialog about importing from a shared folder on windows.

There have been a lot of changes since b3, and hopefully I'll get some time to post details about them this week, so keep an eye on the blog.

BaseElements 3 beta 4

I've just finishing uploading BaseElements 3 beta 4 :

BaseElements_Mac.dmg - fp7 files.
BaseElements_Mac_Runtime.dmg - Runtime.

BaseElements_Win.zip - fp7 files.
BaseElements_Win_Runtime.zip - Runtime.

There is a windows as well as a mac version, and the fp7 version now has sharing enabled for all accounts. The full list of changes is :

  • More plugin error checking and reporting.
  • Fixed an issue where odbc Data Sources weren't being imported.
  • Fixed a false error in OSBC usernames and passwords - thanks Doug Rowe.
  • Fixed a superfluous warning in Layout Fields.
  • Added a dialog about running on PowerPC on Mac.
  • Changed the display of the BaseElements wording on layouts to display properly in FileMaker Go.
  • Built in a Minimum Required FMP version for calculations and script steps.
  • Changed the licencing so that the single user version can be shared with FileMaker Server to a single user.
  • Added more display checkboxes for layout objects to show printing, sliding etc.
  • Added a portal filter calculation portal to the Object layout.
  • Began work on a FileMaker Go Compatability Report.
  • Some plugin fixes and enhancements.

There is a lot in there, so I'll write some more about it in the next day or so, but it's now late here and it's time for dinner :)

One quick note, if you're using it on windows it will only load xml from a local path, not from network drives. ( You can map the path to a network drive and it will work fine. ) This is something in the way that the file paths are handled inside libxslt, and I haven't yet got a solution to that one.

Updated FileMaker New Features for Version 11

Yesterday I updated my list of new FileMaker features for version 11. The list I maintain doesn't include everything (like the new Quick Start Screen, or text highlighting), but includes all of the items that affect our ability to program in FileMaker. So things like new script steps or changes to calculations.

The reason for doing this yesterday was that I was building into BaseElements the ability to recognise which calculation functions and script steps came from which versions of FileMaker. Using this list you can then say that to open this file, you need to use at least a minimum of FMP version X. There is a little more to do in BaseElements though to make it comprehensive, like extra features within certain script steps or layout features like web viewers or filtered portals. However, the functions area is more technical and is built into the XSLT in BaseElements, so I wanted to get that coding done first.

But I stumbled across an interesting bug I'd never noticed before in the FileMaker DDR. To test the various functions, including the Get functions, I built a file with a single calculation that included every possible built in function. Then I ran the DDR for this file and had a look at the details. To understand how the DDR works, when you have a long calculation, FileMaker breaks it into chunks. There are chunks for Fields, Custom Functions, Functions and everything else is a "NoRef" chunk (in other words text).

All of the built in FileMaker functions were reported, correctly, as "FunctionRef" nodes. I had thought that the most likely answer for the collection of the "Get" functions, was that the Get part would be a FunctionRef and the rest would be NoRef. Because in reality, the parameter you pass to the Get function is just a reserved word and the actual function is called simply "Get". I was hoping that it might distinguish the reserved words in some way so I could figure out which ones it used, but I wasn't sure what it would do.

It turns out in FMP 10, and I presume earlier versions, FileMaker reports some of the parameters as "FunctionRef" chunks and ignores some of them, so they end up in the text along with the round brackets and any whitespace and also anything that follows the function. I wasn't sure what the logic for this was, there didn't seem to be any consistency to it. And even stranger it reported one Get reserved word as a Custom Function.

I was all set to report a bug to FMI when I realised I had generated the DDR with FMP 10 and realised I should run the same test with 11. Well in 11, every Get reserved word is reported as a FunctionRef, so I can accurately report on them in BaseElements. So no bug report is in order.

What this does mean is that there could also be other differences and inaccuracies with the DDR in older versions that are fixed in 11, and I'd suggest that if you're running any tool like BaseElements where you utilise the DDR that you should run it from version 11, just to be sure.

I do have a big feature request for the DDR though : That it broke the calculation into more discreet chunks. For example, plugin functions are reported as "FunctionRef" along side every built in function, it would be great to have a "PluginRef" type by itself. Comments and quoted text aren't put into anything separate, it would be handy to have them put into a "TextRef" or CommentRef section. And technically the Get function types aren't really functions, they're reserved words, so a "ReservedWordRef" would be nice. Maybe in v12 :)

BaseElements 3 Hardware and Software Requirements

A few people have asked me about which versions of their OS will BE3 run under. There is a little detail, so I'll include it all here with some explanations and put the specifics into the FAQ section.

Firstly, BE3 is designed for FMP 11 and will close if you attempt to open it under earlier copies of FileMaker Pro. We tend to update BaseElements for the latest features in the OS each time so that we can build on the new capabilities. We realise that not everyone will have the latest version straight away, but runtimes are available that mean you don't need to have the latest FMP to have the latest BE version.

So the general requirements for BaseElements are the same as the requirements for FMP 11.

I have tested BaseElements 3 on the iPad via FileMaker Go and it does load and work. There are some font issues that need to be worked around, and also some issues with the fact that plugins are not available. I'd like to think we could get a version of BaseElements that works, minus importing and translated dialogs for the iPad very soon. We may have a few issues with displaying of data and missing toolbars and menus, but they can also be resolved relatively simply.

I don't think it's feasible to run BE on an iPhone, although it would work the layouts are not designed for that, and it would be a lot of work to design more iPhone friendly layouts. I'm not going to do anything to prevent this, I just won't be working towards an iPhone version I don't think.

BaseElements Plugin Requirements

Plus we have a new plugin in BaseElements that has it's own separate requirements. The basic requirements are the same as for FMP 11 with some additional caveats :

The plugin is currently intel only. It's a non-trivial amount of work to get a PPC version up and running, and given the open source libraries we've used, there is additional work to convert those. Considering Apple last shipped a PPC Mac nearly 4 years ago, I would expect the number of users that this affects to be small and shrinking. That said, we are open sourcing the plugin and will consider a PPC version if there is demand for one.

The plugin should work as far back as FMP 8.5 at least. Initially we're not testing this far back ourselves, as BE requires FMP 11 anyway, so any other testing we do will come later.

We've been testing on Win XP and Win 7, but not Vista, even though Vista is a supported OS for FMP 11. There isn't though anything to suggest it won't run on Vista.

At present it's a client side only plugin, so can't be installed on FMS. This is definitely something we're looking to do in a future version although it may not make it for 3.0.0. I'd like to have a server side import process working in BE3.

Citrix and TS should work fine, although we haven't tested. If you have any feedback about this or can report success or failure, we'd love to know about it. We will have more options for plugin feedback once the open source code is released.

Syndicate content