Transactional Logs

[Neil Hargreaves]

Hi Brendan,

I’m loving Tapforms and have been using its capabilities to manage the flight records for my drone business for the last three years.

Our Civil Aviation Safety Authority has introduced a requirement that any flight records be indelible, or, if they are able to be edited, that there is an indelible record of any changes.

I wondered if TapForms keeps a transactional log behind the scenes, and, if so, how we might be able to access its content?

Thanks,

Neil

[Brendan]

Hi Neil,

There’s a revision history that’s kept behind the scenes, but it’s just an implementation detail of the CouchbaseLite database engine I built Tap Forms on. It gets added to every time any change is made to a record. And only 20 revisions are kept at most. It’s not really easy to get access to the revision history though.

However, I’m working on a function to let individual forms be protected so that you can prevent someone from deleting records. It’s not rock hard security as anyone could open up the database engine and modify it directly outside of Tap Forms. Well, unless you encrypt the database document that is.

It’s still in development and hasn’t even gone out to my beta testers yet. But it is coming. So maybe that will help you out with this issue.

[Sam Moffatt]

TapForms uses CouchbaseLite behind the scenes and that exposes a CouchDB compatible stream. Each change that happens to a document in TapForms is propagated to a change log which can be used for replication or in your case, recording each individual change that happens.

There are a few complications here:

• This change log isn’t guaranteed to be comprehensive. Some changes maybe squished together for the sake of efficiency and historic records may be removed to reclaim space. This may happen manually if you click “Compact Database” under preferences or it may happen automatically.
• If you are recording this via an iOS device and then sync’ing to your Mac using the CouchDB based replication options (P2P, Cloudant, CouchDB) then your iOS device will only send the latest change it had. If you have make multiple changes out in the field to a document and then sync them, your Mac will only get the latest change.
• Getting to this log is non-trivial. TapForms uses a randomised port each time the app starts per database which means you have to figure out what that port is via Bonjour/mDNS before you can get to the API. If you sync to an external CouchDB server, getting to the log is a little easier.

I’m not sure how detailed CASA want to be, I suspect by default TapForms is actually more conformant than they’d care to think. If you don’t have iOS, or you’re fine with only one remote change making it back to your Mac, then you can use this API to extract out the raw documents and changes from the TapForms document. In theory you could extract the changes from the iOS devices via the same API but that’s a little more complicated.

If you are interested in setting up a CouchDB instance, you can have TapForms replicate to it and then use it’s change log as your source of truth. If your devices are always online and the CouchDB server is always online then it will obviously replicate the changes as you go automatically.

Ok, so how do we get the changes? The first step is to get an application called “Discovery” from the App Store. Discovery is a Bonjour/mDNS/DNS-SD browser that will let you find instances of TapForms on your personal network. Once you’ve downloaded it, open it up and scroll down to the bottom to find the entry “_tapforms-sync._tcp.” and expand it out. It should list all of the instances of TapForms running on your network with an entry each for the documents that are open. It looks like this:

If you have more than one document open, you’ll need to know the ID of the document that you’re interested in. You can find this in the Preferences pop over for your document under the “Sync” tab

Once you’ve got this document ID, you’ll need to expand it out and copy the line that has the IP address on it, in my case “10.0.1.145:57343”. You’ll need to use this with the database ID to get the change log.

Next up, grab “Terminal” from “Applications” > “Utilities” and run a command like this:

curl -k https://[YOUR_IP_ADDRESS:YOUR_PORT]/[YOUR_DB_ID]/_changes?include_docs=true

For me with the example above, it looks like this:

curl -k https://10.0.1.145:57343/db-f8365d0d8a99446db370477295e705fe/_changes?include_docs=true

When you run that, it’ll spit out a raw transaction log of all of the changes that this TapForms instance is currently retaining.

You can checkout the CouchDB docs on the _changes endpoint for some more details about how it works and it’s limitations.

Now the complication of this is that you need to find the new port each time you open and close TapForms. I tried to build some automation for this because I was curious about building an automatic network backup tool to find TapForms instances on my network and automatically backup their change logs but I never got it finished and it never worked reliably. What I do is I actually use a CouchDB server on my local network and replicate the changes to there. I use that to then build automatic hourly snapshots of my database using GIT.

I have two tools I use to keep backups, the first is a Time Machine based backup solution which I have put up on GitHub. I don’t think this will work for you because that method is inherently lossy. The second one I built scrapes the CouchDB and pulls out the internal revisions of the documents to store to disk. I don’t have that online yet, I need to integrate it properly into the little framework I’ve built but it isn’t a transaction log per se but a per record version history. The challenge with both of these is that inherently they don’t offer non-repudiation and are themselves in theory editable. You control all of the devices and at that point there is no audit pathway. Short of relying upon a third party service that is fully managed by someone else, I’m not sure you could assure an auditor of the veracity of your logs.

Attachments:

You must be logged in to view attached files.

[Author]

Posts