Give field objects more control over how their values are persisted

[mscott]

The primary goals of this ticket are to give field classes more control over how their instances' values are persisted.

One thing that used to be a special case, but must be generalized by this branch, are Entity fields. Previously, only the Entity field was "allowed" to store entity references, and although custom field classes could store nearly arbitrarily complex data structures, they could not reliably and easily store structures that contained entity references.

By making Entity fields just one variation of fields that may store entities, and allowing fields to store more than one entity reference while retaining bidirectional linking and referential integrity, the breadth of possibilities for structures stored in field values becomes much wider.

Work is being done in source:/branches/fieldstates/Schevo

[mscott]

Todo:

• Database class becomes Database2, handling format 2 database files; Database is just an alias for it.
• Database1 subclass of Database2 that handles format 1 database files.
• Format conversion function to convert a database from format 1 to format 2.
• "schevo db convert DBFILE" to convert database from format 1 to format 2.
• Unit tests common to both formats must run with both Database2 and Database1 engines.
• Unit tests specific to one format must run with just the engine it's designed for.

[mscott]

We also need to resolve the backwards-compatibility issue with regards to the field_factories_with_underscores vs. fieldFactoriesWithCamelCase as described in detail at r3057.

I'm personally still +1 for putting in camelCase aliases, but making those aliases loudly warn of deprecation when those particular factories are invoked.

For instance, foo = f.hashed_value() would be acceptable and silent, and foo = f.hashedValue() would be acceptable but would loudly proclaim that the more correct incantation is f.hashed_value().

[mscott]

SchevoXml also needs fixing, since it uses EntityRef? but that class no longer exists in the form that SchevoXml used it for. It needs its own variation.

[mscott]

EntityList? needs to be added to the test_field unit test.

[mscott]

Actually, the Entity tests in test_field are not the same as the rest of the tests. We should just factor out that test into test_field_entity, to correspond with test_field_entitylist.

[mscott]

Tests are now split up to run as many tests as possible with both the format 2 engine and the format 1 engine.

[mscott]

Format 1 engine is in a fairly stable state now. Todo for format 1 engine:

• make sure test_field_entitylist fails with format 1
• write internal tests for format 1 in test_transaction to correspond with the format 2 tests
• tests to make sure that a schema with a may_store_entities field type other than Entity will fail when loaded with the format 1 engine

[mscott]

test_field_entitylist updated with UnsupportedFieldType? checks.

test_transaction updated with format 1 internal structure checks.

test_on_delete updated with format 1 internal structure checks.

[mscott]

List of things that the format converter needs to do (crossposted from http://groups.google.com/group/schevo/browse_frm/thread/b0de0363aadecc98)

• Convert all (extent_id, oid) tuples to Placeholder instances in entity fields and indices.
• For all entities, create a related_entities structure to store sets of related entities.
• Bump format from 1 to 2.
• Do everything in one schevo.store transaction that either succeeds or fails.

To be determined:

• Get rid of entity_field_ids structure, since that is no longer necessary now that related_entities is used.

[mscott]

See r3099 for latest format converter work; it's ready to test on production databases now.

[mscott]

Replying to mscott:

See r3099 for latest format converter work; it's ready to test on production databases now.

The "NOTE" in the commit message for r3099 was written late at night and is not pertinent; please disregard.

[mscott]

Replying to mscott:

To be determined: * Get rid of entity_field_ids structure, since that is no longer necessary now that related_entities is used.

Let's just keep this in format 2 for now. The issue is that database1 currently subclasses database2 and overrides what is needed.

Perhaps format 2 could be updated in the future to not use entity_field_ids, and strip them from a format 2 database if it finds them. This could happen if database1 just provided its own separate implementation independent of database2. database2 would then be free to do whatever it wanted.

At the very least I will check to see where entity_field_ids is accessed but never used, such as in _create_entity, _delete_entity and _update_entity.

All the rest we can tackle in a "format2cleanup" branch, such as using sets instead of tuples or lists where appropriate, and so forth.

[mscott]

Replying to mscott:

I'm personally still +1 for putting in camelCase aliases, but making those aliases loudly warn of deprecation when those particular factories are invoked. For instance, foo = f.hashed_value() would be acceptable and silent, and foo = f.hashedValue() would be acceptable but would loudly proclaim that the more correct incantation is f.hashed_value().

This is the strategy that I will implement.

[mscott]

Replying to mscott:

SchevoXml also needs fixing, since it uses EntityRef? but that class no longer exists in the form that SchevoXml used it for. It needs its own variation.

SchevoXml will be deprecated for the following reasons:

• It is not in active production use.
• It does not use a file format that is useful for anything other than exporting and importing entire Schevo databases.
• It would require a lot of effort to make it work with Schevo 3.1.
• It would require effort on the part of custom field authors in order to be useful as a general export/import tool.
• The return on that effort investment would be at most minimal, at worst zero.

Instead, individual applications should have their own XML export and import mechanisms that use file formats more closely tied to the application domain itself.

[mscott]

Ready for review and merge.

[mscott]

pobrien reviewed this branch, made a slight modification to the format conversion routine in r3110 and to the GTK navigator and widget set in r3109, and noted that the branch is working well with databases generated by his commercial app.

Merging this branch to trunk and closing this ticket.

Open new ticket(s) and create new branches with specific changes needed in the future.

[mscott]

Merged at r3112.