Allow use of schevo.store and Durus backends

[mscott]

They all act very similarly, they all have different release schedules, they all have pros and cons. We might as well choose between the three of them at will. (Zodb will be tackled in a separate ticket)

Much of this work will be based on source:branches/backend that lekma was working on, and will take a test-driven approach to covering all the bases.

Its main goal is to allow the latest Durus to be used as a backend, so that Schevo can benefit from its recently-implemented memory reduction strategies.

General plan of action:

• Given a backend class, yield test subclasses (properly named so we know where they came from), using both version 1 and version 2 database engines.
• Move usage of schevo.store into a backend that's included with Schevo. Include test runners for it.
• Modify command line scripts to work with backends properly.
• Create a new distribution SchevoDurus with a Durus-based backend. Make Durus eggs available for OSX and Windows. Include test runners.
• Create a "schevo db copy" command that will copy a database to a different backend. For migrating existing databases.

[mscott]

All the storage-related tests are now written out as base classes rather than test classes.

See schevo.test.test_backend_schevo_store for the canonical usage against the built-inschevo.store backend.

[mscott]

Replying to mscott:

All the storage-related tests are now written out as base classes rather than test classes. See schevo.test.test_backend_schevo_store for the canonical usage against the built-inschevo.store backend.

This is no longer the case for the built-in Schevo tests. Instead, they are regular test classes, but they subclass some base classes that let you say which backend you'd like to use.

So, running "nosetests" within Schevo will run all the storage-related tests against the "schevo.store" backend, but running "nosetests" within SchevoDurus will run all the storage-related tests against the "durus" backend, with both the "shelf" and "file" storage types.

[mscott]

Some musings on the use of backends in command-line tools...

We need a way to specify the backend. In lekma's branch, he proposed a global command-line argument that specified the backend. The default was "schevo" (which, in this branch, would be "schevo.store"). These command lines would look like this, assuming the durus backend was being used:

$ schevo --backend=durus db create --app=foo test.db
$ schevo --backend=durus shell test.db
$ schevo -Bdurus db create -afoo test.db
$ schevo -Bdurus shell test.db

This seems reasonable, however I think we need to extend it a little further.

We should allow listing of available backends with their options:

$ schevo backends
Schevo 3.1a1

Available backends and their options are listed below.

durus
=====
Durus-based backend.
Provided by SchevoDurus 1.0a1
--backend-storage=TYPE
   The storage type used by Durus for new files.  Default: "shelf"
--backend-cache-size=SIZE
   The size of the in-memory object cache.  Default: 100000

schevo.store (default)
====================
Built-in backend.
Provided by Schevo 3.1a1
--backend-cache-size=SIZE
   The size of the in-memory object cache.  Default: 100000

We should first scan the command-line for a backend, and if not, determine the default ('schevo.store' for now). After determining the backend, load its specific option parser, mesh it with the default global option parser. Convert the whole mess into the backend_name and backend_opts arguments that various schevo.database calls require.

Eventually, we'll want to make sure that we can detect the backend used to create a file that we want to do something with.

• write a function to determine the backend used to create a file
• if backend and existing filename given, and they match, use backend opts specified
• if backend and existing filename given, and they do not match, stop and show error about mismatch
• if existing filename given, but no backend given, use the backend that we determined created the file, with default backend opts

Programmatically speaking, it will be useful to

Speaking of that, we should definitely split "schevo.database:open" into "create" and "open" as lekma did in his branch. The method signatures would be as follows:

def create(filename, backend='schevo.store', backend_opts=dict(), ...)
def open(filename, backend_opts=dict(), ...)

Notice how "open" should not even require a backend name. Schevo should be able to determine this automatically by having each available backend inspect the file and determine if that backend should handle it. In other words, you should never have to do anything beyond "schevo shell foo.db" to open foo.db in a Python shell -- *unless* you wanted to tweak specific backend options like cache size.

That's all for now...

[mscott]

I'm silly. The global options available for the schevo command-line tool aren't separated out from the rest of the options.

So the above examples would be:

$ schevo db create --backend=durus --app=foo test.db
$ schevo shell --backend=durus test.db
$ schevo db create -Bdurus -afoo test.db
$ schevo shell -Bdurus test.db

Also, in terms of keeping the implementation concise, I think for now we should do something like this for specifying backend arguments:

$ schevo db create --backend=durus --backend-args=storage=shelf --app=foo test.db
$ schevo shell --backend-args=cache_size=200000 test.db

[mscott]

Things left to do for this branch before initial testing (not necessarily all inclusive):

• Command line tools
  • Update existing tools to properly use new backend-based schevo.database functions
• schevo.database functions
  • Split out open into open and create (partially done)
  • Update inject to use backends
  • Finish database auto-detection (partially done)
    • Allows you to use schevo.database.open(filename) without specifying a backend name.
    • (untested) functions already written to determine file type from first 128 bytes.

Things to do before merging:

• Docstrings for new code
• Docstrings for altered code
• Documentation updates
• Durus unit test database caching? (this worked, except when reopen() was used)
• ZODB backend?
• Test/update SchevoWsgi et al to make sure it works

[mscott]

"initial testing" == "initial review and testing by pobrien"

:)

[mscott]

schevo.database:open has been split into open and create.

[mscott]

Need to get database caching in the SchevoDurus unit tests -- they all pass on OSX but in XP, its NTFS implementation really bogs down and makes it excruciating to wait for all of its tests to pass.

[mscott]

Replying to mscott:

Things left to do for this branch before initial testing (not necessarily all inclusive): - Command line tools - Update existing tools to properly use new backend-based schevo.database functions

Done.

- schevo.database functions - Split out open into open and create (partially done) - Update inject to use backends - Finish database auto-detection (partially done) - Allows you to use schevo.database.open(filename) without specifying a backend name. - (untested) functions already written to determine file type from first 128 bytes.

Done.

Things to do before merging: - Docstrings for new code - Docstrings for altered code - Documentation updates - Durus unit test database caching? (this worked, except when reopen() was used) - ZODB backend? - Test/update SchevoWsgi et al to make sure it works

Some more for this list:

• definitely do Durus database caching somehow so we can confidently test under Windows
• schevo.database.duplicate function and according schevo db duplicate command-line tool to duplicate databases from one backend to another.

[mscott]

In lekma's backend-60 branch, he stored the database label within the backend's storage itself. This is a much better idea than a transient label, so let's make sure that gets into this branch too.

[mscott]

Database labels are now persistent.

The twitabit example now works correctly.

[mscott]

Zodb to be covered in a separate ticket.

[mscott]

Database cache for SchevoDurus tests works now. Tests pass in OSX and Windows XP, and without excessive resource consumption or filesystem abuse on either platform.

Will merge this 2007-09-04.

[mscott]

Zodb fixed quickly enough to cover it in this ticket :)

Merged at r3509.

Thanks, pobrien and lekma, for testing and suggestions. Nice to get this one merged.