Changeset 3509

Show
Ignore:
Timestamp:
09/04/07 21:19:49 (1 year ago)
Author:
mscott
Message:

Merge #60 to trunk.

Schevo now uses a lightweight backend storage plugin system for handling storage of internal database structures within a Python object-oriented database.

The new SchevoDurus distribution provides the new durus backend. SchevoDurus is now a requirement of Schevo itself as durus is now the default storage engine due to its decreased startup time when opening packed databases. It uses a Durus 3.7-schevo1, a lightly-patched version of Durus 3.7 that fixes Windows and Unicode-related bugs. We expect to require version 3.8 when that is released.

The new SchevoZodb? distribution provides the new zodb backend, which uses ZODB3 3.7.0.

The Schevo package itself provides the schevo.store backend, which will continue to be included with Schevo to provide backward-compatibility with existing Schevo databases and to provide for running unit tests using in-memory databases that do not write files to disk.

The unit testing infrastructure in schevo.test now allows you to specify the backend storage engine for running tests, and also has the ability to export the standard battery of Schevo unit tests in a way that allows overriding of the backend. This is used by SchevoDurus and SchevoZodb? to run the standard Schevo tests against each of those backend. See source:trunk/Durus/schevodurus/test/test_durus_backend.py for a usage example.

Functions in schevo.database have been updated to support the new backend system. You should not have to change client code for typical use cases. See source:trunk/Schevo/schevo/database.py for more information on the API changes.

schevo.database.open now only opens existing databases. Use schevo.database.create to create new ones. You should change client code to handle this change if your application is responsible for creating new databases and not just opening existing ones.

schevo.database.open no longer accepts a label argument for specifying transient database labels. Database labels are now persistent. You may specify a label for a database when creating it with schevo.database.create. You may relabel an existing database using schevo.label.relabel.

The new schevo.database.copy function copies internal structures verbatim from one database to another, and is typically used to copy an existing database using the schevo.store backend to a new database using the durus or zodb backend.

Command line tools now accept arguments for specifying the backend and optional backend arguments. The new schevo db copy command is a command-line interface to the schevo.database.copy function. See SchevoCommandLineTool for further information.

Files:

Legend:

Unmodified
Added
Removed
Modified
Copied
Moved

  • trunk/Policy/schevopolicy/test/base.py

    r3383 r3509  
    4444        _use_db_cache = False 
    4545 
    46         def _open(self, suffix=''): 
     46        def _open(self, suffix='', reopening=False): 
    4747            db = super(TestClass, self)._open(suffix) 
    4848            db_name = 'db' + suffix 

  • trunk/Policy/schevopolicy/wrapper.py

    r3426 r3509  
    131131 
    132132    def _open_file(self, db_alias, db_filename): 
    133         db = database.open(db_filename, label=db_alias) 
     133        db = database.open(db_filename) 
    134134        return db 
    135135 

  • trunk/Schevo/doc/SchevoCommandLineTool.txt

    r3191 r3509  
    5454Common options 
    5555-------------- 
     56 
     57The following options are available to all commands. 
    5658 
    5759**-h**, **--help**: 
    …… 
    6668    optimized Python modules. 
    6769 
     70**-B NAME**, **--backend=NAME**: 
     71 
     72    Use the named storage backend when creating or opening a 
     73    database. Use the `schevo backends`_ command to view a list of 
     74    backends and their available arguments. 
     75 
     76    You do not need to specify a backend if you desire to use the 
     77    default ``durus`` backend. You also do not need to specify a 
     78    backend when using an existing database as Schevo has an 
     79    auto-detection mechanism that determines it for you. 
     80 
     81**-A ARGS**, **--backend-args=ARGS**: 
     82 
     83    Pass the given comma-separated arguments to the backend. Use the 
     84    `schevo backends`_ command to view a list of backends and their 
     85    available arguments. 
     86 
     87 
     88schevo backends 
     89--------------- 
     90 
     91Usage: ``schevo backends`` 
     92 
     93Prints a list of available storage backends and their available 
     94arguments. 
     95 
    6896 
    6997schevo db compare 
    …… 
    96124 
    97125    Convert to a specific format.  (Default: latest format.) 
     126 
     127 
     128schevo db copy  
     129-------------- 
     130 
     131Usage: ``schevo db copy [options] SRCFILE DESTFILE`` 
     132 
     133Copies the internal structures verbatim from `SRCFILE` to 
     134`DESTFILE`. 
     135 
     136`SRCFILE`: The database file to copy the internal structures from. 
     137 
     138`DESTFILE`: The empty file to copy internal structures to. 
     139 
     140Backend options given apply to `DESTFILE`. The backend for `SRCFILE` 
     141is determined automatically. 
    98142 
    99143 

  • trunk/Schevo/doc/SchevoInternalDatabaseStructures.txt

    r3112 r3509  
    1515 
    1616  'SCHEVO': PersistentDict{ 
     17      'label': <db-label>,          [*] 
    1718      'format': <db-format>, 
    1819      'version': <schema-version>, 
    …… 
    3839                              }, 
    3940                          }, 
    40                       'related_entities': PersistentDict{       [2+] 
     41                      'related_entities': PersistentDict{       [**] 
    4142                          <field-id>: <related-entity-set>, 
    4243                          ..., 
    …… 
    6667      } 
    6768 
    68 ``[2]`` indicates that the structure is only present in databases of format 
    69 2 or higher. 
     69``[*]``: Labels are not present in older databases, and a default 
     70label is assumed if a persistent label is not assigned. 
     71 
     72``[**]``: The related entities structure is only present in databases 
     73of format 2 or higher. 
    7074 
    7175 

  • trunk/Schevo/schevo/database.py

    r3306 r3509  
    77from schevo.lib import optimize 
    88 
    9  
    109from schevo import database1 
    1110from schevo import database2 
     11from schevo.error import DatabaseAlreadyExists, DatabaseDoesNotExist 
    1212from schevo.field import not_fget 
    1313from schevo import icon 
     14from schevo.label import relabel 
    1415from schevo.store.connection import Connection 
    1516from schevo.store.file_storage import FileStorage 
     17from schevo.trace import log 
    1618 
    1719 
    …… 
    3133 
    3234 
    33 def convert_format(filename_or_fp, format): 
    34     """Convert database to a new storage format. 
    35  
    36     - `filename_or_fp`: The filename or file pointer to convert. 
    37     - `format`: The format to convert to. 
    38     """ 
    39     if isinstance(filename_or_fp, basestring): 
    40         # If it's a string, assume it's a filename. 
    41         fs = FileStorage(filename_or_fp) 
    42         # Since it's an actual filesystem file, close the FileStorage after 
    43         # the entire conversion is done. 
    44         close_when_done = True 
    45     else: 
    46         # Otherwise, assume it's a file-like object. 
    47         fs = FileStorage(fp=filename_or_fp) 
    48         # Since StringIO and similar file-liike objects' values cannot be 
    49         # inspected after closing, do not explicitly close the FileStorage 
    50         # object when conversion is done. 
    51         close_when_done = False 
     35def convert_format(filename, backend_name=None, backend_args={}, format=None): 
     36    """Convert database to a new internal structure format. 
     37 
     38    - `filename`: Filename of the database to convert. 
     39    - `backend_name`: (optional) Name of the backend to use when 
     40      opening the database. 
     41    - `backend_args`: (optional) Arguments to pass to the backend. 
     42    - `format`: (optional) Format to convert internal structure to. 
     43      If not given, the most recent format available will be used. 
     44    """ 
     45    backend = new_backend(filename, backend_name, backend_args) 
    5246    # Check the format of the database. 
    53     conn = Connection(fs) 
    54     root = conn.get_root() 
     47    root = backend.get_root() 
    5548    # XXX: Better error checking might be handy. 
     49    # XXX: So might data structure verification. 
    5650    schevo = root['SCHEVO'] 
    5751    original_format = schevo['format'] 
     52    if format is None: 
     53        format = max(format_converter) 
    5854    # Convert one version at a time, ensuring that failures result in a 
    5955    # rollback to the database's original state. 
    6056    try: 
    61         try: 
    62             for new_format in xrange(original_format + 1, format + 1): 
    63                 converter = format_converter[new_format] 
    64                 converter(conn) 
    65         except: 
    66             conn.abort() 
    67             raise 
    68         else: 
    69             conn.commit() 
    70     finally: 
    71         if close_when_done: 
    72             fs.close() 
     57        for new_format in xrange(original_format + 1, format + 1): 
     58            converter = format_converter[new_format] 
     59            converter(backend) 
     60    except: 
     61        backend.abort() 
     62        raise 
     63    else: 
     64        backend.commit() 
     65 
     66 
     67def copy(src_filename, dest_filename, dest_backend_name, dest_backend_args={}): 
     68    """Copy internal structures verbatim from a source database to a 
     69    new destination database. 
     70 
     71    To see progress of the copy operation, turn on tracing as 
     72    described in `schevo.trace` 
     73 
     74    - `src_filename`: Filename of the source database.  Schevo must be 
     75      able to open the source database using backend autodetection, 
     76      and by using default backend arguments. 
     77    - `dest_filename`: Filename of the destination database. This file 
     78      may exist if it is in the format used by the destination 
     79      backend, but must not already contain a Schevo database. 
     80    - `dest_backend_name`: Name of the backend to use when creating 
     81      the destination database. 
     82    - `dest_backend_args`: (optional) Arguments to pass to the 
     83      backend. 
     84    """ 
     85    src_backend = new_backend(src_filename) 
     86    # Make sure the source backend is in the proper format. 
     87    assert log(1, 'Checking source', src_filename) 
     88    src_root = src_backend.get_root() 
     89    if 'SCHEVO' not in src_root: 
     90        src_backend.close() 
     91        raise DatabaseDoesNotExist( 
     92            'No schevo database in file %r.' % src_filename) 
     93    if src_root['SCHEVO']['format'] != 2: 
     94        src_backend.close() 
     95        raise DatabaseFormatMismatch('Database must be in format 2.') 
     96    # Make sure the destination backend does not have a database. 
     97    assert log(1, 'Checking destination', src_filename) 
     98    dest_backend = new_backend( 
     99        dest_filename, dest_backend_name, dest_backend_args) 
     100    dest_root = dest_backend.get_root() 
     101    if 'SCHEVO' in dest_root: 
     102        src_backend.close() 
     103        dest_backend.close() 
     104        raise DatabaseAlreadyExists( 
     105            'Schevo database already in file %r.' % dest_filename) 
     106    assert log(1, 'Start copying structures.') 
     107    d_btree = dest_backend.BTree 
     108    d_pdict = dest_backend.PDict 
     109    d_plist = dest_backend.PList 
     110    s_btree = src_backend.BTree 
     111    assert log(2, 'Creating SCHEVO key.') 
     112    src_SCHEVO = src_root['SCHEVO'] 
     113    dest_SCHEVO = dest_root['SCHEVO'] = d_pdict() 
     114    assert log(2, 'Copying lightweight structures.') 
     115    if 'label' in src_SCHEVO: 
     116        dest_SCHEVO['label'] = src_SCHEVO['label'] 
     117    dest_SCHEVO['format'] = 2 
     118    dest_SCHEVO['version'] = src_SCHEVO['version'] 
     119    dest_SCHEVO['schema_source'] = src_SCHEVO['schema_source'] 
     120    dest_SCHEVO['extent_name_id'] = d_pdict( 
     121        src_SCHEVO['extent_name_id'].iteritems()) 
     122    assert log(2, 'Copying extents.') 
     123    dest_extents = dest_SCHEVO['extents'] = d_pdict() 
     124    def copy_btree(src): 
     125        """Used for copying indices structure.""" 
     126        copy = d_btree() 
     127        for key, value in src.iteritems(): 
     128            if isinstance(value, s_btree): 
     129                value = copy_btree(value) 
     130            copy[key] = value 
     131        return copy 
     132    for extent_id, src_extent in src_SCHEVO['extents'].iteritems(): 
     133        extent_name = src_extent['name'] 
     134        assert log(2, 'Creating extent', extent_name) 
     135        dest_extent = dest_extents[extent_id] = d_pdict() 
     136        assert log(2, 'Copying lightweight structures for', extent_name) 
     137        dest_extent['entity_field_ids'] = src_extent['entity_field_ids'] 
     138        dest_extent['field_id_name'] = d_pdict( 
     139            src_extent['field_id_name'].iteritems()) 
     140        dest_extent['field_name_id'] = d_pdict( 
     141            src_extent['field_name_id'].iteritems()) 
     142        dest_extent['id'] = src_extent['id'] 
     143        dest_extent['len'] = src_extent['len'] 
     144        dest_extent['name'] = src_extent['name'] 
     145        dest_extent['next_oid'] = src_extent['next_oid'] 
     146        assert log(2, 'Copying', len(src_extent['entities']), 'entities in', 
     147            extent_name) 
     148        dest_entities = dest_extent['entities'] = d_btree() 
     149        for entity_oid, src_entity in src_extent['entities'].iteritems(): 
     150            assert log(3, 'Copying', entity_oid) 
     151            dest_entity = dest_entities[entity_oid] = d_pdict() 
     152            dest_entity['rev'] = src_entity['rev'] 
     153            dest_entity['fields'] = d_pdict(src_entity['fields'].iteritems()) 
     154            dest_entity['link_count'] = src_entity['link_count'] 
     155            src_links = src_entity['links'] 
     156            dest_links = dest_entity['links'] = d_pdict() 
     157            for key, value in src_links.iteritems(): 
     158                links = dest_links[key] = d_btree() 
     159                links.update(src_links[key].iteritems()) 
     160            dest_entity['related_entities'] = d_pdict( 
     161                src_entity['related_entities'].iteritems()) 
     162        assert log(2, 'Copying indices for', extent_name) 
     163        dest_extent['index_map'] = d_pdict(src_extent['index_map'].iteritems()) 
     164        dest_extent['normalized_index_map'] = d_pdict( 
     165            src_extent['normalized_index_map'].iteritems()) 
     166        dest_indices = dest_extent['indices'] = d_btree() 
     167        for index_spec, src_index_data in src_extent['indices'].iteritems(): 
     168            unique, src_index_tree = src_index_data 
     169            dest_indices[index_spec] = (unique, copy_btree(src_index_tree)) 
     170        assert log(2, 'Done copying', extent_name, '-- committing to disk') 
     171        dest_backend.commit() 
     172    # Finalize. 
     173    assert log(1, 'Close source.') 
     174    src_backend.close() 
     175    assert log(1, 'Pack destination.') 
     176    dest_backend.pack() 
     177    assert log(1, 'Close destination.') 
     178    dest_backend.close() 
     179 
     180 
     181def create(filename, backend_name, backend_args={}, 
     182           schema_source=None, schema_version=None, initialize=True, 
     183           format=None, label=u'Schevo Database'): 
     184    """Create a new database and return it. 
     185 
     186    - `filename`: Filename of the new database. 
     187    - `backend_name`: Name of the backend to use when creating the 
     188      database. 
     189    - `backend_args`: (optional) Arguments to pass to the backend. 
     190    - `schema_source`: (optional) Schema source code to synchronize 
     191      the new database with. If `None` is given, the database will 
     192      exist but will contain no extents. 
     193    - `schema_version`: (optional) Version of the schema being used to 
     194      create the database.  If `None` is given, `1` is assumed. 
     195    - `initialize`: `True` (default) if the new database should be 
     196      populated with initial values defined in the schema. 
     197    - `format`: (optional) Internal structure format to use.  If 
     198      `None` is given, the latest format will be used. 
     199    - `label`: (optional) The label to give the new database. 
     200    """ 
     201    backend = new_backend(filename, backend_name, backend_args) 
     202    # Make sure the database doesn't already exist. 
     203    root = backend.get_root() 
     204    if 'SCHEVO' in root: 
     205        backend.close() 
     206        raise DatabaseAlreadyExists( 
     207            'Use evolve, update, or open on this database.') 
     208    # Continue creating the new database. 
     209    Database = format_dbclass[format] 
     210    db = Database(backend) 
     211    db._sync( 
     212        schema_source = schema_source, 
     213        schema_version = schema_version, 
     214        initialize = initialize, 
     215        ) 
     216    # Apply label. 
     217    relabel(db, label) 
     218    # Install icon support. 
     219    icon.install(db) 
     220    db._on_open() 
     221    return db 
    73222 
    74223 
    …… 
    131280 
    132281 
    133 def inject(filename, schema_source, version): 
    134     """Inject a new schema and schema version into a database file. DANGEROUS! 
     282def inject(filename, schema_source, version, 
     283           backend_name=None, backend_args=None): 
     284    """Inject a new schema and schema version into a database 
     285    file. DANGEROUS! 
    135286 
    136287    PLEASE USE WITH CAUTION; this is not intended to be used in normal course 
    …… 
    143294 
    144295    - `filename`: Filename of database to inject new schema into. 
    145     - `schema_source`: The new schema source to inject into the database. 
    146     - `version`: The new version number of the database schema to inject. 
    147     """ 
    148     fs = FileStorage(filename) 
    149     conn = Connection(fs) 
    150     root = conn.get_root() 
     296    - `schema_source`: The new schema source to inject into the 
     297      database. 
     298    - `version`: The new version number of the database schema to 
     299      inject. 
     300    - `backend_name`: (optional) Name of the backend to use when 
     301      opening the database, if auto-detection is not desired. 
     302    - `backend_args`: (optional) Arguments to pass to the backend. 
     303    """ 
     304    backend = new_backend(filename, backend_name, backend_args) 
     305    root = backend.get_root() 
    151306    schevo = root['SCHEVO'] 
    152307    schevo['schema_source'] = schema_source 
    153308    schevo['version'] = version 
    154     conn.commit() 
    155     fs.close() 
    156  
    157  
    158 def open(filename=None, schema_source=None, schema_version=None, 
    159          initialize=True, label='', fp=None, cache_size=100000, 
    160          format_for_new=None): 
    161     """Return an open database by opening an existing database or creating 
    162     a new one. 
    163  
    164     If a new database is created, it is assumed that the `schema_source` given 
    165     is schema version 1. 
    166  
    167     - `filename`: Filename of database to create or open, or None if using `fp`. 
    168     - `schema_source`: Schema source to create a new database with, or None 
    169       if not creating a new database. 
    170     - `schema_version`: Schema version to create a new database with, if 
    171       skipping evolution from version 1. 
    172     - `initialize`: If True, will create initial values in the database if 
    173       creating a new database. 
    174     - `label`: The label of the database, to be used for a return value when 
    175       the open database instance is passed to `schevo.label.label`. 
    176     - `fp`: A file-like object, such as a StringIO instance, to use instead of 
    177       opening a file in the filesystem.  None if using `filename`. 
    178     - `cache_size`: The number of Python objects that the stoage backend 
    179       will cache in memory before re-ghosting old objects. 
    180     - `format_for_new`: The version number of the database format to use when 
    181       creating a new database, or None to use the default format. 
    182     """ 
    183     # Create a FileStorage instance for the database based on `fp` or 
    184     # `filename` arguments. 
    185     if fp is not None: 
    186         fs = FileStorage(fp=fp) 
     309    backend.commit() 
     310    backend.close() 
     311 
     312 
     313def new_backend(filename, backend_name=None, backend_args=None): 
     314    """Return a new database backend instance for a file. 
     315 
     316    - `filename`: Name of the file to open with the backend. 
     317    - `backend_name`: Name of the backend to use.  If the file already 
     318      exists and `None` is given, then the backend is auto-detected if 
     319      possible, or an exception is raised.  If the file does not 
     320      already exist and `None` is given, an exception is raised. 
     321    - `backend_args`: (optional) Arguments to pass to the backend. 
     322    """ 
     323    from schevo.backend import backends 
     324    if backend_name: 
     325        # Determine backend class by name. 
     326        BackendClass = backends[backend_name] 
     327        additional_args = {} 
    187328    else: 
    188         fs = FileStorage(filename) 
    189     # Create a Connection instance for the file storage, with a specific 
    190     # cache size. 
    191     conn = Connection(fs, cache_size) 
     329        # In absence of name, determine backend class by file. 
     330        BackendClass = None 
     331        for BC in backends.itervalues(): 
     332            usable = BC.usable_by_backend(filename) 
     333            if usable: 
     334                BackendClass = BC 
     335                usable, additional_args = usable 
     336                break 
     337    if BackendClass is None: 
     338        raise IOError('No suitable backends found for %r' % filename) 
     339    # Convert backend args to a dictionary. 
     340    if backend_args is None: 
     341        backend_args = {} 
     342    elif isinstance(backend_args, basestring): 
     343        backend_args = BackendClass.args_from_string(backend_args) 
     344    backend_args.update(additional_args) 
     345    return BackendClass(filename, **backend_args) 
     346 
     347 
     348def open(filename, backend_name=None, backend_args={}): 
     349    """Open an existing database and return it. 
     350 
     351    - `filename`: Name of the file containing the database. 
     352    - `backend_name`: (optional) Name of the backend to use if 
     353      auto-detection is not desired. 
     354    - `backend_args`: (optional) Arguments to pass to the backend. 
     355    """ 
     356    backend = new_backend(filename, backend_name, backend_args) 
     357    # Make sure the database already exists. 
     358    root = backend.get_root() 
     359    if 'SCHEVO' not in root: 
     360        backend.close() 
     361        raise DatabaseDoesNotExist('Use create to create a new database.') 
    192362    # Determine the version of the database. 
    193     root = conn.get_root() 
    194     if 'SCHEVO' in root: 
    195         schevo = root['SCHEVO'] 
    196         format = schevo['format'] 
    197     else: 
    198         format = format_for_new 
     363    schevo = root['SCHEVO'] 
     364    format = schevo['format'] 
    199365    # Determine database class based on format number. 
    200366    Database = format_dbclass[format] 
    201367    # Create the Database instance. 
    202     db = Database(conn) 
    203     if label: 
    204         db.label = label 
    205     # Synchronize it with the given schema source. 
    206     db._sync( 
    207         schema_source, schema_version=schema_version, initialize=initialize) 
     368    db = Database(backend) 
     369    db._sync() 
    208370    # Install icon support and finalize opening of database. 
    209371    icon.install(db) 

  • trunk/Schevo/schevo/database1.py

    r3306 r3509  
    1515from schevo.field import Entity as EntityField 
    1616from schevo.placeholder import Placeholder 
    17 from schevo.store.btree import BTree 
    18 from schevo.store.persistent_dict import PersistentDict as PDict 
    1917from schevo.trace import log 
    2018 
    …… 
    6563        links_created = [] 
    6664        lc_append = links_created.append 
     65        BTree = self._BTree 
     66        PDict = self._PDict 
    6767        try: 
    6868            if oid is None: 
    …… 
    111111                else: 
    112112                    relaxed = None 
    113                 _index_add(extent_map, index_spec, relaxed, oid, field_values) 
     113                _index_add(extent_map, index_spec, relaxed, oid, field_values, 
     114                           BTree) 
    114115                ia_append((extent_map, index_spec, oid, field_values)) 
    115116            # Update links from this entity to another entity. 
    …… 
    178179            deletes.update([(extent_name_id[del_entity_cls.__name__], del_oid) 
    179180                            for del_entity_cls, del_oid in tx._known_deletes]) 
    180         for (other_extent_id, other_field_id), others in links.items(): 
     181        for (other_extent_id, other_field_id), others in links.iteritems(): 
    181182            for other_oid in others: 
    182183                if (other_extent_id, other_oid) in deletes: 
    …… 
    291292            # Return all of them. 
    292293            assert log(2, 'Return all oids.') 
    293             return entity_maps.keys() 
     294            return list(entity_maps.keys()) 
    294295        extent_name_id = self._extent_name_id 
    295296        indices = extent_map['indices'] 
    296         assert log(3, 'indices.keys()', indices.keys()) 
    297297        normalized_index_map = extent_map['normalized_index_map'] 
    298         assert log(3, 'normalized_index_map.keys()', 
    299                    normalized_index_map.keys()) 
    300298        entity_field_ids = extent_map['entity_field_ids'] 
    301299        field_name_id = extent_map['field_name_id'] 
    …… 
    346344                if field_value not in branch: 
    347345                    # No matches found. 
    348                     assert log(3, field_value, 'not found in', branch.keys()) 
    349346                    match = False 
    350347                    break 
    …… 
    353350                # Now we're at a leaf that matches all of the 
    354351                # criteria, so return the OIDs in that leaf. 
    355                 results = branch.keys() 
     352                results = list(branch.keys()) 
    356353        else: 
    357354            # Fields aren't indexed, so use brute force. 
    …… 
    407404        lc_append = links_created.append 
    408405        ld_append = links_deleted.append 
     406        BTree = self._BTree 
    409407        try: 
    410408            # Get old values for use in a potential inversion. 
    …… 
    479477                else: 
    480478                    relaxed = None 
    481                 _index_add(extent_map, index_spec, relaxed, oid, field_values) 
     479                _index_add(extent_map, index_spec, relaxed, oid, field_values, 
     480                           BTree) 
    482481                ia_append((extent_map, index_spec, oid, field_values)) 
    483482            # Update links from this entity to another entity. 
    …… 
    520519                _index_remove(_e, _i, _o, _f) 
    521520            for _e, _i, _r, _o, _f in indices_removed: 
    522                 _index_add(_e, _i, _r, _o, _f) 
     521                _index_add(_e, _i, _r, _o, _f, BTree) 
    523522            for other_entity_map, links, link_key, oid in links_created: 
    524523                del links[link_key][oid] 
    …… 
    532531        """Create or update Schevo structures in the database.""" 
    533532        root = self._root 
    534         if 'SCHEVO' not in root.keys(): 
     533        PDict = self._PDict 
     534        if 'SCHEVO' not in root: 
    535535            schevo = root['SCHEVO'] = PDict() 
    536536            schevo['format'] = 1 

  • trunk/Schevo/schevo/database2.py

    r3382 r3509  
    2828import schevo.schema 
    2929from schevo.signal import TransactionExecuted 
    30 from schevo.store.btree import BTree 
    31 from schevo.store.persistent_dict import PersistentDict as PDict 
    32 from schevo.store.persistent_list import PersistentList as PList 
    3330from schevo.trace import log 
    3431from schevo.transaction import ( 
    …… 
    3734 
    3835class Database(base.Database): 
    39     """Schevo database, format 2, using schevo.store as an object store. 
     36    """Schevo database, format 2. 
    4037 
    4138    See doc/SchevoInternalDatabaseStructures.txt for detailed information on 
    …… 
    4340    """ 
    4441 
    45     label = 'Schevo Database' 
    46  
    4742    # By default, don't dispatch signals.  Set to True to dispatch 
    4843    # TransactionExecuted signals. 
    …… 
    5348    write_lock = dummy_lock 
    5449 
    55     def __init__(self, connection): 
     50    def __init__(self, backend): 
    5651        """Create a database. 
    5752 
    58         - `connection`: The Durus connection to use. 
     53        - `backend`: The storage backend instance to use. 
    5954        """ 
    60         self.connection = connection 
    61         self._root = connection.get_root() 
     55        self.backend = backend 
     56        self._BTree = backend.BTree 
     57        self._PDict = backend.PDict 
     58        self._PList = backend.PList 
     59        self._root = backend.get_root() 
    6260        # Shortcuts to coarse-grained commit and rollback. 
    63         self._commit = connection.commit 
    64         self._rollback = connection.abort 
     61        self._commit = backend.commit 
     62        self._rollback = backend.rollback 
    6563        # Keep track of schema modules remembered. 
    6664        self._remembered = [] 
    …… 
    9896            p.pop().close() 
    9997        assert log(1, 'Closing storage.') 
    100         self.connection.storage.close() 
    101         del self.connection 
     98        self.backend.close() 
    10299        remembered = self._remembered 
    103100        while remembered: 
    …… 
    227224        """Pack the database.""" 
    228225        if os.environ.get('SCHEVO_NOPACK', '').strip() != '1': 
    229             self.connection.pack() 
     226            self.backend.pack() 
    230227 
    231228    def populate(self, sample_name=''): 
    …… 
    245242    def version(self): 
    246243        return self._root['SCHEVO']['version'] 
     244 
     245    def _get_label(self): 
     246        SCHEVO = self._root['SCHEVO'] 
     247        if 'label' not in SCHEVO: 
     248            # Older database, no label stored in it. 
     249            return u'Schevo Database' 
     250        else: 
     251            return SCHEVO['label'] 
     252 
     253    def _set_label(self, new_label): 
     254        if self._executing: 
     255            raise error.DatabaseExecutingTransaction( 
     256                'Cannot change database label while executing a transaction.') 
     257        self._root['SCHEVO']['label'] = unicode(new_label) 
     258        self._commit() 
     259 
     260    label = property(_get_label, _set_label) 
     261    _label = property(_get_label, _set_label) 
    247262 
    248263    def _append_change(self, typ, extent_name, oid): 
    …… 
    327342        links_created = [] 
    328343        lc_append = links_created.append 
     344        BTree = self._BTree 
     345        PDict = self._PDict 
    329346        try: 
    330347            if oid is None: 
    …… 
    369386                else: 
    370387                    relaxed = None 
    371                 _index_add(extent_map, index_spec, relaxed, oid, field_values) 
     388                _index_add(extent_map, index_spec, relaxed, oid, field_values, 
     389                           BTree) 
    372390                ia_append((extent_map, index_spec, oid, field_values)) 
    373391            # Update links from this entity to another entity. 
    …… 
    438456            deletes.update([(extent_name_id[del_entity_cls.__name__], del_oid) 
    439457                            for del_entity_cls, del_oid in tx._known_deletes]) 
    440         for (other_extent_id, other_field_id), others in links.items(): 
     458        for (other_extent_id, other_field_id), others in links.iteritems(): 
    441459            for other_oid in others: 
    442460                if (other_extent_id, other_oid) in deletes: 
    …… 
    472490                    other_extent_map = extent_maps_by_id[other_extent_id] 
    473491                    if other_oid in other_extent_map['entities']: 
    474                         other_entity_map = other_extent_map['entities'][other_oid] 
     492                        other_entity_map = other_extent_map[ 
     493                            'entities'][other_oid] 
    475494                        links = other_entity_map['links'] 
    476495                        other_links = links[link_key] 
    …… 
    528547        # If no more transactions have relaxed this index, enforce it. 
    529548        if not txns: 
     549            BTree = self._BTree 
    530550            for _extent_map, _index_spec, _oid, _field_values in added: 
    531                 _index_validate(_extent_map, _index_spec, _oid, _field_values) 
     551                _index_validate(_extent_map, _index_spec, _oid, _field_values, 
     552                                BTree) 
    532553 
    533554    def _entity(self, extent_name, oid): 
    …… 
    599620            btree = entity_links.get(key, {}) 
    600621            if return_count: 
    601                 count = len(btree.keys()) # XXX Optimization opportunity. 
    602                 assert log(2, 'returning len(btree.keys())', count) 
     622                count = len(btree) 
     623                assert log(2, 'returning len(btree)', count) 
    603624                return count 
    604625            el