_ ____ ___ ____ / \ | _ \_ _| | _ \ ___ ___ ___ / _ \ | |_) | | | | | |/ _ \ / __/ __| / ___ \| __/| | | |_| | (_) | (__\__ \ /_/ \_\_| |___| |____/ \___/ \___|___/
Copyright ©2020 - Mad Penguin Consulting Limited
class Manager (UserDict ):
Manager is a dictionary like object the holds references to all the databases currently registered with the running instance. If you only ever reference one database then technically you can skip this object and just use the Database object.
METHODS
def __getitem__(self, name) -> Database
Get a reference to an already open database Returns the Database object associated with the supplied name
PARAMETERS
  • name [str] - the name of database
def __init__(self) -> None
Initialise this object
    def database(self, name, [path], [config]) -> Database
    Open a database creating it if necessary Returns a reference to an open Database
    PARAMETERS
    • name [str] - an arbitrary name to reference the database by
    • path [str / default=None] - the path to the database files
    • config [Config / default=None] - a dictionary containing configuration specifics for this database
    def exit_handler(self) -> None
    Make sure all databases are closed

      class Database (UserDict ):
      The Database object models the top-level container for a collection of tables and indexes, each Database object maps to an LMDB database object. When you open a database there are a variety of low-level (LMDB) database settings that can be applied, it's worth understanding what some of them do as when you come to productionise your system, they will make a difference. Specifically you will want to tweak 'map_size' which is the maximum allowable size of your database, and possibly max_dbs if you are going to open large numbers of tables at the same time. sync: If False, don’t flush system buffers to disk when committing a transaction. This optimization means a system crash can corrupt the database or lose the last transactions if buffers are not yet flushed to disk. The risk is governed by how often the system flushes dirty buffers to disk and how often sync() is called. However, if the filesystem preserves write order and writemap=False, transactions exhibit ACI (atomicity, consistency, isolation) properties and only lose D (durability). I.e. database integrity is maintained, but a system crash may undo the final transactions. Note that sync=False, writemap=True leaves the system with no hint for when to write transactions to disk, unless sync() is called. map_async=True, writemap=True may be preferable. lock: If False, don’t do any locking. If concurrent access is anticipated, the caller must manage all concurrency itself. For proper operation the caller must enforce single-writer semantics, and must ensure that no readers are using old transactions while a writer is active. The simplest approach is to use an exclusive lock so that no readers may be active at all when a writer begins. subdir: If True, path refers to a subdirectory to store the data and lock files in, otherwise it refers to a filename prefix. create: False, do not create the directory path if it is missing. writemap: If True, use a writeable memory map unless readonly=True. This is faster and uses fewer mallocs, but loses protection rom application bugs like wild pointer writes and other bad updates into the database. Incompatible with nested transactions. Processes with and without writemap on the same environment do not cooperate well. metasync: If False, flush system buffers to disk only once per transaction, omit the metadata flush. Defer that until the system flushes files to disk, or next commit or sync(). This optimization maintains database integrity, but a system crash may undo the last committed transaction. I.e. it preserves the ACI (atomicity, consistency, isolation) but not D (durability) database property. readahead: If False, LMDB will disable the OS filesystem readahead mechanism, which may improve random read performance when a database is larger than RAM. map_async: When writemap=True, use asynchronous flushes to disk. As with sync=False, a system crash can then corrupt the database or lose the last transactions. Calling sync() ensures on-disk database integrity until next commit. max_readers: Maximum number of simultaneous read transactions. Can only be set by the first process to open an environment, as it affects the size of the lock file and shared memory area. Attempts to simultaneously start more than this many readtransactions will fail. max_dbs: Maximum number of databases available. If 0, assume environment will be used as a single database. map_size: Maximum size database may grow to; used to size the memory mapping. If database grows larger than map_size, an exception will be raised and the user must close and reopen Environment. On 64-bit there is no penalty for making this huge (say 1TB). Must be <2GB on 32-bit. Default values for these settings come from the CONFIG class variable, please note that currently we do NOT support the 'readonly' option. It does not appear that this option works properly with transactions on sub-databases and until we can work out why, please avoid read-only databases.
      CLASS PROPERTIES
      CONFIG =
      {
          "sync": true,
          "lock": true,
          "subdir": true,
          "create": true,
          "writemap": true,
          "metasync": false,
          "readahead": true,
          "map_async": true,
          "max_readers": 64,
          "max_dbs": 64,
          "map_size": 2147483648
      }
      PROPERTIES
      def isopen(self) -> bool
      Return True is the database is currently open
        def map_size(self) -> int
        Return the currently mapped database size
          def name(self) -> None
          Return the unique name (uuid) of this database for replication
            def read_transaction(self) -> TXN
            Return a read-only transaction for use with "with"
              def storage_allocated(self) -> None
              Return the amount of storage space pre-allocated to this database, assuming the underlying filesystem supports 'sparse' storage, this allocation will not reflect the amount of disk space 'actually' used. (see 'storage_used')
                def write_transaction(self) -> TXN
                Return a write-transaction for this database
                  METHODS
                  def __getitem__(self, name) -> Table
                  Shortcut to self.table
                  PARAMETERS
                  • name [str] - the name of the table to recover
                  def __init__(self) -> None
                  Initialise this object
                    def __repr__(self) -> str
                    Return a string representation of a Database object
                      def close(self) -> None
                      Close this database if it is open
                        def configure(self, config) -> Database
                        Adjust the database configuration Return a reference to the current database instance
                          def drop(self, name, [txn]) -> None
                          Drop (delete) a database table
                          PARAMETERS
                          • name [str] - name of table to drop
                          • txn [TXN / default=None] - an optional transaction
                          def open(self, path) -> Database
                          Open the database Returns a reference to the Database object
                          PARAMETERS
                          • path [str] - the location of the database files
                          def reopen(self) -> None
                          Reopen a database and all of it's tables After calling set_mapsize to resize the database, individual database handles are potentially invalidated, hence the save option is to reopen everything.
                            def storage_used(self, [txn]) -> None
                            Return a tuple that represents the amount of storage space consumed by this database. The first entry in the tuple is the number of bytes occupied by data, the second is a tuple representing the size (in bytes) and name of each table in the database.
                            PARAMETERS
                            • txn [TXN / default=None] - an optional transaction to wrap this operation
                            def sync(self, [force]) -> None
                            Force a database sync
                            PARAMETERS
                            • force [bool / default=True] - if True make the flush synchronous
                            def table(self, table_name, [codec], [integerkey], [txn]) -> Table
                            Returns the table associated with the supplied name
                            PARAMETERS
                            • table_name [str] - the name of the table to recover
                            def tables(self, [all], [txn]) -> None
                            Generate a list of tables available in this database
                            PARAMETERS
                            • all [bool / default=False] - if True also show hidden / structural tables
                            • txn [TXN / default=None] - an optional transaction

                            class Table (UserDict , Compression , Serialiser ):
                            The Table class is used to wrap access to individual database tables and incorporates semi-transparent compression / decompression on a per table basis. Compression libraries are pluggable and implemented in the Compression class. o APPEND_MODE - when appending a record we know that the new key will be the highest value in the table so we can take advantage of LMDB's "append" mode. If you never want to use this option, set pynndb2.Table.APPEND_MODE to False. NOTE: compatibility issue, we've switched to our own version of "ObjectId", if you set this to True and have data that uses both BSON/ObjectId and the new ObjectId, you will have a problem. (data loss)
                            CLASS PROPERTIES
                            APPEND_MODE = "False"
                            PROPERTIES
                            def isopen(self) -> bool
                            Return True if this table is open
                              def read_transaction(self) -> TXN
                              Use with "with" to begin a Read-Only transaction
                                def write_transaction(self) -> TXN
                                Return a write transaction for this database (compatibility only)
                                  METHODS
                                  def __init__(self, database, name) -> None
                                  Intantiate a table instance bases on the name of the table required. A reference to the containing database is also required so the table can back-reference the database environment.
                                  PARAMETERS
                                  • database [Database] - a reference to the containing database object
                                  • name [str] - the name of the table to reference
                                  def __repr__(self) -> str
                                  Generate a string representation of this object, by default we include the table name and the table status, i.e. whether it is open or not.
                                    def __setitem__(self, name, conf) -> None
                                    Create an entry for an index with the specified name
                                    PARAMETERS
                                    • name [str] - the name of the index to create
                                    • conf [Config] - configuration options for the index we're creating
                                    def append(self, doc, [txn]) -> Doc
                                    Append a new record to this table
                                    PARAMETERS
                                    • doc [Doc] - the data to append
                                    • txn [TXN | WriteTransaction / default=None] - an optional transaction object
                                    def close(self) -> None
                                    Close a table by essentially losing all references to it
                                      def delete(self, keyspec, [txn]) -> None
                                      Delete one or more records from the database based on a key specification that should reference one or more records by primary key.
                                      PARAMETERS
                                      • keyspec [OID | OIDS | Doc] - we accept either a key, a list of keys or a Doc, keys may be str or bytes
                                      • txn [TXN / default=None] - an optional transaction
                                      def drop(self, index_name, [txn]) -> None
                                      Drop an index from the current table
                                      PARAMETERS
                                      • index_name [str] - the name of the index to drop
                                      • txn [TXN | WriteTransaction / default=None] - an optional transaction
                                      def droptable(self, [txn]) -> None
                                      Drop the current table, this will empty the table, remove all the indexes, remote the table itself, and remove all associated metadat.
                                      PARAMETERS
                                      • txn [TXN | WriteTransaction / default=None] - a write transaction to wrap this operation
                                      def empty(self, [txn]) -> None
                                      Remove all data from the current table leaving the indexing structure in-tact
                                      PARAMETERS
                                      • txn [TXN | WriteTransaction / default=None] - an optional transaction
                                      def ensure(self, index_name, func, [duplicates], [force], [txn]) -> Index
                                      Ensure that the specified index exists, if it does by default do nothing. If the index does not exist, or if the 'force' flag is true, the index will be (re)created using the new index function. The "func" parameter can take one of two forms, it can either be a Python format string (the only option in v1) or it can be a complete python function if prefixed with a 'def'. So for example as a format string;
                                      func = '{name}'         # index by name
                                      func = '{name}|{age}'   # index by name + age
                                      func = '{age:03d}'      # index by age with leading zero for correct numerical sort order
                                      
                                      Or if you want to use a function which allows for more flexibility;
                                      func = 'def func(doc): return "{:03d}".format(doc["age"]).encode()'
                                      
                                      For a complete working example, the natural order is in descending on age, but when iterating using either of the example indexes, you should see the order as ascending order of age.
                                      #!/usr/bin/env python
                                      from pynndb import Manager, Doc
                                      from shutil import rmtree
                                      rmtree('.database')
                                      db = Manager().database('database', '.database')
                                      people = db.table('people')
                                      people.append(Doc({'name': 'Tom', 'age': 21}))
                                      people.append(Doc({'name': 'Harry', 'age': 19}))
                                      people.ensure('by_age_fs', '{age:03d}')
                                      people.ensure('by_age_func', 'def func(doc): return "{:03d}".format(doc["age"]).encode()')
                                      [print(person.doc) for person in people.find()]
                                      print('--')
                                      [print(person.doc) for person in people.find('by_age_fs')]
                                      print('--')
                                      [print(person.doc) for person in people.find('by_age_func')]
                                      
                                      PARAMETERS
                                      • index_name [str] - the name of the required index
                                      • func [str] - a description of how index keys should be generated
                                      • duplicates [bool / default=False] - whether this is a duplicate index or not
                                      • force [bool / default=False] - whether to re-index the index if it already exists
                                      • txn [TXN | WriteTransaction / default=None] - an optional transaction
                                      def filter(self, [index_name], [lower], [upper], [expression], [context], [page_size], [inclusive], [suppress_duplicates], [txn]) -> None
                                      * Paging To use the paging function, you need to supply both a page_size and context. If the context is None and the page_size is positive, paging will start from the begininning of theindex, and if the context is None and the page size is negative, paging will start from the end. A positive page_size moves forwards through the index based on the context and a negative page size moves backwards. For forward paging, the context will be the last result from the previous page, and for moving backwards, the context will be the first result from the previous page. * lambda The "expression" parameter should be a lambda (or function) which will receive the document and return a True/False based on whether the document should be included in the search results. For example;
                                          filter(expression=lambda doc: doc['age'] > 19)
                                      
                                      PARAMETERS
                                      • index_name [str / default=None] - the name of an index to search on, or None to use the primary key
                                      • lower [Doc / default=None] - the record at the lower end of the range
                                      • upper [Doc / default=None] - the record at the upper end of the range
                                      • expression [Callable[[Doc], bool] / default=None] - an lambda expression to filter the results
                                      • context [FilterResult / default=None] - a paging context to determine where to start tge next page (see comments)
                                      • page_size [int / default=0] - maximum number of records to return
                                      • inclusive [bool / default=True] - if set to True, include the keys at each end, i.e. use <=|=> rather than <|>
                                      • suppress_duplicates [bool / default=False] - no duplicate keys, return only unique key values and ignore duplicates
                                      • txn [TXN / default=None] - an optional transaction
                                      def find(self, [index_name], [expression], [limit], [txn]) -> None
                                      Find records in this table either in natural (date) order, or in index order
                                      PARAMETERS
                                      • index_name [str / default=None] - an optional index name for ordering
                                      • expression [str / default=None] - the expression to filter the sort on
                                      • limit [int / default=MAXSIZE] - the maximum number of records to return
                                      • txn [TXN / default=None] - an optional transaction
                                      def first(self, [index_name], [txn]) -> None
                                      Return the first record in the table or None if there are no records
                                      PARAMETERS
                                      • index_name [str / default=None] - the name of the index to use (defaults to primary)
                                      • txn [TXN / default=None] - an optional transaction
                                      def get(self, oid, [txn]) -> Doc
                                      Recover a single record from the database based on it's primary key
                                      PARAMETERS
                                      • oid [None] - primary key of record to recover
                                      • txn [TXN | WriteTransaction / default=None] - an optional active transaction
                                      def indexes(self, [txn]) -> None
                                      Generate a list if indexs (names) available for this table
                                      PARAMETERS
                                      • txn [TXN / default=None] - an optional transaction
                                      def last(self, [index_name], [txn]) -> None
                                      Return the last record in the table or None if there are no records
                                      PARAMETERS
                                      • index_name [str / default=None] - the name of the index to use (defaults to primary)
                                      • txn [TXN / default=None] - an optional transaction
                                      def open(self, [compression_type], [compression_level], [codec], [integerkey], [txn]) -> Table
                                      Open this table and make it available for use, if the compression type is set to anything other than NONE, the following the call the table will be set to read and write data using the selected compression mechanism, and any data in the table will be compressed.
                                      PARAMETERS
                                      • compression_type [CompressionType / default=CompressionType.NONE] - the type of compression to use
                                      • compression_level [int / default=None] - the compression level to set
                                      • txn [TXN | WriteTransaction / default=None] - an optional transaction to wrap this request
                                      def range(self, [index_name], [lower], [upper], [keyonly], [inclusive], [limit], [page_number], [nodups], [txn]) -> None
                                      Find all records within a range of keys, optionally including keys at each end and optionally returning just the keys rather than the entire record.
                                      PARAMETERS
                                      • index_name [str / default=None] - an optional index name, if no index is supplied, use primary keys
                                      • lower [Doc / default=None] - the record at the lower end of the range
                                      • upper [Doc / default=None] - the record at the upper end of the range
                                      • keyonly [bool / default=False] - if set to True, only returns keys rather than the entire records
                                      • inclusive [bool / default=True] - if set to True, include the keys at each end, i.e. use <=|=> rather than <|>
                                      • limit [int / default=MAXSIZE] - maximum number of records to return
                                      • page_number [int / default=0] - index of the page (starts from 0) with a size `limit` to return results from
                                      • nodups [bool / default=False] - no duplicate keys, return only unique key values and ignore duplicates
                                      • txn [TXN / default=None] - an optional transaction
                                      def records(self, [txn]) -> int
                                      Return the number of records in this table
                                      PARAMETERS
                                      • txn [TXN / default=None] - an transaction to wrap the operation
                                      def reindex(self, index_name, [txn]) -> None
                                      Reindex the named index, assuming the index exists. The index is first emptied and then each record is reindexed, for a large table this can take some time and will lock the database while in progress.
                                      PARAMETERS
                                      • index_name [str] - the name of the index to reindex
                                      • txn [TXN / default=None] - a write transaction to wrap the operation
                                      def reopen(self, [txn]) -> None
                                      ReOpen a table, used following a change to the map size. Everything should be the same, we just need new database handles.
                                        def replicated(self, txn) -> None
                                        Return True if the current transaction is going to be marked for replication
                                          def save(self, doc, [txn]) -> None
                                          Update the current record in the table
                                          PARAMETERS
                                          • doc [Doc] - the record to update
                                          • txn [TXN | WriteTransaction / default=None] - an optional transaction
                                          def seek(self, index_name, doc, [limit], [page_number], [keyonly], [txn]) -> None
                                          Return a selection of records from the selected table matching the template record provided in "doc". Should return a maximum of 1 record for unique indexes.
                                          PARAMETERS
                                          • index_name [str] - the name of the index to search
                                          • doc [Doc] - the template record to find
                                          • limit [int / default=MAXSIZE] - the maximum number of results to return
                                          • page_number [int / default=0] - index of the page (starts from 0) with a size `limit` to return results from
                                          • keyonly [bool / default=False] - return a Cursor object relating to the key instead of the data item
                                          • txn [TXN / default=None] - an optional transaction
                                          def seek_one(self, index_name, doc, [txn]) -> Doc
                                          Find the first matching record from an index
                                          PARAMETERS
                                          • index_name [str] - the name of the index to search
                                          • doc [Doc] - the template record to find
                                          • txn [TXN / default=None] - an optional transaction
                                          def storage_used(self, [txn]) -> int
                                          Return the amount of storage space used by data contained within this table
                                          PARAMETERS
                                          • txn [TXN / default=None] - optional transaction to wrap this operation
                                          def tail(self, [key], [txn]) -> None
                                          Generates a sequence of records starting from the key after the primary key supplied. If no key is supplied, all records are returned, if a misssing key is supplied, no records are returned. Typically use this against the last-seen key to access new keys since the last check. TODO: add index to allow tailing based on indexes
                                          PARAMETERS
                                          • key [OID / default=None] - the key to start from
                                          • txn [TXN / default=None] - an optional transaction

                                          class Index
                                          The Index class models individual indexes of which a table may have zero or more. There are two crucial parameters supplied when setting up an index and are passed via "conf". o dupsort this controls whether the index will allow duplicate values to be inserted o func this is the anonymous function used to generate key values from the date record
                                          METHODS
                                          def __init__(self, table, name, conf) -> None
                                          When we instantiate an Index we primarily need a back-reference to the database we're working with, the name of the index we're instantiating, and a definition of the indexing function. The indexing function is held in the 'func' item in the conf dictionary. By default the indexing function will be a python format string, hoever you can also supply a 'proper' python function if you prefix it with 'def '
                                          PARAMETERS
                                          • database [None] - a reference to the database we're working with
                                          • name [str] - the name of the index we're creating a reference to
                                          • conf [Config] - a configuration dict containing information specific to this dictionary
                                          def anonymous(text) -> Callable
                                          An function used to generate anonymous functions for database indecies
                                          PARAMETERS
                                          • text [str] - a Python lambda function
                                          def anonymous_full(text) -> Callable
                                          An function used to generate anonymous functions for database indecies
                                          PARAMETERS
                                          • text [str] - a Python lambda function
                                          def delete(self, doc, txn) -> None
                                          Delete an entry from this index
                                          PARAMETERS
                                          • doc [Doc] - record associated with the index entry to delete
                                          • txn [TXN] - an optional transaction
                                          def drop(self, txn) -> None
                                          Remove all entries from this index and then remove the index
                                          PARAMETERS
                                          • txn [TXN] - an optional transaction
                                          def empty(self, txn) -> None
                                          Remove all entries from this index
                                          PARAMETERS
                                          • txn [TXN] - an optional transaction
                                          def get(self, doc, txn) -> None
                                          Get an entry from this index
                                          PARAMETERS
                                          • doc [Doc] - the record template for the data to retrieve
                                          • txn [TXN] - an optional transaction
                                          def index_path(table_name, index_name) -> str
                                          Produce an index "path" name for this index based on the table name and index name
                                            def map_key(self, doc) -> str
                                            Return the key derived from the supplied record for this particular index
                                            PARAMETERS
                                            • doc [Doc] - the record from which we want to derive a key
                                            def open(self, txn) -> None
                                            Open the index and make it available to the table txn = an optional transaction
                                              def put(self, doc, txn) -> None
                                              Put a new entry in this index, used when createing new records
                                              PARAMETERS
                                              • doc [Doc] - the document associated with this index entry
                                              • txn [TXN] - an optional transaction
                                              def put_cursor(self, cursor, txn) -> None
                                              Put a new index entry based on a Cursor rather than a Doc object. This is here mainly to make "reindex" more elegant / readable.
                                              PARAMETERS
                                              • cursor [Cursor] - an LMDB Cursor object
                                              • txn [TXN] - an optional transaction
                                              def records(self, [txn]) -> int
                                              Return the number of records in this index
                                              PARAMETERS
                                              • txn [TXN / default=None] - an optional transaction
                                              def save(self, old_doc, new_doc, txn) -> None
                                              Update a pre-existing index entry, we need both the old version of the record in order to remove the old index entries, and the new record to generate and insert the new ones.
                                              PARAMETERS
                                              • old_doc [Doc] - the previous version of the record
                                              • new_doc [Doc] - the new version of the record
                                              • txn [TXN] - an optional transaction

                                              class JournalType (Enum ):
                                              CLASS PROPERTIES
                                              APPEND = "A"
                                              CREATE = "C"
                                              DELETE = "D"
                                              ENSURE = "E"
                                              REMOVE = "R"
                                              SNAPXX = "S"
                                              UPDATE = "U"
                                              class Doc
                                              The Doc object is used to encapsulate data records before they are written to the database and after they are read from the database. It performs a number of crucial functions including; o Associating a data buffer with a key (OID) o Tracking updated attributes between the last read record and the current buffer o Tracking deleted attributes o Serialising and deserialising data between the database and user buffers o Providing a dict and object-like interfaces to interact with the data Note that this interface is 'slower' than using a raw 'dict' as in version 1, however the use of 'Doc' facilitates the new 'C' extension (which should be a drop-in replacement for 'Doc') which should be much faster, and this in turn facilitates the new GOBJECT serialisation mechanism that should provide a 2-10x performance increase.
                                              PROPERTIES
                                              def changed(self) -> bool
                                              Return True is the Doc has changed since it was read from the database. Changes are indicated by something in (rem) which signifies a field has been deleted or something in (upd) which signifies something has been added or updated.
                                                def doc(self) -> None
                                                Return the new buffer as a dictionary, i.e. the old + updates
                                                  def key(self) -> str
                                                  Return the current key (oid) for this document
                                                    METHODS
                                                    def __bool__(self) -> None
                                                    Always return True for a Doc object
                                                      def __contains__(self, key) -> bool
                                                      Return True if the field exists in this instance.
                                                      PARAMETERS
                                                      • key [str] - the key to look for
                                                      def __delitem__(self, key) -> None
                                                      Delete the data item associated with key
                                                      PARAMETERS
                                                      • key [str] - data item name
                                                      def __eq__(self, o) -> None
                                                      Equal operator for a Doc object
                                                        def __getattr__(self, key) -> Any
                                                        Return the data item associated with 'key' (dropping the '_' prefix)
                                                        PARAMETERS
                                                        • key [str] - data item name prefixed with an underscore
                                                        def __getitem__(self, key) -> Any
                                                        Return the data item associated with 'key'
                                                        PARAMETERS
                                                        • key [str] - data item name
                                                        def __init__(self, [doc], [oid], [dat], [integerkey]) -> None
                                                        Instantiate a Doc object either based on a Dict or Cursor object. In the event a Cursor object is supplied, the (upd) field is set to cursor.value() and (oid) is set to cursor.key(). If a dict is supplied, (upd) is set to the value of the dict supplied, and oid is either set to the supplied oid or left as None by default.
                                                        PARAMETERS
                                                        • doc [Dict / default=None] - the data to associate with this object
                                                        • oid [bytes | str / default=None] - a primary key to associate with this object
                                                        def __len__(self) -> int
                                                        Return the current length of the output buffer
                                                          def __ne__(self, o) -> None
                                                          Not Equal operator for a Doc object
                                                            def __repr__(self) -> str
                                                            Return a string representation of the Doc object
                                                              def __setattr__(self, key, val) -> None
                                                              Set the data item associated with 'key' (dropping the '_' prefix)
                                                              PARAMETERS
                                                              • key [str] - data item name prefixed with an underscore
                                                              • val [Any] - actual data item to associate with key
                                                              def __setitem__(self, key, val) -> None
                                                              Set the data item associated with 'key' (dropping the '_' prefix)
                                                              PARAMETERS
                                                              • key [str] - data item name prefixed with an underscore
                                                              • val [Any] - actual data item to associate with key
                                                              def get(self, table, txn) -> None
                                                              Populate this structure by reading data from the database Returns a reference to self
                                                              PARAMETERS
                                                              • db [None] - open database handle
                                                              • txn [TXN] - active database transaction
                                                              def items(self) -> ItemsView
                                                              Return the equivalent of dict.items() for our current buffer
                                                                def journal_entry(self, jtype, table_name) -> None
                                                                Return a Journal entry for the current object
                                                                  def keys(self) -> KeysView
                                                                  Return the equivalent of dict.keys() for our current buffer
                                                                    def pop(self, key) -> None
                                                                    Delete the named attribute
                                                                      def put(self, table, [append], [txn]) -> None
                                                                      Store the current output buffer back in the database
                                                                      PARAMETERS
                                                                      • db [None] - open database handle
                                                                      • append [bool / default=False] - whether to use "append" mode
                                                                      • txn [TXN / default=None] - active database transaction
                                                                      def snapshot_entry(self, jtype, table_name) -> None
                                                                      Return a Snapshot entry for the current object
                                                                        def values(self) -> ValuesView
                                                                        Return the equivalent of dict.values() for our current buffer

                                                                          ___ _ _ _ ___ _ / _ \| |__ (_) ___ ___| |_|_ _|__| | | | | | '_ \| |/ _ \/ __| __|| |/ _` | | |_| | |_) | | __/ (__| |_ | | (_| | \___/|_.__// |\___|\___|\__|___\__,_| |__/
                                                                          Copyright ©2020 - Mad Penguin Consulting Limited
                                                                          class ObjectId
                                                                          An ObjectId is a 14-byte unique identifier consisting of: - a 8-byte value representing the nano seconds since the Unix epoch - a 4-byte machine identifier - a 2-byte process id This code is based on the ObjectId class found in the Python BSON module.
                                                                          PROPERTIES
                                                                          def binary(self) -> bytes
                                                                          14-byte binary representation of this ObjectId.
                                                                            def generation_time(self) -> float
                                                                            Return the generation time of this ObjectId as a float (timestamp)
                                                                              METHODS
                                                                              def __eq__(self, other) -> bool
                                                                              Boolean Equality check: EQ
                                                                                def __ge__(self, other) -> bool
                                                                                Boolean Equality check: GE
                                                                                  def __getstate__(self) -> bytes
                                                                                  Return value of object for pickling. needed explicitly because __slots__() defined.
                                                                                    def __gt__(self, other) -> bool
                                                                                    Boolean Equality check: GT
                                                                                      def __hash__(self) -> int
                                                                                      Get a hash value for this :class:`ObjectId`
                                                                                        def __init__(self, [oid]) -> None
                                                                                        Generate a new ObjectId either from a pre-existing sequence if bytes, or based on information from this machine including the time, the machine identifier, and the current process id.
                                                                                        PARAMETERS
                                                                                        • oid [bytes | str / default=None] - can be a bytes(14) or a str(28) which should represent an ObjectId
                                                                                        def __le__(self, other) -> bool
                                                                                        Boolean Equality check: LE
                                                                                          def __lt__(self, other) -> bool
                                                                                          Boolean Equality check: LT
                                                                                            def __ne__(self, other) -> bool
                                                                                            Boolean Equality check: NE
                                                                                              def __setstate__(self, value) -> None
                                                                                              explicit state set from pickling
                                                                                                def __str__(self) -> str
                                                                                                Return the value id as a hex string

                                                                                                  class CompressionType (Enum ):
                                                                                                  Enumeration of the different compression types we've implemented. Note that although we support these types, the only ones that will be available are the ones we have back-end library support for. Compression libraries are not included in the dependency list for PyNNDB so it's up to you to install any libraries you wish to use. o ZSTD the Zstandard compression algorithm (pip install zstandard) o SNAPPY the Snappy compression library by Google (pip install python-snappy)
                                                                                                  CLASS PROPERTIES
                                                                                                  NONE = "None"
                                                                                                  SNAPPY = "snappy"
                                                                                                  ZSTD = "zstd"
                                                                                                  class Compression
                                                                                                  This class encapsulates all compression activity and is used as a base class for 'Table'. It's designed in such a way that any number of compression libraries can be used (or not) on a table by table basis, with potentially different types of compression being used on different tables.
                                                                                                  METHODS
                                                                                                  def __init__(self) -> None
                                                                                                  When we instantiate this class, all it does is to set up a default compressor and decompressor in advance of being told which (if any) compression library to use for this table.
                                                                                                    def _get(self, dat) -> Dict
                                                                                                    Base read routine, run any incoming data through the selected decompressor then convert it to 'str'
                                                                                                    PARAMETERS
                                                                                                    • dat [bytes] - the incoming binary data
                                                                                                    def _put(self, doc) -> None
                                                                                                    Base write routine, pass all data through the currently selected compressor for this table.
                                                                                                    PARAMETERS
                                                                                                    • doc [Doc] - object to write to the table
                                                                                                    def close(self) -> None
                                                                                                    Reset the compression routines (to "no" compression) for this table, called by Table.close to make sure any subsequent Table.open has a known / consistent starting state.
                                                                                                      def compress_existing_data(self, [txn]) -> None
                                                                                                      Compress all pre-existing data in this table and is called by Table.open when compression has been turned on for the first time. Table operations rely on all data either being compressed, or uncompressed, so interrupting this routine would not be advisable. If possible, always run "open" within a write transaction.
                                                                                                      PARAMETERS
                                                                                                      • txn [TXN / default=None] - a write transaction to wrap the operation
                                                                                                      def compressed(self, [txn]) -> bool
                                                                                                      Determine whether this table is set to be compressed or not, returns True if the table is compressed.
                                                                                                      PARAMETERS
                                                                                                      • txn [TXN / default=None] - a transaction to wrap the table read
                                                                                                      def compression_select(self, compression_type, [level], [txn]) -> bool
                                                                                                      Select the compression type for this table, this routine is called as a part of table open is compression has been requested. We return True if the compression setting has changed (and as a result we need to compress existing data) or False if nothing has changed.
                                                                                                      PARAMETERS
                                                                                                      • compression_type [CompressionType] - the type of compression to request
                                                                                                      • level [int / default=3] - the compression level to apply
                                                                                                      • txn [TXN / default=None] - a write transaction to wrap the operation
                                                                                                      def compression_type(self, [txn]) -> CompressionType
                                                                                                      Return the compression type that has been appled to this table, or return an empty string if no compression is available.
                                                                                                      PARAMETERS
                                                                                                      • txn [TXN / default=None] - a transaction to wrap the table read
                                                                                                      def open(self, txn) -> None
                                                                                                      Called as part of Table.open, if compression is enabled, set up the required aspects of the chosen compression library in advance of any IO requests.
                                                                                                      PARAMETERS
                                                                                                      • txn [TXN] - a write transaction to wrap the operation
                                                                                                      def zstd_train(self, [training_record_count], [training_samples], [training_dict_size], [threads], [txn]) -> None
                                                                                                      This is a ZSTD specific routine to generate a training dict to aid in the compression process. Compression will work without this, however in theory you will get better compression if you provide some training. Note that once you have run the training routine and generated a compression dict, this is stored permemantly in the database meta data and is "required" to decompress data that has been compressed using this dict. If you lose (or overwrite) this "compression dict", it will render your compressed data useless. Any attempt to re-train a table should raise an exception.
                                                                                                      db = Manager()['mydb'].open('.database')
                                                                                                      people = db['people'].open()
                                                                                                      (add data)
                                                                                                      people.zstd_train(10)
                                                                                                      people.close()
                                                                                                      people.open(CompressionType.ZSTD, 22)
                                                                                                      
                                                                                                      The last line will re-open the table with compression enabled, using the compression dict generated by 'zstd_train', then compress all existing data using this dict.
                                                                                                      PARAMETERS
                                                                                                      • training_record_count [int / default=None] - the number of records to sample from the current table
                                                                                                      • training_samples [list / default=None] - if supplied these will be used instead of reading data from the table
                                                                                                      • training_dict_size [int / default=4096] - the maximum size allowable for the compression dictionary
                                                                                                      • threads [int / default=-1] - the number of threads to use for training
                                                                                                      • txn [TXN | WriteTransaction / default=None] - an optional transaction to wrap the operation

                                                                                                      _ _ _ ___ ___ _ __(_) __ _| (_)___ ___ _ __ / __|/ _ \ '__| |/ _` | | / __|/ _ \ '__| \__ \ __/ | | | (_| | | \__ \ __/ | |___/\___|_| |_|\__,_|_|_|___/\___|_|
                                                                                                      Copyright ©2020 - Mad Penguin Consulting Limited
                                                                                                      class SerialiserType (Enum ):
                                                                                                      Serialisers are pluggable via this module, currently we support the following; o UJSON the default historical serialiser o ORJSON the new guy on the block
                                                                                                      CLASS PROPERTIES
                                                                                                      JSON = "json"
                                                                                                      NONE = "none"
                                                                                                      ORJSON = "orjson"
                                                                                                      UJSON = "ujson"
                                                                                                      class Serialiser
                                                                                                      All attempts to serialise or de-serialise data come through this point. Any new serialiser that supports "loads" and "dumps" can simplu be imported and plugged into __init__ and that should be all there is to it.
                                                                                                      PROPERTIES
                                                                                                      def codec(self) -> SerialiserType
                                                                                                      Return the serialiser currently in use for this table
                                                                                                        METHODS
                                                                                                        def __init__(self, [codec], [txn]) -> None
                                                                                                        Set up handlers for serialisation and de-serialisation.
                                                                                                        PARAMETERS
                                                                                                        • codec [SerialiserType / default=SerialiserType.NONE] - module that will supply "dumps" and "loads" methods
                                                                                                        def deserialise(self, blob) -> dict
                                                                                                        Generic deserialiser interface
                                                                                                          def serialise(self, doc) -> bytes
                                                                                                          Generic serialiser interface

                                                                                                            class MetaData
                                                                                                            This is a wrapper for the __metadata__ table, for which there should be 'one' per database. This contains additional information including index definitions and other persistent information pertaining to tables or indeed to the database itself. This can also be used to store user defined information, however be careful to avoid key namespace overlaps. Currently we are using keys of the form; o _(table_name)_(index_name)_ to store information about indexes o _(table_name)@config to store table specific conguration o _(table_name)!zstd_cdict to store the zstd training dictionary o __uuid__ unique identifier for this machine
                                                                                                            METHODS
                                                                                                            def __init__(self, database) -> None
                                                                                                            The MetaData class is initialised with a database reference so it can get access to the current working environment and metadata table.
                                                                                                            PARAMETERS
                                                                                                            • database [Database] - the database that holds the metadata table we'll be working with
                                                                                                            def fetch(self, name, [txn]) -> None
                                                                                                            Fetch all the keys (and values) from the metadata table that match the name of the table supplied.
                                                                                                            PARAMETERS
                                                                                                            • name [str] - name of the table to recover keys from
                                                                                                            • txn [TXN / default=None] - a read or write transaction to wrap the operation
                                                                                                            def fetch_config(self, table_name, [txn]) -> Doc
                                                                                                            Fetch the configuration dictionary from the metadata table for the named table. This dictionary will contain table specific settings such as the compression mechanism currently being employed on the table.
                                                                                                            PARAMETERS
                                                                                                            • table_name [str] - the name of the table we want the configuration for
                                                                                                            • txn [TXN / default=None] - a transaction to wrap the operation
                                                                                                            def fetch_index(self, table_name, index_name, [txn]) -> Doc
                                                                                                            Fetch the index definition for the index in the named table. This is an internal routine primarily used when opening a table to recover the names of the indexes associated with that table. Returns a Doc object containing a 'conf' key
                                                                                                            PARAMETERS
                                                                                                            • table_name [str] - name of the table the index is associated with
                                                                                                            • index_name [str] - the name of the index to recover information on
                                                                                                            • txn [TXN / default=None] - a read or write transaction to wrap the operation
                                                                                                            def fetch_int(self, key, [txn]) -> int
                                                                                                            Recover an integer value from the metadata store based on a key
                                                                                                            PARAMETERS
                                                                                                            • key [str] - the name of the specific key to store
                                                                                                            • txn [TXN / default=None] - a transaction to wrap the operation
                                                                                                            def fetch_key(self, key, [txn]) -> Doc
                                                                                                            Recover a key from the metadata. This is for recovering database specific information.
                                                                                                            PARAMETERS
                                                                                                            • key [str] - the name of the specific tag we want
                                                                                                            • txn [TXN / default=None] - a transaction to wrap the operation
                                                                                                            def fetch_tag(self, table_name, tag, [txn]) -> Doc
                                                                                                            Recover a tag from the metadata table for a given table and tag name. It works against binary blobs as opposed to formatted JSON so is suited to 1storing things like compression dictionaries. (which was initially why it was added)
                                                                                                            PARAMETERS
                                                                                                            • table_name [str] - the name of the table we're fetching tags for
                                                                                                            • tag [str] - the name of the specific tag we want
                                                                                                            • txn [TXN / default=None] - a transaction to wrap the operation
                                                                                                            def open(self, [txn]) -> None
                                                                                                            Open the __metadata__ table and make it available for IO
                                                                                                              def path_config(self, table_name) -> bytes
                                                                                                              This is used to generate a key for methods that operate on configuration metadata. On being passed a table name, this routine returns the key associated with the config item in the metadata table for the named table.
                                                                                                              PARAMETERS
                                                                                                              • table_name [str] - the name of the table we need to generate a key for
                                                                                                              def path_index(self, table_name, index_name) -> bytes
                                                                                                              Build a key for the __metadata__ table to access an index definition Returns the assembled key in bytes format ready to be used by get/put
                                                                                                              PARAMETERS
                                                                                                              • table_name [str] - name of the table the index is associated with
                                                                                                              • index_name [str] - the name of the index
                                                                                                              def path_tag(self, table_name, tag) -> bytes
                                                                                                              Return the primary key for a given tag in the metadata table. Tags are based on or relative to a table name and a tag name.
                                                                                                              PARAMETERS
                                                                                                              • table_name [str] - the name of the table we're fetching tags for
                                                                                                              • tag [str] - the name of the specific tag we want
                                                                                                              def remove(self, name, [txn]) -> None
                                                                                                              Remove all keys from the database for the current table. This is used by 'droptable' to clean any latent meta data from the metadata table when a table is dropped.
                                                                                                              PARAMETERS
                                                                                                              • name [str] - name of the table to remove keys from
                                                                                                              • txn [TXN / default=None] - a write transaction to wrap the operation
                                                                                                              def remove_index(self, table_name, index_name, [txn]) -> None
                                                                                                              Remove an index definition, used either when a table or an index is dropped
                                                                                                              PARAMETERS
                                                                                                              • table_name [str] - name of the table the index is associated with
                                                                                                              • index_name [str] - the name of the index
                                                                                                              • txn [TXN / default=None] - a write transaction to wrap the operation
                                                                                                              def remove_key(self, key, [txn]) -> None
                                                                                                              Remove a key, used to remove database specific information.
                                                                                                              PARAMETERS
                                                                                                              • key [str] - the key of the information to remove
                                                                                                              • txn [TXN / default=None] - a write transaction to wrap the operation
                                                                                                              def reopen(self, [txn]) -> None
                                                                                                              Re-Open the __metadata__ table and make it available for IO
                                                                                                                def store_config(self, table_name, value, [txn]) -> None
                                                                                                                Store the configuration dictionary back in the metadata table for the named table. Typically this is handled internally for example when compression is enabled on the table.
                                                                                                                PARAMETERS
                                                                                                                • table_name [str] - the name of the table we're storing the configuration for
                                                                                                                • txn [TXN / default=None] - a transaction to wrap the operation
                                                                                                                def store_index(self, table_name, index_name, value, [txn]) -> None
                                                                                                                Store the index definition for the index in the named table. This is an internal routine primarily used when creating an index store index and names and definitions for the index being created.
                                                                                                                PARAMETERS
                                                                                                                • table_name [str] - name of the table the index is associated with
                                                                                                                • index_name [str] - the name of the index
                                                                                                                • txn [TXN / default=None] - a write transaction to wrap the operation
                                                                                                                def store_int(self, key, val, [txn]) -> int
                                                                                                                Store a key against an integer in the metadata table, returning the value stored.
                                                                                                                PARAMETERS
                                                                                                                • key [str] - the name of the specific key to store
                                                                                                                • val [int] - the value to associate with the key
                                                                                                                • txn [WriteTransaction / default=None] - a transaction to wrap the operation
                                                                                                                def store_key(self, key, val, [txn]) -> None
                                                                                                                Store a key in the metadata table for a given name. This is for storing database specific information.
                                                                                                                PARAMETERS
                                                                                                                • key [str] - the name of the specific key to store
                                                                                                                • val [Doc] - the value to associate with the key
                                                                                                                • txn [TXN / default=None] - a transaction to wrap the operation
                                                                                                                def store_tag(self, table_name, tag, value, [txn]) -> None
                                                                                                                Store a tag in the metadata table for a given table and tag name. This was initially added to facilitate the storage of compression dict's (as binary blobs).
                                                                                                                PARAMETERS
                                                                                                                • table_name [str] - the name of the table we're fetching tags for
                                                                                                                • tag [str] - the name of the specific tag we want
                                                                                                                • txn [TXN / default=None] - a transaction to wrap the operation

                                                                                                                class FilterResult
                                                                                                                Wrapper for results returned from the "filter" API call, use the "doc" method to access the resulting data.
                                                                                                                PROPERTIES
                                                                                                                def doc(self) -> Doc
                                                                                                                Returns the uncompressed data associted with this search result.
                                                                                                                  def raw(self) -> bytes
                                                                                                                  Returns the raw serialised data directly from the KV store
                                                                                                                    METHODS
                                                                                                                    def __init__(self, table, index, cursor, txn) -> None
                                                                                                                    Instantiate a FilterResult instance, used by the filter() API call to return a result to the caller. If we're searching on a primary key then we effectively have the data we want already, but if we're searching on an index, we implement a lazy-loader, so we store the real primary key OID, then use it to recover the actual record when the user calls the "doc" method. Properties: o count the number of duplicate keys that apply to this result o key the key used to acquire this result
                                                                                                                    PARAMETERS
                                                                                                                    • index [Index] - the index we were searching on (None for primary key)
                                                                                                                    • cursor [Cursor] - the LMDB cursor object we're using
                                                                                                                    • txn [TXN] - a transaction to wrap this operation

                                                                                                                    class Cursor
                                                                                                                    The Cursor class is a simple wrapper for the LMDB cursor object, it's primary goal is to ensure that functions returning with "keyonly" set employ proper Python strings rather than 'bytes' arrays. This class provides the following properties; key - the current key val - the associated value which is the key of the associated data in the main table count - the number of duplicates that exist for this key value
                                                                                                                    METHODS
                                                                                                                    def __init__(self, index, cursor) -> None
                                                                                                                    Simply set the "key" and "val" attributes of the object to the values we want to make available to the library user.
                                                                                                                    PARAMETERS
                                                                                                                    • index [Index] - a reference to the index this cursor object relates to
                                                                                                                    • cursor [LMDBCursor] - an LMDB cursor object which has a reference to both the data key and value
                                                                                                                    def fetch(self, [txn]) -> Doc
                                                                                                                    Recover the data item that is associated with this key
                                                                                                                    PARAMETERS
                                                                                                                    • txn [TXN / default=None] - an optional transaction to wrap this operation