Manager is a dictionary like object the holds references to all the databases currently registered with the running instance. If you only ever reference one database then technically you can skip this object and just use the Database object.

The Database object models the top-level container for a collection of tables and indexes, each Database object maps to an LMDB database object. When you open a database there are a variety of low-level (LMDB) database settings that can be applied, it's worth understanding what some of them do as when you come to productionise your system, they will make a difference. Specifically you will want to tweak 'map_size' which is the maximum allowable size of your database, and possibly max_dbs if you are going to open large numbers of tables at the same time. sync: If False, don’t flush system buffers to disk when committing a transaction. This optimization means a system crash can corrupt the database or lose the last transactions if buffers are not yet flushed to disk. The risk is governed by how often the system flushes dirty buffers to disk and how often sync() is called. However, if the filesystem preserves write order and writemap=False, transactions exhibit ACI (atomicity, consistency, isolation) properties and only lose D (durability). I.e. database integrity is maintained, but a system crash may undo the final transactions. Note that sync=False, writemap=True leaves the system with no hint for when to write transactions to disk, unless sync() is called. map_async=True, writemap=True may be preferable. lock: If False, don’t do any locking. If concurrent access is anticipated, the caller must manage all concurrency itself. For proper operation the caller must enforce single-writer semantics, and must ensure that no readers are using old transactions while a writer is active. The simplest approach is to use an exclusive lock so that no readers may be active at all when a writer begins. subdir: If True, path refers to a subdirectory to store the data and lock files in, otherwise it refers to a filename prefix. create: False, do not create the directory path if it is missing. writemap: If True, use a writeable memory map unless readonly=True. This is faster and uses fewer mallocs, but loses protection rom application bugs like wild pointer writes and other bad updates into the database. Incompatible with nested transactions. Processes with and without writemap on the same environment do not cooperate well. metasync: If False, flush system buffers to disk only once per transaction, omit the metadata flush. Defer that until the system flushes files to disk, or next commit or sync(). This optimization maintains database integrity, but a system crash may undo the last committed transaction. I.e. it preserves the ACI (atomicity, consistency, isolation) but not D (durability) database property. readahead: If False, LMDB will disable the OS filesystem readahead mechanism, which may improve random read performance when a database is larger than RAM. map_async: When writemap=True, use asynchronous flushes to disk. As with sync=False, a system crash can then corrupt the database or lose the last transactions. Calling sync() ensures on-disk database integrity until next commit. max_readers: Maximum number of simultaneous read transactions. Can only be set by the first process to open an environment, as it affects the size of the lock file and shared memory area. Attempts to simultaneously start more than this many readtransactions will fail. max_dbs: Maximum number of databases available. If 0, assume environment will be used as a single database. map_size: Maximum size database may grow to; used to size the memory mapping. If database grows larger than map_size, an exception will be raised and the user must close and reopen Environment. On 64-bit there is no penalty for making this huge (say 1TB). Must be <2GB on 32-bit. Default values for these settings come from the CONFIG class variable, please note that currently we do NOT support the 'readonly' option. It does not appear that this option works properly with transactions on sub-databases and until we can work out why, please avoid read-only databases.

{
    "sync": true,
    "lock": true,
    "subdir": true,
    "create": true,
    "writemap": true,
    "metasync": false,
    "readahead": true,
    "map_async": true,
    "max_readers": 64,
    "max_dbs": 64,
    "map_size": 2147483648
}

The Table class is used to wrap access to individual database tables and incorporates semi-transparent compression / decompression on a per table basis. Compression libraries are pluggable and implemented in the Compression class. o APPEND_MODE - when appending a record we know that the new key will be the highest value in the table so we can take advantage of LMDB's "append" mode. If you never want to use this option, set pynndb2.Table.APPEND_MODE to False. NOTE: compatibility issue, we've switched to our own version of "ObjectId", if you set this to True and have data that uses both BSON/ObjectId and the new ObjectId, you will have a problem. (data loss)

func = '{name}'         # index by name
func = '{name}|{age}'   # index by name + age
func = '{age:03d}'      # index by age with leading zero for correct numerical sort order

func = 'def func(doc): return "{:03d}".format(doc["age"]).encode()'

#!/usr/bin/env python
from pynndb import Manager, Doc
from shutil import rmtree
rmtree('.database')
db = Manager().database('database', '.database')
people = db.table('people')
people.append(Doc({'name': 'Tom', 'age': 21}))
people.append(Doc({'name': 'Harry', 'age': 19}))
people.ensure('by_age_fs', '{age:03d}')
people.ensure('by_age_func', 'def func(doc): return "{:03d}".format(doc["age"]).encode()')
[print(person.doc) for person in people.find()]
print('--')
[print(person.doc) for person in people.find('by_age_fs')]
print('--')
[print(person.doc) for person in people.find('by_age_func')]

    filter(expression=lambda doc: doc['age'] > 19)

The Index class models individual indexes of which a table may have zero or more. There are two crucial parameters supplied when setting up an index and are passed via "conf". o dupsort this controls whether the index will allow duplicate values to be inserted o func this is the anonymous function used to generate key values from the date record

The Doc object is used to encapsulate data records before they are written to the database and after they are read from the database. It performs a number of crucial functions including; o Associating a data buffer with a key (OID) o Tracking updated attributes between the last read record and the current buffer o Tracking deleted attributes o Serialising and deserialising data between the database and user buffers o Providing a dict and object-like interfaces to interact with the data Note that this interface is 'slower' than using a raw 'dict' as in version 1, however the use of 'Doc' facilitates the new 'C' extension (which should be a drop-in replacement for 'Doc') which should be much faster, and this in turn facilitates the new GOBJECT serialisation mechanism that should provide a 2-10x performance increase.

An ObjectId is a 14-byte unique identifier consisting of: - a 8-byte value representing the nano seconds since the Unix epoch - a 4-byte machine identifier - a 2-byte process id This code is based on the ObjectId class found in the Python BSON module.

Enumeration of the different compression types we've implemented. Note that although we support these types, the only ones that will be available are the ones we have back-end library support for. Compression libraries are not included in the dependency list for PyNNDB so it's up to you to install any libraries you wish to use. o ZSTD the Zstandard compression algorithm (pip install zstandard) o SNAPPY the Snappy compression library by Google (pip install python-snappy)

This class encapsulates all compression activity and is used as a base class for 'Table'. It's designed in such a way that any number of compression libraries can be used (or not) on a table by table basis, with potentially different types of compression being used on different tables.

db = Manager()['mydb'].open('.database')
people = db['people'].open()
(add data)
people.zstd_train(10)
people.close()
people.open(CompressionType.ZSTD, 22)

Serialisers are pluggable via this module, currently we support the following; o UJSON the default historical serialiser o ORJSON the new guy on the block

All attempts to serialise or de-serialise data come through this point. Any new serialiser that supports "loads" and "dumps" can simplu be imported and plugged into __init__ and that should be all there is to it.

This is a wrapper for the __metadata__ table, for which there should be 'one' per database. This contains additional information including index definitions and other persistent information pertaining to tables or indeed to the database itself. This can also be used to store user defined information, however be careful to avoid key namespace overlaps. Currently we are using keys of the form; o _(table_name)_(index_name)_ to store information about indexes o _(table_name)@config to store table specific conguration o _(table_name)!zstd_cdict to store the zstd training dictionary o __uuid__ unique identifier for this machine

Wrapper for results returned from the "filter" API call, use the "doc" method to access the resulting data.

The Cursor class is a simple wrapper for the LMDB cursor object, it's primary goal is to ensure that functions returning with "keyonly" set employ proper Python strings rather than 'bytes' arrays. This class provides the following properties; key - the current key val - the associated value which is the key of the associated data in the main table count - the number of duplicates that exist for this key value