dballe database functions

class dballe.DB

DB-All.e database access.

Many methods are the same in dballe.DB and dballe.Transaction. The versions in dballe.DB are implemented by automatically creating a temporary transaction and running the equivalent dballe.Transaction method inside it.

dballe.DB objects are not constructed explicitly, but via one of the DB.connect() or DB.connect_test() class methods.

Examples:

# Connect to a database and run a query
db = dballe.DB.connect_from_file("db.sqlite")
query = {"latmin": 44.0, "latmax": 45.0, "lonmin": 11.0, "lonmax": 12.0}

# The result is a dballe.Cursor (dballe.CursorData in this case), which can
# be iterated to get results as dict objects.
with db.transaction() as tr:
    for row in tr.query_data(query):
        print(row["lat"], row["lon"], row["var"], row["variable"].format("undefined"))

# Insert 2 new variables in the database
db.insert_data({
    "lat": 44.5, "lon": 11.4,
    "level": dballe.Level(1),
    "trange": dballe.Trange(254),
    "date": datetime.datetime(2013, 4, 25, 12, 0, 0),
    "B11101": 22.4,
    "B12103": 17.2,
})
attr_insert_data(varid: int, attrs: Dict[str, Any])

Insert new data attributes into the database

attr_insert_station(varid: int, attrs: Dict[str, Any])

Insert new constant station data attributes into the database

attr_query_data(varid: int) → Dict[str, Any]

query data attributes

attr_query_station(varid: int) → Dict[str, Any]

query constant station data attributes

attr_remove_data(varid: int, attrs: Iterable[str])

Remove attributes from data

attr_remove_station(varid: int, attrs: Iterable[str])

Remove attributes from constant station data

connect(url: str) → dballe.DB

create a DB to access a database identified by a DB-All.e URL

connect_from_file(name: str) → dballe.DB

create a DB to access a SQLite file

connect_from_url(url: str) → dballe.DB

create a DB to access a database identified by a DB-All.e URL (deprecated, use connect instead)

connect_test() → dballe.DB

Create a DB for running the test suite, as configured in the test environment

disappear()

Remove all DB-All.e tables and data from the database, if possible

get_default_format() → str

get the default DB format

import_messages(messages: Union[dballe.Message, Sequence[dballe.Message], Iterable[dballe.Message], dballe.ImporterFile], report: str=None, import_attributes: bool=False, update_station: bool=False, overwrite: bool=False)

Import one or more Messages into the database.

Parameters:
  • messages –
  • report – the network name to use for importing the data. If left to None, the network is selected automatically from the message type
  • import_attributes – if set to True, requests the variable attributes to also be imported.
  • update_station – if set to True, station information is merged with existing one in the database. If false (default), station information is imported only when the station did not exist in the database.
  • overwrite – if set to True, causes existing information already in the database to be overwritten. If false (default), trying to import a message which contains data already present in the database causes the import to fail.
  • varlist – if set to a string in the same format as the varlist query parameter, only imports data whose varcode is in the list.
insert_data(record: Union[Dict[str, Any], dballe.Cursor, dballe.Data], can_replace: bool=False, can_add_stations: bool=False) → Dict[str, int]

Insert data values in the database

The return value is a dict that always contains ana_id mapped to the station ID just inserted, and an entry for each varcode inserted mapping to the database ID of its value.

insert_station_data(record: Union[Dict[str, Any], dballe.Cursor, dballe.Data], can_replace: bool=False, can_add_stations: bool=False) → Dict[str, int]

Insert station values in the database

The return value is a dict that always contains ana_id mapped to the station ID just inserted, and an entry for each varcode inserted mapping to the database ID of its value.

is_url(url: str) → bool

Checks if a string looks like a DB-All.e DB url

query_data(query: Dict[str, Any]) → dballe.CursorData

Query the data in the database

Returns:a cursor to iterate the query results (see dballe.CursorDataDB)
query_messages(query: Dict[str, Any]) → dballe.CursorMessage

Query the database returning the matching data as Message objects

This can also be used to export messages to a file. For example:

exporter = dballe.Exporter("BUFR")
with open("file.bufr", "wb") as outfile:
    for row in tr.query_messages(...):
        outfile.write(exporter.to_binary(row.message))

See: dballe.Exporter and dballe.CursorMessage.

query_station_data(query: Dict[str, Any]) → dballe.CursorStationData

Query the constant station data in the database

Returns:a cursor to iterate the query results (see dballe.CursorStationDataDB)
query_stations(query: Dict[str, Any]) → dballe.CursorStation

Query the stations in the database

Returns:a cursor to iterate the query results (see dballe.CursorStationDB)
query_summary(query: Dict[str, Any]) → dballe.CursorSummary

Query the summary of the results of a query

Returns:a cursor to iterate the query results (see dballe.CursorSummaryDB)
remove(query: Dict[str, Any])

Remove data variables from the database (deprecated)

remove_all()

Remove all data from the database

remove_data(query: Dict[str, Any])

Remove data variables from the database

remove_station_data(query: Dict[str, Any])

Remove station variables from the database

reset(repinfo_file: str=None)

Reset the database, removing all existing Db-All.e tables and re-creating them empty.

set_default_format(format: str)

set the default DB format

transaction(readonly: bool=False) → dballe.Transaction

Create a new database transaction

vacuum()

Perform database cleanup operations

class dballe.Transaction

DB-All.e transaction

A Transaction is used to execute DB operations in an all-or-nothing fashion. In fact, most DB methods are implemented using a short-lived temporary transaction.

You cannot have more than one active dballe.Transaction for each dballe.DB. An attempt to start a second one will result in an exception being raised. Note that dballe.DB functions like dballe.Transaction.insert_data() create a temporary transaction to run, and so they will also fail if a transaction is currently open. The general idea is that all database work should be done inside a transaction.

Transactions run using the REPEATABLE READ isolation level of the underlying database. This usually means that modifications performed inside a transaction are not visible to other database connections until the transaction is committed. If a transaction is rolled back, all changes done with it are undone.

Transactions can also be used as context managers, like this:

with db.transaction() as t:
    for i in range(10):
        t.insert({
            "lat": 44.5 + i, "lon": 11.4 + i, "level": (1,),
            "trange": (254,), "date": datetime.datetime(2013, 4, 25, 12, 0, 0),
            "B11101": 22.4 + i, "B12103": 17.2
        })

The dballe.Transaction methods are the same as those in dballe.DB. The version in dballe.DB is implemented by automatically creating a temporary transaction and running the dballe.Transaction method inside it.

attr_insert_data(varid: int, attrs: Dict[str, Any])

Insert new data attributes into the database

attr_insert_station(varid: int, attrs: Dict[str, Any])

Insert new constant station data attributes into the database

attr_query_data(varid: int) → Dict[str, Any]

query data attributes

attr_query_station(varid: int) → Dict[str, Any]

query constant station data attributes

attr_remove_data(varid: int, attrs: Iterable[str])

Remove attributes from data

attr_remove_station(varid: int, attrs: Iterable[str])

Remove attributes from constant station data

commit()

commit the transaction

import_messages(messages: Union[dballe.Message, Sequence[dballe.Message], Iterable[dballe.Message], dballe.ImporterFile], report: str=None, import_attributes: bool=False, update_station: bool=False, overwrite: bool=False)

Import one or more Messages into the database.

Parameters:
  • messages –
  • report – the network name to use for importing the data. If left to None, the network is selected automatically from the message type
  • import_attributes – if set to True, requests the variable attributes to also be imported.
  • update_station – if set to True, station information is merged with existing one in the database. If false (default), station information is imported only when the station did not exist in the database.
  • overwrite – if set to True, causes existing information already in the database to be overwritten. If false (default), trying to import a message which contains data already present in the database causes the import to fail.
  • varlist – if set to a string in the same format as the varlist query parameter, only imports data whose varcode is in the list.
insert_data(record: Union[Dict[str, Any], dballe.Cursor, dballe.Data], can_replace: bool=False, can_add_stations: bool=False) → Dict[str, int]

Insert data values in the database

The return value is a dict that always contains ana_id mapped to the station ID just inserted, and an entry for each varcode inserted mapping to the database ID of its value.

insert_station_data(record: Union[Dict[str, Any], dballe.Cursor, dballe.Data], can_replace: bool=False, can_add_stations: bool=False) → Dict[str, int]

Insert station values in the database

The return value is a dict that always contains ana_id mapped to the station ID just inserted, and an entry for each varcode inserted mapping to the database ID of its value.

query_data(query: Dict[str, Any]) → dballe.CursorData

Query the data in the database

Returns:a cursor to iterate the query results (see dballe.CursorDataDB)
query_messages(query: Dict[str, Any]) → dballe.CursorMessage

Query the database returning the matching data as Message objects

This can also be used to export messages to a file. For example:

exporter = dballe.Exporter("BUFR")
with open("file.bufr", "wb") as outfile:
    for row in tr.query_messages(...):
        outfile.write(exporter.to_binary(row.message))

See: dballe.Exporter and dballe.CursorMessage.

query_station_data(query: Dict[str, Any]) → dballe.CursorStationData

Query the constant station data in the database

Returns:a cursor to iterate the query results (see dballe.CursorStationDataDB)
query_stations(query: Dict[str, Any]) → dballe.CursorStation

Query the stations in the database

Returns:a cursor to iterate the query results (see dballe.CursorStationDB)
query_summary(query: Dict[str, Any]) → dballe.CursorSummary

Query the summary of the results of a query

Returns:a cursor to iterate the query results (see dballe.CursorSummaryDB)
remove(query: Dict[str, Any])

Remove data variables from the database (deprecated)

remove_all()

Remove all data from the database

remove_data(query: Dict[str, Any])

Remove data variables from the database

remove_station_data(query: Dict[str, Any])

Remove station variables from the database

rollback()

roll back the transaction

class dballe.Explorer

Browser for a summary of DB-All-e database of message contents

all_levels

get all level values

all_reports

get all report values

all_stations

get all stations

all_stats

get the stats for all values

all_tranges

get all time range values

all_varcodes

get all varcode values

levels

get all the level values currently selected

query_summary() → dballe.CursorSummarySummary

Get the currently selected Explorer summary information

Returns:a cursor to iterate the query results (see dballe.CursorSummarySummary)
query_summary_all() → dballe.CursorSummarySummary

Get all the Explorer summary information.

Returns:a cursor to iterate the query results (see dballe.CursorSummarySummary)
rebuild()

Empty the Explorer and start adding new data to it.

Returns an ExplorerUpdate context manager object that can be used to add data to the explorer in a single transaction.

reports

get all the report values currently selected

set_filter()

Set a new filter, updating all browsing data

stations

get all the stations currently selected

stats

get stats for the currently selected values

to_json()

Serialize the contents of this explorer to JSON.

Only the global summary is serialized: the current query is not preserved.

tranges

get all the time range values currently selected

update()

Start adding new data to the Explorer without clearing it first.

Returns an ExplorerUpdate context manager object that can be used to add data to the explorer in a single transaction.

varcodes

get all the varcode values currently selected

class dballe.DBExplorer

Browser for a summary of DB-All-e database of message contents

all_levels

get all level values

all_reports

get all report values

all_stations

get all stations

all_stats

get the stats for all values

all_tranges

get all time range values

all_varcodes

get all varcode values

levels

get all the level values currently selected

query_summary() → dballe.CursorSummaryDBSummary

Get the currently selected Explorer summary information

Returns:a cursor to iterate the query results (see dballe.CursorSummaryDBSummary)
query_summary_all() → dballe.CursorSummaryDBSummary

Get all the Explorer summary information.

Returns:a cursor to iterate the query results (see dballe.CursorSummaryDBSummary)
rebuild()

Empty the Explorer and start adding new data to it.

Returns an ExplorerUpdate context manager object that can be used to add data to the explorer in a single transaction.

reports

get all the report values currently selected

set_filter()

Set a new filter, updating all browsing data

stations

get all the stations currently selected

stats

get stats for the currently selected values

to_json()

Serialize the contents of this explorer to JSON.

Only the global summary is serialized: the current query is not preserved.

tranges

get all the time range values currently selected

update()

Start adding new data to the Explorer without clearing it first.

Returns an ExplorerUpdate context manager object that can be used to add data to the explorer in a single transaction.

varcodes

get all the varcode values currently selected

class dballe.ExplorerUpdate

Manage updates to an Explorer

add_db()

Add the summary of the contents of the given database to the Explorer.

add_explorer()

Add the contents of the given Explorer or DBExplorer to the Explorer.

add_json()

Add the contents of the given JSON string to the Explorer.

add_messages()

Add dballe.Message objects to the explorer.

It takes the same messages argument of dballe.DB.import_messages

class dballe.DBExplorerUpdate

Manage updates to an Explorer

add_db()

Add the summary of the contents of the given database to the Explorer.

add_explorer()

Add the contents of the given Explorer or DBExplorer to the Explorer.

add_json()

Add the contents of the given JSON string to the Explorer.

add_messages()

Add dballe.Message objects to the explorer.

It takes the same messages argument of dballe.DB.import_messages