Data sources

1Presentation

Temma offers a mechanism for managing access to data sources, making it easy to connect to them and use them in a unified way, while still allowing you to call on capabilities specific to each source.

Unified operation means that most data sources can be used in the same way, to write and read key/value pairs. Data sources used in this way can be interchanged (for example, a Memcache access replaced by MySQL or Redis; or a Beanstalkd message queue transparently replaced by AWS SQS).

2Configuration

To open a connection to a data source, you need to configure it in the etc/temma.php file (see configuration documentation). 

<?php

return [
    // application configuration
    'application' => [
        'dataSources' => [
            'db'      => 'mysql://user:passwd@localhost/my_base',
            'ndb'     => 'redis://localhost',
            'cache'   => 'memcache://localhost',
            'file'    => 'file:///var/data/temma',
            's3'      => 's3://ACCESS_KEY:SECRET_KEY@REGION/my_bucket',
            'sqs'     => 'sqs://ACCESS_KEY:SECRET_KEY@my_queue',
            'mq'      => 'beanstalk://localhost/my_queue',
            'sms'     => 'smsmode://API_KEY',
            'slack'   => 'slack://hooks.slack.com/services/TXX/BYY/ZZ',
            'discord' => 'discord://discord.com/api/webhooks/ABC/XYZ',
            'push'    => 'pushover://APP_TOKEN',
            'myDS1'   => '[MyDataSource]mydsn://SOME_PARAM',
            'myDS2'   => '[\Namespace\OtherDataSource]otherdsn://OTHER_PARAM',
        ]
    ]
];
  • Line 7: Connection to MySQL.
  • Line 8: Connection to Redis.
  • Line 9: Connection to Memcache.
  • Line 10: Access to local files.
  • Line 11: Connection to Amazon S3.
  • Line 12: Connection to Amazon SQS.
  • Line 13: Connection to Beanstalkd.
  • Line 14: Connection to smsmode.
  • Line 15: Connection to Slack.
  • Line 16: Connection to Discord.
  • Line 17: Connection to Pushover.
  • Line 18: Use the MyDataSource object as data source.
  • Line 19: Use the \Namespace\OtherDataSource object as data source.

See the documentation for each data source to learn more about its configuration.

In controllers, data sources are directly accessible using the names defined in the configuration: $this->db, $this->cache, $this->s3, etc.

Objects accessing the dependency injection component must pass through the dataSources registry, which can be used with object-oriented writing or as an array: 

$db = $this->_loader->dataSources->db;
$db = $this->_loader->dataSources['db'];

3Unified access

Data sources offer a number of common methods for accessing key/value pairs. Some methods are not always usable (for example, Memcache, Amazon SQS or Beanstalkd don't allow you to search through stored data), but the principles are broadly the same.

There are three sets of methods:

  1. General methods (find out if data exists, delete data, etc.).
  2. Methods for reading and writing complex data. Data is serialized (usually in JSON format, unless the data source provides its own serialization mechanism).
  3. Methods for reading and writing raw data. These methods manipulate strings which are stored directly.

For simplicity's sake, complex data can be accessed using associative array syntax.

4Table-type access

4.1Existence of data

Use the isset() function.

if (isset($this->source['key1']))
    TµLog::l("The 'key1' data exists.");

4.2Reading data

The data is deserialized.

$data = $this->source['key1'];

4.3Writing data

The data is serialized.

$this->source['key1'] = $data;

4.4Deleting data

Use the unset() function.

unset($this->source['key1']);

4.5Get the number of stored data

Use the count() function.

$nbr = count($this->source);

5General methods

5.1isSet()

isSet(string $key) : bool

Returns true if the key $key refers to data that exists.

Example:

// Check if the "key1" key exists
if (!$this->db->isSet('key1'))
    TµLog::l("Unknown data.");

// alternate writing
if (!isset($this->db['key1']))
    TµLog::l("Unknown data.");

5.2remove()

remove(string $key) : void

Deletes the data referenced by the key $key.

Example:

// delete the key "key1"
$this->cache->remove('key1');

5.3mRemove()

mRemove(array $keys) : void

Deletes all data whose keys are listed in the $keys array.

Example:

// delete the keys "key1" and "key2"
$this->cache->mRemove(['key1', 'key2']);

5.4clear()

clear(string $pattern) : void

Deletes all data whose key corresponds to the parameter supplied. Depending on how the data source works, the parameter may be a prefix or a regular expression mask.

Example:

// delete all files in the "images" directory
$this->files->clear('images/*');

5.5flush()

flush() : void

Deletes all data stored in the data source.

Example:

// delete all files stored in an Amazon S3 bucket
$this->s3->flush();

6Management of complex serialized data

search(string $pattern, bool $getValues=false) : array

Returns the list of keys corresponding to the first parameter. Depending on how the data source works, the parameter may be a prefix or a regular expression mask.

If the second parameter is set to true, the list returned contains the (deserialized) data values associated with each key.

Examples:

// retrieves the list of file names in the "users" directory
$users = $this->files->search('users/*');
foreach ($users as $login)
    TµLog::l("User: $login");

// retrieves the list of files in the "users" directory,
// with their contents deserialized
$users = $this->files->search('users/*', true);
foreach ($users as $path => $user) {
    $login = mb_substr($path, mb_strlen('users/'));
    TµLog::l("User   : $login");
    TµLog::l("Age    : " . $user['age']});
    TµLog::l("Address: " . $user['address']);
}

6.2get()

get(string $key, mixed $defaultOrCallback=null, mixed $options=null) : mixed

Returns the (deserialized) data whose key is supplied as the first parameter.

If the data does not exist in the data source, the second parameter is used:

  • if it contains scalar data, this is used as the default value and returned by the method;
  • if it contains a function, the function is executed. Its return value is added to the data source, associated with the supplied key, and returned by the method. In this case, the third parameter can receive options that are used to store the data.

Examples:

// retrieve cached data
$colors = $this->cache->get('app:colors');

// alternate writing
$color = $this->cache['app:colors'];

// idem, but if the data is not cached, a default value is returned
$colors = $this->cache->get('app:colors', ['blue', 'red', 'green']);
// $colors = ['blue', 'red', 'green'];

// Retrieve cached data. If the data is not cached, a function is called
// to fetch the data from the database (via a DAO). The cached data is
// saved and returned. The data will be available for subsequent accesses
// to the cache.
$user = $this->cache->get($userId, function() use ($userId) {
    return $this->_dao->get($userId);
});

6.3mGet()

mGet(array $keys) : array

Takes a list of keys as parameter, and returns an associative array with the keys and their (deserialized) contents.

Example:

// retrieves (deserialized) data from "key1" and "key2" keys
$data = $this->db->mGet(['key1', 'key2']);
foreach ($data as $key => $datum) {
    TµLog::l("Name: '$key' - size: '{$datum['size']}'.");
}

6.4set()

set(string $key, mixed $value=null, mixed $options=null) : mixed

Writes (serialized) data to the data source.

  • 1st parameter: Key of data to be created or updated.
  • 2nd parameter: Data content (scalar or complex).
  • 3rd parameter: Option used for recording, depending on the type of data source.

Example:

// add data to cache
$this->cache->set('app:color', 'blue');

// alternate writing
$this->cache['app:color'] = 'blue';

// create a text file on Amazon S3 with public read rights
$this->s3->set('text/introduction.txt', $texte, [
    'public'   => true,
    'mimetype' => 'text/plain',
]);

6.5mSet()

mSet(array $data, mixed $options=null) : int

Writes multiple (serialized) data.

  • 1st parameter: Associative array whose keys are the identifiers of the data to be written, and whose associated values are the contents of the data to be written.
  • 2nd parameter: Option used for all records, depending on data source type.

Example:

// write multiple (serialized) keys to a Redis database
$this->ndb->mSet([
    'key1' => 'value 1',
    'key2' => ['value 2.1', 'value 2.2', 'value 2.3'],
    'key3' => [
        'key3.1' => 'value 3.1',
        'key3.2' => 'value 3.2',
    ],
]);

7Raw data management

7.1find()

find(string $pattern, bool $getValues=false) : array

Returns the list of keys corresponding to the first parameter. Depending on how the data source works, the parameter may be a prefix or a regular expression mask.

If the second parameter is set to true, the list returned contains the (raw) data values associated with each key.

If the second parameter is set to false, this method is identical to the search() method.

Examples:

// retrieves the list of file names in the "images" directory
$keys = $this->files->find('images/*');
foreach ($keys as $key)
    TµLog::l("File: '$key'.");

// retrieves the list of files in the "images" directory,
// with their raw contents
$keys = $this->files->find('images/*', true);
foreach ($keys as $key => $value) {
    $size = mb_strlen($value, 'ascii');
    TµLog::l("File: '$key' - size: '$size'.");
}

7.2read()

read(string $key, mixed $defaultOrCallback=null, mixed $options=null) : mixed

Returns the data whose key is supplied as the first parameter.

If the data does not exist in the data source, the second parameter is used:

  • if it contains scalar data, this is used as the default value and returned by the method;
  • if it contains a function, the function is executed. Its return value is added to the data source, associated with the supplied key, and returned by the method. In this case, the third parameter can receive options that are used to store the data.

Example:

// Retrieving a cached CSS file. If the file is not cached,
// it is read and added to the cache with a lifetime of 10 minutes.
$user = $this->cache->read('style.css', function() {
    $css = file_get_contents('/path/to/style.css');
    return ($css);
}, 600);

7.3mRead()

mRead(array $keys) : array

Takes a list of keys as parameter, and returns an associative array with the keys and their contents.

Example:

// read multiple raw data from Redis
$data = $this->ndb->mRead(['key1', 'key2', 'key3']);

7.4copyFrom()

copyFrom(string $key, string $localPath, mixed $defaultOrCallback=null, mixed $options=null) : bool

Retrieves data and saves it in a local file.

If the data doesn't exist, the third and fourth parameters are used to return a default value or to generate the data (see the read method).

Example:

// retrieves a file from Amazon S3 and saves it locally
$this->s3->copyFrom('images/user1.jpg', '/var/data/img/login.jpg');

7.5mCopyFrom()

mCopyFrom(array $keys) : int

Takes as parameter an associative array whose keys are the identifiers of the data to be retrieved, and whose associated values are the paths of the files to which the data is to be written.

Example:

// retrieves multiple files from Amazon S3 and saves them locally
$this->s3->mCopyFrom([
    'images/user1.jpg' => '/var/data/img/login1.jpg',
    'images/user2.jpg' => '/var/data/img/login2.jpg',
]);

7.6write()

write(string $key, string $value, mixed $options=null) : mixed

Writes data to the data source.

  • 1st parameter: Key for data to be created or updated.
  • 2nd parameter: Textual (or binary) data content.
  • 3rd parameter: Possible option used for recording, depending on the type of data source.

Examples:

// writes data to a Redis server
$this->ndb->write('linux:year', '1991');

// writes data to Redis, with a lifetime of 10 minutes
$this->ndb->write('linux:year', '1991', 600);

7.7mWrite()

mWrite(array $data, mixed $options=null) : int

Writes multiple data.

  • 1st parameter: Associative array whose keys are the identifiers of the data to be written, and whose associated values are the contents of the data to be written.
  • 2nd parameter: Option used for all records, depending on data source type.

Example:

// writes multiple data to a Redis server
$this->ndb->mWrite([
    'qnx'      => '1984',
    'nextstep' => '1988',
    'solaris'  => '1990',
    'linux'    => '1991',
]);

7.8copyTo()

copyTo(string $key, string $localPath, mixed $options=null) : mixed

Writes data to the data source, copying the contents of a local file.

  • 1st parameter: Key to data to be created or updated.
  • 2nd parameter: Path to source file.
  • 3rd parameter: Possible option used for recording, depending on the type of data source.

Example:

// copy a local file to Amazon S3
$this->ndb->copyTo('css/style.css', '/var/data/css/style.css');

7.9mCopyTo()

mCopyTo(array $data, mixed $options=null) : int

Writes multiple data sets, copying the contents of several local files.

  • 1st parameter: Associative array whose keys are the identifiers of the data to be written, and whose associated values are the paths to the source files.
  • 2nd parameter: Option used for all records, depending on data source type.

Example:

// copy multiple local files to Amazon S3
$this->ndb->mCopyTo([
    'css/style.css'    => '/var/data/css/style.css',
    'js/app.js'        => '/var/data/js/app.js',
    'images/login.jpg' => '/var/data/img/login.jpg',
]);

8Connection management

Usually, you don't need to manage connections. On first access (read or write), the object connects to the data source. When the object is destroyed, disconnection is automatic.

However, there are rare cases where you may wish to explicitly open or close the connection. The connect(), reconnect() and disconnect() methods are there for this purpose.

Not all data sources manage connections and disconnections. For example, the File source doesn't make any connections (it accesses the desired files when reading or writing). The connect/disconnect methods can then still be called, but they do nothing.

8.1connect()

connect() : void

Connects to the data source.

If the connection has already been made, nothing happens.

8.2reconnect()

reconnect() : void

Reconnects to the data source.

If the connection was not open, it will be opened. If a connection was already open, it is first closed before being opened again.

8.3disconnect()

disconnect() : void

Closes the connection to the data source.