GNUrc

From ProgClub
Jump to: navigation, search

GNUrc is the GNU Remote Control software. That's the software that aids in the maintenance of your thermostats. For other projects see projects.

Status

v1.1 released, v2.0 under development.

Motivation

To seek operational efficiencies in air-conditioners thereby saving electricity and improving comfort.

Administration

Contributors

Members who have contributed to this project. Newest on top.

All contributors have agreed to the terms of the Contributor License Agreement. This excludes any upstream contributors who tend to have different administrative frameworks.

Upstream (GNU) contributors: GNU Remote Control Contributors

Copyright

Copyright 2015, Contributors.

License

Licensed under the AGPL.

Resources

Downloads

See the home page for downloads.

Source code

See the home page for source code.

Links

Jargon

Following are definitions for abbreviations and acronyms that we use:

svn
Subversion version control software
i18n
intl
Internationalization/translation
SQL
Structured Query Language
API
Application Programming Interface
MVC
Model/View/Controller
ORM
Object/Relational Mapping
BOM
Business Object Model
HTML
HyperText Markup Language
HTTP
Hypertext Transfer Protocol
XSRF
Cross-Site Request Forgery
AJAX
Asynchronous JavaScript and XML

Data format specification

In this section we document various data types and data formats used within the system.

Langtag format

Languages in the i18n subsystem are identified by a value known within the GNUrc system as a "langtag", short for "language tag".

The langtag format is defined by RFC-5646, Tags for Identifying Languages.

See the webstat langagg report for examples of encountered Accept-Language HTTP headers.

Translation content-types

The translation content-types indicate the type of content in the translation subsystem:

html
Content is HTML content, including HTML elements and HTML entities. HTML content is not preprocessed or converted before inclusion in HTML content. It is assumed to be safe (e.g. no malicious <script> elements, etc...) This type of content cannot be put directly into HTML attributes.
attr
Content is suitable for use in HTML attributes. I.e. no HTML elements (angle brackets) but HTML entities are supported. Note that attr content may be included in HTML attributes and therefore quote marks (both double and single) should be properly encoded as &quot; and &apos;. This HTML Attribute content is not preprocessed or converted before inclusion in HTML content. It is assumed to be safe (e.g. no malicious <script> elements, etc...)
text
Content is in plain text format. Will be converted to HTML before inclusion in HTML content. This means that no markup is supported by this content type.
safe
Safe content is textual content that does not contain any HTML element or HTML entity characters, i.e. <, >, and & -- this safe content can be included in other content, particularly HTML content, without HTML or other encoding as it cannot include unsafe characters.

Database specification

Database data types

You can read more about MySQL data types.

Name MySQL Name Size (Bytes) Min Max
ID8 TINYINT 1 -128 127
ID16 SMALLINT 2 -32,768 32,767
ID24 MEDIUMINT 3 -8,388,608 8,388,607
ID32 INT 4 -2,147,483,648 2,147,483,647
ID64 BIGINT 8 -9,223,372,036,854,775,808 9,223,372,036,854,775,807
Bool TINYINT(1) 1 0 1
Int8 TINYINT 1 -128 127
UInt8 TINYINT UNSIGNED 1 0 255
Int16 SMALLINT 2 -32,768 32,767
UInt16 SMALLINT UNSIGNED 2 0 65,535
Int24 MEDIUMINT 3 -8,388,608 8,388,607
UInt24 MEDIUMINT UNSIGNED 3 0 16,777,215
Int32 INT 4 -2,147,483,648 2,147,483,647
UInt32 INT UNSIGNED 4 0 4,294,967,295
Int64 BIGINT 8 -9,223,372,036,854,775,808 9,223,372,036,854,775,807
UInt64 BIGINT UNSIGNED 8 0 18,446,744,073,709,551,615
Single FLOAT 4
Double DOUBLE 8
Date DATE 3 1000-01-01 9999-12-31
Time TIME 3 -838:59:59 838:59:59
Datetime DATETIME 8 1000-01-01 00:00:00 9999-12-31 23:59:59
Timestamp TIMESTAMP 4 1970-01-01 00:00:01 2038-01-19 03:14:07
Char CHAR
Varchar VARCHAR
Longtext LONGTEXT
Enum ENUM( ..., ... )
Hash CHAR 32

Database collations

We use the following collations for string data. You can read more about MySQL Character Sets and Collations.

Name MySQL Name Bytes/Char Case-sensitive SQL
BINARY binary 1 Yes character set binary
ASCII_BIN ascii_bin 1 Yes charset ascii collate ascii_bin
ASCII ascii_general_ci 1 No charset ascii collate ascii_general_ci
UTF8_BIN utf8_bin 3 Yes charset utf8 collate utf8_bin
UTF8 utf8_general_ci 3 No charset utf8 collate utf8_general_ci

Database tables

The database has been completely replaced for v2.0.

v2_exception_log table

The v2_exception_log table is a registry of unhandled exceptions that have occurred in the application deployment.

Column Type Size Collation Null Default Extra
id ID24 No AUTO_INCREMENT
timestamp Timestamp No CURRENT_TIMESTAMP
previous_id ID24 No 0
type Varchar 64 ASCII No
message Longtext UTF8 No
code Int32 No 0
const Varchar 128 ASCII_BIN No
file Varchar 256 ASCII_BIN No
line UInt32 No 0
message_stack Longtext UTF8 No
trace Longtext UTF8 No
trace_data Longtext UTF8 No

v2_exception_log.previous_id column

The ID of the previous exception, if any; otherwise zero.

v2_exception_log.type column

The name of the PHP exception class.

v2_exception_log.message column

The message of the PHP exception.

v2_exception_log.code column

The exception code. When -ve values are error numbers.

v2_exception_log.const column

When code is -ve this field is the ERROR CONST label at the time of the error. See declaring error numbers for details of format.

v2_exception_log.file column

The full path to the PHP file that raised the exception. Note that only ASCII file/path names are supported.

v2_exception_log.line column

The line number of the expression that raised the exception.

v2_exception_log.message_stack column

All exception messages in a row. I.e. previous exception messages followed by consequent exception messages.

v2_exception_log.trace column

The callstack at time of exception as a string with one function call per line.

v2_exception_log.trace_data column

The callstack at time of exception as a JSON data structure representing the callstack.

v2_url table

The URL table stores URLs encountered by the system and indicates if redirection to particular URLs is permitted or not.

Column Type Size Collation Null Default Extra
hash Char 40 ASCII No
url Longtext ASCII_BIN No
redirect Bool No 0

v2_url.hash column

The SHA1 hash of the url column.

v2_url.url column

A URL. Supported URL schemes are 'http' and 'https'.

v2_url.redirect column

True if redirection to the URL is permitted, false otherwise.

v2_intl_language table

The v2_intl_language table defines languages supported by the user interface.

Column Type Size Collation Null Default Extra
langtag Varchar 64 ASCII No
fallback Varchar 64 ASCII No 'en'
english_name Varchar 64 UTF8 No
local_name Varchar 64 UTF8 No
active Bool No 0

v2_intl_language.langtag column

A language identifier in langtag format.

v2_intl_language.fallback column

A language identifier in langtag format that indicates the 'fallback' translation for this language. That is, which translation will be used if there is no translation for the current language. The default value is 'en', which means fall back to the default English content.

v2_intl_language.english_name column

The name of this language, in English.

v2_intl_language.local_name column

The name of this language, in its own language.

v2_intl_language.active column

True if language is available in the user-interface, false otherwise. Languages are created as deactivated during development and after translation is complete the language is activated making it a language available to users.

v2_intl_context table

The v2_intl_context table defines message contexts for use in translation.

Column Type Size Collation Null Default Extra
context Varchar 64 ASCII No
seq Int24 No

v2_intl_context.context column

The name of the translation context.

v2_intl_context.seq column

The order of this record, i.e. its sequence.

v2_intl_message table

The v2_intl_message table supports messages which are English language content available for translation.

Column Type Size Collation Null Default Extra
type Enum( html, attr, text, safe ) 4 No
context Varchar 64 ASCII No
hash Hash 32 ASCII No
content Longtext UTF8 No
seq ID24 No AUTO_INCREMENT
seq UInt24 No

v2_intl_message.type column

The message's translation content-types. I.e. html, attr, text, safe.

v2_intl_message.context column

The message's message context. Message contexts are a label that indicates the context of a message. For example the message contexts 'Nationality' and 'Language' could be used as the context for the message 'Japanese'; without contexts messages might be ambiguous.

v2_intl_message.hash column

The message's hash is an MD5 hash of the normalized content. See below for an explanation of the normalization process.

v2_intl_message.content column

The message's content is a normalized English message available for translation. Normalization is the process of removing leading and trailing whitespace and converting duplicate space characters into a single space character. Message content is normalized before being hashed and/or stored.

v2_intl_message.seq column

This is an auto-increment field which indicates the sequence the message contents was inserted.

v2_intl_message.words column

The words column stores the number of English words in the message content.

v2_intl_translation table

The v2_intl_translation table contains native language translations of message content.

Column Type Size Collation Null Default Extra
langtag Varchar 64 ASCII No

v2_intl_translation.langtag column

A language identifier in langtag format.

Functional specification

The functional specification describes what the project does.

Administration features

Administration dashboard

The administration dashboard (admin-home.php) is an administrator's home page. From the administration dashboard an administrator can access:

  1. User administration
  2. Translation administration
  3. Group administration
  4. Error reporting

User administration

Translation administration

Group administration

Error reporting

Translation features

Standard features

Role-based security

We use role-based security to limit system functions to particular classes of users. There are three user roles and any given user can be in any combination of the roles:

  1. Administrator
  2. Translator
  3. Standard

Administrators have access to administration functions. Non-administrators do not have access to administration functions. Translators have access to translation functions. Non-translators do not have access to translation functions. All users have access to the standard functions.

Internationalization

The i18n functional spec is documented on the Language Administration Help page.

Backend features

Error logging

Error log testing

Technical specification

The technical specification describes how the project works.

Model-view-controller

GNUrc 2.0 implements the Model-View-Controller (MVC) design pattern.

Software layers

The GNUrc software is layered:

Layer Directory MVC role
Function Libraries /src/1-lib
Objects /src/2-obj Model/View
Data Access Layer (DAL) /src/3-dal Model
Object/Relational Mapping (ORM) /src/4-orm Model
Business Object Model (BOM) /src/5-bom Model
View helpers /src/6-view View
Ajax Controllers /src/7-ajax Controller
Page Controllers /web Controller

Each layer depends on (up to) all of the previous layers. So controllers use views, BOM, ORM, DAL, objects, and libraries. Views use BOM, ORM, DAL, objects, and libraries. The BOM uses the ORM, DAL, objects, and libraries. And so on. Generally controllers should call on the services of the BOM rather than calling on the services of the DAL directly, so the BOM encapsulates business logic and mediates it into the DAL. If higher layers can encapsulate functionality in lower layers that is a good thing to do.

Services and modules

In order to facilitate a team of developers working on the code base at the same time we implement a software module pattern. Note that this is a "political" rather than "technical" or "logical" categorisation of functionality. We separate our code into modules that developers can "own" while they're working on the project. The idea is to avoid changing files that other developers are working on so as to reduce conflicts.

The idea of software modules is to split functionality into parts which can be managed by one developer at a time. For example consider the Data Access Layer (DAL) service. We could have one DAL object that had methods for manipulating thermostats, users, and groups. The problem is that if Miguel is working on thermostats and Federico is working on users, if they both edit the single DAL class then they run the risk of overwriting each others' work. By splitting functionality for thermostats, users, and groups into DAL modules each of our developers can work on their modules without interfering with code used by other developers.

So "services" are an interface into a collection of modules, and "modules" are a bunch of features/functionality implemented by one developer at a time.

The software module pattern is employed by the following services:

Service Modules
DAL /src/3-dal/*/module
BOM /src/5-bom/module
AJAX Controller /src/7-ajax/module
Page Controller /web

Directory structure

The software has the following directory structure starting in the base:

/

The base directory is /path/to/your/gnurc.

  • .svn-ignore -- files for Subversion (svn) to ignore
    e.g.: $ svn propset svn:ignore -RF .svn-ignore .
  • config.example.php -- an example config file
  • config.php -- the production config file
    e.g.: $ cp config.example.php config.php; vim config.php

/dat

The /dat directory is for data files.

/etc

The /etc directory is for miscellaneous scripts.

/etc/dbscripts

The /etc/dbscripts directory is an svn:externals for the database scripts: 

svn://svn.sv.gnu.org/remotecontrol/branches/development/dbscripts

/src

The /src directory is for most of the source code.

  • include.php -- the main include file for loading the GNUrc software components.
  • test.php -- the main include file for use by PHPUnit unit tests.

/src/1-lib

The /src/1-lib directory contains function libraries.

/src/2-obj

The /src/2-obj directory contains PHP classes used by GNUrc that do not fit in another category.

/src/3-dal

The /src/3-dal directory contains data access layer facilities.

  • GrcDal.php -- the include file for the Data Access Layer; defaults to MySQL/PDO API.
  • GrcDalModule.php -- a base class for DAL Modules.
/src/3-dal/mysql-pdo

The /src/3-dal/mysql-pdo directory contains data access layer facilities for the MySQL/PDO database connectivity.

/src/3-dal/mysql-pdo/module

The /src/3-dal/mysql-pdo/module directory contains DAL modules.

/src/4-orm

The /src/4-orm directory contains Object/Relational Mappers. The ORM strategy that we employ is known as the Active Record pattern. An Active Record is a class that mediates data between the program (a PHP object) and the database (a record in a table). As a general rule (which can be broken) the Active Record is supposed to be CRUD only, any other business logic and/or validation should be implemented as a business process in the Business Object Model.

/src/5-bom

The /src/5-bom directory contains an interface into "business process" via the Business Object Model (GrcBom):

  • GrcBom -- the business process interface
/src/5-bom/module

The /src/5-bom/module directory contains business process modules accessed via the BOM.

/src/6-view

The /src/6-view directory contains libraries of helper functions for view composition:

/src/7-ajax

The /src/7-ajax directory contains AJAX controllers.

  • GrcAjax -- a service interface into AJAX operations (modules)
/src/7-ajax/module

The /src/7-ajax/module directory contains AJAX modules exposed by the GrcAjax service.

/test

The /test directory contains Unit Tests for the project.

/test/php

The /test/php directory contains test cases that demonstrate a PHP feature.

/test/src

The /test/src directory contains test cases that test a program feature.

/web

The /web directory contains front-facing components. Those are components which are accessible via HTTP(S). Each PHP file in the /web directory is known as a Page Controller. In addition to Page Controllers the /web directory includes:

Managing errors

The error management subsystem is comprised of the /src/2-obj/GrcError.php class and the 'err' and 'Error' functions in /src/1-lib/01-api.php.

Errors are managed in two modes: first they are 'declared' and second they are 'raised'.

Declaring error numbers

To declare a possible error call the err()->define function. The first argument is the name of the error known as the error const. The error const has up to 6 parts separated by double-underscore:

  1. 'ERROR'
  2. type (e.g. 'LIB', 'OBJ', etc.)
  3. module/class (e.g. 'API', 'VALIDATION')
  4. function name
  5. variable name
  6. state (e.g. 'NOT_NULL', 'INVALID')

The second argument is the error message. The error messages will be translated if necessary and can include variables using the '%variable%' notation of the i18n subsystem.

Raising particular errors

After you have declared an error like this: 

err()->define( 'ERROR__LIB__EXAMPLE', 'An error with %param% occurred.' );

you can raise it like this: 

throw Error( ERROR__LIB__EXAMPLE, 'param', 'value', intl_context, previous_exception );

where intl_context is an i18n translation context and previous_exception is a previous exception if any.

Security features

In addition to role-based security we protect from XSRF and SQL-injection attacks.

XSRF protection

To prevent XSRF attacks we configure a session token that must be included in all HTTP POST submissions.

SQL-injection protection

To prevent SQL injection attacks we take care to escape inputs when building SQL strings and/or use parameters with our database API.

Character encoding

All textual content is encoded, stored, and transmitted as UTF-8.

HTML charset

Read about character encodings in HTML for information about how to declare the UTF-8 charset in various HTML versions.

HTTP Content-Type

When sending HTTP replies the Content-Type should be specified with a charset parameter as described here.

For HTML the Content-Type would be: 

Content-Type: text/html;charset=utf-8

For JSON the Content-Type would be: 

Content-Type: application/json;charset=utf-8

And so on.

Internationalization (i18n) API

Internationalization functions defined in /src/1-lib/45-intl.php are the interface into the i18n API. They expose functions for the 01n paradigm and for table-heading/HTML/HTML-attribute/text/safe-text content types. See the functional specification for details on the 01n paradigm and content types.

In addition to the above API the HTML composition system supports automatic translation of content via the following methods:

  • GrcHtmlElement->add_nbsp: add HTML after converting spaces to &nbsp;
  • GrcHtmlElement->add_html: add HTML content (no escaping)
  • GrcHtmlElement->add_h01n: add HTML 01n paradigm
  • GrcHtmlElement->add_attr: add HTML attribute content (no escaping, angle brackets should not be present)
  • GrcHtmlElement->add_a01n: add HTML attribute 01n paradigm
  • GrcHtmlElement->add_text: add text content (HTML characters escaped)
  • GrcHtmlElement->add_t01n: add text 01n paradigm
  • GrcHtmlElement->add_safe: add safe text content (content must not include HTML special chars)
  • GrcHtmlElement->add_s01n: add safe text 01n paradigm

Web interface

admin-home.php

The admin-home.php

Notes

Notes for implementers

If you are interested in incorporating this software into your project, here's what you need to know:

Notes for developers

If you're looking to set up a development environment for this project here's what you need to know:

Setting up a development environment

$ svn co svn+ssh://USERNAME@svn.savannah.gnu.org/remotecontrol/branches/development/jj5-mvc gnurc-jj5-mvc

Note: substitute USERNAME with your Savannah username. Make sure your firewall has ports 22 and 3690 open. 

$ cd gnurc-jj5-mvc
$ cp config.example.php config.php

Tasks

TODO

Things to do, in rough order of priority:

  • Document system design
  • Generate task list
  • Allocate tasks to developers

Done

Stuff that's done. Latest stuff on top.

  • JE 2015-05-07: database specification progress
  • JE 2015-04-01: created project page
Retrieved from "https://www.progclub.org/wiki/mediawiki/index.php?title=GNUrc&oldid=4842"