Ciliatus

Documentation

The Essentials

Animals

The goal of Ciliatus is to make your animals' life happier and your's easier. Animals are connected to their Terrarium and store information about their health history and vital parameters.

You can define weighing and feeding schedules and easily document weights and feedings. As soon as your schedule tells you to feed or weigh you will get notified on your dashboard or even on your smart phone.

The biography feature allows you to document everything about your animal in your self defined categories. Use cases include laying of eggs, sheddings, strange behaviour and more.

From the data collected about your animal Ciliatus can generate comprehensive care sheets.
Care sheets show the latest feeding and weight history, biography entries and environment data from the animal's terrarium in a printable format, ready to be taken along to the vet in case of emergency.

Terraria

Terraria are the core of monitoring and automation. They have related components like valves and pumps and use action sequences to orchestrate it's components into automated processes.

They also have sensors which store a time series of readings to keep track of humidity and temperature.

Ciliatus automation focuses on keeping a terrarium's vital parameters within limits by applying action sequences once sensor readings come close to or exceed critical levels.

Control Units

A Control Unit can be virtually anything that is able to submit sensor readings via an HTTP request or manage physical components like pumps and valves. The recommended software is the Ciliatus Control Unit, which can be easily set up on any Windows or Linux system. It's main focus lies on using a Raspberry Pi to control components or read from sensors via I2C or GPIO.

Ciliatus offers an API to store sensor readings and associating them with their respective sensor and therefore with a terrarium and animal.

It is recommended to create special users for each Control Unit because they only require a special subset of permissions to submit sensor readings and fetch desired states. You should grant the following permissions:

  • grant_api-list
  • grant_api-read
  • grant_api-write:controlunit
  • grant_api-write:sensorreading
  • grant_api-fetch:desired_states

See also:

Sensors

Sensors

Sensors are split into two categories:

  • Physical sensors
  • Logical sensors

The physical sensor means the actual device you place in your terrarium and connect to a controlunit.

A logical sensor on the other hand describes the types of sensor included in a physical sensor. For example a humidity and temperature sensor combined within one physical appliance.

Sensor readings can only be of one type and are therefore always related to a logical sensor - not the physical sensor.

A logical sensor can have multiple thresholds. A threshold defines a reference value and whether the sensor's reading should stay below or above this value. The threshold's starts_at property defines a time of day from which on the threshold is active. The currently active threshold per sensor is determined by sorting them by the threshold's starts_at property in descending order and selecting the first one which's starts_at property is lower then the current time of day.

If a logical sensor's raw value is not within the currently active threshold's bounds, it will be considered as critical and a critcal state will be created.

Components

Introduction

In Ciliatus components describe everything you want to use for automation. There are fixed component types like and Valves and Pumps, but you can also define own components types to integrate custom components in your processes.

Pumps and Valves

To actually automate tasks such as irrigating you need actors like pumps and valves. They each have their own component type are automatically used in irrigation action sequences. See also Automation - Action Sequences.

The idea behind these two components is that you have an array of terraria, each one with a valve to allow water to flow to a water nozzle within the terrarium and multiple valves connected to one pump. Of course you can also have a pump for each valve.

Custom Components

To allow more flexibility you can also define custom component types with their own properties, possible states and intentions. In our demo environment for example we have custom component types for fans to cool and heating lamps. You can think of a custom component type as a template to create components by.

To enable automation you can allow custom components to be leveraged by defining their intentions, such as increasing humidity. See also Components - Intentions.

Intentions

Each acting component (valves, pumps, custom components) has one or more intentions. An intention defines a terrarium parameter and how the component can influence this parameter. By default valves and pumps have the intention to increase humidity. You could for example create a custom component type for your heating lamps with the intention to heat. Ciliatus can now automatically determine that it needs to activate a terrarium's heating lamp when the temperature rises to far.

Monitoring

Sensor readings

A sensor reading represents a value of a logical sensor at a given time.

When submitting sensor readings you have to assign group IDs so Ciliatus knows which readings belong together. This means every time you start to retrieve and submit sensor data you have to generate a new random UUID as a group id (see example).

This allows Ciliatus to:

  • Calculate averages for a terrarium at a given time if you have multiple sensors of the same type installed
  • Match readings of different types for a terrarium to a single point in time if two related readings (e.g. humidity and temperature of the same terrarium) are not submitted at the exact same time to avoid gaps in graphs.

This is an example procedure of how you could submit sensor data to Ciliatus with two logical sensors.

  • Generate new group id
  • Read sensor 1
  • Submit sensor 1 (see API example)
  • Read sensor 2
  • Submit sensor 2
  • Sleep for n seconds
  • Go to step 1

The API call has to be pointed as a POST request to the endpoint /api/v1/sensorreadings.

{
    "group_id": "<random uuid>",
    "logical_sensor_id": "22c2db80-526f-11e6-a3b4-9f0c220ec948",
    "rawvalue": 23.5902932
}

All sensor readings will be readable as a graph in the terrarium details. The last few hours (or what you defined in your config) are displayed as graphs over your terrarium card's title image.

Among other features submitting of sensor readings is implemented in python in Ciliatus Control Unit.

Why no time series database?

Ciliatus, despite it's complexity, should be as easy to set up as possible. Therefore time series databases like the implemented InfluxDB are only used to push metrics to. Ciliatus itself will always query the MySQL database.

Maybe a way to query time series databases internally will be implemented in the future, but there are no concrete plans at the moment.

Critical states

Critical states are created and recovered by evaluating logical sensor readings and comparing them to their sensor's assigned thresholds. Upon creation a critical state is considered "soft". If the reading stays above or below it's threshold for the sensor's soft_state_duration_minutes time (or environment variable DEFAULT_SOFT_STATE_DURATION_MINUTES if the property is null) the soft-flag will be removed and notifications will be sent out.

The Critical state will call check_notifications_enabled on the model it belongs to. (e.g. a LogicalSensor). This model will either check it's own notifications_enabled property or again asks the model it belongs to by calling check_notifications_enabled on it, until a model with check_notifications_enabled is found. If none is found, no notifications will be sent out. When the sensor is in healthy condition again, the critical state will again check whether it should send out notifications, send them if necessary and removes itself.

Notifications

Ciliatus allows you to send out notifications for critical sensor readings and general messages like reminders via different notification providers. You can use the already implemented Telegram provider or create your own.

Telegram

Telegram allows you to create bots by using their master bot BotFather. Detailed instructions can be found in the Telegram documentation.

After you have obtained your bot's authorization token, set the TELEGRAM_BOT_TOKEN and TELEGRAM_BOT_NAME variables in .env. To be able to receive bot messages, you now have to enable Telegram's webhook feature. This will ask Telegram to redirect all messages directed to your bot via POST to the Ciliatus API. Learn how to set up your webhook here and make sure to set your TELEGRAM_WEBHOOK_TOKEN variable first. The webhook URL will be https://yourdomain.com/telegram/<TOKEN>.

After you finished the setup, you can go to you user settings and start the Telegram setup for your account.

Troubleshooting

  • 1. No messages reach my server

This can have multiple causes. First make sure the webhook URL is correct and the token is defined in your .env

Telegram will only push you messages if you have a valid SSL certificate. You can either use a certificate signed by a public CA (e.g. you can obtain Let's Encrypt certificates for free) or use a self signed certificate. Make sure you use the certificate parameter when calling Telegram's setWebhook method when using self signed certificates, like it's explained in their documentation.

If you have multiple SSL certificates for multiple domains on one host it could be required that your server serves the certificate of your ciliatus domain first, in case Telegram can't handle SNI requests (I'm not 100% on this yet)

  • 2. I don't receive messages from my bot

Make sure you have contacted your bot first. Telegram bots can only talk to you, if you have initiated the conversation with /start. Simply follow the Telegram setup in your Ciliatus user settings.

Automation

Actions

An action describes a state for a component (e.g. turn Valve or Pump on). Actions are arranged within an action sequence to fulfill certain intentions.

Action sequences

An action sequence describes a sequence of actions. For example an action sequence to irrigate a terrarium contains two actions per default:

  • Open a valve (for 10 minutes)
  • Start a pump (for 10 minutes)

Each action is sorted within it's sequence using the sequence_sort_id property. When fetching desired states, each scheduled action sequence will evaluate it's actions using this property and check if the dependencies to run this action are met (wait_for_started_action_id, wait_for_finished_action_id properties).

Action Sequence Schedules

Action sequence schedules are used to schedule action sequences. They can be executed as one time schedules or daily.

Schedules are automatically generated for one time action sequences, for example when clicking on irrigation from the terrarium card. The schedule will be markes as run once and removed after it completes.

Action Sequence Triggers

Action sequence triggers are used to start an action sequence once the selected sensor's value reaches a certain state. You can also define how long the sensor value has to stay above/below this value and how often the action sequence is allowed to be started by this trigger within a certain timeframe.

You can use triggers to react to situations before a critical state occurs. To automatically react to critical states you should use Intentions.

For example: "Start ActionSequence once LogicalSensor is greater/lesser than reference value for duration minutes, but not more often then every timeout minutes and only between timeframestart and timeframeend o'clock"

Action Sequence Intentions

When assigning an intention to an action sequence you thereby define it's usage. You can select a type of sensor reading (e.g. humidity) and the intended effect the action sequence has on this value (e.g. lower humidity).

Intentions are triggered as soon as a critical state is active and not in soft state.

For example: "If CriticalState is active because sensor reading type is higher/lower than it's threshold find ActionSequence with ActionSequenceIntention to increase/decrease this value, but not more often then every timeout minutes and only between timeframestart and timeframeend o'clock"

Desired states

Once an action sequence is running a Control Unit can fetch it's components' desired states. When a control unit queries Ciliatus to fetch desired states for components controlled by this Control Unit, every action which should run is copied to a RunningAction. When the action's duration is up relative to the RunningAction's started_at property, it's finished_at property will be set - if all action's associated with this schedule are finished, they're RunningActions will be deleted.

Control Units can query ciliatus to receive a list of components which should be turned on at this time. Therefore Control Units don't need to implemented any kind of logic.

This means if a Control Unit controls a component which's state is affected by an action sequence (e.g. Humidify a terrarium -> start Pump) it's desired state will be returned. If no desired state is returned for a component it should be off.

It is advised to have a safe guard in place to turn off components in case the Ciliatus API is not reachable.

API calls need to be pointed as a GET request to the endpoint /api/v1/controlunits/<id>/fetch_desired_states. Results look like this:

{
  "http_code": 200,
  "data": {
    "Valve": {
      "6d522760-526b-11e6-9bbe-7d56150a362c": "running"
    },
    "Pump": {
      "68678610-526b-11e6-b6a0-090ef00e70e6": "running"
    }
  },
  "meta":[]
}

Take a look at the Ciliatus Control Unit.

Suggestions

Every sunday you will receive suggestions on your dashboard recommending the creation of action sequences and schedules.

They are generated by evaluating critical states and measuring when they occur the most and whether the amount of critical states at a specific time of day is repeatingly over the threshold.

API

Authentication

All API endpoints use OAuth authentication using Laravel Passport.

To authenticate yourself, a Control Unit or any other application go to user settings and create a security token in the security tab.

Your HTTP requests have to submit the generated token using an Authorization header:

Authorization: Bearer YourToken
Endpoints

This explains the basic structure of the API URLs. For more details regarding REST APIs, please refer to the Laravel documentation.

For this example we will be using the Animal model.

Index all entities

  • Endpoint: /animals
  • Method: GET
  • Parameters:
    boolean raw - Removes pagination and returns all animals. Requires grant_api-list:raw permission.
  • Returns: Paginated list of animals
  • Requires ability: grant_api-list, optional without pagination: grant_api-list:raw

Show single entity

  • Endpoint: /animals/$id
  • Method: GET
  • Parameters:
    uuid id - Animal ID
  • Returns: Transformed animal object
  • Requires ability: grant_api-read

Create entity

  • Endpoint: /animals
  • Method: POST
  • Parameters: Depending on the model, usually at least:
    string name
  • Returns: Success/Error details and id of the new model
  • Requires ability: grant_api-write:animal

Delete entity

  • Endpoint: /animals/$id
  • Method: DELETE
  • Parameters:
    uuid id - Animal ID
  • Returns: Success/Error details
  • Requires ability: grant_api-write:animal

Update

  • Endpoint: /animals/$id
  • Method: PUT/PATCH
  • Parameters:
    uuid id - Animal ID
  • Returns: Success/Error details
  • Requires ability: grant_api-write:animal

Error Codes

  • 200 - OK
  • 401 - Unauthorized
  • 404 - Model not found
  • 422 - Unprocessable Entity (in case of missing parameters, etc)
  • 500 - Server Error
Responses

The base JSON the API returns consists of data or error and meta:

{
    "data": {},
    "error": {},
    "meta": {}
}

data field

The data field can contain a single model, or a paginated list of models. In the later case you will also find pagination information in the meta section. See the components' wiki pages for detailed information.

Usually, data contains more details when requesting an explicit model (show) instead of indexing.

meta field

redirect contains redirect information for the ciliatus frontend. This is useful upon creating a new model and you don't know the model's id yet to redirect from the frontend.

redirect: {
    "url": "https://yourdomain.com/animals",
    "delay": 200
}

pagination field

pagination contains information about the current page, number of pages, number of items and items per page

pagination: {
    "total_items": 147,
    "total_pages": 10,
    "current_page": 1,
    "per_page": 15
}

error field

error contains information in case the request was not successful. The error_code is a model controller and event specific code. The namespace (part before the x) identifies the model controller and the event identifier (part after the x) the event. The event identifiers 101 - 1FF are reserved for common events like model not found. 201 - FFF are custom events.

error: {
    "error_code": "",
    "message": ""
}
Monitoring Ciliatus

Ciliatus exposes internal performance metrics via an HTTP API endpoint at /api/v1/system/health.
The metrics include execution times, notification and sensor reading rates and a list of the longest running requests.

You can find a monitoring plugin for Icinga 2 on Icinga Exchange or directly on GitHub.

Other features

API.AI / Voice control

You can query Ciliatus using simple voice commands using the free platform API.AI.

To enable this feature you need create an account at API.AI. You need to configure the agent in resources/assets/api.ai/YOUR_LANGUAGE/agent.json and replace YOUR_CILIATUS_HOST with your server's URL, YOUR_BEARER_TOKEN with a personal access token you create in user settings and YOUR_APP_ID with your app's ID or another random string. Then you can import the Ciliatus agent for your language by compressing the folder to a ZIP and importing in the API.AI settings. If you want to use multiple languages you need to create and import and agent for every language.

You might need to enable and configure the API endpoint on API.AI in the Fulfilment settings again after importing.

It is also required that you train your agent for your language and your animal and terrarium names. Otherwise it might not recognize your animals by name. The german agent is used regularly in our environment and we will try to update the trained agent from time to time, the other languages only received basic training. Please share your trained agents in other languages. Thank you.

After that you can enable the API.AI feature by setting an environment variable in .env for every language and the matching agent's Client access token. For german and english for example the config will look like this:

API_AI_ACCESS_TOKEN_de=RandomIdForGermanAgent
API_AI_ACCESS_TOKEN_en=RandomIdForEnglishAgent

Currently there are three intents implemented.

Animal health - Examples: "How is animal name", "Is animal name ok?", ...
Schedules animal feeding - Examples: "When do I have to feed animal name", "When is animal name feeding due", ...
Get today's feedings - Examples: "Who do I have to feed today?", "Who is hungry?", ...

InfluxDB

InfluxDB is an open source time series database with no external dependencies. It's useful for recording metrics, events, and performing analytics.

Ciliatus can write to InfluxDB upon creating sensor readings. This allows you to connect a multitude of third-party applications to evaluate and process your data, like Grafana.

To use InfluxDB you just need to create a database and set up the INFLUX_* keys in your .env file. Ciliatus requires you to use an SSL certificate for secure communication with InfluxDB.

Installation and Setup

Overview

View diagram

Installation

The easiest way to install Ciliatus is to clone the latest build from GitHub:

git clone -b release/2.0 --single-branch https://github.com/ciliatus/ciliatus.git

After all files have been downloaded you need to set some permissions for the PHP framework Laravel on which Ciliatus is built to work properly:

cd ciliatus
sudo chown -R www-data:www-data .
sudo find . -type f -exec chmod 644 {} \;
sudo find . -type d -exec chmod 755 {} \;
sudo chmod -R ug+rwx ./bootstrap/cache

To allow Laravel to route HTTP requests you need to set up your web server accordingly.

Apache
Options +FollowSymLinks
RewriteEngine On

RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^ index.php [L]
Nginx
location / {
    try_files $uri $uri/ /index.php?$query_string;
}

If you're having trouble with the basic setup of your webserver you can consult the Laravel documentation or the excellent e-learning platform Laracasts

Setup

After you completed the installation it's time to configure some environment variables. Make sure you're still in your Ciliatus root directory.

Let's start with downloading all required libraries. Make sure you have Composer installed on your system before continuing. To install the libraries, call this command:

composer update

Ciliatus stores it's basic configuration in a file called .env. You already have an example file in your root directory called .env.example. Just rename it to .env. Now generate your own unique app key. The app key is required for everything related to encryption within your application. For example user sessions.

php artisan key:generate

Now open your .envfile and complete the basic configuration. We will go over the basic steps now. If you like in depth information about this file, please consult the Laravel documentation.

App section

You can turn APP_DEBUG on by setting it to true for easier debugging, but make sure to turn it off before making your site accessible from the internet.
APP_URL should be set to your websites base URL, e.g. https://mydomain.com.

Database section

Configure your database connection here. Using MySQL/MariaDB is strongly recommended. Other drivers are untested with Ciliatus. More

Cache, Session, Queue section

It is strongly recommended to use a caching driver. To use the default driver redis please configure redis first. See your distribution's or hoster's manual for that. Ciliatus also plays well with memcached as an alternative to redis.

To configure redis you need to setup the EDIS_ prefixed .envvariables but i will probably work with the default settings if you just installed redis without additional configuration.

If you don't want to use any caching driver, set CACHE_DRIVER to file.

Telegram section

If you intend to use a Telegram bot for push notifications, you can configure your authentication settings here. Please consult the Telegram section for more information.

Broadcasting and Push section

To make full use of Ciliatus' real-time functionality it is required to use a push service for real-time events. The recommended service is Pusher. If you don't want to use event broadcasting, set BROADCAST_DRIVER to log.

If you want to use something else than Pusher, you'll need to configure the credentials resources/assets/js/echo.js

Migrations

Next up you need to populate your database. Again go to your Ciliatus root directory and run

php artisan migrate
Passport

Laravel Passport is used to authenticate API requests with Ciliatus. You need to setup Passport before continuing. This is again a straight forward artisan command:

php artisan passport:install
Finalizing

After completing your configuration you can start setting up Ciliatus from your browser. Just copy the content of your app key from .env and go to https://mydomain.com/setup/yourappkeyhere.

Schedules

Ciliatus uses artisan schedules to run timed tasks. These tasks include:

  • Creating/deleting/updating critical states
  • Sending scheduled notifications like alerts
  • Rebuilding caches to accelerate page load times

The simplest way to achieve this is to set up a cronjob.

# m h  dom mon dow   command
  * *   *   *   *    php /var/www/ciliatus/artisan schedule:run
Update
Updating to v2.0
  • 1. Backup your files and database
  • 2. Overwrite all existing files with the files from v2.0
  • 3. Run artisan to update and convert your database:
php artisan ciliatus:update:v2.0
  • 4. Update dependencies with composer:
composer update

Help

Troubleshooting

Most problems can be identified by checking your webserver's log files as well as the laravel log located in storage/logs/laravel.log

This overview will give basic tips for a Ciliatus installation on Ubuntu 16.04 using PHP 7.0 (FPM).

Telegram problems? Look here.

PusherException: There is missing dependant extensions - please ensure both cURL and JSON modules are installed

Install the missing dependancies like this:

apt-get install php7.0-curl php7.0-json
RuntimeException: The only supported ciphers are AES-128-CBC and AES-256-CBC with the correct key lengths

You might have forgotten to generate an application key:

php artisan key:generate.
Class 'Memcached' not found

Make sure you have installed Memcache for php: apt-get install memcached or change the CACHE_DRIVER in your .env file to file.

If the error occurs during installation run optimize after fixing the issue.

php artisan optimize
Could not establish Memcached connection.

Make sure your memcache service is running.

service memcache status
QueryException: could not find driver (SQL: select * from properties where type = SetupCompleted)

You probably forgot to install the php driver for your chosen database provider.

In case of MySQL/MariaDB you can install them like this:

apt-get install php7.0-mysql
BroadcastException in PusherBroadcaster.php: 404 NOT FOUND

If you don't intend to use Pusher or another push service, you'll have to change the default broadcast driver to log. You should do this in your .env file by changing the variable BROADCAST_DRIVER.

laravel/framework vX.y.z requires ext-mbstring * -> the requested PHP extension mbstring is missing from your system.

Install mbstring.

apt-get install php7.0-mbstring
phpunit/phpunit X.y.z requires ext-dom * -> the requested PHP extension dom is missing from your system.

Install the php XML extension:

apt-get install php7.0-xml
GD Library extension not available with this PHP installation.

Install the php GD extension:

apt-get install php7.0-gd
Contact

Feel free to contact me at info@ciliatus.io.

Keep in mind that this is an open source project and I work full time, so it may take some time until I'm able to get back to you. I'll do my best to answer your questions as fast as possible.