Ciliatus

Documentation

Getting started - v1.10-beta

Installation

The easiest way to install Ciliatus is to clone the latest build from GitHub:

git clone -b release/1.10 --single-branch https://github.com/ciliatus/ciliatus.git

After all files have been downloaded you need to set some permissions for the PHP framework Laravel on which Ciliatus is built to work properly:

cd ciliatus
sudo chown -R www-data:www-data .
sudo find . -type f -exec chmod 644 {} \;
sudo find . -type d -exec chmod 755 {} \;
sudo chmod -R ug+rwx ./bootstrap/cache

To allow Laravel to route HTTP requests you need to set up your web server accordingly.

Apache
Options +FollowSymLinks
RewriteEngine On

RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^ index.php [L]
Nginx
location / {
    try_files $uri $uri/ /index.php?$query_string;
}

If you're having trouble with the basic setup of your webserver you can consult the Laravel documentation or the excellent e-learning platform Laracasts

Setup

After you completed the installation it's time to configure some environment variables. Make sure you're still in your Ciliatus root directory.

Let's start with downloading all required libraries. Make sure you have Composer installed on your system before continuing. To install the libraries, call this command:

composer update

Ciliatus stores it's basic configuration in a file called .env. You already have an example file in your root directory called .env.example. Just rename it to .env. Now generate your own unique app key. The app key is required for everything related to encryption within your application. For example user sessions.

php artisan key:generate

Now open your .envfile and complete the basic configuration. We will go over the basic steps now. If you like in depth information about this file, please consult the Laravel documentation.

App section

You can turn APP_DEBUG on by setting it to true for easier debugging, but make sure to turn it off before making your site accessible from the internet.
APP_URL should be set to your websites base URL, e.g. https://mydomain.com.

Database section

Configure your database connection here. Using MySQL/MariaDB is strongly recommended. Other drivers are untested with Ciliatus. More

Cache, Session, Queue section

It is strongly recommended to use a caching driver. To use the default driver redis please configure redis first. See your distribution's or hoster's manual for that. Ciliatus also plays well with memcached as an alternative to redis.

To configure redis you need to setup the EDIS_ prefixed .envvariables but i will probably work with the default settings if you just installed redis without additional configuration.

If you don't want to use any caching driver, set CACHE_DRIVER to file.

Telegram section

If you intend to use a Telegram bot for push notifications, you can configure your authentication settings here. Please consult the Telegram section for more information.

Broadcasting and Push section

To make full use of Ciliatus' real-time functionality it is required to use a push service for real-time events. The recommended service is Pusher. If you don't want to use event broadcasting, set BROADCAST_DRIVER to log.

Migrations

Next up you need to populate your database. Again go to your Ciliatus root directory and run

php artisan migrate
Passport

Laravel Passport is used to authenticate API requests with Ciliatus. You need to setup Passport before continuing. This is again a straight forward artisan command:

php artisan passport:install
Finalizing

After completing your configuration you can start setting up Ciliatus from your browser. Just copy the content of your app key from .env and go to https://mydomain.com/setup/yourappkeyhere.

Schedules

Ciliatus uses artisan schedules to run timed tasks. These tasks include:

  • Creating/deleting/updating critical states
  • Sending scheduled notifications like alerts
  • Rebuilding caches to accelerate page load times
  • The simplest way to achieve this is to set up a cronjob.

    # m h  dom mon dow   command
      * *   *   *   *    php /var/www/ciliatus/artisan schedule:run
    Update

    Updating to v1.10-beta

    • 1. Overwrite all existing files with the files from v1.10-beta
    • 2. Update dependencies:
      composer update
    • 3. Update database:
      php artisan migrate

    Components

    Introduction

    In Ciliatus components describe everything you want to manage, monitor or automate. There are fixed component types like Animals, Terraria and Physical Sensors, but you can also define own components types to integrate custom components in your processes.

    Animals

    The goal of Ciliatus is to make your animals' life happier and your's easier. The Animals component stores all your animals, documents their health and history and connects them to their terrarium.

    You can define weighing and feeding schedules and easily document weights and feedings. As soon as your schedule tells you to feed or weigh you will get notified on your dashboard or event on your smartphone if you set it up.

    The biography feature allows you to document everything about your animal in your self defined categories. Use cases include laying of eggs, sheddings, strange behaviour and more.

    From the data collected about your animal Ciliatus can generate comprehensive care sheets.
    Care sheets show the latest feeding and weight history, biography entries and environment data from the animal's terrarium in a printable format, ready to be taken along to the vet in case of emergency.

    Terraria

    Terraria are the core of monitoring and automation. They store sensor readings, have related components like valves and pumps and use action sequences to orchestrate it's components into automated processes.

    Ciliatus automation focuses on keeping a terrarium's vital parameters within limits by applying action sequences once sensor readings come close to critical levels.

    Control Units

    A control unit can be virtually anything that is able to submit sensor readings via an HTTP request. Ciliatus offers an API to store sensor readings and associating them with their respective sensor and therefore with a terrarium and animal.

    It is recommended to create special users for each control unit because they don't require only a special subset of permissions to submit sensor readings and fetch desired states. You should grant the following permissions:

  • grant_api-list
  • grant_api-read
  • grant_api-write:controlunit
  • grant_api-write:sensorreading
  • grant_api-fetch:desired_states
  • See also:

  • Monitoring - Sensor readings
  • Automation - Action sequences
  • Sensors

    Sensors are split into two categories:

  • Physical sensors
  • Logical sensors
  • The physical sensor means the actual device you place in your terrarium and connect to a controlunit.

    A logical Sensor on the other hand describes the types of sensor included in a physical sensor. For example a humidity and temperature sensor combined within one physical appliance.

    Sensor readings can only be of one type and are therefore always related to a logical sensor - not the physical sensor.

    A logical sensor can have multiple thresholds. A threshold defines a reference value and whether the sensor's reading should stay below or above this value. The threshold's starts_at property defines a time of day from which on the threshold is active. The currently active threshold per sensor is determined by sorting them by the threshold's starts_at property in descending order and selecting the first one which's starts_at property is lower then the current time of day.

    If a logical sensor's raw value is not withing the currently active threshold's bounds, it will be considered as critical a critcal state will be created.

    Pumps and Valves

    To actually automate tasks such as irrigating you need actors like pumps and valves. They each have their own component type are automatically used in irrigation action sequences. See also Automation - Action Sequences.

    The idea behind these two components is that you have an array of terraria, each one with a valve to allow water to flow to a water nozzle within the terrarium and multiple valves connected to one pump. Of course you can also have a pump for each valve.

    Generic Components

    To allow more flexibility you can also define your own component types with their own properties and possible states. In our demo environment for example we have a custom component for a fan to cool and a heating lamp. You can think of a generic component type as a template to create components by.

    To allow automation you can also define intentions for your component types, such as irrigating and heating. See also Components - Intentions.

    Intentions

    Each acting component (valves, pumps, generic components) has one or more intentions. An intention defines a terrarium parameter and how the component can influence this parameter. By default valves and pumps have the intention to irrigate. You could for example create a generic component type for your heating lamps with the intention to heat. Ciliatus can now automatically determine that it needs to activate a terrarium's heating lamp when temperature get critical.

    Monitoring

    Sensor readings

    A sensor reading represents a value of a logical sensor at a given time.

    When submitting sensor readings you have to assign group IDs so Ciliatus knows which readings belong together. This means every time you start to retrieve and submit sensor data you have to generate a new random UUID as a group id (see example).

    This allows Ciliatus to:

  • Calculate averages for a terrarium at a given time if you have multiple sensors of the same type installed
  • Match readings of different types for a terrarium to a single point in time if two related readings (e.g. humidity and temperature of the same terrarium) are not submitted at the exact same time to avoid gaps in graphs.
  • This is an example procedure of how you could submit sensor data to Ciliatus with two logical sensors.

  • Generate new group id
  • Read sensor 1
  • Submit sensor 1 (see API example)
  • Read sensor 2
  • Submit sensor 2
  • Sleep for n seconds
  • Go to step 1
  • The API call has to be pointed as a POST request to the endpoint /api/v1/sensorreadings.

    {
        "group_id": "<random uuid>",
        "logical_sensor_id": "22c2db80-526f-11e6-a3b4-9f0c220ec948",
        "rawvalue": 23.5902932
    }

    All sensor readings will be readable as a graph in the terrarium details. The last few hours (or what you defined in your config) are displayed as graphs over your terrarium card's title image.

    Among other features submitting of sensor readings is implemented in python in Ciliatus Controlunit.

    Critical states

    Critical states are created and recovered by evaluating logical sensor readings and comparing them to their sensor's assigned thresholds. Upon creation a critical state is considered "soft". If the reading stays above or below it's threshold for the sensor's soft_state_duration_minutes time (or environment variable DEFAULT_SOFT_STATE_DURATION_MINUTES if the property is null) the soft-flag will be removed and notifications will be sent out.

    The Critical state will call check_notifications_enabled on the model it belongs to. (e.g. a LogicalSensor). This model will either check it's own notifications_enabled property or again asks the model it belongs to by calling check_notifications_enabled on it, until a model with check_notifications_enabled is found. If none is found, no notifications will be sent out. When the sensor is in healthy condition again, the critical state will again check whether it should send out notifications, send them if necessary and removes itself.

    Notifications

    Ciliatus allows you to send out notifications for critical sensor readings and general messages like reminders via different notification providers. You can use the already implemented Telegram provider or create your own.

    Telegram

    Telegram allows you to create bots by using their master bot BotFather. Detailed instructions can be found in the Telegram documentation.

    After you have obtained your bot's authorization token, set the TELEGRAM_BOT_TOKEN and TELEGRAM_BOT_NAME variables in .env. To be able to receive bot messages, you now have to enable Telegram's webhook feature. This will ask Telegram to redirect all messages directed to your bot via POST to the Ciliatus API. Learn how to set up your webhook here and make sure to set your TELEGRAM_WEBHOOK_TOKEN variable first. The webhook URL will be https://yourdomain.com/telegram/<TELEGRAM_WEBHOOK_TOKEN>.

    After you finished the setup, you can go to you user settings and start the Telegram setup for your account.

    Troubleshooting

  • 1. No messages reach my server
  • This can have multiple causes. First make sure the webhook URL is correct and the token is defined in your .env

    Telegram will only push you messages if you have a valid SSL certificate. You can either use a certificate signed by a public CA (e.g. you can obtain Let's Encrypt certificates for free) or use a self signed certificate. Make sure you use the certificate parameter when calling Telegram's setWebhook method when using self signed certificates, like it's explained in their documentation.

    If you have multiple SSL certificates for multiple domains on one host it could be required that your server serves the certificate of your ciliatus domain first, in case Telegram can't handle SNI requests (I'm not 100% on this yet)

  • 2. I don't receive messages from my bot
  • Make sure you have contacted your bot first. Telegram bots can only talk to you, if you have initiated the conversation with /start. Simply follow the Telegram setup in your Ciliatus user settings.

    Automation

    Action Sequences
    Actions

    An action describes a state for a component (e.g. turn Valve or Pump on). Actions are arranged within an action sequence to fulfill certain intentions.

    Action sequences

    An action sequence describes a sequence of actions. For example an action sequence to irrigate a terrarium contains two actions per default:

  • Open a valve (for 10 minutes)
  • Start a pump (for 10 minutes)
  • Each action is sorted within it's sequence using the sequence_sort_id property. When fetching desired states, each scheduled action sequence will evaluate it's actions using this property and check if the dependencies to run this action are met (wait_for_started_action_id, wait_for_finished_action_id properties).

    Action Sequence Schedules

    Action sequence schedules are used to schedule action sequences. They can be executed as one time schedules or daily.

    Schedules are automatically generated for one time action sequences, for example when clicking on irrigation from the terrarium card. The schedule will be markes as run once and removed after it completes.

    Action Sequence Triggers

    Action sequence triggers are used to start an action sequence once the selected sensor's value reaches a certain state. You can also define how long the sensor value has to stay above/below this value and how often the action sequence is allowed to be started by this trigger within a certain timeframe.

    You can use triggers to react to situations before a critical state occurs. To automatically react to critical states you should use Intentions.

    For example: "Start ActionSequence once LogicalSensor is greater/lesser than reference value for duration minutes, but not more often then every timeout minutes and only between timeframestart and timeframeend o'clock"

    Action Sequence Intentions

    When assigning an intention to an action sequence you thereby define it's usage. You can select a type of sensor reading (e.g. humidity) and the intended effect the action sequence has on this value (e.g. lower humidity).

    Intentions are triggered as soon as a critical state is active and not in soft state.

    For example: "If CriticalState is active because sensor reading type is higher/lower than it's threshold find ActionSequence with ActionSequenceIntention to increase/decrease this value, but not more often then every timeout minutes and only between timeframestart and timeframeend o'clock"

    Desired states

    Once an action sequence is running a control unit can fetch it's components' desired states. When a control unit queries Ciliatus to fetch desired states for components controlled by this control unit, every action which should run is copied to a RunningAction. When the action's duration is up relative to the RunningAction's started_at property, it's finished_at property will be set - if all action's associated with this schedule are finished, they're RunningActions will be deleted.

    Controlunits can query ciliatus to receive a list of components which should be turned on at this time. Therefore control units don't need to implemented any kind of logic.

    This means if a control unit controls a component which's state is affected by an action sequence (e.g. Humidify a terrarium -> start Pump) it's desired state will be returned. If no desired state is returned for a component it should be off.

    It is advised to have a safe guard in place to turn off components in case the Ciliatus API is not reachable.

    API calls need to be pointed as a GET request to the endpoint /api/v1/controlunits/<id>/fetch_desired_states. Results look like this:

    {
      "http_code": 200,
      "data": {
        "Valve": {
          "6d522760-526b-11e6-9bbe-7d56150a362c": "running"
        },
        "Pump": {
          "68678610-526b-11e6-b6a0-090ef00e70e6": "running"
        }
      },
      "meta":[]
    }

    Take a look at the Ciliatus Control unit.

    Suggestions

    Every sunday you will receive suggestions on your dashboard recommending the creation of action sequences and schedules.

    They are generated by evaluating critical states and measuring when they occur the most and whether the amount of critical states at a specific time of day is repeatingly over the threshold.

    API

    Authentication

    All API endpoints use OAuth authentication using Laravel Passport.

    To authenticate yourself, a control unit or any other application go to user settings and create a security token in the security tab.

    Your HTTP requests have to submit the generated token using an Authorization header:

    Authorization: Bearer YourToken
    Endpoints

    This explains the basic structure of the API URLs. For more details regarding REST APIs, please refer to the Laravel documentation.

    For this example we will be using the Animal model.

    Index all entities

  • Endpoint: /animals
  • Method: GET
  • Parameters:
    boolean raw - Removes pagination and returns all animals. Requires grant_api-list:raw permission.
  • Returns: Paginated list of animals
  • Requires ability: grant_api-list, optional without pagination: grant_api-list:raw
  • Show single entity

  • Endpoint: /animals/$id
  • Method: GET
  • Parameters:
    uuid id - Animal ID
  • Returns: Transformed animal object
  • Requires ability: grant_api-read
  • Create entity

  • Endpoint: /animals
  • Method: POST
  • Parameters: Depending on the model, usually at least:
    string name
  • Returns: Success/Error details and id of the new model
  • Requires ability: grant_api-write:animal
  • Delete entity

  • Endpoint: /animals/$id
  • Method: DELETE
  • Parameters:
    uuid id - Animal ID
  • Returns: Success/Error details
  • Requires ability: grant_api-write:animal
  • Update

  • Endpoint: /animals/$id
  • Method: PUT/PATCH
  • Parameters:
    uuid id - Animal ID
  • Returns: Success/Error details
  • Requires ability: grant_api-write:animal
  • Error Codes

  • 200 - OK
  • 401 - Unauthorized
  • 404 - Model not found
  • 422 - Unprocessable Entity (in case of missing parameters, etc)
  • 500 - Server Error
  • Responses

    The base JSON the API returns consists of data or error and meta:

    {
        "data": {},
        "error": {},
        "meta": {}
    }

    data field

    The data field can contain a single model, or a paginated list of models. In the later case you will also find pagination information in the meta section. See the components' wiki pages for detailed information.

    Usually, data contains more details when requesting an explicit model (show) instead of indexing.

    meta field

    redirect contains redirect information for the ciliatus frontend. This is useful upon creating a new model and you don't know the model's id yet to redirect from the frontend.

    redirect: {
        "url": "https://yourdomain.com/animals",
        "delay": 200
    }

    pagination field

    pagination contains information about the current page, number of pages, number of items and items per page

    pagination: {
        "total_items": 147,
        "total_pages": 10,
        "current_page": 1,
        "per_page": 15
    }

    error field

    error contains information in case the request was not successful. error_code is a model specific code - do not confuse this with HTTP codes.

    error: {
        "error_code": "",
        "message": ""
    }
    Monitoring Ciliatus

    Ciliatus exposes internal performance metrics via an HTTP API endpoint at /api/v1/system/health.
    The metrics include execution times, notification and sensor reading rates and a list of the longest running requests.

    Other features

    API.AI / Voice control

    You can query Ciliatus using simple voice commands using the free platform API.AI.

    To enable this feature you need create an account at API.AI. You need to configure the agent in resources/assets/api.ai/YOUR_LANGUAGE/agent.json and replace YOUR_CILIATUS_HOST with your server's URL, YOUR_BEARER_TOKEN with a personal access token you create in user settings and YOUR_APP_ID with your app's ID or another random string. Then you can import the Ciliatus agent for your language by compressing the folder to a ZIP and importing in the API.AI settings. If you want to use multiple languages you need to create and import and agent for every language.

    You might need to enable and configure the API endpoint on API.AI in the Fulfilment settings again after importing.

    It is also required that you train your agent for your language and your animal and terrarium names. Otherwise it might not recognize your animals by name. The german agent is used regularly in our environment and we will try to update the trained agent from time to time, the other languages only received basic training. Please share your trained agents in other languages. Thank you.

    After that you can enable the API.AI feature by setting an environment variable in .env for every language and the matching agent's Client access token. For german and english for example the config will look like this:

    API_AI_ACCESS_TOKEN_de=RandomIdForGermanAgent
    API_AI_ACCESS_TOKEN_en=RandomIdForEnglishAgent

    Currently there are three intents implemented.

    Animal health - Examples: "How is animal name", "Is animal name ok?", ...
    Schedules animal feeding - Examples: "When do I have to feed animal name", "When is animal name feeding due", ...
    Get today's feedings - Examples: "Who do I have to feed today?", "Who is hungry?", ...

    InfluxDB

    InfluxDB is an open source time series database with no external dependencies. It's useful for recording metrics, events, and performing analytics.

    Ciliatus can write to InfluxDB upon creating sensor readings. This allows you to connect a multitude of third-party applications to evaluate and process your data, like Grafana.

    To use InfluxDB you just need to create a database and set up the INFLUX_* keys in your .env file. Ciliatus requires you to use an SSL certificate for secure communication with InfluxDB.

    Help

    Troubleshooting

    Most problems can be identified by checking your webserver's log files as well as the laravel log located in storage/logs/laravel.log

    This overview will give basic tips for a Ciliatus installation on Ubuntu 16.04 using PHP 7.0 (FPM).

    Telegram problems? Look here.

    PusherException: There is missing dependant extensions - please ensure both cURL and JSON modules are installed

    Install the missing dependancies like this:

    apt-get install php7.0-curl php7.0-json

    RuntimeException: The only supported ciphers are AES-128-CBC and AES-256-CBC with the correct key lengths

    You might have forgotten to generate an application key:

    php artisan key:generate.

    Class 'Memcached' not found

    Make sure you have installed Memcache for php: apt-get install memcached or change the CACHE_DRIVER in your .env file to file.

    If the error occurs during installation run optimize after fixing the issue.

    php artisan optimize

    Could not establish Memcached connection.

    Make sure your memcache service is running.

    service memcache status

    QueryException: could not find driver (SQL: select * from properties where type = SetupCompleted)

    You probably forgot to install the php driver for your chosen database provider.

    In case of MySQL/MariaDB you can install them like this:

    apt-get install php7.0-mysql

    BroadcastException in PusherBroadcaster.php: 404 NOT FOUND

    If you don't intend to use Pusher or another push service, you'll have to change the default broadcast driver to log. You should do this in your .env file by changing the variable BROADCAST_DRIVER.

    laravel/framework vX.y.z requires ext-mbstring * -> the requested PHP extension mbstring is missing from your system.

    Install mbstring.

    apt-get install php7.0-mbstring

    phpunit/phpunit X.y.z requires ext-dom * -> the requested PHP extension dom is missing from your system.

    Install the php XML extension:

    apt-get install php7.0-xml

    GD Library extension not available with this PHP installation.

    Install the php GD extension:

    apt-get install php7.0-gd
    Contact

    Feel free to contact me at info@ciliatus.io.

    Keep in mind that this is an open source project and I work full time, so it may take some time until I'm able to get back to you. I'll do my best to answer your questions as fast as possible.