Python Interval Program for OSPi

= Python Interval Program for OpenSprinkler Pi =

Background
This program, written by Dan Kimberling, is a Python port of the Arduino based micro-controller OpenSprinkler interval program originally written by Ray.

It is designed to run on a Raspberry Pi with a current release of the Raspbian operating system installed.

Despite the difference in spelling, the “Pi“ in the name Raspberry Pi is derived from Python and this was a logical choice for the programming language since it is included with the operating system.

A major goal in the design of this program was to avoid having to install additional software or require the user to compile code. The program uses very basic, standard Python modules plus the Python GPIO module which is also included with recent distros of Raspbian. The only other non-standard Python module is web.py which enables the web interface of the program including a built-in web server. Web.py is in the public domain and is included in the program download as the /web directory. You will also need to have git installed on your Pi but no other software is required.

History
The initial release was announced by Ray on the OpenSprinkler forum Jun 09, 2013. It was equivalent to version 1.8.2 of the micro-controller firmware. This was followed by a series of updates based on user feedback:

September 10 2013 Additions, bug fixes:
 * 1) Added a per-station "Ignore rain" option that allows a station to operate during rain delay or if a rain sensor detects rain.
 * 2) Modified the program to use the HTTP port setting from the Options page.
 * 3) Improved the way the program tracks current time. This simplified the code and should eliminate some timing bugs.
 * 4) Edited Denny's init.d startup script to remove IP address and port settings no longer needed.

August 25 2013 Additions, bug fixes:
 * 1) Implemented improved installation and update methods using GitHub.
 * 2) Modified the program to automatically create default config files on new installations. This also prevents existing settings from being overwritten.
 * 3) Added a "Run now" button to the programs page. Allows a schedule program to be started at any time. This overrides (stops) any running program.
 * 4) Added a readout of the Raspberry Pi's CPU temperature to the home page.
 * 5) Fixed a bug that would allow a station to be stopped / started without a password ueing the HTML API.
 * 6) Fixed a bug that would display an incorrect start day for a schedule program.

August 1 2013 Revved to firmware V 1.8.3
 * 1) Added Sequential/Concurrent option.
 * 2) Added a function to detect Pi board rev and auto-configure GPIO pins for rev 1 boards.
 * 3) Fixed a bug in manual mode that would cause any zone with a master association to stop the master when turned off, even if another station with a master association was still running.
 * 4) Changed how ospi.py handles master zone associations. The program should now work with more than 3 expansion boards (untested in hardware but at least 5 expansion boards, 64 stations, work in software).

July 21 2013
 * 1) Fixed a bug that kept an in progress program running after it was disabled.
 * 2) Added error checking to prevent an 'lg' KeyError
 * 3) When a new irrigation program was added, it became program 1 instead of being added at the end of the list. - fixed.
 * 4) When Rain Delay was set, running stations did not stop. - Fixed.
 * 5) Added a 1.5s delay in the screen refresh of manual Mode to allow active stations and last run log time to update.

July 19 2013
 * 1) Eliminated over 100 lines of redundant code. The code is now much closer to the micro-controller version. Manual Mode and Run-once now rely on the main loop algorithm. This eliminates potential conflicts and makes the code easier to maintain. The program should now be more stable and have fewer bugs although the UI is a little slower.
 * 2) Changed bit-wise operations to make them more reliable.
 * 3) Station names now accept Unicode characters allowing names to be entered in any language.
 * 4) Faveicon now appears on all pages.
 * 5) A small bug in the display of Master valve off time in the program preview has been fixed. The off time was 1 minute short.
 * 6) A file named 'sd_reference.txt' has been added to the OSPi directory. It contains a list with descriptions of the values contained in the global settings dictionary variable (gv.sd) which holds most settings for the program. These values are kept in memory and also stored in the file OSPi/data/sd.json to persist across system restarts. This reference file is for the benefit of anyone who wishes to tinker with the code.

July 10 2013
 * 1) Fixed a bug that prevented zones 9+ from running.
 * 2) The Run once program was not observing the station delay setting - Fixed
 * 3) Made the sd variable an attribute of the gv module. All references to sd... are now gv.sd... This should potentially fix several bugs, Specifically the Rain delay seems to be working properly now.
 * 4) The Graph Programs time marker was not recognizing the time zone setting from the Options page - fixed.
 * 5) Time displayed on the last run line of the main page was not correct - fixed.
 * 6) Added a faveicon which will help distinguish the OpenSprinkler tabs on the browser.
 * 7) Added an import statement and file which provide a stub for adding user written custom functions to the interval program without modifying the program itself

Jun 26 2013
 * 1) Last run logging is now working for manual mode when an optional time value is selected, even if more that one station is started.
 * 2) Fixed a bug that prevented the home page display from updating when running irrigation programs.
 * 3) Includes a fix from Samer that allows the program preview time marker to update properly.

Jun 20, 2013
 * 1) Changed the way ospi.py handles time. It now uses the time zone setting from the OpenSprinkler options page. It also eliminates the auto daylight savings time adjustment that was causing problems for some users.
 * 2) Fixes a bug mentioned on the forum that caused Samer's app to not update in program mode.
 * 3) Fixes a bug that caused a program to re-start after the "Stop all stations" button was clicked.
 * 4) A partial fix for "last run" logging. Still needed to get manual mode with an optional time setting working.
 * 5) Added a docstring at the top of the ospi.py file with the release date for version tracking.

Jun 19, 2013
 * 1) Applied Samer Albahra's patch so that the program will work with Samer's mobile web app. Per forum discussion: http://rayshobby.net/phpBB3/viewtopic.php?f=2&t=154&start=40#p781

Installation
Note: If you are updating an existing installation of the program, see the Updating the program section below.

Requirements:

 * 1) A suitable SD memory card that has been flashed with an up to date rev of the Raspbian operating system. See the Raspberry Pi Quick Start Guidefor instructions. Be sure the system is properly set up with the correct time zone for your location.
 * 2) An internet connection

Install git:
With the Raspberry Pi connected to the internet, open a terminal window (command line). enter the following command: sudo apt-get install git After a couple of seconds, you may be asked if you want to proceed, answer yes by typing y and the installation should finish within a minute or two.

Quick Instructions:
Be sure you are in the Pi user's home directory (pi@raspberrypi ~ $). Clone the OSPi directory from GitHub by issuing the following command:

git clone https://github.com/Dan-in-CA/OSPi

The program is now ready to run. To start the program execute: cd OSPi sudo python ospi.py

Download the program from GitHub.
 Open a terminal window or connect to the pi via ssh. Type the command:

git clone https://github.com/Dan-in-CA/OSPi

followed by the enter key.

The program file should start downloading from the GitHub server.

If you get an error message, carefully retype the command and double check that it is exactly as shown above. Also be sure the pi is connected to the internet.

Check that the OSPi directory was created in the proper location by typing ls (that's lower case LS). You should see the directory (OSPi) listed as a sub-directory.

Start the program for testing
 Log into the OSPi sub-directory by typing: cd OSPi followed by the enter key. start the program by typing the following command: sudo python ospi.py followed by the enter key. You should see a message from the program “Starting main loop” and a URL.

Open the program in a web browser
On a device connected to your local network, enter the URL of your Raspberry Pi with port number 8080. For example, on my test system, the URL would be http://192.168.1.22:8080. The URL of your Raspberry Pi will be different. Don't forget to include the colon before the port number.

You should now see the interval program's home page in your browser. Congratulations! The program is now installed and running.

Using the native web interface
Instructions for using the built in web interface can be found at the Ray's Hobby web site.

From an installation that was created using git
 Log into your OSPi directory and issue the command: git pull The updates should be installed automatically and your settings should remain intact.</li> Reboot the Pi: sudo reboot Or click the Reboot button on the program's home page. You should now be running the latest version of the program.</li> </ol>

Note: the September 10 2013 update may cause an error. But the solution is fairly easy.

The Sept. 10 2013 update now uses the HTTP port setting from the Options page of the native web interface. If you are running a server on your Pi that is using port 80 you will get an error when you restart the program. For example if you are using Samer's Mobile Web App and have the Apache server installed.

The problem is caused by an old default setting that is stored in the file OSPi/data/sd.json. When the program starts, it tries to use that port 80 setting and creates a conflict. The solution is: From a terminal window and in the Pi home directory (pi@raspberrypi ~ $) issue the following command. sudo nano OSPi/data/sd.json This will open the offending file in the nano text editor. Use the cursor keys (arrow keys) to move to "htp": 80 and change 80 to the port number you have been using for the program, e.g. 8080. Then use control+o to save the file and control+x to exit the editor.

Restart the pi with: sudo reboot

From an older version not installed using git
If you are updating the program from an installation that was not installed using git, or was installed prior to August 25, 2013, follow the instructions bellow. You can check the date of the files in your installation by issuing the command: ls -l OSPi The dates of the files will appear in the listing as e.g.: -rw-r--r-- 1 pi  pi   40512 Aug 25 16:08 ospi.py

 install git: sudo apt-get install git After a couple of seconds, you may be asked if you want to proceed, answer yes by typing y and the installation should finish in a minute or two.</li> Rename your existing OSPi directory to something like OSPi-old: mv OSPi OSPi-old </li> Clone the new OSPi directory from git hub: git clone https://github.com/Dan-in-CA/OSPi This will create a new directory named OSPi containing the latest program files.</li> Copy your data (config) files from your old installation to the new one: cp -f ./OSPi-old/data/*.* ./OSPi/data/ </li> If you have a customized ospi_addon.py file, copy it to the new installation: cp -f ./OSPi-old/ospi_addon.py ./OSPi/ </li> Reboot the Pi and you should now be running the latest update with your old settings intact.</li> sudo reboot </ol>

Starting the program automatically
Once the program is installed, you will probably want it to start automatically when the Raspberry Pi boots up. This will ensure that the program will be running even after a power outage.

There are a couple of ways to accomplish this. The simplest is to add some start-up commands to a file named “rc.local” which is in a directory named “/etc”.

Quick instructions:
Edit the file /etc/rc.local and add the following before the “exit 0” line: cd /home/pi/OSPi/ python ospi.py & </li>
 * 1) Start the OSPi interval program

Reboot the Pi. sudo reboot </li></ol>

Detailed instructions:
Nano is the name of a light weight text editor that is included with the Raspbian operating system. It is used to edit various configuration files including the rc.local file that can be used to start the Python interval program when the Pi boots up.

In a terminal window type the following command:

sudo nano /etc/rc.local

followed by the enter key.

This will open the file in the editor.

You can use the keyboard arrow keys to move around and the Enter key to add new lines in the editor.

Use the down arrow key on your keyboard to move the cursor (which appears as a highlighted box) down past most of the text to a blank line just before the line “exit 0”.</li>

Carefully type in the following:

cd /home/pi/OSPi/ python ospi.py &
 * 1) Start the OSPi interval program

Be sure to include all spaces and punctuation just as shown above.

The first line is a comment that is there as a reminder of what the code does but doesn't actually perform any action.

The third line is what actually starts the interval program. If you ever need to disable the automatic start, you can “comment out” this line by editing the file and placing a # character at the start of the line. You can then re-enable automatic start by removing the # character.</li>

After you have entered the lines as shown above, save the file by typing <tt>Ctrl + o</tt>. (that's the control key and the letter o together) followed by the Enter key.</li> Exit the editor by typing <tt>Ctrl + x</tt>.</li> <li>Once you have exited the editor, reboot the Pi by typing: sudo reboot followed by the Enter key.</li></ol> After the Pi has rebooted you should be able to go to the URL of your Pi from a device connected to your local network and access the interval program's web interface.

If you open a terminal window on the Pi, or access the Pi via ssh you will not see any indication that the program is running. It started as “root” and is not visible to the standard Pi user. You can use the Pi as if the program wasn't running but be aware that a lot of additional activity may cause timing problems with the interval program.

Starting ospy.py from a script in /etc/init.d
Here's an alternate way to automatically start opsy.py on boot instead of using /etc/rc.local. This is the preferred method and a file (ospi.sh) is included with the program distribution to make setup easy.

The advantage of using an /etc/init.d script is that you can easily stop, start, and check the status of ospi.py. Since /etc/rc.local is only executed on boot up, it's a little awkward to stop ospi.py and start it again without rebooting or a typing bunch of commands. The script was adapted from the Debian skeleton template in /etc/init.d.

<ol><li>Move the script file to /etc/init.d and rename it to ospi (without the .sh extension). Log into the OSPi directory and run the command: sudo mv ospi.sh /etc/init.d/ospi </li> <li>Make the script executable: sudo chmod +x /etc/init.d/ospi </li> <li>Activate auto start on boot: sudo update-rc.d ospi defaults </li></ol> If you want to disable the auto-start, use: sudo update-rc.d ospi remove

If you are developing new features in the code you will find the restart command (see below) a quick way to check your changes.

Check status, start, stop, and restart ospi.py
If you are using the ospi script in /etc/init.d, as described above, you can check if the interval program is running by executing the command: /etc/init.d/ospi status

To start the interval program, execute: sudo /etc/init.d/ospi start

To stop the interval program, execute: sudo /etc/init.d/ospi stop

To quickly restart the program after making changes, execute: sudo /etc/init.d/ospi restart

Logging
The logging feature included with the Python Interval program is disabled by default. This is to prevent the uncontrolled accumulation of log data. It also eliminates unnecessary redundancy if you are using another logging method such as the one provided by Samer's web app.

To enable logging:
Click the “Log Options” button at the upper right of the log page. You will be presented with a dialog containing a check box for enabling the feature.

Set the maximum number of records to keep:
It is set to 100 records by default but you can change the value to whatever you like. When the maximum number of records has been reached, new records will continue to be added to the top of the list and the oldest records deleted from the bottom.

Setting the maximum to zero (no maximum) will allow records to accumulate until all available storage space is filled. This may effect system performance if the number of records becomes very large but it has been tested with over 1100 records without a noticeable problem. A practical setting might be the number of records you would get in a period of time such as a week.

Archiving log data:
The interval program's logging feature stores data in a spread sheet friendly, comma separated values, (csv) text file. The data file is set up to be easily downloaded from the Raspberry Pi to another device, e. g. a laptop. This provides a means of keeping a permanent archive of irrigation data in manageable size chunks.

You can download the log file by pointing your web browser to: [URL of your pi]/static/log/water_log.csv for example on my system the url is: http://192.168.1.22/static/log/water_log.csv Your browser will offer to save the file to your local device. Once the file has been downloaded it can be imported in to your favorite spread sheet program for viewing, sorting, etc.

Add-On Programs
In keeping with the open source, do it yourself, spirit of the OpenSprinkler Pi, some users have been adding new features to the Python program. This is highly encouraged. Among the reasons for choosing Python as the programming language for this project are it's ease of learning and readable code.

It is possible to add functionality to the Python interval program without modifying the program itself by writing a “module” that will be automatically imported into the program when it starts. All Python programs are considered modules and can be imported by other programs.

The program currently includes code which will try to import a module named “ospi_addon” if it exists.

There is also an example file located in the OSPi directory named “ospi_addon.py” which you can use as the starting point for creating your own custom feature(s) for the interval program.

You can test drive the example add-on program by pointing your browser to: [URL of your Pi]/c1 that's /c and a numeral one.