05-01-2018, 07:25
(This post was last modified: 05-01-2018, 09:16 by Palmyweather.)
The following is written as a proof of concept and not as a step by step guide. There may be some missing information from this outline. A good understanding of web development and how the WeatherDuino Pro2+ system works would be recommended. No further support will be offered. This concept will make reference to “Cumulus” and “Cumulus MX” software but could be adapted for other weather software.
This has been written following a recent post made on the forum about graphing extra sensors form the Air Quality Monitor. ( http://www.meteocercal.info/forum/Thread...art-option ).
Goals:
· Get extra sensor data uploaded and graphed by personal website server (including soil/leaf and air).
· Gain control of how my graphs look and can be interacted on my personal website.
· Eliminate Cumulus Toolbox.
· Reduce uploading bandwidth to website external host.
After adding soil, leaf and air quality sensors to my WeatherDuino Pro2+ system, I wanted to get the data not only displayed as numbers on my website, but also graphed using HighCharts or JPGraphs. By doing this I have more control over how the data is displayed and not relying on an external graphing website.
Through the Cumulus forum, these two graphing methods (HighCharts and JPGraphs) are documented. Simply put, they read a file that has been up loaded to the website to make the graphs. This is not the place to go into detail on how this works as more information can be found elsewhere.
I found a way whereby using PHP script, an already existing file on the website server can be written to; adding a new line at the bottom of the file. This is the starting point for the project.
Every time the PHP script is run, the file will continue to grow without my control. Research was done and it was found that with PHP, a file can also remove a line from within an existing file.
Using this information and PHP scripts, a file was created which also contained the Cumulus tags the software uses to populate values in a file. Cumulus after processing the file, will upload the data to the web server – but at this stage, it is only a single line of “real time” data. The file which is set up includes the original data string that the graphing scripts are already using, but the additional data fields for the extra sensors have also been added to the end – this will become easier to see in a moment.
Within Cumulus MX Internet settings, there is an option called “Custom HTTP” which can call a web URL a set interval. This calls the file that Cumulus had just uploaded and executes the PHP code to write a new line and delete an old line on the set file.
Now all the data is in one file ready for the graphing scripts to read the file. The extra fields need to be added into the settings of the chosen graphing method.
Let us take a look as some of the code:
The above code is the file that is uploaded by Cumulus MX every one minute after it processes the file (adding the numbers to the fields).
NOTE: This is an example for my own personal weather station and how I have the “WeatherDuino Pro2+” settings configured. Therefore the sensor numbers may not be relevant to your system and I may have more or less sensors listed than you require.
After Cumulus processes the data and uploaded the file, this is what the data filed will now look like populated with data:
This is the same file which Cumulus MX will call every minute to run the file.
The result when the file is run, a new line of code is added to the “realtime.log” file for the graphing code to read later on. The “realtime.log” file has 2880 lines of code already in the file, this equates to two days worth of data logged every one minute. You can make this more or less depending on how much data you want to display. It is important to ensure there is already data in the file for the PHP script to remove a line.
This is where you can start to see some benefits: If the “realtime.log” file containing two days worth of data was to be uploaded every one minute by Cumulus Toolbox, that equates to 750kb file being uploaded every time. However by doing it this way, Cumulus uploads a file as small as 1.5kb every minute – a big saving on bandwidth/data usage.
The above image is an example of where “Cumulus MX” will process and upload the PHP script to the web server.
The above image is an example of where “Cumulus MX” will call the PHP script that it previously uploaded.
Within the graphing script you choose to use, the additional sensors are going to need to be added to all instances of where the data is being read. For example with HighCharts, there are two instances where the extra sensors need to be listed to make this work.
File Name: realtimeLogParser.php
File Name: ExtraSensors.JS – Settings added for the JS file related to the graph
Now all that is needed is to design the graphs within the JavaScript file with the desired data of your choosing.
Simple... right? Well it did take me a long time to get this right. At the bottom of this page is an example of the graphed output. Finally, Cumulus Toolbox can be turned off and is not needed to get this sort of extra data to a web server.
What if you are not running “Cumulus MX” with the “Custom HTTP” option? This could be done with “Cumulus 1” as well by processing and uploading the file to the web server and getting Cumulus Toolbox to call the PHP file from the website. OR, Add a refresh META tag or JavaScript line to refresh the PHP page every 1 minute and leave a web browser window open with the PHP page open and automatically refreshing (probably not desirable however).
I have found this so successful; I am also using the same concept to upload/write my “Day File” which my website also uses – saves uploading every day a large file.
A live example can be viewed at http://palmyweather.co.nz/extrasensors
I hope that this will be useful for someone.
Kind regards,
PalmyWeather
This has been written following a recent post made on the forum about graphing extra sensors form the Air Quality Monitor. ( http://www.meteocercal.info/forum/Thread...art-option ).
Goals:
· Get extra sensor data uploaded and graphed by personal website server (including soil/leaf and air).
· Gain control of how my graphs look and can be interacted on my personal website.
· Eliminate Cumulus Toolbox.
· Reduce uploading bandwidth to website external host.
After adding soil, leaf and air quality sensors to my WeatherDuino Pro2+ system, I wanted to get the data not only displayed as numbers on my website, but also graphed using HighCharts or JPGraphs. By doing this I have more control over how the data is displayed and not relying on an external graphing website.
Through the Cumulus forum, these two graphing methods (HighCharts and JPGraphs) are documented. Simply put, they read a file that has been up loaded to the website to make the graphs. This is not the place to go into detail on how this works as more information can be found elsewhere.
I found a way whereby using PHP script, an already existing file on the website server can be written to; adding a new line at the bottom of the file. This is the starting point for the project.
Every time the PHP script is run, the file will continue to grow without my control. Research was done and it was found that with PHP, a file can also remove a line from within an existing file.
Using this information and PHP scripts, a file was created which also contained the Cumulus tags the software uses to populate values in a file. Cumulus after processing the file, will upload the data to the web server – but at this stage, it is only a single line of “real time” data. The file which is set up includes the original data string that the graphing scripts are already using, but the additional data fields for the extra sensors have also been added to the end – this will become easier to see in a moment.
Within Cumulus MX Internet settings, there is an option called “Custom HTTP” which can call a web URL a set interval. This calls the file that Cumulus had just uploaded and executes the PHP code to write a new line and delete an old line on the set file.
Now all the data is in one file ready for the graphing scripts to read the file. The extra fields need to be added into the settings of the chosen graphing method.
Let us take a look as some of the code:
PHP Code:
<?php
// Set Data string with Cumulus webtags for processing in order of which the graphing software will read the file.
$data = "<#date> <#timehhmmss> <#temp> <#hum> <#dew> <#wspeed> <#wlatest> <#bearing> <#rrate> <#rfall> <#press> <#currentwdir> <#beaufortnumber> <#windunit> <#tempunitnodeg> <#pressunit> <#rainunit> <#windrun> <#presstrendval> <#rmonth> <#ryear> <#rfallY> <#intemp> <#inhum> <#wchill> <#temptrend> <#tempTH> <#TtempTH> <#tempTL> <#TtempTL> <#windTM> <#TwindTM> <#wgustTM> <#TwgustTM> <#pressTH> <#TpressTH> <#pressTL> <#TpressTL> <#version> <#build> <#wgust> <#heatindex> <#humidex> <#UV> <#ET> <#SolarRad> <#avgbearing> <#rhour> <#forecastnumber> <#isdaylight> <#SensorContactLost> <#wdir> <#cloudbasevalue> <#cloudbaseunit> <#apptemp> <#SunshineHours> <#CurrentSolarMax> <#IsSunny>
//End of Original List. Below is the new extra sensors being added
<#ExtraTemp7> <#ExtraHum7> <#ExtraTemp6> <#ExtraHum6> <#ExtraTemp1> <#ExtraHum1> <#SoilTemp1> <#SoilTemp2> <#LeafTemp1> <#LeafTemp2> <#SoilMoisture1> <#SoilMoisture2> <#SoilMoisture3> <#SoilMoisture4> <#LeafWetness1> <#LeafWetness2> <#battery>\n\n"; //Include new line return at end of string
$fp = fopen('realtime.log', 'a'); //open file
fwrite($fp, $data); // write new data to bottom of file
//Remove first (old) line
$file = "realtime.log"; //Open file
$f = fopen($file, 'r'); //Read file
$line = fgets($f);
fclose($f);
$contents = file($file, FILE_IGNORE_NEW_LINES);
$first_line = array_shift($contents);
file_put_contents($file, implode("\n", $contents)); //Delete first line from newline point
?>The above code is the file that is uploaded by Cumulus MX every one minute after it processes the file (adding the numbers to the fields).
NOTE: This is an example for my own personal weather station and how I have the “WeatherDuino Pro2+” settings configured. Therefore the sensor numbers may not be relevant to your system and I may have more or less sensors listed than you require.
After Cumulus processes the data and uploaded the file, this is what the data filed will now look like populated with data:
PHP Code:
$data = "4/01/2018 20:54:55 19.0 96 18.4 7.9 7.2 67 0.2 10.4 1003.76 ENE 2 km/h C hPa mm 71.6 -0.44 30.8 30.8 11.2 33.2 44 19.0 -0.3 28.3 13:35 17.7 03:52 12.3 15:43 43.3 15:41 1012.02 00:11 1003.69 20:49 3.0.0 3043 16.6 19.0 25.2 0.0 0.00 0 88 0.4 24 1 0 E 266 ft 20.4 1.9 0 0 2.2 8 20.6 90 19.4 93 23.3 21.7 20.6 20.0 97 99 33 66 5 9 13.07\n\n";
This is the same file which Cumulus MX will call every minute to run the file.
The result when the file is run, a new line of code is added to the “realtime.log” file for the graphing code to read later on. The “realtime.log” file has 2880 lines of code already in the file, this equates to two days worth of data logged every one minute. You can make this more or less depending on how much data you want to display. It is important to ensure there is already data in the file for the PHP script to remove a line.
This is where you can start to see some benefits: If the “realtime.log” file containing two days worth of data was to be uploaded every one minute by Cumulus Toolbox, that equates to 750kb file being uploaded every time. However by doing it this way, Cumulus uploads a file as small as 1.5kb every minute – a big saving on bandwidth/data usage.
The above image is an example of where “Cumulus MX” will process and upload the PHP script to the web server.
The above image is an example of where “Cumulus MX” will call the PHP script that it previously uploaded.
Within the graphing script you choose to use, the additional sensors are going to need to be added to all instances of where the data is being read. For example with HighCharts, there are two instances where the extra sensors need to be listed to make this work.
File Name: realtimeLogParser.php
Code:
// Fields of realtime.txt file
// Use the same names as the corresponding web tags
$rfields = array(
"date","time","temp","hum","dew","wspeed","wlatest","bearing","rrate","rfall","press","currentwdir","beaufortnumber","windunit","tempunitnodeg","pressunit","rainunit","windrun","presstrendval","rmonth","ryear","rfallY","intemp","inhum","wchill","temptrend","tempTH","TtempTH","tempTL","TtempTL","windTM","TwindTM","wgustTM","TwgustTM","pressTH","TpressTH","pressTL","TpressTL",version","build","wgust","heatindex","humidex","UV","ET","SolarRad”,"avgbearing","rhour", "forecastnumber", "isdaylight","SensorContactLost","wdir","cloudbasevalue","cloudbaseunit","apptemp","SunshineHours","CurrentSolarMax","IsSunny"
//End of Original List. Below is the new extra sensors being added
,"ExtraTemp7","ExtraHum7","ExtraTemp6","ExtraHum6","ExtraTemp1","ExtraHum1","SoilTemp1","SoilTemp2","LeafTemp1","LeafTemp2","SoilMoisture1","SoilMoisture2","SoilMoisture3","SoilMoisture4","LeafWetness1","LeafWetness2","battery"
);File Name: ExtraSensors.JS – Settings added for the JS file related to the graph
Code:
// Fields of realtime.txt file
// Use the same names as the corresponding web tags
fields = {
date: 0, time: 1, temp: 2, hum: 3, dew: 4, wspeed: 5, wlatest: 6, bearing: 7, rrate: 8, rfall: 9,press: 10, currentwdir: 11, beaufortnumber: 12, windunit: 13, tempunitnodeg: 14, pressunit: 15,rainunit: 16, windrun: 17, presstrendval: 18, rmonth: 19, ryear: 20, rfallY: 21, intemp: 22,inhum: 23, wchill: 24, temptrend: 25, tempTH: 26, TtempTH: 27, tempTL: 28, TtempTL: 29, windTM: 30,TwindTM: 31, wgustTM: 32, TwgustTM: 33, pressTH: 34, TpressTH: 35, pressTL: 36, TpressTL: 37,version: 38, build: 39, wgust: 40, heatindex: 41, humidex: 42, UV: 43, ET: 44, SolarRad: 45,avgbearing: 46, rhour: 47, forecastnumber: 48, isdaylight: 49, SensorContactLost: 50, wdir: 51,cloudbasevalue: 52, cloudbaseunit: 53, apptemp: 54, SunshineHours: 55, CurrentSolarMax: 56, IsSunny: 57
//End of Original List. Below is the new extra sensors being added
, ExtraTemp7: 58, ExtraHum7: 59, ExtraTemp6: 60, ExtraHum6: 61, ExtraTemp1: 62, ExtraHum1: 63, SoilTemp1: 64, SoilTemp2: 65, LeafTemp1: 66, LeafTemp2: 67, SoilMoisture1: 68, SoilMoisture2: 69, SoilMoisture3: 70, SoilMoisture4: 71, LeafWetness1: 72, LeafWetness2: 73, battery: 74
},Now all that is needed is to design the graphs within the JavaScript file with the desired data of your choosing.
Simple... right? Well it did take me a long time to get this right. At the bottom of this page is an example of the graphed output. Finally, Cumulus Toolbox can be turned off and is not needed to get this sort of extra data to a web server.
What if you are not running “Cumulus MX” with the “Custom HTTP” option? This could be done with “Cumulus 1” as well by processing and uploading the file to the web server and getting Cumulus Toolbox to call the PHP file from the website. OR, Add a refresh META tag or JavaScript line to refresh the PHP page every 1 minute and leave a web browser window open with the PHP page open and automatically refreshing (probably not desirable however).
I have found this so successful; I am also using the same concept to upload/write my “Day File” which my website also uses – saves uploading every day a large file.
A live example can be viewed at http://palmyweather.co.nz/extrasensors
I hope that this will be useful for someone.
Kind regards,
PalmyWeather
it!
![[Image: thumbsup.gif]](http://www.meteocercal.info/forum/images/icons/thumbsup.gif){kind=icon}
