Recently a random crashing issue has surfaced on the forum regarding some of the OpenSprinkler 2.0 development boards that we sent out a few weeks ago. The background is that we have started shipping a development version of 2.0 for any recent order of the assembled OpenSprinkler. The symptom is that some of the units started crashing and losing network connection, and this seems to be always correlated with the timing running crazy off the charts. Initially I thought this issue may have been caused by a bad RTC crystal, which leads to incorrect timing, and which in turn causes the controller to get stuck. Then at some point I realized all the problematic cases happened on a particular PCB version dated Nov 2012. This helped me narrow down the search range. After a thorough comparison to the more recent version, I discovered that the problem is actually caused by a noise coupling issue with DS1307 RTC. Specifically, according to the datasheet (right image below), the region around traces between the 32.768kHz crystal and DS1307 should be shielded by a ground trace. If this is not implemented, the crystal may pick up noise from nearby traces and cause the clock to run significantly off the charts. When I designed OpenSprinkler PCBs, I had not paid attention to this recommended layout at all. Jeese, why did’t I read the datasheet carefully!
After discovering the issue, I panicked a little bit. Since I didn’t know about the issue previously, and there are hundreds of boards in production at SeeedStudios, am I screwed? It turns out that the situation is not as bad as I thought. First of all, I usually add a ground plane to fill the empty spaces between traces. As a result, on most PCB versions, there is indeed a ground plane around the traces from the crystal to the RTC. Below are examples of the traces on OpenSprinkler 1.4s, the more recent version 2.0, and the version in production at SeeedStudios:
Although they don’t match the recommended layout exactly, they are pretty good at shielding the crystal pins from picking up noise. You know, I have always questioned about those ground planes and wondered whether it’s worth my effort of adding them. Well, in this case it did save me from potentially deep trouble.
Now, in comparison, on the Nov 2012 version of 2.0, the ground plane is missing around that region (probably because there is not enough space between the traces to allow a ground plane). As a result, here is the PCB image:
Not only it is missing the ground plane in that region, but also there are all kinds of signal traces nearby, which makes is extra easy to pick up noise. This is bad, bad, bad. Now, that being said, even with the careless design, it’s not like you are guaranteed to have the problem. It more of a reliability thing: it increase the chance of noise coupling, but in many cases the issue probably doesn’t show up. In fact I was trying to reproduce the problem today on a particular PCB version that has no ground plane. After testing 8 of these, one showed a problem of extremely fast clock (5x faster than normal). And it turned out that I forgot to cut the crystal leads, which make them act like antenna to happily pick up noise. After cutting the leads, the clock went back to normal speed. This probably explains why the problem didn’t surface earlier even though I have never paid attention to the PCB trace around RTC previously.
The DIY version 1.42u is also missing the ground traces around that region. So I won’t be surprised if users have encountered a clock speed problem (although so far I have not received any report of it). In case this happens, an easy thing to try is to re-solder the 32.768kHz crystal and try to push it down to the PCB holes as much as you can. This will help reduce the length of the crystal pins and in turn reduce the chance of picking up noise. Another possibility is to simply solder the crystal directly onto the IC pins,
So overall this is a very valuable learning experience for me. I am grateful to those who reported the issue on the forum, and helped me figure out the problem as quickly as I can. There are about 10 of those Nov 2012 assembled boards out there, and I am happy to replace them for free. With that, it’s now time for me to revisit the datasheets. Never overlook them again!
Ray have you come across issues where the RTC hour or some other registers of 1307 going corrupt ? It might be a rare event but can happen frequently if there are frequent power failures.
Have any of your users reported incidents where time totally went wrong, no not fast or slow as caused by issues related to load capacitance of crystal or noise issues.
This issue is documented in Maxims site http://www.maximintegrated.com/app-notes/index.mvp/id/504 The section is titled data loss/data corruption.
This problem has been nagging me since over a year because its random and even the clamping diode did not help. I’ve found a solution but it might still need testing and usage over months to confirm its really solved.
Thanks for the pointer. Very useful document. I haven’t received further reports of RTC problem since that one batch, and not any report of data corruption problems so far. Where did you get your RTC? I’ve had issues in the past with RTCs sourced from China. Since then I’ve been very careful at using only trusted vendors.
I too have heard that there are lots of counterfeit rtcs from china. I sourced it from an indian vendor and am not sure if it came from china. But the interesting thing is that I successfully made a test rig where I can simulate data corruption by rapid on offs with a relay, so at least I now know how to corrupt the rtc so that I can confirm if it happens to all or just counterfeits. This issue was raised long time back in a picbasic thread http://www.picbasic.co.uk/forum/showthread.php?t=14647 but no clear working solution is given. If my solution works reliably and passes the ultimate time test I’ll blog and document it in my upcoming open hardware project.
Cool, I will be very interested in hearing your findings. Thanks for the pointer.
Just forgot, what was the issue you had from rtcs from china ?
The issue was that it causes the microcontroller to hang reading RTC from the I2C lines. This is often correlated with another effect that upon power-on for the first time, the RTC returns a seemingly random time (whereas the authentic ones return a fixed time such as 01-01 2000 00:00).
I have found software i2c seems to be more reliable as you have the possibility of easily handling hung up conditions. The ones which I tested are giving 00:00:00 1-1-00 when I remove the battery and pu it back.
Facing some issue is DS1307 ardunio based clock.
i observed my clock is running 10 min faster. i have perform sketch burning process for sync time again from PC. But this time date/time not syncing with PC time and CLOCK RUNNING THREE TIME FASTER like raise. i have performed all troubleshooting step what i can do. rechecked, wiring circuit, replace crystal DS1307 chip, battery, fixed crystal body with ground etc. even try different version of code also. On serial monitor i have found wrong date/time is printing and two different value of date/time showing like:
0:8:23 Date 31 (Wrong time and day)
165:166:342 Date 232 (Not able to understand what is this)
there is two problem i am facing.
1) Clock running three time faster. This case clock is not usable for me
2) Date/Time not syncing with PC time after burning sketch
You can check my video which are showing clock running very fast
https://youtu.be/JxhtqnobwhE
Photo album:
https://photos.google.com/u/1/share/AF1QipMMUaFD0StJG9CDjFB55GEf2hR3gJ1Cuf7AW-CKZTWj9dGcsvQofFVZzz_l4hKULw?key=eFNpbkVKNTZPcDYxZGNFZHk5OHo4NUN3ZlU2X3F3
Please help… Thank you so much in advance
Even though you said you have replaced DS1307 chip, I would still recommend checking it again — some of the early problems I had with clock running much faster turns out to be actually caused by counterfeit DS1307 chips. The noise issue actually wasn’t the key cause. So check the chip and make sure you get it from a reliable source.
Thanks Ray for your reply.
it’s was solved after given ground to crystal and makes ground plan to this.
I am using DS1307 in my circuit. It works fine for around 2-3 weeks and thereafter, it stops incrementing the time. I found this with 3 boards. I am thinking, due to some reason, the oscillator stops working. Of course, I got DS1307 from a local supplier and perhaps it is imported from China. Anybody has faced this type of problem and found any solution.
i have same issue