Month: February 2018

It’s been just under 5 months since I simultaneously synchronized ten DS3231/DS3231M RTCs as part of a long-term experiment to measure their drift. Of the ten, seven are crystal-based DS3231 chips, while three are DS3231M chips. Since they’re all on the same breadboard connected to the same power supply, all of them have been subject to the same physical conditions of temperature, movement, voltage, etc. insofar as I can control for them in my apartment.

In the table below “Number” is the identifying number of each chip I arbitrarially asigned to uniquely identify each one, “Type” refers to its type (all the DS3231 modules are marked as the wide-temperature-range SN type, while the DS3231Ms are listed as M. The official DS3231M chip (#2) I received directly from Maxim is marked with an asterix.), “Offset” is the aging offset in register 0x10 expressed in decimal form, “PPM” is the stability in parts per million, and “Drift” is the number of seconds the clock has drifted since the start. For both the PPM and Drift columns, a positive value indicates that the RTC has run faster than the NTP-synchronized system clock while a negative value indicates the RTC has run slower than the system clock.

All results were collected over a 26 minute period starting 12904549 seconds after the clocks were first synchronized. Each clock was measured three times and the resulting values averaged and rounded to two decimal places. Keep in mind that this dataset consists of just two data points (zero drift at the start, and the measured drift now) for each clock: unlike Dan, who continuously collected data and made many nice graphs, I set the clocks and essentially ignored them for five months.

Anyway, I digress. Here’s the results:

Number	Type	Offset	PPM	Drift
0	SN	-6	0.19	2.46
1	SN	0	-0.69	-8.96
2	M*	0	-1.62	-20.85
3	M	0	-3.06	-39.54
4	M	0	-2.76	-35.65
5	SN	0	0.16	2.07
6	SN	0	0.01	0.10
7	SN	-15	0.33	4.32
8	SN	0	0.08	0.98
9	SN	0	-0.05	-0.60

Some commentary, bullet-pointed for your reading pleasure:

• All clocks are within their advertised tolerances (2 ppm for the crystal-based clocks and 5 ppm for the MEMS-based clocks).
• Five of the seven crystal-based DS3231 chips run fast, while two run slow. All three of the MEMS-based DS3231M chips run slow.
• Clock #6 has essentially no drift whatsoever. There’s nothing particularly noteworthy about it: just luck of the draw.
• The Raspberry Pi used to set the clocks in September was also used to measure the offset today. It has run continuously since the clocks were set, and has been synchronized continuously to another Raspberry Pi (+/- 0.1 ms), which was in turn synchronized to GPS using a Motorola Oncore UT+ receiver (+/- 150 ns). Time errors on the Pis are negligible.
• I have two PCA9548A I2C switches that allow me to wire up all the clock chips and switch between them using software commands rather than needing to physically move wires around. This makes life easy.
• At the start of the measurement, I had observed each of the clocks’ outputs using my oscilloscope and compared them to a GPS-synchronized PPS signal. Clocks #0 and #7 drifted faster than the others but were still within the advertised specs (unfortunately I didn’t write down how fast they were drifting and have since forgotten). I adjusted the aging register until the short-term drift was minimal; the results are acceptable, though I note they drifted the most of any of the crystal-based clocks.
• All ten clocks are part of a cheap module available on eBay from various Chinese sellers. The module comes with either a DS3231 or DS3231M chip (the sellers don’t sort them so you can get either type at random) of various vintages. The oldest I’ve seen is from 2006. None of the chips seem to be new, with various smudges and wear visible on the face of the chip, so they’re likely pulled from old equipment and reused on these boards. Even so, they work well.
• Each board also comes with a 24C32A I2C EEPROM, which is nice, but not strictly necessary. I use them for storing the aging offset in case I remove the backup battery and want to tune it again without using the oscilloscope.
• The boards also come with a holder for a backup coin cell battery. Critically, the boards are also wired with a 2N4148 diode and a 200-ohm series resistor feeding the positive terminal of the battery, presumably for use with a (not included) LIR2032 rechargeable coin cell. If you use a non-rechargeable CR2032 coin cell you must remove either the diode or resistor or else the circuit will try to charge the coin cell battery, which can damage the battery. I’ve removed the battery holder entirely from one or two other test boards and the charging circuit works well to charge a backup supercapacitor, but the charging circuit must be removed or disabled if you use non-rechargeable coin cell batteries.
• The data here is just a brief summary; I have more detailed data in a spreadsheet that’s available upon request.

My ensemble of DS3231 and DS3231M RTCs has been running since early September 2017 and I’m looking to start gathering data on their performance in the near future. Thus, I’ve taken one of the DS3231 RTCs from the ensemble (leaving ten) and done some experiments with it to figure out how best to read the time from the clock chips at the highest precision I can using a Raspberry Pi.

This is complicated by three issues:

1. Although the DS3231 RTC has an internal 15-bit counter that counts individual crystal ticks, that counter is not accessible by the user. Instead, we can only access the timekeeping registers with a precision of one second.
2. Due to weird historical quirks that trace back to the original MC146818 RTC used in the first PC/AT standard, the hwclock utility sets the RTC so there’s a half-second difference between the system clock and the RTC.
3. With the reference NTP daemon running on a stock Raspbian system, the kernel enters an “eleven minute mode” where it will periodically set the RTC’s clock to the NTP-synchronized system clock. This is undesirable, but turning it off requires recompiling the kernel and I’m lazy.

First Issue

To address the first issue, I use the hwclock -c command, which repeatedly reads the timing registers on the RTC until the second changes. With the help of an oscilloscope and logic analyzer, I’ve confirmed that the seconds register (hex value 0x0, or “00h” in the notation the datasheet uses) is updated on the next read after the 1 Hz square wave output falls.

It’s worth quoting this part of the DS3231 datasheet:

On an I2C START or address pointer incrementing to location 00h, the current time is transferred to a second set of registers. The time information is read from these secondary registers, while the clock may continue to run. This eliminates the need to reread the registers in case the main registers update during a read.

This means that nothing weird will happen if the 1 Hz counter ticks in the middle of a read and the next read from the seconds register after the tick will have the latest value. Perfect. This means we can mark the new second with a precision of +/- 1 I2C read packet, which is ~1 ms when using hwclock to read the data. Not bad.

Second Issue

For the second issue, we need to account for a half-second offset between the system clock and the RTC. If we let the kernel automatically set the RTC using the “eleven minute mode”, the RTC is a half-second ahead of the system clock. The output of hwclock -c looks like this:

hw-time system-time freq-offset-ppm tick
1517492769 1517492768.514489 -3 -0
1517492780 1517492779.514081 -3 -0
1517492791 1517492790.513630 -3 -0
1517492802 1517492801.513253 -3 -0

That is, the RTC is 0.486747 ahead of the system clock. The reported precision is far too high: I’ve measured a difference of 3-4 milliseconds between the value reported by hwclock and difference between the RTC’s 1 Hz pulse and the UTC-aligned one-pulse-per-second (PPS) signal from a timing GPS receiver.

However, if we set the RTC manually using hwclock -w,the system clock is a half-second ahead of the RTC (at least until the eleven minute mode resets the RTC). Here’s an example:

hw-time system-time freq-offset-ppm tick
1517493232 1517493232.502916
1517493244 1517493244.502209 -59 -1
1517493255 1517493255.502850 -3 -0
1517493266 1517493266.502547 -11 -0
1517493277 1517493277.502220 -15 -0

Keep this in mind if you decide to do a similar test. For consistency, I used hwclock -w to set all the clocks in the ensemble simultaneously. One nice thing about the DS3231 and DS3231M is that it resets its internal 15-bit fractional seconds counter whenever the seconds register is written, so once set you only need to account for the half-second offset and not any leftover fractional seconds. (See the “Clock and Calendar” section of the datasheet: “The countdown chain is reset whenever the seconds register is written.”)

The RTC’s 1 Hz pulse goes high 500 ms after the seconds register is written, with the falling edge (which delineates the seconds) occurring 500 ms after that. Thus, the output of the 1 Hz pulse is synchronized to the seconds boundary (modulo the half-second offset discussed above) whenever the RTC is set.

This is rather interesting, as I had assumed the 1 Hz output from the RTC to simply be the buffered, divided-by-32768 output of the crystal with no additional processing, but it’s actually something the chip can adjust the timing of based on user input. That’s really cool and could come in handy when measuring the phase drift of two or more such clocks (keeping in mind that the 1 Hz pulse is aligned to the closest crystal tick).

As an experiment, I removed the battery from the RTC and reset it to factory defaults, enable the 1 Hz output, and measured its offset relative to the GPS PPS pulse (it was about 115 ms, but it was free-wheeling After setting the RTC with hwclock -w, the rising edge was within 1 ms of the GPS pulse. Yup, it works.

Let’s look at this visually (click to enlarge):

Third Issue

The solution to this issue is easy: I basically ignore it. I have two PCA9548A I2C multiplexers connected to my Raspberry Pi’s I2C bus, so I can programmatically switch between each RTC (all of which have the same fixed I2C address, necessitating the multiplexers). Thus, I can rapidly scan through each of the RTCs, gather data from each, and finally set the multiplexer to a position where there is no RTC.

The probability of the kernel setting the RTC on its eleven minute mode schedule during this quick burst of activity is non-zero but low enough that I don’t worry. If I wanted to turn off the eleven minute mode entirely, I could recompile the kernel with that option disabled, but I’m lazy.

Conclusion

Based on measurements of both the software and hardware timing, I can conservatively read the time from the RTC within +/- 10 ms. That’s both better than I expected and more than adequate for my testing. Going into this, I was expecting around 200-500 ms precision.

The major (and I use that relatively) issue I face is the half-second offset. Since I know I used hwclock -w to set all the RTCs simultaneously, I know they were all set 0.5 seconds behind the system time at the moment they were set. Also, since the output of hwclock -c is based on the tick of the RTC clock, the system clock would show as 0.5 seconds ahead. Later, when I measure the difference between the RTC and system clock I need to make sure to subtract 0.5 seconds from the reported system time to get a proper comparison.

Lastly, datasheets have a lot of interesting details that are really easy to overlook. Having a logic analyzer (I really like Saleae) and a cheap timing GPS with a PPS output (an old Motorola Oncore UT+ I use for my house NTP server; it’s old but keeps ticking along) makes it really easy and fun to explore these details. I enjoyed digging into the behavior of the hwclock software and DS3231 hardware and hope this information is of some use to you.