Fix leap second expiry and truncation

* Makefile, NEWS, tzfile.5, zic.8: Document the change.
* zic.c (ZIC_VERSION_PRE_2013, ZIC_VERSION, comment_leapexpires):
Remove.  All uses removed.
(infile): Don’t bother parsing #expires lines any more;
only Expires lines matter now.
(struct timerange): New member leapexpiry.
(limitrange, writezone): Set it.
(writezone): If leap seconds are generated, append a no-op
leap correction indicating when leap seconds expire.
Generate version 4 files (with a warning if -v) if
the leap second table is truncated from below, or expires.
Fix bug if thisleapcnt changes after thisleaplim is set.

Makefile

@@ -152,8 +152,10 @@ REDO=		posix_right
 # The EXPIRES_LINE value matters only if REDO's value contains "right".
 # If you change EXPIRES_LINE, remove the leapseconds file before running "make".
 # zic's support for the Expires line was introduced in tzdb 2020a,
-# and EXPIRES_LINE defaults to 0 for now so that the leapseconds file
-# can be given to older zic implementations.
+# and was modified in tzdb 2021b to generate version 4 TZif files.
+# EXPIRES_LINE defaults to 0 for now so that the leapseconds file
+# can be given to pre-2020a zic implementations and so that TZif files
+# built by newer zic implementations can be read by pre-2021b libraries.
 EXPIRES_LINE=	0
 
 # To install data in text form that has all the information of the TZif data,

NEWS

@@ -39,10 +39,24 @@ Unreleased, experimental changes
     seconds error than with an hour error, so zic -L no longer
     truncates output in this way.
 
+    Instead, when zic -L is given the "Expires" directive, it now
+    outputs the expiration by appending a no-change entry to the leap
+    second table.  Although this should work well with most TZif
+    readers, it does not conform to Internet RFC 8536 and some pickier
+    clients (including tzdb 2017c through 2021a) reject it, so
+    "Expires" directives are currently disabled by default.  To enable
+    them, set the EXPIRES_LINE Makefile variable.  If a TZif file uses
+    this new feature it is marked with a new TZif version number 4.
+
+    zic -L LEAPFILE -r @LO no longer no longer generates an invalid
+    TZif file that omits leap second information for the range LO..B
+    when LO falls between two leap seconds A and B.  Instead, it
+    generates a TZif version 4 file that represents the
+    previously-missing information.
+
     The TZif reader now allows the leap second table to begin with a
     correction other than -1 or +1, and to contain adjacent
-    transitions with equal corrections.  This supports possible
-    future extensions to the TZif format.
+    transitions with equal corrections.  This supports TZif version 4.
 
     Fix bug that caused 'localtime' etc. to crash when TZ was
     set to a all-year DST string like "EST5EDT4,0/0,J365/25" that does

tzfile.5

@@ -30,10 +30,11 @@ The magic four-byte ASCII sequence
 identifies the file as a timezone information file.
 .IP *
 A byte identifying the version of the file's format
-(as of 2017, either an ASCII NUL, or
+(as of 2021, either an ASCII NUL,
 .q "2",
+.q "3",
 or
-.q "3" ).
+.q "4" ).
 .IP *
 Fifteen bytes containing zeros reserved for future use.
 .IP *
@@ -134,13 +135,15 @@ the first value of each pair gives the nonnegative time
 (as returned by
 .BR time (2))
 at which a leap second occurs;
-the second is a signed integer specifying the
+the second is a signed integer specifying the correction, which is the
 .I total
 number of leap seconds to be applied during the time period
 starting at the given time.
 The pairs of values are sorted in ascending order by time.
 Each transition is for one leap second, either positive or negative;
 transitions always separated by at least 28 days minus 1 second.
+The first entry's correction is +1 (or \(mi1, for a hypothetical leap
+second table where the first leap second was negative).
 .IP *
 .B tzh_ttisstdcnt
 standard/wall indicators, each stored as a one-byte boolean;
@@ -212,6 +215,16 @@ from 0 through 24.
 Second, DST is in effect all year if it starts
 January 1 at 00:00 and ends December 31 at 24:00 plus the difference
 between daylight saving and standard time.
+.SS Version 4 format
+For version-4-format TZif files,
+the first leap second transition can have a correction that is neither
++1 nor \(mi1, to support TZif files with reduced timestamp range.
+Also, if two or more leap second transitions are present and the last
+entry's correction equals the previous one, the last entry
+denotes the expiration of the leap second table instead of a leap second;
+timestamps after this expiration are unreliable in that future
+releases will likely add leap second entries after the expiration, and
+the added leap seconds will change how post-expiration timestamps are treated.
 .SS Interoperability considerations
 Future changes to the format may append more data.
 .PP

zic.8

@@ -31,7 +31,8 @@ zic \- timezone compiler
 The
 .B zic
 program reads text from the file(s) named on the command line
-and creates the time conversion information files specified in this input.
+and creates the timezone information format (TZif) files
+specified in this input.
 If a
 .I filename
 is
@@ -213,6 +214,15 @@ code designed for older
 output formats.  These compatibility issues affect only timestamps
 before 1970 or after the start of 2038.
 .PP
+The output contains a truncated leap second table,
+which can cause some older TZif readers to misbehave.
+This can occur if the
+.B "\*-L"
+option is used, and either an Expires line is present or
+the
+.B "\*-r"
+option is also used.
+.PP
 The output file contains more than 1200 transitions,
 which may be mishandled by some clients.
 The current reference client supports at most 2000 transitions;
@@ -720,19 +730,6 @@ The
 and
 .B HH:MM:SS
 fields give the expiration timestamp in UTC for the leap second table;
-If there is no expiration line,
-.B zic
-also accepts a comment
-.q "#expires \fIE\fP ...\&"
-where
-.I E
-is the expiration timestamp as a decimal integer count of seconds
-since the Epoch, not counting leap seconds.
-However, the
-.q "#expires"
-comment is an obsolescent feature,
-and the leap second file should use an expiration line
-instead of relying on a comment.
 .SH "EXTENDED EXAMPLE"
 Here is an extended example of
 .B zic

zic.c

@@ -15,9 +15,6 @@
 #include <stddef.h>
 #include <stdio.h>
 
-#define	ZIC_VERSION_PRE_2013 '2'
-#define	ZIC_VERSION	'3'
-
 typedef int_fast64_t	zic_t;
 #define ZIC_MIN INT_FAST64_MIN
 #define ZIC_MAX INT_FAST64_MAX
@@ -681,9 +678,6 @@ static zic_t hi_time = MAXVAL(zic_t, TIME_T_BITS_IN_FILE);
 /* The time specified by an Expires line, or negative if no such line.  */
 static zic_t leapexpires = -1;
 
-/* The time specified by an #expires comment, or negative if no such line.  */
-static zic_t comment_leapexpires = -1;
-
 /* Set the time range of the output to TIMERANGE.
    Return true if successful.  */
 static bool
@@ -1354,8 +1348,7 @@ infile(const char *name)
 			++nfields;
 		}
 		if (nfields == 0) {
-		  if (name == leapsec && *buf == '#')
-		    sscanf(buf, "#expires %"SCNdZIC, &comment_leapexpires);
+			/* nothing to do */
 		} else if (wantcont) {
 			wantcont = inzcont(fields, nfields);
 		} else {
@@ -1975,6 +1968,7 @@ struct timerange {
   int defaulttype;
   ptrdiff_t base, count;
   int leapbase, leapcount;
+  bool leapexpiry;
 };
 
 static struct timerange
@@ -1986,7 +1980,7 @@ limitrange(struct timerange r, zic_t lo, zic_t hi,
     r.count--;
     r.base++;
   }
-  while (0 < r.leapcount && trans[r.leapbase] < lo) {
+  while (1 < r.leapcount && trans[r.leapbase + 1] <= lo) {
     r.leapcount--;
     r.leapbase++;
   }
@@ -1997,6 +1991,7 @@ limitrange(struct timerange r, zic_t lo, zic_t hi,
     while (0 < r.leapcount && hi + 1 < trans[r.leapbase + r.leapcount - 1])
       r.leapcount--;
   }
+  r.leapexpiry = 0 <= leapexpires && leapexpires - 1 <= hi;
 
   return r;
 }
@@ -2107,9 +2102,36 @@ writezone(const char *const name, const char *const string, char version,
 	rangeall.base = rangeall.leapbase = 0;
 	rangeall.count = timecnt;
 	rangeall.leapcount = leapcnt;
+	rangeall.leapexpiry = false;
 	range64 = limitrange(rangeall, lo_time, hi_time, ats, types);
 	range32 = limitrange(range64, INT32_MIN, INT32_MAX, ats, types);
 
+	/* TZif version 4 is needed if a no-op transition is appended to
+	   indicate the expiration of the leap second table, or if the first
+	   leap second transition is not to a +1 or -1 correction.  */
+	for (pass = 1; pass <= 2; pass++) {
+	  struct timerange const *r = pass == 1 ? &range32 : &range64;
+	  if (pass == 1 && !want_bloat())
+	    continue;
+	  if (r->leapexpiry) {
+	    if (noise)
+	      warning(_("%s: pre-2021b clients may mishandle"
+			" leap second expiry"),
+		      name);
+	    version = '4';
+	  }
+	  if (0 < r->leapcount
+	      && corr[r->leapbase] != 1 && corr[r->leapbase] != -1) {
+	    if (noise)
+	      warning(_("%s: pre-2021b clients may mishandle"
+			" leap second table truncation"),
+		      name);
+	    version = '4';
+	  }
+	  if (version == '4')
+	    break;
+	}
+
 	fp = open_outfile(&outname, &tempname);
 
 	for (pass = 1; pass <= 2; ++pass) {
@@ -2117,7 +2139,7 @@ writezone(const char *const name, const char *const string, char version,
 		register int	thisleapi, thisleapcnt, thisleaplim;
 		struct tzhead tzh;
 		int currenttype, thisdefaulttype;
-		bool locut, hicut;
+		bool locut, hicut, thisleapexpiry;
 		zic_t lo;
 		int old0;
 		char		omittype[TZ_MAX_TYPES];
@@ -2148,6 +2170,7 @@ writezone(const char *const name, const char *const string, char version,
 			toomanytimes = thistimecnt >> 31 >> 1 != 0;
 			thisleapi = range32.leapbase;
 			thisleapcnt = range32.leapcount;
+			thisleapexpiry = range32.leapexpiry;
 			locut = INT32_MIN < lo_time;
 			hicut = hi_time < INT32_MAX;
 		} else {
@@ -2157,6 +2180,7 @@ writezone(const char *const name, const char *const string, char version,
 			toomanytimes = thistimecnt >> 31 >> 31 >> 2 != 0;
 			thisleapi = range64.leapbase;
 			thisleapcnt = range64.leapcount;
+			thisleapexpiry = range64.leapexpiry;
 			locut = min_time < lo_time;
 			hicut = hi_time < max_time;
 		}
@@ -2177,7 +2201,6 @@ writezone(const char *const name, const char *const string, char version,
 		}
 
 		thistimelim = thistimei + thistimecnt;
-		thisleaplim = thisleapi + thisleapcnt;
 		if (thistimecnt != 0) {
 		  if (ats[thistimei] == lo_time)
 		    locut = false;
@@ -2279,6 +2302,7 @@ writezone(const char *const name, const char *const string, char version,
 		}
 		if (pass == 1 && !want_bloat()) {
 		  thisleapcnt = 0;
+		  thisleapexpiry = false;
 		  thistimecnt = - (locut + hicut);
 		  thistypecnt = thischarcnt = 1;
 		  thistimelim = thistimei;
@@ -2289,7 +2313,7 @@ writezone(const char *const name, const char *const string, char version,
 		tzh.tzh_version[0] = version;
 		convert(utcnt, tzh.tzh_ttisutcnt);
 		convert(stdcnt, tzh.tzh_ttisstdcnt);
-		convert(thisleapcnt, tzh.tzh_leapcnt);
+		convert(thisleapcnt + thisleapexpiry, tzh.tzh_leapcnt);
 		convert(locut + thistimecnt + hicut, tzh.tzh_timecnt);
 		convert(thistypecnt, tzh.tzh_typecnt);
 		convert(thischarcnt, tzh.tzh_charcnt);
@@ -2347,6 +2371,7 @@ writezone(const char *const name, const char *const string, char version,
 		if (thischarcnt != 0)
 			fwrite(thischars, sizeof thischars[0],
 				      thischarcnt, fp);
+		thisleaplim = thisleapi + thisleapcnt;
 		for (i = thisleapi; i < thisleaplim; ++i) {
 			register zic_t	todo;
 
@@ -2370,6 +2395,15 @@ writezone(const char *const name, const char *const string, char version,
 			puttzcodepass(todo, fp, pass);
 			puttzcode(corr[i], fp);
 		}
+		if (thisleapexpiry) {
+		  /* Append a no-op leap correction indicating when the leap
+		     second table expires.  Although this does not conform to
+		     Internet RFC 8536, most clients seem to accept this and
+		     the plan is to amend the RFC to allow this in version 4
+		     TZif files.  */
+		  puttzcodepass(leapexpires, fp, pass);
+		  puttzcode(thisleaplim ? corr[thisleaplim - 1] : 0, fp);
+		}
 		if (stdcnt != 0)
 		  for (i = old0; i < typecnt; i++)
 			if (!omittype[i])
@@ -2779,7 +2813,7 @@ outzone(const struct zone *zpfirst, ptrdiff_t zonecount)
 	** Generate lots of data if a rule can't cover all future times.
 	*/
 	compat = stringzone(envvar, zpfirst, zonecount);
-	version = compat < 2013 ? ZIC_VERSION_PRE_2013 : ZIC_VERSION;
+	version = compat < 2013 ? '2' : '3';
 	do_extend = compat < 0;
 	if (noise) {
 		if (!*envvar)
@@ -3154,12 +3188,6 @@ adjleap(void)
 		last = corr[i] += last;
 	}
 
-	if (leapexpires < 0) {
-	  leapexpires = comment_leapexpires;
-	  if (0 <= leapexpires)
-	    warning(_("\"#expires\" is obsolescent; use \"Expires\""));
-	}
-
 	if (0 <= leapexpires) {
 	  leapexpires = oadd(leapexpires, last);
 	  if (! (leapcnt == 0 || (trans[leapcnt - 1] < leapexpires))) {