HDF version 4.2.16-2 released on 2023-06-30
====================================================

INTRODUCTION

This document describes the differences between this release and the
HDF 4.2.15 release.  It is written for people who are familiar with
previous releases of HDF and wish to migrate to this version of HDF.

Note that the HDF4 documentation will be updated at the time of
each final release and can be found on the HDF4 support page at:

    https://portal.hdfgroup.org/display/HDF4/HDF4

The official HDF4 releases can be obtained from:

    https://portal.hdfgroup.org/display/support/Download+HDF4

If you have any questions or comments, please send them to the HDF Help Desk:

     help@hdfgroup.org

CONTENTS

- New features and changes
  -- Configuration
- Support for new platforms and compilers
- Bugs fixed since HDF 4.2.16
- Bugs fixed since HDF 4.2.15
  -- Configuration
  -- Library
  -- Utilities
- Documentation
- Platforms Tested
- Known problems


New features and changes
========================
    Configuration:
    -------------
    - The Autotools no longer strip -g from production CFLAGS

      configure would strip -g from CFLAGS when --enable-production was
      specified. This is no longer the case.

      (DER - 2023/03/01)

    - The Autotools now set -DNDEBUG/-UNDEBUG via build mode

      Several makefiles set -DNDEBUG for all build modes. This is now set
      via the build mode configure options. Production sets -DNDEBUG and
      development sets -UNDEBUG.

      (DER - 2023/03/01)

    - Tools are now built by default in CMake

      The option to build and test the HDF4 tools hdiff, hrepack, hdfimport,
      and dumper was disabled by default. It is now enabled by default.

      This option also controls the ability to build the netCDF tools.

      (DER - 2023/02/28)

    - Added an option to disable building the netCDF tools ncdump and ncgen

      HDF4 builds netCDF 2.3.2 versions of ncdump and ncgen. These can conflict
      with the official versions so an option to disable building them has
      been added.

        CMake:        HDF_ENABLE_NETCDF_TOOLS
        Autotools:    --enable-netcdf-tools

      This option can be set independently of the option that controls netCDF
      name mangling (HDF_ENABLE_NETCDF / --enable-netcdf) as no C library
      functionality is affected by that option.

      This option is enabled by default. HDF4_BUILD_TOOLS must be enabled to
      build the netCDF tools.

      (DER - 2023/02/28)

    - Removed checks for flex/lex and bison/yacc in the Autotools

      flex and bison are only needed to regenerate the ncgen parser files.
      This is done via bin/genparser and not using Makefiles, so the checks
      for these programs have been removed from configure (they were never
      checked for in CMake).

      (DER - 2023/02/23)

    - Removed VOID and VOIDP from hdfi.h

      These were typedef'd to void and void *, respectively. Although they did
      not appear in the public API, they were used in the examples.

      (DER - 2023/02/05)

    - Added Intel OneAPI compiler support to the configuration files

      The CMake toolchain file for Intel was updated to allow the
      LLVM based compilers to be chosen. The autotools config for
      Intel check for the new compiler name; icx.

      (ADB - 2023/02/02)

    - Updated the warnings flags for compilers with files for gcc,
      clang and Intel compilers.

      As explained in a README file in config, the warnings are generally
      version-specific based on a common file. Warnings that are no longer
      an issue are configured to generate an error if it is accidentally
      introduced. Other compilers could be added.

      (ADB - 2023/01/31)

    - The FreeBSD Autotools config was updated to work like MacOS and Linux

      The config/freebsd file was updated to try a variety of compilers and
      the default was set to 'cc', which should map to clang. Fortran is less
      well tested due to its issues with 64-bit systems, but the default
      of 'f77' was left alone.

      It may be necessary to specify --with-jpeg=/usr/local if libjpeg was
      installed via the package manager.

      (DER - 2023/01/25)

    - The vattr.h header file is no longer distributed

      This header file was empty aside from including hdf.h and vg.h.

      (DER - 2023/01/22)

    - Better Autotools compiler support on Linux and MacOS

      The machine configuration files have been split into machine and
      compiler files, which have been updated with additional warning
      flags and compiler support.

      Both Linux and MacOS now support:

        * gcc
        * clang
        * icc/icx

      (DER - 2023/01/20)

    - The Fortran interface is considered deprecated

      The Fortran interface is broken on 64-bit systems (see the known
      problems section) and is currently disabled by default. It may
      be removed in a future version of the library or replaced with an
      alternative Fortran wrapper, possibly as early as 4.2.17.

      (DER - 2023/01/15)

    - Fixed-width HDF4 types like int16 now map to C99 types under the hood

      In an effort to make HDF4 more maintainable and "just work" for more
      platforms, the scheme where HDF4 fixed-width integer types like int16
      were mapped to C native types like int and short has been replaced
      with a system that maps them to equivalent C99 types.

      (DER - 2023/01/15)
    
    - Changed how maintainer mode works in the Autotools

      The macro has changed from AM_MAINTAINER_MODE to
      AM_MAINTAINER_MODE([enable]) in master and AM_MAINTAINER_MODE([disable])
      at release time. The main change is that build dependencies will be ON
      by default in master so out-of-date configure, Makefile.in will be
      regenerated when their input files (configure.ac, Makefile.am, etc.)
      change.

      We previously used the macro with no parameter since we checked in the
      Autotools-generated files. This allowed switching on maintainer mode
      with a command-line option, but left maintainer mode off to avoid
      regenerating the build files when checking out from git did not
      preserve timestamps, making the files look out-of-date. Now that we no
      longer check in Autotools-geneated files, we can turn the
      dependencies on.

      Detailed comments about the behavior of this macro have been added to
      configure.ac where we define the macro.

      (DER - 2023/01/13)

    - Removed the hdf/util/MacProjects directory

      It's unclear what build system these were for, but they have been
      unmaintained for some time and have been removed. MacOS users can
      build with the Autotools or CMake.

      (DER - 2023/01/13)

    - Dropped support for many obsolete platforms

      HDF4 uses pre-defined machine configurations instead of configure tests
      to include header files, set up typedefs, etc. In the future, the
      library will be adapted to use a flexible configure test scheme like
      HDF5, but in the meantime we have simplified the machine configuration
      header by dropping configurations for the following platforms:

        - Irix / MIPS
        - Itanium
        - Intel i860
        - Fujitsu VP
        - Convex
        - "K&R Sun" and Solaris on x86
        - DEC Alpha
        - Apollo
        - HP-UX / HP-9000
        - Motorola m88k
        - NeXT
        - MacOS on PowerPC chipsets
        - IBM RS/6000

      In addition, many #defines have been updated to reflect modern usage,
      particularly on Windows and MacOS.

      (DER - 2023/01/13)

    - Added new option to build jpeg, libaec and zlib inline with CMake.

      Using the CMake FetchContent module, the external filters can populate
      content at configure time via any method supported by the ExternalProject
      module. Whereas ExternalProject_Add() downloads at build time, the
      FetchContent module makes content available immediately, allowing the
      configure step to use the content in commands like add_subdirectory(),
      include() or file() operations.

      The HDF options (and defaults) for using this are:
          BUILD_JPEG_WITH_FETCHCONTENT:BOOL=OFF
          JPEG_USE_LOCALCONTENT:BOOL=OFF
          BUILD_SZIP_WITH_FETCHCONTENT:BOOL=OFF
          LIBAEC_USE_LOCALCONTENT:BOOL=OFF
          BUILD_ZLIB_WITH_FETCHCONTENT:BOOL=OFF
          ZLIB_USE_LOCALCONTENT:BOOL=OFF

      The CMake variables to control the path and file names:
          JPEG_TGZ_ORIGPATH:STRING
          JPEG_TGZ_ORIGNAME:STRING
          LIBAEC_TGZ_ORIGPATH:STRING
          LIBAEC_TGZ_ORIGNAME:STRING
          ZLIB_TGZ_ORIGPATH:STRING
          ZLIB_TGZ_ORIGNAME:STRING

      See the CMakeFilters.cmake and config/cmake/cacheinit.cmake files for usage.

      (ADB - 2023/01/13)

    - Removed all references to C++

      HDF4 does not have a C++ wrapper and, like HDF5, we do not support
      building the library with a C++ compiler since the languages are
      diverging.

      Despite this, some parts of the build system checked for a C++ compiler,
      manipulated CXXFLAGS, etc. This was mainly in files that were brought
      over from HDF5. This build system code has been removed.

      As a part of this C++ cleanup, the extern "C" ifdefs in the header files
      no longer test for c_plusplus.

      (DER - 2023/01/13)

    - The Fortran interface is disabled by default

      The Fortran interface is unsafe on 32-bit systems as it stores pointers
      in 32-bit integers. It will be removed in a future release, but for now
      it will be disabled by default in both CMake and the Autotools.

      (DER - 2023/01/12)

    - Add new CMake configuration variable HDF4_USE_GNU_DIRS

      HDF4_USE_GNU_DIRS (default OFF) selects the use of GNU Coding Standard install
      directory variables by including the CMake module, GNUInstallDirs(see CMake
      documentation for details). The HDF_DIR_PATHS macro in the HDFMacros.cmake file
      sets various PATH variables for use during the build, test and install processes.
      By default, the historical settings for these variables will be used.

      (ADB - 2022/10/21)

    - Corrected path searched by CMake find_package command

      The install path for cmake find_package files had been changed to use
        "share/cmake"
      on all platforms. However, the trailing "hdf4" directory was not removed
      at that time. This "hdf4" additional directory has been removed from all
      the install paths.

      (ADB - 2021/09/27)

    - Corrected pkg-config compile script

      It was discovered that the position of the "$@" argument for the command
      in the compile script may fail on some platforms and configurations. The
      position of the "$@"command argument was moved to a position before the
      pkg-config sub command.

      (ADB - 2021/08/30)

    - CMake option to use AEC (open source SZip) library instead of SZip

      The open source AEC library is a replacement library for SZip. In
      order to use it for hdf5 the libaec CMake source was changed to add
      "-fPIC" and exclude test files. Autotools does not build the
      compression libraries within hdf5 builds. New option USE_LIBAEC is
      required to compensate for the different files produced by AEC build.

      (ADB - 2020/04/22, OESS-65)


Support for new platforms and compilers
=======================================

    - CMake added support for clang 12.0.0 with MSVC-like command line in VS2019,
          and for VS2022 with clang 15.0.1 and with Intel oneAPI 2022

    - Ubuntu 22.04, Linux 5.15.0 with GCC 11.3.0 and clang version 14.0.0-1

    - Ubuntu 20.04, Linux 5.13.0 with GCC 9.4.0 and clang version 10.0.0-4

    - macOS M1 11.7.3, Darwin 20.6.0 with Apple clang version 13.0.0 and
                                     Intel icc/icpc/ifort version 2021.8.0

    - macOS Big Sur 11.7.3, Darwin 20.4.0 with Apple clang version 12.0.5 and
                                          Intel icc/icpc/ifort version 2021.7.1


Bugs fixed since HDF 4.2.16
===========================

    - Fixed an external file bug introduced in 4.2.16

      Fixing HDFFR-1607 introduced a bug where the flag that indicates whether
      the external file directory has been changed from the previous location
      was not reset properly.
      Reset the flag after building a filename path for an external element.

      Fixes HDFFR-1609

    - Build fortran with -fallow-argument-mismatch w/ gcc 10+

      The gfortran configuration checks checked the version via a variable that
      was never set, preventing -fallow-argument-mismatch from being set w/
      gcc 10+, which would prevent building the fortran interface.

      Fixes HDFFR-1582

    - JNI SDSreaddata functions for float and double used mismatched buffer macros

      The SDreaddata_double and SDreaddata_float function created data buffers with
      PIN_DOUBLE_ARRAY_CRITICAL and PIN_FLOAT_ARRAY_CRITICAL macros then tried to
      free the buffers with UNPIN_DOUBLE_ARRAY and UNPIN_FLOAT_ARRAY macros. This caused
      free() call to use an invalid pointer.

      Change the macros PIN_DOUBLE_ARRAY_CRITICAL to PIN_DOUBLE_ARRAY and
      PIN_FLOAT_ARRAY_CRITICAL to PIN_FLOAT_ARRAY.

      Github issue #355


Bugs fixed since HDF 4.2.15
===========================

    - Applied a user's patch to fix a failure with SDwritedata

      The access to an internal element was not terminated in one of the SD
      APIs causing the subsequent call to SDwritedata to fail.  The user's
      patch ended access to the element.

      (BMR - 2023/03/01, HDFFR-1573)

    - Corrected the setting of directory for external file

      When application called HXsetdir after a read of an external data, the
      new setting of the external directory did not have any effect because
      the library would just use the currently opened file.  The library now
      closes the opened external file and reopens it at the new location if
      the external directory is changed.

      (BMR - 2023/02/22, HDFFR-1607)

    - Correct theCMake generated pkg-config file

      The pkg-config file generated by CMake had the order and placement of the
      libraries wrong. Also added support for debug library names.

      Changed the order of Libs.private libraries so that dependencies come after
      dependents. Did not move the compression libraries into Requires.private
      because there was not a way to determine if the compression libraries had
      supported pkconfig files. Still recommend that the CMake config file method
      be used for building projects with CMake.

      (ADB - 2023/02/16)

    - Fixed a small memory leak in the atom code

      The atom list hast table of each atom group list was not cleaned up at
      library shutdown, leading to a small memory leak (at most, a few kB).

      (DER - 2023/01/29)

    - Specifying CC on the command line works correctly in the Autotools

      The CC variable was processed incorrectly in the Autotools build files
      which led to minimal CFLAGS, etc. flags being specified when anything
      other than CC=gcc was specified.

      (DER - 2023/01/20)

    - --enable-production in the Autotools works correctly

      The variable that was checked when processing --enable-production in
      the Autotools was spelled incorrectly, leading an Autotools build
      to default to a "user defined" build that specified minimal flags
      in CFLAGS, etc.

      (DER - 2023/01/20)

    - A segfault is replaced by a failure

      A segfault had occurred when reading a user's file.  Examining the file
      contents showed that the value for the number of fields in one of the
      vdata had been corrupted and became larger than the maximum number of
      fields allowed in a vdata.  The library now detects such situation and
      fails instead of producing a segfault.

      (BMR - 2023/01/19, HDFFR-1593)

    - jpeglib.h `boolean` type no longer conflicts with windows.h

      jpeglib.h defines a `boolean` type that conflicts with the `boolean`
      type defined in windows.h. In previous versions of the library, this
      could cause compilation errors when the types clashed.

      We now ensure that `HAVE_BOOLEAN` is defined before jpeglib.h when
      building on Windows, which avoids the redefinition.

      (DER - 2022/01/15)



Documentation
=============


Platforms Tested
================
This version has been tested in the following platforms:

(Format:
   uname -s, uname -r
   uname -v, uname -p, uname -m)

    Linux 5.15.0-1026-aws            gcc (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
    #30-Ubuntu SMP x86_64 GNU/Linux  GNU Fortran (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
    Ubuntu 22.04                     Ubuntu clang version 14.0.0-1ubuntu1
                                     (cmake and autotools)

    Linux 5.15.0                     GNU gcc (GCC) 11.3.0
    Ubuntu SMP x86_64 GNU/Linux      GNU Fortran (GCC) 11.3.0-
    Ubuntu 22.04                     Ubuntu clang version 10.0.0-4
      (mingw64)                      (cmake and autotools)

    Linux 5.13.0-1031-aws            GNU gcc (GCC) 9.4.0-1ubuntu1
    #35-Ubuntu SMP x86_64 GNU/Linux  GNU Fortran (GCC) 9.4.0-1ubuntu1
    Ubuntu 20.04                     clang version 10.0.0-4ubuntu1
                                     (cmake and autotools)

    Linux 4.18.0-348.7.1.el8_5       gcc (GCC) 8.5.0 20210514 (Red Hat 8.5.0-4)
    #1 SMP x86_64 GNU/Linux          GNU Fortran (GCC) 8.5.0 20210514 (Red Hat 8.5.0-4)
    CentOS8                          clang version 12.0.1 (Red Hat 12.0.1)
                                     (cmake and autotools)

    Linux 3.10.0-1160.36.2.el7.ppc64 gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39)
    #1 SMP ppc64 GNU/Linux           g++ (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39)
    (echidna)                        GNU Fortran (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39)
                                     IBM XL C/C++ V13.1
                                     IBM XL Fortran V15.1
 
    Linux 3.10.0-327.18.2.el7.x86_64 GNU C (gcc) and Fortran (gfortran) compilers:
    #1 SMP x86_64, GNU/Linux             Version 4.8.5 20150623 (Red Hat 4.8.5-4)
    jelly/moohan)                        Version 4.9.3, 5.3.0, 6.2.0, 7.1.0, 8.3.0
                                     Intel(R) C (icc) and Fortran (ifort) compilers:
                                        Version 17.0.0.098 Build 20160721
                                     pgcc and pgf90 17.10-0 64-bit target
                                         on x86-64 Linux -tp haswell

    Linux 2.6.32-754.11.1.el6.x86_64  gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-16)
    #1 SMP, x86_64                    GNU Fortran (GCC) 4.4.7 20120313 (Red Hat 4.4.7-16)
    (platypus)                        icc (ICC) 17.0.0.098 Build 20160721
                                      ifort (IFORT) 17.0.0.098 Build 20160721
                                      pgcc and pgf90 17.10-0 64-bit target
                                         on x86-64 Linux -tp nehalem

    macOS Apple M1 11.7.3            Apple clang version 13.0.0 (clang-1300.0.29.30)
    Darwin 20.6.0 arm64              gfortran GNU Fortran (Homebrew GCC 12.2.0) 12.2.0
    (macmini-m1)                     Intel icc/icpc/ifort version 2021.8.0 20221120

    macOS Big Sur 11.7.3             Apple clang version 12.0.5 (clang-1205.0.22.9)
    Darwin 20.4.0 x86_64             gfortran GNU Fortran (Homebrew GCC 12.2.0) 12.2.0
    (bigsur-1)                       Intel icc/icpc/ifort version 2021.7.1 20221019

    macOS High Sierra 10.13.6        Apple LLVM version 10.0.0 (clang-1000.10.44.4)
    64-bit                           gfortran GNU Fortran (GCC) 6.3.0
    (bear)                           Intel icc/icpc/ifort version 18.0.4.233 20190416

    macOS Sierra 10.12.6              Apple LLVM version 9.0.0 (clang-900.39.2)
    64-bit                            gfortran GNU Fortran (GCC) 7.4.0
    (kite)                            Intel icc/icpc/ifort version 17.0.2

    Mac OS X El Capitan 10.11.5      Apple clang version 7.3 from Xcode 7.3
    Darwin 15.6.0 x86_64             gfortran GNU Fortran (GCC) 5.2.0
    (osx1011test)                    Intel icc and ifort version 15.0.3

    Windows 10 x64                  Visual Studio 2019 w/ clang 12.0.0
                                        with MSVC-like command-line (C/C++ only - cmake)
                                    Visual Studio 2019 w/ Intel C/C++/Fortran oneAPI 2023 (cmake)
                                    Visual Studio 2022 w/ clang 15.0.1
                                        with MSVC-like command-line (C/C++ only - cmake)
                                    Visual Studio 2022 w/ Intel C/C++/Fortran oneAPI 2023 (cmake)

Known problems
==============
o  The Fortran interface does not work on 64-bit systems as it stores addresses
   in memory as Fortran INTEGER values, which are typically 32-bit. The
   Fortran interface is currently disabled by default due to this. It should
   only be enabled on 32-bit systems.

o  Builds with the autotools option --enable-hdf4-xdr fail on Solaris and
   on IBM ppc64 with the xlc compiler.  The option should not be used on
   those platforms.

o  When building HDF4 in 32-bit mode on 64-bit Linux platforms with the -m32
   flag, many tests fail with errors converting data.  We have not seen these
   test failures building on 32-bit systems.  The problem seems to be caused
   specifically by building with the -m32 flag on 64-bit systems.  It will be
   investigated;  the -m32 flag on 64-bit systems should not be used with this
   HDF4 release.  

o  A number of tools and tests fail to free small amounts of memory if they
   are just going to exit anyway.  This doesn't appear to be a memory leak
   in the core library.  We're planning to fix these leaks in future releases,
   likely by modifying these tools to have a single point of return, where
   resource cleanup will be carried out. 

o  CMake files do not behave correctly with paths containing spaces.
   Do not use spaces in paths because the required escaping for handling spaces
   results in very complex and fragile build files.
   ADB - 2019/05/07

o  Several Fortran examples print "^@" when displaying strings (for example,
   names of the attributes). This happens because Fortran application
   doesn't know the length of the strings passed from the C library.
   EIP - 2015-01-11, HDFFR-1477

o  CMake fails to set the full path to the install location on Windows:
    The configuration file for examples, HDF4_Examples.cmake, must be updated
    with the correct value by editing the file or using the INSTALLDIR option.
    This issue is because of spaces in the path.
   ADB - 2014/02/03

o  CMake "make install" fails installing the tools:
    Use CPack to create an install package.
   ADB - 2014/02/03

o  CMake does not install these man pages:
    hdf.1, ncdump.1, ncgen.1
   AKC/BMR - 2014/02/02

o  On IBM PowerPC 64, hdftest fails when gcc 4.4.6 is used with -O3 optimization
   level.

o  When building on Linux/UNIX platforms, the szip shared library files must
   be in the system library path.  This can be done by adding a link to
   the libsz.* files in the /usr/lib folder or by adding the library
   location to the LD_LIBRARY_PATH environment variable.
      Ex. export LD_LIBRARY_PATH=path_to_szip_lib:$LD_LIBRARY_PATH
   Optionally, one can use the static szip library files by adding '-static'
   to the CFLAGS environment variable.

o  Existing data written by an HDF4 Library prior to HDF 4.2r2:
   When a one-dimensional SDS and a dimension scale have
   the same name, subsequent accesses to the dimension scale or to the
   SDS might produce undesired results because the libraries could not
   distinguish between the two objects.  In the case of writing, data
   might even be corrupted.  For example, SDS data might be written to a
   dimension variable or vice versa. (bugzilla #624)

   HDF4 Library Releases 4.2r2 and later make a distinction between an SDS
   and a dimension variable.  However, as with older versions, these recent
   versions are unable to detect such conflicts in files created by earlier
   releases.  It is therefore STRONGLY recommended to check for such name
   duplication before working with data created with a pre-4.2r2 library.

   The functions SDgetnumvars_byname and SDnametoindices are provided
   to help detect such name conflicts and select the correct object to
   access, respectively; see the HDF Reference Manual entries for
   further details.
   FB - 2009/01/26
   BMR - revised 2011/06/24

o  N-bit compression is not supported with Fortran APIs.

o  Using both fill-value and compression on SD datasets does not work.

o  When using PGI compilers, make sure that the JPEG library is also compiled
   with a PGI C compiler; linking with a JPEG library built with gcc causes
   JPEG library tests to fail.  To bypass the problem:

       x Set LIBS flag to $PGI_JPEG_INSTALL_DIR/lib/libjpeg.a
         where $PGI_JPEG_INSTALL_DIR points to the installation directory
         for the PGI-compiled JPEG library:

         setenv LIBS $PGI_JPEG_INSTALL_DIR/lib/libjpeg.a

       x Use the --with-jpeg=$PGI_JPEG_INSTALL_DIR configure flag to
         configure with the PGI-compiled JPEG library:

         ./configure --with-jpeg=$PGI_JPEG_INSTALL_DIR --with-zlib....

o  In order for the API SDgetdatasize to get the correct compressed size
   of the data, the dataset needs to be closed (SDendaccess) or read
   (SDreaddata) after being written and before SDgetdatasize is called.
   BMR - 2008/11/22

