FAQs for the INTERMAGNET web site

What data should I send to INTERMAGNET?

To join INTERMAGNET you need to be able to send provisional real-time minute mean data to one of the Geomagnetic Information Nodes within 72 hours of recording. You also need to be able to send definitive data within the deadline set in INTERMAGNET’s annual call for data. If you are able, INTERMAGNET would very much welcome some other data products:

How do I get a real-time data feed from INTERMAGNET?

INTERMAGNET data is available via FTP. Automated processes can interrogate the INTERMANGET FTP site frequently to discover whether new data is available and to download it if it is. This offers users the ability to retrieve data from INTERMAGNET in real-time. You will need a username and password to gain access to the FTP site – to request this access contact NRCan.geomag-webmaster-geomag-webmaster.RNCan@canada.ca

What are the INTERMAGNET data types?

INTERMAGNET has used two data formats for the dissemination of non-definitive data: INTERMAGNET Minute Mean Format (IMF, see http://www.intermagnet.org/formats/Imfv123e_e.php); and IAGA-2002 format (see www.ngdc.noaa.gov/IAGA/vdat/iagaformat.html). The IMF format is being phased out (from 2013 data in IMF format will no longer be distributed from the INTERMAGNET web site). The data types in the two formats are:

INTERMAGNET Data Types
Format Data Type Description
IMF Reported Data are defined as: the raw data obtained from the IMO, either by satellite, computer link, or other means. It will be formatted in either version IMFV2.8N (binary) or IMFV1.2N (ASCII) without any BRM (Baseline Reference Measurements), or other modifications applied to it.
IMF Adjusted Data are defined as: the Reported data with BRM, spike removal, timeshifts, and/or other modifications applied to it. It is emphasized that only one (1) adjusted version of the data would be allowed, to be completed within 7 days of receipt of the Reported data to prevent the proliferation of multiple versions of the Adjusted data.
IMF Quasi-definitive Data are defined as data that have been corrected using provisional baselines. Produced soon after their acquisition, their accuracy is intended to be very close to that of an observatory's definitive data product. 98% of the differences between quasi-definitive and definitive data (X, Y, Z) monthly mean values should be less than 5nT.
IMF Definitive Data are defined as the final adopted data values. Definitive data will only be distributed by the institution responsible for the observatory.
IAGA Variation The IAGA-2002 format does not describe the data types
IAGA Provisional The IAGA-2002 format does not describe the data types
IAGA Quasi-definitive The IAGA-2002 format does not describe the data types
IAGA Definitive Data The IAGA-2002 format does not describe the data types

The two data types quasi-definitive and definitive are well defined and have the same meaning when used in either of the two data types. The remaining data types are less clearly defined. Some computer software uses the IMF reported data type interchangeably with the variation data type, also the IMF adjusted data type with the IAGA-2002 provisional data type, and this is the policy used in INTERMAGNET.

What is quasi-definitive data?

INTERMAGNET has defined a standard for a new data type called quasi-definitive. As the name implies the data should be close to the expected definitive value, but is to be delivered more rapidly than an observatories annual definitive data. This initiative will be useful for a number of scientific activities, but with the Swarm satellite mission near its start date, the data will be particularly useful in providing ground truth data for the satellite’s magnetometers.

Quasi-definitive data are (H, D, Z) or (X, Y, Z) 1-minute data:

  • Corrected using temporary baselines
  • Made available less than 3 months after their acquisition
  • Such that the difference between the quasi-definitive and definitive (X, Y, Z) monthly means is less than 5 nT for every month of the year

Point c is checked a posteriori by comparing quasi-definitive and definitive data from the previous year.

How do I create quasi-definitive data?

Provided you can keep to the timeliness and accuracy definitions for Quasi-Definitive (QD) data, you are free to create QD data using any method. Two institutes have been creating quasi-definitive data for a number of years (British Geological Survey, BGS and Institut de Physique du Globe de Paris, IPGP) and the methods they use are described in the FAQ as examples of methods that you might adopt in your own institute.

How do I send my observatory data to INTERMAGNET in near real-time?

The best way to do this is to use the IAGA-2002 data format (www.ngdc.noaa.gov/IAGA/vdat/iagaformat.html) which allows small fragments of a day file to be transmitted to the INTERMAGNET web service. Using this approach, some observatories are sending data as frequently as once every five minutes, allowing it to be available for distribution from the INTERMAGNET web site within twenty minutes of recording.

What is the web service for delivering data to INTERMAGNET?

The Paris and Edinburgh GIN both offer a web service for sending data to INTERMAGNET. The web service can be used interactively, but it is more likely that you will want to automate the delivery of data (possibly integrating the transmission to INTERMAGNET into your existing data processing). The Edinburgh GIN web service is documented here (account required): http://app.geomag.bgs.ac.uk/GINFileUpload/ - for access contact the Edinburgh GIN manager (e_ginman@bgs.ac.uk). For access and information on the Paris GIN web service, contact the Paris GIN manager (vmaury@ipgp.fr).

What is the BGS method for creating Quasi-Definitive Data?

Note: We can use this method because we have great faith in the stability of the variometer instruments. It would be much more difficult if the baselines were trending more rapidly. As a rough rule of thumb use the following procedure on a year’s worth of historic data to check whether this method is suitable:

  • Calculate an extrapolated baseline for a period of time where you already have a definitive baseline, starting the ‘extrapolation’ at the time of the first absolute observation. Record the difference between the extrapolated baseline and the true (definitive) baseline at the time of the next absolute observation.
  • Repeat this calculation for a number of absolute observations throughout the year

If these differences are less than 2.5nT, then your baselines should be stable enough to use this method.

The BGS system produces QD data on a next day basis. The variometer data (both one-minute and one-second) and the daily baseline values are stored in separate files. Data from these files are combined to produce baseline corrected data. This allows changes to the baseline to be easily updated.

There are two parts to the production of quasi-definitive data, which incorporates both manual and automated operations.

  1. Daily data processing and quality control procedures for variometer data
    • During the day and on a next day basis plots are viewed by the duty processor. These are magnetograms, F difference (or closing error) plots, individual component comparison plots (where an observatory has more than one system installed, such as in the UK). These plots are automatically produced or can be regenerated manually.
    • Any errors in the variometer data are either removed, or, in the case of the UK where backup systems are running, replaced with an appropriate set of data. For real time products, the data processing software will carry out the latter automatically following changes to configuration files.
    • Every working day morning any required adjustments to the variometer data for the previous day (or three days following the weekend) is completed.
  2. Processing of absolute observation measurements and production of baseline values
    • Absolute observations are recorded and processed using a Java program, which will also read in the variometer data and output the spot baseline values (that is the absolute measurement minus the variometer reading at the time of the measurement). This is carried out as soon as possible after absolute measurements are made.
    • A separate FORTRAN plotting program is used to view the spot baselines against the current continuous daily baseline values. The plot includes the daily mean F difference or closing error as well as the daily mean temperature in the variometer chamber. When new observations are available this plot is assessed to make sure the current baseline is still well within the QD definition.
    • Once a month, or sooner if required (see above), the recent spot baseline values and other absolute observation data are analysed using an Excel spreadsheet. The continuous baselines are fitted using a series of piecewise polynomial fits, which are subjectively chosen following the removal of outliers. Outliers are removed based on a set of rules, which takes into account the collimation errors as well as any other obvious factors.
    • Adjustments are made to the piecewise polynomials as required and the baseline updated. All previous versions of the baseline file are saved using a simple version control system.
    • Daily baseline values are created for the complete year. i.e. including a projection into the future based on the baselines on the day of computation. Account can be taken of any current trend (if any) however where this is the case it is even more important to repeat the process with new observations as soon as they are available, thus reducing the number of days of predicted baseline values.

Combining the two parts enables:

  • Real time and next day processing to be carried out automatically. The processes read in the most up-to-date baselines (part 2) and the corrected one-minute (or one-second) variometer data (part 1) to produce real time and next day data that should meet the QD definition that has been established by INTERMAGNET.
  • The QD data to be prepared and delivered to INTERMAGNET manually on a next (working) day basis following the completion of all necessary checks as described.

What is the IPGP method for creating Quasi-Definitive Data?

Quasi-definitive data are calculated every month, shortly after the end of the month. The processing is very similar to that for producing definitive data, except that the time interval for the baseline calculation is shorter (a few months instead of one year or more) and the data preprocessing (despiking, etc.) is less thorough.

The IPGP method can be applied even if baselines have irregular variations provided they meet the INTERMAGNET standards (Peltier and Chulliat, 2010).

The processing is made of the following three steps, repeated after each month M:

  1. Variation data of month M are pre-processed in order to remove errors and spikes, correct for jumps (if any) and replace gaps by data from a back-up magnetometer (if available). This step is usually completed within a few working days after the end of month M, although for some observatories it can take longer.
  2. A temporary baseline is calculated from 1st December of the previous year to the last day of month M, for each component of the field. Absolute data are visually inspected and outliers are removed prior to the baseline calculation. If needed, absolute data prior to 1st December of the previous year are included in order to extend the time interval of the temporary baseline. (This happens if, for example, M is at the beginning of the year and/or too few absolute data are available at the beginning of the interval.) IPGP baselines are calculated using cubic smoothing splines with a constant smoothing parameter (csaps function of the MatlabTM software, which relies on an algorithm described in de Boor, 1978).
  3. Baseline-corrected 1s and 1min data are calculated from 1st January of the current year to the last day of month M. A final quality control is performed at this step, by visual inspection of the time series of components and scalar differences. Once validated, the baseline-corrected data for month M are reformatted within the IAGA quasi-definitive data format and delivered to INTERMAGNET.

References:

  • Peltier, A. and A. Chulliat, On the feasibility of promptly producing quasi-definitive magnetic observatory data, Earth Planets Space, 62(2), e5-e8, doi:10.5047/eps.2010.02.002, 2010.
  • De Boor, C., A Practical Guide to Splines, Springer-Verlag, New York, 1978.