Voir la source de User Manual 4.1 Data management system


== Introduction ==
=== Scope ===
The scope of this chapter is to present data management. This section presents the three modules of the data management system originally provided by Orekit :

* how does the data management system work?
* how to set it up?
* how to use it?
* how to add data to what already exists?
* what are the pros and cons of this system?

=== Javadoc ===
The data objects are available in the package <code>fr.cnes.sirius.patrius.data</code>.

{| class="wikitable"
|-
! scope="col"| Library
! scope="col"| Javadoc
|-
| Orekit
|[{{JavaDoc4.1}}/fr/cnes/sirius/patrius/data/package-summary.html Package fr.cnes.sirius.patrius.data]
|}

=== Links ===
{{specialInclusion prefix=$theme_sub section="Links"/}}

=== Useful Documents ===
{{specialInclusion prefix=$theme_sub section="UsefulDocs"/}}

=== Package Overview ===
The data loading process is organized through three main objects.

The <code>DataProvider</code> classes handle data sources. Each one of them has a particular type of source it can browse. The <code>DirectoryCrawler</code> performs a bottom-first search in a directory tree. The <code>ZipJarCrawler</code> works alike, but inside a compressed file. The <code>ClassPathCrawler</code> handles a list of data files and/or compressed files that are in the classpath (it can not search recursively like the <code>DirectoryCrawler</code> though). Finally, the <code>NetworkCrawler</code> works like the <code>ClassPathCrawler</code>, although in its case, it has a list of URLs instead of files. There is no limit to the number of DataProviders a program can use at once.

The Providers are listed and put to work through the <code>DataProvidersManager</code> singleton. This is the single point of access to the data management system. It contains a list of Providers that are queried every time a user needs data.

The various crawlers provide streams to the <code>DataLoader</code>. From these streams, the DataLoaders can reconstruct data that was stored in files (either compressed or not), even if some files come from different sources. These streams effectively separate the machine world from the program world, because they hide the former to the latter. Therefore, parsing data from a new format only means creating a loader, and being able to read another kind of file means creating a <code>DataProvider</code>. Note that the DataLoaders usually serve as a facade for the higher layers of the program.

== Features Description ==

=== Data providers ===

==== Default provider ====
The data management system can use a system-wide property, <code>orekit.data.path</code>, as an entry point for default data. This default data must be file-based (either a file system entry point or a java resource) and either a directory or a zip/jar file.
Setting a default provider is not mandatory, and must be done explicitly by :

* setting a value to <code>orekit.data.path</code>,
* calling <code>addDefaultProviders</code> on the data provider manager.

==== Adding a provider ====
{{specialInclusion prefix=$theme_sub section="Provider"/}}

==== Using the data management system ====
The data management system main operation is through the <code>feed</code> method. This method takes a <code>DataLoader</code>, and a regexp string matching the name of files the <code>DataLoader</code> is able to process. In this method call :

* the <code>DataProviders</code> list is traversed in the priority order.
* the first <code>DataProvider</code> providing a file matching the regexp is the one (and only) used to feed the <code>DataLoader</code>.

==== Adding new data ====
{{specialInclusion prefix=$theme_sub section="Adding"/}}

=== File formats ===
Patrius can read a variety of files:
* '''Static potential files'''
These files contains static potential coefficients up to a certain order and degree and are used to compute Earth (or any other body) static potential perturbation.

* '''Variable potential files'''
Theses files contains variable potential coefficients up to a certain order and degree and are used to compute Earth (or any other body) variable potential perturbation.

* '''Geomagnetic coefficients files'''
These files contains geomagnetic coefficients and are used to compute Earth (or any other body) geomagnetic field.

* '''Ionospheric coefficients files'''
These files contains ionospheric data and are used to compute Earth ionospheric correction.

* '''Solar activity files'''
These files contains solar flux and geomagnetic coefficients are used to get solar activity in order to compute drag perturbation.

* '''Earth Orientation Parameters files'''
These files contains earth orientation parameters (polar motion, LOD, etc.) used in frames transformation.

* '''TAI-UTC shift files'''
These files contains TAI-TUC shift.

* '''Orbital data files'''
These files are used to store orbital ephemeris

* '''Third body ephemeris files'''
These files are used to store third-body orbital ephemeris (Sun, Moon, etc.)

The following sections describe all file readable in PATRIUS. When the file format is not described anywhere, a short description is detailed below the tab.

==== Static Potential ====

Static potential coefficients files contains potential coefficients up to a certain order and degree.<br>
Warning: at very high order and degree (> 100), some numerical quality issues can appear and results may be degraded.

{| class="wikitable"
|-
! scope="col"| Data type
! scope="col"| Data provider
! scope="col"| Format
! scope="col"| Reader
|-
|'''GRGS'''
|Not direct link. This link provides GRGC potential coefficients [http://grgs.obs-mip.fr/grace/variable-models-grace-lageos/interactive-tools/Check-individual-solutions]
|[http://grgs.obs-mip.fr/grace/variable-models-grace-lageos/formats]
|[{{JavaDoc4.1}}/fr/cnes/sirius/patrius/forces/gravity/potential/GRGSFormatReader.html GRGSFormatReader]
|-
|'''EGM'''
|[http://earth-info.nga.mil/GandG/wgs84/gravitymod/egm2008/first_release.html]
|See below
|[{{JavaDoc4.1}}/fr/cnes/sirius/patrius/forces/gravity/potential/EGMFormatReader.html EGMFormatReader]
|-
|'''ICGEM'''
|[http://icgem.gfz-potsdam.de/tom_longtime]
|[http://icgem.gfz-potsdam.de/ICGEM-Format-2011.pdf]
|[{{JavaDoc4.1}}/fr/cnes/sirius/patrius/forces/gravity/potential/ICGEMFormatReader.html ICGEMFormatReader]
|-
|'''SHM'''
|Not provided (not used any more, replaced by ICGEM format)
|See below
|[{{JavaDoc4.1}}/fr/cnes/sirius/patrius/forces/gravity/potential/SHMFormatReader.html SHMFormatReader]
|}

'''EGM file format''' (''Column text file'')

Column index:<br>
1:Degree of coefficients<br>
2:Order of coefficients<br>
3:Tesseral-sectorial cosinus coefficient<br>
4:Tesseral-sectorial sinus coefficient<br>
5:Sigma applied to the cosinus<br>
6:Sigma applied to the sinus

[[File:EGM.png|center]]

'''SHM file format''' (''Column text file'')

Column index:<br>
2:Degree of coefficients<br>
3:Order of coefficients<br>
4:Tesseral-sectorial cosinus coefficient<br>
5:Tesseral-sectorial sinus coefficient<br>


[[File:SHM.png|center]]

==== Variable Potential ====

Variable potential coefficients files contains potential coefficients up to a certain order and degree.<br>
Warning: at very high order and degree (> 100), some numerical quality issues can appear and results may be degraded.

FES2004 is used to model oceanic tides.

{| class="wikitable"
|-
! scope="col"| Data type
! scope="col"| Data provider
! scope="col"| Format
! scope="col"| Reader
|-
|'''GRGSRL02'''
|[http://grgs.obs-mip.fr/grace/variable-models-grace-lageos/interactive-tools/Check-individual-solutions]
|[http://grgs.obs-mip.fr/grace/variable-models-grace-lageos/formats]
|[{{JavaDoc4.1}}/fr/cnes/sirius/patrius/forces/gravity/variations/coefficients/GRGSRL02FormatReader.html GRGSRL02FormatReader]
|-
|'''FES2004'''
|[https://www.aviso.altimetry.fr/en/data/products/auxiliary-products/global-tide-fes.html]
|See below
|[{{JavaDoc4.1}}/fr/cnes/sirius/patrius/forces/gravity/tides/coefficients/FES2004FormatReader.html FES2004FormatReader]
|}

'''FES2004 file format''' (''Column text file'')

Column index:<br>
1:Doodson number<br>
2:Darwin number<br>
3:Degree of coefficients <br>
4:Order of coefficients<br>
5:Sinus coefficient positif<br>
6:Cosinus coefficient positif<br>
7:Sinus coefficient negatif<br>
8:Cosinus coefficient negatif<br>
9:Positif coefficient<br>
10:Epsilon positif<br>
11:Negatif coefficient<br>
12:Negatif epsilon

[[File:FES2004.png|center]]

==== Geomagnetic coefficients ====

Geomagnetic coefficients fiels contains geomagnetic coefficients up to a certain order and degree.

{| class="wikitable"
|-
! scope="col"| Data type
! scope="col"| Data provider
! scope="col"| Format
! scope="col"| Reader
|-
|'''COF'''
|IGCRF data: [https://www.ngdc.noaa.gov/IAGA/vmod/igrf.html] WMM data: [https://www.ngdc.noaa.gov/geomag/WMM/soft.shtml]
|See below 
|[{{JavaDoc4.1}}/fr/cnes/sirius/patrius/models/earth/COFFileFormatReader.html COFFileFormatReader]
|}

'''COF file format''' (''Column text file'')

Column index:<br>
1:Degree of coefficients <br>
2:Order of coefficients<br>
3:g coefficient at position n,m<br>
4:h coefficient at position n,m<br>
5:dg coefficient at position n,m<br>
6:dh coefficient at position n,m<br>
g and h are the Gauss coefficients of main geomagnetic model (nT)<br>
dg and dh are the Gauss coefficients of secular geomagnetic model (nT/years)

[[File:COF.png|center]]

==== Ionospheric coefficients ====

Ionospheric coefficients files contains ionospheric coefficient to model the state of the ionosphere.

{| class="wikitable"
|-
! scope="col"| Data type
! scope="col"| Data provider
! scope="col"| Format
! scope="col"| Reader
|-
|'''CCIR12'''
|N/A
|See below
|[{{JavaDoc4.1}}/fr/cnes/sirius/patrius/signalpropagation/iono/R12Loader.html R12Loader]
|-
|'''USK(NEWUSK)'''
|N/A
|See below
|[{{JavaDoc4.1}}/fr/cnes/sirius/patrius/signalpropagation/iono/USKLoader.html USKLoader]
|}

'''CCIR12 file format''' (''Column text file'')

Column index:<br>
1:Year <br>
2:Month<br>
3:Unused<br>
4:Midval<br>
5:Unused<br>
6:Midday<br>


[[File:CCIR12.png]]

'''USK file format''' (''Line text file'')

[[File:NEWUSK.png|center]]

==== Solar activity ====

Solar activity contains solar flux and geomagnetic coefficients for a given timespan. ACSOL and NOAA files usually provide data for several years.

{| class="wikitable"
|-
! scope="col"| Data type
! scope="col"| Data provider
! scope="col"| Format
! scope="col"| Reader
|-
|'''ACSOL'''
|N/A
|See below
|[{{JavaDoc4.1}}/fr/cnes/sirius/patrius/forces/atmospheres/solarActivity/ACSOLFormatReader.html ACSOLFormatReader]
|-
|'''NOAA'''
|N/A
|See below
|[{{JavaDoc4.1}}/fr/cnes/sirius/patrius/forces/atmospheres/solarActivity/NOAAFormatReader.html NOAAFormatReader]
|}

'''ACSOL file format''' (''Column text file'')

Column index:<br>
1:Julian day since 1950<br>
2:Seconds of the day (UTC)<br>
3:Flux at JD + seconds<br>
4-11 are the three-hours AP of the intervals 

[[File:ACSOL.png|center]]

'''NOAA file format''' (''Column text file'')

Column index:<br>
1:Day<br>
2:Flux at the day<br>
3:The background flux <br>
4-11:the three-hours AP of the intervals <br>
12:Year<br>
13:Unused character flag

==== Pole data ====

These files contains earth orientation parameters (polar motion, LOD, etc.) used in frames transformation.<br>
These data are usually valid of a timespan of several days/months, although there is no theoretical limit.


{| class="wikitable"
|-
! scope="col"| Data type
! scope="col"| Data provider
! scope="col"| Format
! scope="col"| Reader
|-
|'''Bulletin B'''
|[ftp://hpiers.obspm.fr/iers/bul/bulb_new]
|[ftp://hpiers.obspm.fr/iers/bul/bulb_new/bulletinb.pdf]
|[{{JavaDoc4.1}}/fr/cnes/sirius/patrius/frames/configuration/eop/BulletinBFilesLoader.html BulletinBFilesLoader]
|-
|'''EOP 05 C04'''
|[ftp://hpiers.obspm.fr/iers/eop/eopc04_05]
|See below
|[{{JavaDoc4.1}}/fr/cnes/sirius/patrius/frames/configuration/eop/EOP05C04FilesLoader.html EOP05C04FilesLoader]
|-
|'''EOP 08 C04'''
|[https://datacenter.iers.org/web/guest/eop/-/somos/5Rgv/version/212]
|See below
|[{{JavaDoc4.1}}/fr/cnes/sirius/patrius/frames/configuration/eop/EOP08C04FilesLoader.html EOP08C04FilesLoader]
|-
|'''Datas "Rapid and Prediction" (TXT)'''
|[http://maia.usno.navy.mil/ser7]
|[http://maia.usno.navy.mil/ser7/readme.finals]
|[{{JavaDoc4.1}}/fr/cnes/sirius/patrius/frames/configuration/eop/RapidDataAndPredictionColumnsLoader.html RapidDataAndPredictionColumnsLoader]
|-
|'''Datas "Rapid and Prediction" (XML)'''
|[https://datacenter.iers.org/eop/-/somos/5Rgv/eop?]
|Same datas than the column file but write in the xml format
|[{{JavaDoc4.1}}/fr/cnes/sirius/patrius/frames/configuration/eop/RapidDataAndPredictionXMLLoader.html RapidDataAndPredictionXMLLoader]
|}

'''EOP 05 C04 and EOP 08 C04 files format''' (''Column text file'')

Column index:<br>
1:Date at 0h(UTC)<br>
2:Modified Julian Day<br>
3:x(")<br>
4:y(")<br>
5:UT1-UTC(s)<br>
6:LOD(s)<br>
7:DPsi(")<br>
8:DEps(")<br>
9:x error (")<br>
10:y error (")<br>
11:UT1-UTC error (s)<br>
12:LOD error(s)<br>
13:Dpsi error(")<br>
14:DEpsilon error(")<br>
x and y are the coordinales of the Celestial Ephemeris Pole.<br>
UT1: Universal time, the time of the earth clock<br>
LOD: Length Of Day is the excess of revolution time

[[File:EOP08C04.png|center]]

==== TAI-UTC shift====

These files contains TAI-TUC shift. Official data contains shift since the beginning of space era (1961).

{| class="wikitable"
|-
! scope="col"| Data type
! scope="col"| Data provider
! scope="col"| Format
! scope="col"| Reader
|-
|'''Gap TAI-UTC'''
|[ftp://hpiers.obspm.fr/iers/bul/bulc/UTC-TAI.history]
|See below
|[{{JavaDoc4.1}}/fr/cnes/sirius/patrius/time/UTCTAIHistoryFilesLoader.html UTCTAIHistoryFilesLoader]
|}


'''TAI-UTC file format''' (''Column text file'')

Column index:<br>
1:Begining date-Ending date<br>
2:Difference TAI-UTC between the begining date and the ending date

[[File:TAI-TUC.png|center]]

==== Orbital data ====

These files are used to store orbital ephemeris.<br>
TLE provides orbit at one date. Ephemeris is then retrieved thanks to SGP4/SDP4 propagation model.<br>
SP3 files provide orbit over an extended time span (ex: 1 day).

{| class="wikitable"
|-
! scope="col"| Data type
! scope="col"| Data provider
! scope="col"| Format
! scope="col"| Reader
|-
|'''TLE'''
|[https://www.projectpluto.com/pluto/mpecs/pseudo.htm]
|[https://www.mmto.org/obscats/tle.html]
[{{JavaDoc4.1}}/fr/cnes/sirius/patrius/propagation/analytical/tle/TLESeries.html TLESeries]
|-
|'''SP3'''
|GPS and GLONASS only [ftp://cddis.gsfc.nasa.gov/gnss/products]
|[ftp://igs.org/pub/data/format/sp3_docu.txt]
|[{{JavaDoc4.1}}/fr/cnes/sirius/patrius/files/sp3/SP3File.html SP3File]
|}

Note: * SP3 files can be written by anyone. They are however mainly used ton provide GPS and GLONASS ephemeris.

==== Third body ephemeris ====

Third body ephemeris files contains ephemeris for third bodies (Moon, Sun, Jupiter) over an extended time span (ex: 1 or several days).

{| class="wikitable"
|-
! scope="col"| Data type
! scope="col"| Data provider
! scope="col"| Format
! scope="col"| Reader
|-
|'''JPL'''
|[https://ssd.jpl.nasa.gov/pub/eph/planets/SunOS]
|[https://ssd.jpl.nasa.gov/pub/eph/planets/Linux]
|[{{JavaDoc4.1}}/fr/cnes/sirius/patrius/bodies/JPLEphemeridesLoader.html JPLEphemeridesLoader]
|}

== Getting Started ==
{{specialInclusion prefix=$theme_sub section="GettingStarted"/}}

== Contents ==
=== Interfaces ===
The data package includes the following interfaces :

'''Data'''

{| class="wikitable"
|-
! scope="col"| Class
! scope="col"| Summary
! scope="col"| Javadoc
|-
|'''DataLoader'''
|Interface for loading data files from DataProvider data providers.
|[{{JavaDoc4.1}}/fr/cnes/sirius/patrius/data/DataLoader.html ...]
|-
|'''DataProvider'''
|Interface for providing data files to DataLoader file loaders.
|[{{JavaDoc4.1}}/fr/cnes/sirius/patrius/data/DataProvider.html ...]
|}

=== Classes ===
The data package includes the following classes :

'''Data'''

{| class="wikitable"
|-
! scope="col"| Class
! scope="col"| Summary
! scope="col"| Javadoc
|-
|'''DataProvidersManager'''
|This class is the single point of access for all data loading features.
|[{{JavaDoc4.1}}/fr/cnes/sirius/patrius/data/DataProvidersManager.html ...]
|-
|'''DirectoryCrawler'''
|This class handles data files recursively starting from a root directories.
|[{{JavaDoc4.1}}/fr/cnes/sirius/patrius/data/DirectoryCrawler.html ...]
|-
|'''NetworkCrawler'''
|This class handles a list of URLs pointing to data files or zip/jar on the net.
|[{{JavaDoc4.1}}/fr/cnes/sirius/patrius/data/NetworkCrawler.html ...]
|-
|'''ZipJarCrawler'''
|This class browses all entries in a zip/jar archive in filesystem or in classpath.
|[{{JavaDoc4.1}}/fr/cnes/sirius/patrius/data/ZipJarCrawler.html ...]
|}

== [[File:lightBulb.png]] Tips &amp; Tricks ==
=== Strengths ===

* Lightweight implementation. The providers never load data, they merely provide streams on demand to the loaders.
* Scalable for using data from several heterogeneous sources.
* Scalable for new data types : the user only needs to create a new <code>DataLoader</code> implementation to use a new data type in this system.

=== Weaknesses ===

* The user must be aware the data loading overhead happens any time a <code>DataLoader</code> is fed, so the user must manage its loaders so that they are fed only once.
* Several sources for the same type of data cannot be used, since only the last provider added is used to feed data to a loader - unless the user manages the providers list accordingly, knowing one can only add elements or reset the whole list.
* The regexp is the only way to match a data file and a <code>DataLoader</code>.
* As of today, the data management system is a thread-hostile singleton : a multithreaded application shares the same providers list for all threads, and it may deadlock on a concurrent access!