Corresponding authors: Jennifer McWhorter (
Academic editor:
The program captures wave, wind, and temperature data in real-time, updating every 30 minutes.
The data are collected by several redundant pathways.
The majority of our buoy data are transmitted via iridium. The path is shown in the following link which depicts the offshore buoy transmitting the data to iridium satellite, then to the Department of Defense iridium gateway in Honolulu and back to SIO or Amazon Cloud as appropriate. (
For a select number of pier or near-shore stations the data are transmitted via network to CDIP. (
An internal compact flash card stores the data, available upon recovery.
The data are managed at the SIO/CDIP server. Once ingested, CDIP processes and quality controls these data. The data are stored on disk in ASCII, NetCDF, and SQL formats. Back-up occurs hourly locally, daily offsite at the UCSD Supercomputer Center and biannually to Amazon Cloud.
A sophisticated suite of automated and human quality control procedures are developed, as defined in the QARTOD manual (
All errors causing an exception are handled by the following:
logged in a daily errors file
error exception emailed to the CDIP software team
categorized by error type and station at the end of each month to provide an error summary table.
flagged and annotated in the NetCDF file as appropriate
When there are critical errors involving a buoy offsite or a station that has not updated within 3 hours, the software team is not only notified via email but, a designated watch person is also paged.
Only those data that pass all the QC tests are transmitted to the National Data Buoy Center (NDBC) & the National Weather Service (NWS).
The above quality control procedure can be monitored at:
N/A
Once data have been acquired, processed, and quality controlled, CDIP makes the complete data set available. (Near-real time, approximately 3 minutes after the data are transmitted)
N/A
N/A
N/A
CDIP Shares data in a variety of file formats.
FM 65 XML - Used for the real-time data push to the NDBC. FM 65 format is described here
NetCDF - A self-describing, machine-independent data format that support the creation, access, and sharing of array-oriented scientific data, available from the CDIP site
ASCII - Text file that are easily read and parsed by people and programs via the web, available from the CDIP site, e.g.,
All of the above.
All of CDIP's data sets are described by detailed metadata, which is continuously updated and available online in a number of formats. FGDC-compliant metadata are included, in both HTML and XML formats. The metadata for any specific data set are accessible from the station pages in the historic section of the website. In addition to the standard web pages, static XML metadata files are available for download or harvesting from a web-accessible folder (
FGDC metadata consists of seven main sections, five of which do not need to be included if they do not apply to the data set in question. For CDIP metadata, two sections are omitted Spatial_Reference_Information and Spatial_Data_Organization_Information - because they only apply to datasets that include spatial data. (Although CDIP's metadata contains spatial info - deployment positions - the data sets themselves do not.)
Thus CDIP metadata consists of five sections:
Identification_Information
Data_Quality_Information
Entity_and_Attribute_Information
Distribution_Information
Metadata_Reference_Information
Many of the fields in the content standard are defined as free text, and can contain links to other resources. CDIP's metadata takes full advantage of this fact, linking to relevant documents and pages on the CDIP website wherever possible. This is the most efficient and effective approach because CDIP's online documentation is extensive and covers most of the topics addressed in the FGDC standard. By linking directly to CDIP's web resources redundancy is avoided and the metadata are ensured to be up-to-date. This same approach is used in defining CDIP's entity and attribute information.
CDIP's FGDC metadata is generated by querying our 'archive' MySQL database and passed through the US Geological Service’s utility 'mp':
CDIP’s NetCDF files have ISO 19115 compliment metadata which are generated with custom FORTRAN scripts.
CDIP’s data sets are described by detailed metadata in a number of formats:
FGDC XML -
ISO 19115 XML - NetCDF
HTML - CDIP website has extensive documentation
FGDC and ISO 19115 metadata are both accepted standards and mandated by the US Federal Government.
Julie Thomas - Employee 39 years, Principal Investigator/Program Manager
Darren Wright - Employee 10 years, Programmer/Analyst
Jen McWhorter - Employee 1 year, Administrative Analyst
The University of California has a process in place for personnel evaluation. These evaluations are on file with UC San Diego Human Resources. All personnel listed have received excellent evaluations.
National Centers for Environmental Information (NCEI) is the federal archive repository. Historic data from CDIP stations are archived monthly and available at NCEI (
Local redundant HDD storage at the CDIP Lab, the UCSD Supercomputer center, Amazon Glacier and NCEI.
Data are indefinitely stored.
All data are publicly available and preserved.
Raw data are decoded and formatted, analyzed and quality controlled.
FGDC standard metadata are available per deposit and transformation. NetCDF files have complete metadata and quality control flags.
Time series and spectral files.
CDIP Access to Data (http://cdip.ucsd.edu/?nav=documents&xitem=product#access)
THREDDS data are organized into Archived and Realtime folders:a.Archived - contains individual folders for all CDIP stations, both active and decommissioned. Each station’s individual Archived folder contains NetCDF files for each separate deployment (e.g. ‘d17.nc’) and an aggregate file (‘historic.nc’) of the full time-span of data for a buoy.b.Realtime - contains single NetCDF files (‘rt.nc’) for CDIP stations that are currently active and transmitting data.i.OPENDAP - provides URL that can be used in Python/Matlab to automatically grab NetCDF file of data from server. Also provides option to download user-specified variables/timeperiods as ASCII or Binary file.ii.HTTPServer - option to download the whole NetCDF file.iii.NCML (NetCDF Markup Language) - XML document used to define a CDM dataset, and to allow user to add/delete/change metadata and variables, or combine data from multiple CDM files.iv.ISO - XML metadata record for each station.v.UDDC (Unidata Data Discovery Convention) - tool to determine how well file metadata conforms to list of recommended metadata attributes.vi.SOS - web service interface which allows querying observations, sensor metadata, and representations of observed features. Defines means to register/remove sensors and insert new sensor observations.NetCDF files for Archived and Realtime data contain identical buoy parameters and variables, with the exception that the ‘historic.nc’ Archived file and the ‘rt.nc’ Realtime file do not contain Directional Displacement (xyz) data.
CDIP Data Access Routine (DAR)
CDIP Website
CDIP FTP
National Data Buoy Center (NDBC) for distribution on their website and dissemination via the Global Telecommunications Service (GTS).
Several federal, state and private companies access CDIP data for distribution using one of the access methods above.
Several federal, state and private companies access CDIP data for distribution using one of the access methods above
CDIP data and products are freely available for public use. When referenced, please provide a link to the CDIP homepage.
Examples:
Standard html:Data courtesy of <a href=http://cdip.ucsd.edu/>CDIP</a>
Offline references, choose the appropriate form from the recommended acknowledgements below.
Short form (figure captions, etc.) "... data from CDIP, Scripps Institution of Oceanography."
Longer form (in text) "...data were furnished by the Coastal Data Information Program, Integrative Oceanography Division, operated by the Scripps Institution of Oceanography."
Full form (acknowledgements at conclusion of papers, etc.) "...data were furnished by the Coastal Data Information Program (CDIP), Integrative Oceanography Division, operated by the Scripps Institution of Oceanography, under the sponsorship of the U.S. Army Corps of Engineers and the California Department of Parks and Recreation."
Data are publicly available.
In general, a data sharing agreement will not be required. However, data should be properly acknowledged.
The one exception is with NOAA Physical Ocean Real Time System (PORTS). A Memorandum of Understanding (MOU) between NOAA PORTS and the US Army Corps, representing CDIP as the funding agency, is signed.
N/A
The funding agency & the University of California, San Diego through a contractual agreement.
Articles: