Coastal Data Information Program (CDIP)

Background Since its inception in November 1975, the Coastal Data Information Program (CDIP) collects near real-time physical environmental data mostly in the coastal US and South Paci ﬁ c, with a focus on waves. CDIP has many partners including industry, federal and state agencies and academia. In all cases, these data are transmitted from the station location to CDIP at the Scripps Institution of Oceanography (SIO), La Jolla, CA where the data are processed and disseminated to interested parties.


1.
The majority of our buoy data are transmitted via iridium. The path is shown in the following link which depicts the offshore buoy transmitting the data to iridium satellite, then to the Department of Defense iridium gateway in Honolulu and back to SIO or Amazon Cloud as appropriate.
For a select number of pier or near-shore stations the data are transmitted via network to CDIP.
An internal compact flash card stores the data, available upon recovery.
Describe how data are managed (IOOS Certification,f 2.) The data are managed at the SIO/CDIP server. Once ingested, CDIP processes and quality controls these data. The data are stored on disk in ASCII, NetCDF, and SQL formats. Back-up occurs hourly locally, daily offsite at the UCSD Supercomputer Center and biannually to Amazon Cloud.
Describe the data quality control procedures that have been applied to the data. (IOOS Certification,f 3.) A sophisticated suite of automated and human quality control procedures are developed, as defined in the QARTOD manual (http://www.ioos.noaa.gov/qartod/waves/welcome.html). In addition, CDIP has also developed further instrument and site specific tests. The tests are summarized in the following When there are critical errors involving a buoy offsite or a station that has not updated within 3 hours, the software team is not only notified via email but, a designated watch person is also paged.
Only those data that pass all the QC tests are transmitted to the National Data Buoy Center (NDBC) & the National Weather Service (NWS).
The above quality control procedure can be monitored at: http://cdip.ucsd.edu/diag If you will be using existing data, state that fact and include where you got it.What is the relationship between the data you are collecting and the existing data?

Expected schedule for data sharing
Adheres to the NOAA Data Sharing Procedural Directive. The System is an operational system; therefore the RICE should strive to provide as much data as possible, in real-time or near real-time, to support the operation of the System. (IOOS Certification,f. 4.) Once data have been acquired, processed, and quality controlled, CDIP makes the complete data set available. (Near-real time, approximately 3 minutes after the data are transmitted) How long will the original data collector/creator/principal investigator retain the right to use the data before opening it up to wider use?

N/A
How long do you expect to keep the data private before making it available? Explain if different data products will become available on different schedules (Ex: raw data vs processed data, observations vs models, etc.)

N/A Coastal Data Information Program (CDIP)
Explain details of any embargo periods for political/commercial/patent reasons? When will you make the data available?

Standards for format and content
Which file formats will you use for your data, and why?
How can the information be accessed? (IOOS Certification, f 1. ii) CDIP Shares data in a variety of file formats.
1. FM 65 XML -Used for the real-time data push to the NDBC. FM 65 format is described here http://www.ndbc.noaa.gov/decode.shtml.

2.
NetCDF -A self-describing, machine-independent data format that support the creation, access, and sharing of array-oriented scientific data, available from the CDIP site http://thredds.cdip.ucsd.edu. 3.
ASCII -Text file that are easily read and parsed by people and programs via the web, available from the CDIP site, e.g., http://cdip.ucsd.edu/?nav=recent What file formats will be used for data sharing?
All of the above.
What metadata/ documentation will be submitted alongside the data or created on deposit/ transformation in order to make the data reusable?
All of CDIP's data sets are described by detailed metadata, which is continuously updated and available online in a number of formats. FGDC-compliant metadata are included, in both HTML and XML formats. The metadata for any specific data set are accessible from the station pages in the historic section of the website. In addition to the standard web pages, static XML metadata files are available for download or harvesting from a webaccessible folder (http://cdip.ucsd.edu/data_access/metadata Many of the fields in the content standard are defined as free text, and can contain links to other resources. CDIP's metadata takes full advantage of this fact, linking to relevant documents and pages on the CDIP website wherever possible. This is the most efficient and effective approach because CDIP's online documentation is extensive and covers most of the topics addressed in the FGDC standard. By linking directly to CDIP's web resources redundancy is avoided and the metadata are ensured to be up-to-date. This same approach is used in defining CDIP's entity and attribute information.

Policies for stewardship and preservation
What is the long-term strategy for maintaining, curating and archiving the data?

Points of contact-Individuals responsible for the data management and coordination across the region (CV's attached); (IOOS Certification f 1. i)
Julie

What procedures does your intended long-term data storage facility have in place for preservation and backup?
Local redundant HDD storage at the CDIP Lab, the UCSD Supercomputer center, Amazon Glacier and NCEI.

How long will/should data be kept beyond the life of the project?
Data are indefinitely stored.

What data will be preserved for the long-term?
All data are publicly available and preserved.
What transformations will be necessary to prepare data for preservation / data sharing?
Raw data are decoded and formatted, analyzed and quality controlled.
What metadata/ documentation will be submitted alongside the data or created on deposit/ transformation in order to make the data reusable?
FGDC standard metadata are available per deposit and transformation. NetCDF files have complete metadata and quality control flags.

What related information will be deposited?
Time series and spectral files. Data are publicly available.

Will a data sharing agreement be required?
In general, a data sharing agreement will not be required. However, data should be properly acknowledged.
The one exception is with NOAA Physical Ocean Real Time System (PORTS). A Memorandum of Understanding (MOU) between NOAA PORTS and the US Army Corps, representing CDIP as the funding agency, is signed.
Are there ethical and privacy issues? If so, how will these be resolved?

N/A
Who will hold the intellectual property rights to the data and how might this affect data access?
The funding agency & the University of California, San Diego through a contractual agreement.