Attending the CoSA annual meeting over the past 12 years has provided insight on the value of the Archives throughout the Country and globally. Preserving our past informs our present and future.
SSD manufacturers need to fix this with a fully integrated solution on chip. Make it so as long as the drive is powered the data integrity is ensured. Yes only a third of the usable capacity available but data is safe. I think that this issue is not advertised on the retail packaging is misleading and creating misery. Refresh needs to be done every 3 months.
RAIN seems very interesting, is RAIN available outside of HPE, NetApp and dell EMC 520 formatted disks as I haven't seen anything outside of NetApp that offers real time protection like ZFS Offers by default (or Synology with integrity enabled) Raid6 is fine but does not do parity check on read (only when consistency check is ran) as long as the 2 disks are up your good and run monthly consistency and patrol read checks (raid5 doesn't protect from bitrot as it only has a 2 vote to work out if data is corrupted so it might fix the data or it might fix parity error, raid6 has 3 votes to work out witch data is correct or incorrect so if 2 votes out of the 3 say the data is incorrect it will then correct it) ZFS by its design protects from bit rot/data error (basically it doesn't trust the disks ecc to actually return a crc error on read error) as it has block level crc on read/write and can detect corrupted data in real time Btrfs and Refs (can detect and correct in real time as long as integrity check is enabled and has raid6 ideally or 2-3way mirror) btrfs (seems only stable in Synology boxes due to there interesting implementation that makes it reliable,, it uses Btrfs on top of Mdraid, as Btrfs isn't very reliable at managing the disk level raid it self, Synology way seems to resolved it) depending on year of Synology box integrity checking might be disabled or enabled on the share by default, you have to create a new share to disable or enable integrity checking ( you can't enable or disable integrity after you've created the share which is a bit unfortunate) but if your using SHR2 or raid6 bit rot shouldn't be an issue as long as your doing raid scrubbing (data scrubbing is not available on ext4 at all or btrfs with integrity disabled on the share, only raid scrubbing) note CPU inside the Synology box is the limiting factor on speed with integrity (and encryption) enabled witch can slow reads and writes down a lot when enabled Microsoft Refs need the integrity check enabled, (normally integrity check is disabled by default, need to use power shell to enable it , only meta data is dual protected by default) It has to be built on top of a storage space 2-3 way mirror (don't use 1-2parity as it might fail to repair and the below can happen) big warning with Refs when integrity is enabled if it can't repair it will silently delete the file and only log even in the event log about it couldn't repair the file and it deleted it!! (so if 1bit is corrupted and it can't repair it it will remove the whole file with no possibility of recovering the file and it silently does it, so is extremely important that you have a hourly/daily/weekly backup rotation running and check the event logs every week for unrecoverable deleted events) Microsoft silly Microsoft implementation of refs is it will delete the whole file if it can't repair the file if integrity is enabled but it won't make space available when it does it and no way to recover free space because it removes the file from name space doesn't actually mark it as free space so you delete the ReFs volume and recreate it to use the empty space again
These are just a few of the errors presented in this video: CCSDS is not a division of NASA. CCSDS is an international organization of space agencies. NASA is a member agency. The National Archives and Records Administration (NARA) was involved in the development of the OAIS Reference Model from the beginning. The Reference Model was not designed just for space data. See 1.7.2., "As this reference model is applicable to all disciplines and organizations that do, or expect to, preserve and provide information in digital form, these terms cannot match all of those familiar to any particular discipline (e.g., traditional Archives, digital libraries, science data centers)." Content Information = Content Data Object + Representation (not representative!) Information. An AIP contains Content Information + Preservation Description Information - not Representation Information and PDI as stated in the video. Dissemination of DIPs is not limited to the Designated Community.
