This has been by-far the best video on long-term, scalable archiving of data for photographers/videographers/musicians I've ever come accross; and I've been looking for a clean, simple, understandable solution for a while now.
Anyone concerned about long term archiving needs to consider the usefulness of the footage over time. For weddings, you could argue that the footage will have lifelong value. In this case, I would give the footage to the client and make it their responsibility. Really, how long should you be expected to be responsible? For corporate clients, I find that after 5 years, most footage is garbage. Before I purge, I check with the client. Sometimes, they want it. Usually they don’t. The value in holding on is repeat business. But with that opportunity comes responsibility. Your system is virtually identical to mine. It provides adequate storage for a decent price.
Wait until you're shooting all 4K ;) This is very similar to my setup, but there's no way I could wait until the end of the year to dump files unless I got a significantly bigger raid. Now I dump every few months to long term backups as I finish projects and run out of space. Also, depending on your Internet speed, Amazon Cloud Drive may be a good option for you as a third layer of backups. $60/year for unlimited space is a ridiculously amazing deal.
Yeah, I hear you on that. How much total media do you bring home from a 4K wedding? Yeah I need to look into Cloud backups. I've never felt like I've needed it, but as it gets cheaper it may become a better option!
Haha its costs me ZERO to have unlimited archival storage. And try to restore a few TB from the cloud. The cloud is impractical for TB worth of recordings. And its RECORDINGS not FOOTAGE. I have NEVER known Video to have footage, NEVER. Only FILM uses footage
I use same but a little bit different. I work on a large scale team of editor, I use G-Technology G-Shuttle XL ev as my main drive to store all the ongoing project, and the data mirror with a smaller cheap consumer drive as a raw backup and later as a long term yearly archiving. Each Video editor Imac use GDock ev connected. My editor copy the data that they want to edit with Gdrive ev from the shuttle xl. Everytime they finish the project they dump back to shuttle and update the mirror drive. As for the Finish project, I use WD cloud and google drive business.
Those Desktop Drives you edit from, they really can't be anywhere near fast enough if you're editing 4K or similar, you should definitely use SSD drives to minimize the time spent rendering and to make the playback smooth
Love the video. Confused about how you deal with the bare Desktops HD's not really being the best long term storage medium? Has there ever been a problem with file corruption on the oldest un-powered drives? Seen lots of other videos, while researching the topic saying that just leaving info in a HD isn't safe long term. Any thoughts? The Idea that I liked best, but also have my concerns about is adding one more bare drive to the long term Archive group, and periodically leap-frogging the data from drive to drive, so it stays fresh. I have my concerns about a copy of a copy of a copy down the road though....
@@Harshinstinct lol. Can’t says I ever got any answers for someone in the know. But I have been doing the exact thing I suggested for the last three years. Lol. With one exception. Instead of leap frogging the data periodically, I have been cloning the drives on a yearly schedule. But once the drive has been cloned I have been keeping the original NOT THE CLONE. Basically using the cloning process to reactivate the old drive. I then format the clone. I did this because of a fear of degradation due to a copy of a copy over time. It has been working thus far. Never had a problem with old archive drives. Problems migrating old projects into newer Premiere Pro version, now that has been a nightmare.
Nice setup. Mines pretty much the same. Followed a lot of Chase Jarvis' method. The only problem I'm having is long-term archiving (over 10 years). I use a docking station (toaster) like you and have over 40 drives. Over the years of sitting on my shelf a few drives died (didn't know about magnetic refreshing until it was too late). The plan now is just to stay up on technology and transfer archived data to newer, larger hard drives. I think it's like 18 months for computing power to double (Moore's Law), which is just short enough to refresh hard drives (recommended to refresh the magnetic signal every 3 or 4 years) that are sitting on the shelf and consolidate them as well. For us small-time filmmakers, this process would seem to be the most economical compared to cloud-based (even at $5/mo) or LTO tapes. Excellent video mate.
Very true. The larger your archive grows the more money you will need to keep it up to date, expand, backup, transfer, test, and store. For small film makers several hard drives is no problem. For larger companies, they don't have time to transfer 1000's of copies footage to update and then 3000 next few year. That's why they store their footage on Optical Archive Storage Disks. More money upfront but less an a larger scale.
Great Video, Instead of a Drobo setup, I use BACKBLAZE online storage $5 a month unlimited storage. Yes backing up online is a slow process but it's offsite, easy access and never run out of storage.
I definitely need to look into online storage. I would use it to replace my long-term archiving system tho. I access the Drobo way too often. It's not lightning fast, but it's way faster than pulling stuff off Backblaze. I do use Google Drive to organize small documents I need to access on both my laptop and desktop.
tHanks for the video I really enjoy using the docking stations for naked drives. I suppose the one concern is that they don't have a dedicated fan....however knowing that I keep an eye on them as I use the station. Same goes for when you are making toast. I use the Orico three bay....but that 4 bay looks like the bomb!
Thanks for the video but I strongly disagree with using spinning HDDs as long-term archival. My company previously used spinning HDDs to archive with a "toaster" type device and after about 5 years pretty much all of the drives were corrupted to some degree when we tried to spin them up again. Bottom line is that spinning HDDs are NOT a long-term solution. As far as I know LTO tape, and other systems like LTO, are still the best long-term solution with cloud services also an option.
Love it! Subscribed. Though I've heard there's a risk of hard drives not working after sitting on the shelf unused long enough... guess that's where the 3rd layer of cloud protection you're talking about in the comments comes in
Thanks Rene! Yes, I've heard that as well. There is really only so much you can do when it comes to long term storage without investing a TON of money in a "professional" long-term storage solution. I'm not contractually obligated to have any of this footage safe, it's really just for my own personal peace of mind. So IF one of those drives were to go bad from sitting, it's not the end of the world.
great video. Wondering if anybody can answer this. So i use imovie and i have a SSD for editing. AFter done editing how do i transfer those files to my mass storage drive. if all movies are really in one file? thank you
I manage and backup everything in google drive. I have a RAID for projects where i am working on and then i just upload everything. Its the cheapest option. Much cheaper than buying all these hard drives.
Nice video. I was wondering why you use a WD drive to edit when you can have the incredible speed of a RAID 5 enclosure? I use two 3TB G-Techs to put all my footage on and those are automatically copied to a 6TB Drive for a backup. When I’m ready to edit I copy the footage for that bride onto my 6 bay Thunderbolt Promise RAID 5. I can edit 4K on my Mac Mini without creating Proxy Files. I now have 3 copies. When I’m down with that wedding I delete my Optimized Media and copy the FCPX Library with the original files and render files onto one of those toasters with two 4TB drives setup as a RAID 1. Then I delete the files on my G-Tech drives and my 6TB backup drive. I just started so I haven’t filled up my 4TB Archive drives yet. Great idea to keep one in my office and the other offsite.
Very good video however your system is still vulnerable to data loss due to bit rot. The RAID systems you are using don’t detect data going bad on disk, and the long term storage has the same problem. Some of the newer file systems, btrfs for example, will detect and correct the problem. Since throwing away all those drives is not a practical solution, you might want to look into a cloud storage solution. Read up on Amazon s3 durability, this will explain the long term storage problem. You might want to store your archive data on Amazon AWS S3 Glacier. Remember, just keep asking yourself - how do I know the data on the disk is good? Take care.
A very very nice setup! I have almost all of the same hardware, I just use different brands for each layer. Lacie for the editing drives, G-Tech for the big RAID, but I use that same "toaster". Love it! Gonna have to start calling it that now. Hahaha!
Awesome job. Very informative...have you researched what would be involved in storing everything (not currently being edited) in the cloud? Or do you just prefer always owning physical copies?
Thanks! I have looked into it in the past, but don't think it would be a viable solution for my active archive. I need to access the footage too often for other projects, and it's super slow with the cloud. Plus the whole hassle of uploading everything. I guess I just don't see a benefit to doing it, other than a 3rd layer of safety.
i spent 2 hours googling hard drive toaster bc i saw this a year or two ago found this video again. probably going with this system bc it’s like 50$ a 1TB
I like the system. But I think you actually don't need the drobo in the first place. aslong as you do not need 10tb of storage every year... I have external SSDs for my current projects (those would be your 4tb mirror system) and I also would recommend to use *two different brands of HDDs* for your toaster. Because it is possible that that two HDDs are manufactured on the same day and go dead almost simultaniously.... Great video!
Your current year archive Drobo isn't backed up in a second location , a better solution would be a second Nas keep both in sync ideally one at a separate location
NAS storage would keep all of your archives/storage live 24/7. For a small-medium business though your solution sounds great, I may copy your toaster idea :) Btw is your Drobos setup in raid? I'm looking into getting an enclosure for archive but I like the idea and safety of just doing a JBOD setup.
Hey! So when you use chronosync to create a nightly duplicate of your edit drive, is the software making an identical “clone” that’s ready to go, or a compressed disk image file that needs to be mounted and restored in order to access. I’m creating my own workflow and I’m on a P.C. using Acronis; looking for a similar setup. I’ve heard clones are slow and have potential risks for the source drive, so I’m wondering if disk image backups are the way to go, for edit drive backups even if file access isn’t immediate. Would love to know these specifics, super helpful video, thank you!
This was a great video but I have a question: 1) I'm an entry level Freelance Motion Graphics Designer and Video Editor which of the ChronoSync apps would you suggest I get? 2) Are there any free alternative apps similar to ChronoSync? Here are the specs for my laptop: MacBook Pro (13-inch, Mid 2012) 2.5 GHz Intel Core i5 16 GB 1600 MHz DDR3 Intel HD Graphics 4000 1536 MB 1 TB SSD (Storage) which replaced the 500 GB HDD that came with this MacBook Pro when I purchased it in 2014 Accessories include: LogicTech MX Master Mouse ASUS MX27AQ 27'' 2560 x 1440 monitor to pair with my MacBook Pro Programs I use: Adobe Premiere Pro CC (often) Adobe After Effects CC (often) Adobe Photoshop CC (semi-often) Adobe Illustrator CC (sparingly) Adobe Indesign CC (sparingly) Adobe Audition CC (sparingly) Adobe Prelude CC (in the process of learning how to use it) Cinema 4D (in the process of learning how to use it) DaVinchi Resolve (in the process of learning how to use it)
Great video Tyler. I tried sync software before but found it had issues and I had to check it so often that it was working that I may as well have been doing it manually. I think archive drives at a friends is overkill. Client has the product. comparatively, up until your project is complete, your entire footage is on two drives side by side on your desk! Burglary or fire will claim both! I back up to network attached RAID connected to my router in another part of the house.
Cool video dude. Can I ask, being 2019, how has your set up changed and also, how would you change about your set up with the introduction of new tech etc?
Great question! I am going to make an update ideo to this video since things have changed slightly since I filmed this one. The main differences to my workflow now are that I use Backblaze to backup all my desktop HDs and my Drobo just in case of a catastrophe where they were all destroyed. Secondly I've been editing off of a Samsung T5 SSD for all projects active projects. Other than that, a lot of this is the same!
Awesome info, thank you for sharing! Question for you: on your long term archive drives, are you keeping all of the raw footage of every single clip from the wedding + all recorded audio, or just the final product delivered to the client?
Hey Richard, thanks for checking out the video! I personally keep everything, just in case. That's more due to my paranoia, and the fact that I hate getting rid of things. I'm not contractually obligated to keep anything, I just like to. That's a personal decision you can make for yourself!
Great stuff! Thanks for the video. What do you use for a more mobile hard drive? Like if you travel or need to go somewhere with your laptop and still want to work on something?
Currently I use a Lacie 1 TB hard drive for anything on the go. Ideally I would load the wedding onto my Desktop drives first, and then make a copy to the external. If we're shooting an out of town wedding, I'll copy everything onto the Lacie (using the same folder structure) and then leave all the footage on the cards so it's still in 2 places.
This may be a beginner question - I tend to dive into things and figure out the details later... like when I've used up 11/12 TB of my G-Raid external hard drive. :0 What about the FinalCut library? Do you keep that as well? I have some past projects that I can archive - and I'm wondering if I keep the library also, or just the original and edited files? Thanks!
Do you have a backup of the drobo in this system throughout the year? What happens if that fails? Thanks for the awesome video, love the level of detail
Hey Evan! Since posting this video we started using Backblaze as an Cloud backup solution. It automatically backs up the Drobo to the cloud in case of a Drobo failure!
This also assumes that you are storing your Lightroom Catalog on your editing drives or locally on your computer? Do you have 1 catalog or do you have multiples stored with each hard drive?
Hi, thanks for the video! Very helpful. Question: I've decided to use Chronosync as well for keeping my editing drives backed up with all the same changes each day. I have a Drive A and a Drive B; I want to make all my changes to Drive A each day, and then just sync those changes to my Drive B at the end of each day (or however often I want). I want any files I delete off Drive A to also be deleted off Drive B. Basically, I want both drives to always be identical in their contents, but my workflow is that I'll always be making changes to Drive A first, and Drive B exists purely as a duplicate. I'm wondering which is the best Operation to use within Chronosync. I'm currently weighing the pros and cons between "Synchronize Bidirectional" or "Mirror Left to Right." It seems like "Mirror Left to Right" is probably what I want, but I may be missing something. What do you use?
Great question Mary! I use "backup left to right". I do this because it gives you the option of what to do with the files as they are deleted off of the 2nd drive. I have mine go to the trash, but you can save them to a special folder if you want. This is nice in case you accidentally delete something off the main drive, you have a 2nd chance to catch it!
Hi thanks for the video.. I see good things here..just struggling with the fact of having just one copy of non active term projects... It's a lot to loose isn't it?
Since posting this video I've started using Backblaze to backup to the cloud as well! It's a great solution as an extra line of defense! It's not practical to edit off of the cloud because it's not fast enough.
Thought this to be really helpful! I am starting to look for a long term set up but I am still working out of a MBP set up with an 2TB external hard drive. Any thoughts about running a back up system from a laptop?
Thanks, I'm glad you found it helpful! You can totally run this same system off of a laptop, I did it that way for years! Pretty much everything would be the exact same process, except you'd probably want portable HDs as your Editing Drives. I used to have a Henge Dock that I would drop my MacBook Pro into and it would instantly connect to all of my drives, and my Drobo. You can setup rules in Chronosync to automatically run a backup/mirror sequence when it detects a drive is attached that hasn't been backed up in a while. No matter what you decide to do, it's super important to have everything in a least 2 places!
tape storage and Sony's ODA is great but can get expensive for video file sizes. All hard drives will eventually fail. But, if you are always giving them a quick run and health test yearly you can catch problems before failure. And copies are essential for this. Eventually all copies can be moved to newer larger hard drives in the future.
Great video, I have a question. The files on your editing drives are only on your desk, so if you were to have a fire/theft, you have lost the files. Do you have a solution to this as files should always be stored in at least 2 physical places?
True, spinning up the disks periodically would be best practice. I'm not obligated to keep anything past a year so it's all really just for me personally wanting to hold onto these files.
I need the Orico to offload footage from the Drobo once it's full. None of the bays are empty in my Drobo. I don't want to be constantly buying bigger drives for the Drobo every time I fill it. Bigger drives are more expensive. The Drobo doesn't allow for specific RAID configurations. They use a "beyond RAID" system that's built into all Drobos that essentially does all that for you. Essentially it fills 4 drives with data and leaves one drive blank as redundancy!
Great stuff bro!!! As always. But here is a question. Why would I want to hold on to a finished wedding file that long? I tell clients I can hold it for a year tops. What are the benefits for holding them forever? I appreciate the time my man. Thanka
Thanks Marcus! How long you keep the raw footage is totally a personal preference. I just could never bring myself to delete a bunch of wedding footage...no real "reason" it's just the way my brain works. If I didn't keep the footage, I wouldn't have been able to help the bride who emailed me about a corrupt file export. Technically I have no contractual obligation to help her 2 years later, but it feels good to help! haha
It's fairly interesting, i also use a drobo and i use a 5 bay Synology NAS with WD red drives in. So i have two 6TB in the NAS and i believe 3 4TB. And in my Drobo i got four 4TB and 1 3TB. I mirror everything that's on my NAS to my DROBO. It's pretty good, but it's also almost full. I would like to start using 10TB WD red hdd's, but they cost a fortune. Your method seem a good option for me. Thank you! :-)
Gotcha. You probably don't need to mirror the Synology to the Drobo, just because it has redundancy built in. That should save you some space! 10TBs would be awesome, but yes, extremely expensive!
It's true that the Drobo gives me redundancy, but now i have actually 3 copy's :P And i have a mini Drobo, with my most important personal pictures. But it makes a lot of noise, so i'm not using that much, just to make copy's now and then. I'm planning on putting 10 TB's in the drobo next year. I think i can wait for now till this year passes. Also, i have other costs to make, sadly enough. But it's all for the setup, so it will be great :-D
What about an EMP pulse/earthquake/alien invasion??? No but seriously, I thought harddrives were not recommended as long term archiving due to needing to be charged every so often or they'll lose the charge. They are also easily physically corruptible.
Haha, it's definitely a little over the top, but I'd rather be safe than sorry. I've never heard of them losing charge, I'll have to look into that! With the drives sitting on my shelf and not plugged in (no spinning disks) they're pretty safe from corruption. No drive is completely safe tho, which is another reason why I have 2 copies of everything.
The charging that your mentioning is SSD not HDD, since HDD is magnetic it has a longer life span of several millions of years, but the motor might fail before then. Which has happened to me a time or two. But I got it up and running and there were my files. I have kept them for up to 30 years with leap frogging.
It keeps one drive blank at all times. So if you have five 4TB drives, you'd have ~16TB of usable space. If you google "Drobo capacity calculator" you can find a tool that gives you the amounts if the drives are different sizes.
Thanks! How long have you been running the Drobo? I've heard some bad stories about them online about their proprietary system, but I also heard they are super easy to use.
Yeah I've heard those stories too, but honestly I've never had an issue. It's been super easy to use, upgrade and transfer to a new Drobo. I got my first Drobo in 2010 or 2011? I upgraded to a newer/faster one last year and the process was seamless. Took the drives out of one and plugged them into the new one and it was up and running in 30 minutes. I've been super happy with it over the years!
This isn't a good long term storage idea. Drives when not spun up for a while can die. So you would need to spin up the drive atleast once a year or more. Tapes are the best for the long term archiving. LTO 6 Drive
True, tapes are definitely ideal for true long term archiving. But for my workflow and budget, that's not a possibility. I am not contractually obligated to keep footage for any of this work for any amount of time, so this system works for me and my budget.
Hm. First, i would be afraid with this system. Hard drives are not for long term storage. It's magnetic medium, it will lose data even if not touched after some years. I wouldn't trust them longer than 3 years. That's why tools like DiskFresh exist, to refresh data, but then it's too much work. I still prefer good BluRay media. Less work and risk in the long term. As for 2 work drives... External and synced overnight? Why not some small NAS too in raid mirror or even with 3 smaller drives? No need for sync overnight, data synced in real time and also you get some read speed benefits (especially with more than 2 drives).
Thanks for sharing your storage workflow, always good to see what others are doing. But I don't think yours is all that good. Every piece of your system is consumer grade. Its cheap, it works, but its not going to hold up to 'industrial' use in the long haul. Especially with 4k. But I do like your archiving system. You have planted a thought that maybe I don't need to dual NAS, one for backup. Maybe each year I could store them on 2 drives like you do... A good system going into 2019: 3 SSDS on your computer. One for the OP/Programs, one for the edit, one for cache. Size will depend on budget/needs. Much bigger NAS to store all the things. You could also use the NAS to backup your Computer. Plus NAS = HUGE personal Cloud storage!
Archiving and work drives are two TOTALLY different items! They are NOT the same. Think about it. You know how many hard drives you need to archiving? NONE!!! As am Emmy award editor for decades and the ONLY. Information I have on drives are projects I am currently working on. Since I use linux and not some other silly o/s, those work drives are configured as RAID-1. I can recall ANY work that I have done for decades in minutes. I DONT use cloud storage as well (ugh). Most photographers & video people OVER COMPLICATE things. Remember work SMARTER not HARDER. You need to ask yourself if you need all that stuff spinning. If its ARCHIVE material, how important is the ability to retrieve it v. Cost & fragility of faster hard drives? Yes get a 10TB drive, put everything on it, then have the drive go bad. Ever tried to restore a 10TB drive?? Get a grip man. Thats way too much work and time. Work SMARTER!!
