DeveloPassion's Newsletter 185 - Restoration
A newsletter discussing Knowledge Management, Knowledge Work, Zen Productivity, Personal Organization, and more!
Hello everyone! I’m Sébastien Dubois, your host. You’re receiving this email because you signed up for DeveloPassion’s Newsletter, or bought one of my products. Thank you for being here with me ✨
If you enjoy this, please forward it to your friends 👍. If this email was forwarded to you, then don’t forget to subscribe and become a supporter. You can also follow me on X, Bluesky, and Mastodon.
Welcome
Another week, another newsletter! I hope that you all had a great one 🤩
This week has been quite eventful (to say the least) in terms of data management challenges. Despite having multiple backups in place for my notes (local backups, remote ones, etc), I found myself in a situation where I nearly lost important data in my Obsidian vault (about two days worth of notes). While I managed to recover it all, this experience reinforced some super important lessons about data management and backup strategies.
In this edition, I'll share what happened, how I recovered, and most importantly, what we can all learn from this incident.
Alright, let's gooooo 🚀
Before You Go Any Further
I want to reach more people. Got 60 seconds? Then share the link with one friend/colleague who would benefit from joining us!
The Lab 🧪
This week hit a new high. I made > $2K of sales in the last 30 days. I'm really psyched about this, and I hope this trend will continue!
I've continued improving my AI Ghostwriter, enhancing its understanding of my communication plan and extending content templates. To further improve output quality, I'm implementing role-playing capabilities that will let the AI analyze content from different perspectives. This should help generate more targeted improvement suggestions.
I've also made significant progress on the Knowledge Management for Beginners course. Module 6 is complete, and I've started working on Module 7! Only one module remains after that. While I haven't recorded the videos yet (planning to do this in batch next week), I'm really proud of the content I've prepared so far, and can't wait to hit the PUBLISH button! ❤️
💡 Don't Miss Out! If you're interested in getting rid of information overwhelm and want to leverage your knowledge, the Knowledge Management for Beginners course will be launching soon. This comprehensive course will help you build solid foundations for managing your knowledge effectively. Join the waitlist and be the first to know when it launches: https://knowledge-management-for-beginners.com
Last but not least, I've decided to learn more about n8n. I installed it locally as well as on a VPS on Hetzner. It was quite simple to deploy. My plan is to use it to automate various parts of my business. I'm really enthusiastic about it, especially given how little it's going to cost me. Also, n8n has many integrations and it's easy to build additional ones if needed. Lastly, it can also integrate AI as part of workflows, and that's really cool and powerful!
Here's a screenshot of the first workflow I've built to try it out:
In this workflow, I check a mailbox every X to detect new sales, identify the product, the revenue, the customer etc, and import or update the contact along with relevant labels into Ghost (where my newsletter is).
It works really well, and should help me spare a ton of time in the future.
The Importance of Robust Backup Strategies
Data loss is one of those things we rarely think about until it happens to us. This week's incident with my Obsidian vault highlighted several critical points about backup strategies:
- Synchronization is NOT Backup: Having your files synchronized across multiple devices can actually propagate data loss if something goes wrong.
- Multiple Backup Systems Aren't Enough: It's not about having multiple backup systems; it's about having working backup systems that you regularly verify.
- Automation Needs Monitoring: Automated backup solutions are great, but they need regular verification to ensure they're working as intended.
- Recovery Plans Matter: Having a clear recovery procedure is as important as having backups. When things go wrong, you need to know exactly what steps to take.
The key lesson? No matter how many backups you have, if you don't verify they're all working, you're still in an unsafe place. Backups are hard, and we need to treat them with the respect they deserve.
I thought I did, but I still nearly lost data. And I don't like that feeling!
Recovering Obsidian notes after a shitstorm
Here's what happened to me this week, and what you can learn from it. I've written a detailed post-mortem about this that you can find here:
It all started when I moved two files from one folder to another. Either I messed up, or Obsidian did, but almost all my notes ended up in the wrong folder. I didn't notice immediately. Then, my auto note mover plugin detected the misplaced files and tried to move them back. This created a perfect storm:
- My notes are stored in Google Drive, which started synchronizing the changes
- The auto note mover kept moving files back
- This generated a storm of file move events
- My notes are also synchronized with Syncthing across devices
- When Syncthing saw the changes, it generated another storm of events
- Obsidian kept freezing
When I tried to fix it, I didn't fully understand what was happening. I moved folders back, but the event storms continued. Files kept reappearing in the wrong location. Then I made things worse by deleting what I thought was the wrong copy – but actually deleted my notes.
The real messy part? Syncthing propagated the deletions across all my devices. And since they were configured for bi-directional sync, it only made things worse.
Despite having multiple backups in place:
- The Local Backup plugin hadn't run for over a month
- Synchronized copies were broken (remember: sync ≠ backup!)
- My Git backup wasn't up to date because I hadn't committed recent changes
Doh!
I ended up losing about 2 days of notes. And that's a lot considering that I spent those working on my course, which is all in Obsidian!
Here's how I recovered:
- Stopped Obsidian
- Stopped Syncthing synchronization
- Let Google Drive stabilize
- Cleaned the vault folder completely
- Reverted local changes to the latest Git commit
- Restored files from Google Drive's trash
- Committed everything back to Git
- Resumed Syncthing synchronization
Total recovery time: half a day.
Moving forward I will:
- Commit changes more regularly to Git
- Regularly verify my backups
- Immediately investigate when Obsidian freezes
- Backup before attempting fixes
- Fix issues completely before restarting Obsidian
- Create new backups after fixes
The critical lesson here is that synchronization is NOT backup. I repeat: SYNCHRONIZATION IS NOT BACKUP. You must have real backups, and you must verify they're working regularly.
Want to learn more about how I normally handle Obsidian backups and synchronization? Read this:
Sebastien Dubois
Knowledge Base Stats 📊
This week, I've created some scripts to analyze my Obsidian vault and capture statistics, in preparation for an article I'm working on. I published those statistics here:
Here are some interesting numbers:
- ~8K notes (actually over 10K when considering my private notes as well)
- 2868405 words, corresponding to ~32 books (!)
- ~6.5K published notes
- ~64K links between notes
- ~12K external links
- 1800+ tags
- 116 Maps of Content
- ~2K unique tags
I'll update these statistics periodically to share insights about my PKM journey. Stay tuned for the article, where I'll explain how I generated these stats and the insights they revealed about my writing habits.
Latest Articles
No new articles this week.
Books
I'm diving into a number of books about GenAI. I'll share some more details over the next few weeks.
Right now, I'm reading "Generative AI in Action" by Amit Bahree.
Quotes of the week
- Clutter is nothing more than postponed decisions
- Everything in this world, including money, operates not on reality, but upon the perception of reality
New & Interesting Links
The 2024 Obsidian Gems of the Year winners have been announced! Check out these amazing community contributions:
While I would have loved to see my work in the top list, I'm genuinely happy for all the winners. Who knows? Maybe the Knowledge Management for Beginners course will make it next year! 😉
Fascinating video on learning hard concepts intuitively:
This ties in perfectly with what I teach about Knowledge Management and learning!
A great introduction to tagging best practices:
A useful tutorial on finding old notes in Obsidian using Dataview:
Tim Miller
A useful command-line tool to save Web pages as single files:
Y2Z
Perfect for archiving content!
GitIngest helps convert GitHub repositories into LLM-friendly Markdown files:
Great for learning from and analyzing codebases!
That's it for today! ✨
About Sébastien
I am Sébastien Dubois. You can follow me on X 🐦 and on BlueSky 🦋.
I am an author, founder, and coach. I write books and articles about Knowledge Work, Personal Knowledge Management, Note-taking, Lifelong Learning, Personal Organization, and Zen Productivity. I also craft lovely digital products . You can learn more about my projects here.
If you want to follow my work, then become a member.
Ready to get to the next level?
To embark on your Knowledge Management journey, consider investing in resources that will equip you with the tools and strategies you need. Check out the Obsidian Starter Kit and the accompanying video course. It will give you a rock-solid starting point for your note-taking and Knowledge Management efforts.
If you want to take a more holistic approach, then the Knowledge Worker Kit is for you. It covers PKM, but expands into productivity, personal organization, project/task management, and more:
If you are in a hurry, then do not hesitate to book a coaching session with me:
