Extract parser and publish it as a separate crate

[ngirard]

Hi,
let me start by expressing my gratitude for your efforts into this nice project !

Your project is the only one (in Rust) I know of that parses Obsidian files properly.
It would be great if you made the parser available to other projects (including the one I'm trying to implement as I'm writing) !
Skimming through your code, that would be the whole contents of references.rs, plus the YAML and Markdown parsing logic embedded within Exporter::run() from lib.rs. I might have missed something, but I guess you'll know better ;-)

Cheers

[zoni]

Hey @ngirard,

Doing this has crossed my mind in the past as well. While it's not as trivial as simply extracting that code and publishing it separately, the refactoring I've done in the last 2 or so versions has made it easier to do something like this.

Partly as a note-to-self, some of the things that should happen for this:

• We should extract parse_obsidian_note. This should probably become the main function people would call from this new library/crate.
• context should be removed from the interface, I think that only makes sense for the recursive parsing obsidian-export does itself. This requires moving the infinite recursion logic elsewhere though (

  obsidian-export/src/lib.rs

  Lines 399 to 402 in 5ff990c

  	if context.note_depth() > NOTE_RECURSION_LIMIT {
  	return Err(ExportError::RecursionLimitExceeded {
  	file_tree: context.file_tree(),
  	});

  ).
• This function should then stop resolving embedded notes itself (

  obsidian-export/src/lib.rs

  Lines 489 to 497 in 5ff990c

  	Some(RefType::Embed) => {
  	let mut elements = self.embed_file(
  	ref_parser.ref_text.clone().as_ref(),
  	context
  	)?;
  	events.append(&mut elements);
  	buffer.clear();
  	ref_parser.transition(RefParserState::Resetting);
  	}

  ). Instead it should return some sort of "Here's an embed to note foo" element. This requires changing the return type to some sort of enum that can distinguish between regular markdown events and special obsidian tags.

I'm unsure how I feel about immediately publishing this logic as a separate crate, partly because most of the tests for the parsing logic are integration tests which rely on more than just this piece of code and so cannot be lifted over without changes.

Would you accept having the entire obsidian-export crate be a dependency on your project, if all of the parsing logic was in a separate module you could use independently of the rest? It's obviously non-ideal because you'd be pulling in more dependencies than strictly necessary, but would let you use my parser without me immediately having to refactor all the tests or needing to keep two crate releases in sync.

[ngirard]

@zoni, thanks for your reply and apologizes for being slow to respond, I'm being a bit overwhelmed with my work.

I'm unsure how I feel about immediately publishing this logic as a separate crate, partly because most of the tests for the parsing logic are integration tests which rely on more than just this piece of code and so cannot be lifted over without changes.

Please take your time, there's no rush here on my side.

Would you accept having the entire obsidian-export crate be a dependency on your project, if all of the parsing logic was in a separate module you could use independently of the rest? It's obviously non-ideal because you'd be pulling in more dependencies than strictly necessary, but would let you use my parser without me immediately having to refactor all the tests or needing to keep two crate releases in sync.

My project has different needs than Obsidian-export (I'm aiming at watching for file changes within a vault and maintaining an index in an SQLite database), so based on my first look to your code, even if you made quick arrangements for me, I'm not even sure I could benefit from them immediately.

I'd suggest taking the long run at your own pace, refactoring your code when you can. It wouldn't be a problem at all to me if you needed several months before publishing your parser as a separate crate. I'll celebrate when it's out !

Cheers