Dependency and compatibility info in addon template #4
Comments
|
there was some talk about using homebrew for this, but I kept backing away I propose something like this: the readme can have key/value pairs that ofxaddons will recognize and add dependency: https://github.com/YCAMInterlab/ofxTimeline Then a download script will go fetch the rest of the required addons. Maybe you can also specify target platforms This developers can state what they are going for. What do you think @atduskgreg? On Tue, Jul 24, 2012 at 12:45 AM, Christoph Buchner <
|
|
I want to investigate some file format that replaces the current
and probably others, something similar to the current install.xml but this file can be parsed by ofxaddons to look for dependencies and also El 23/07/12 22:54, James George escribió:
|
|
The advantage I see with @obviousjim's approach is that this info being in/parsed from the README, we won't have to duplicate it somewhere else, so that it is readily available for both humans and computers. Also, I think it would be easier for addon maintainers to add this to a README than to construct an xml/json/whatever file, and we want to make it as easy as possible, right? or would your file also be just key:value pairs, arturo? Arturo's approach would be much more powerful of course, in that we can also store more complicated, "internal", info. @arturoc: by next release, you mean 0073, right? 'cause 0072 is way late, already... The beauty of ofxaddons is, when addons get published/updated (btw, do you crawl all the addons regularly?), we can check if the README or Arturo's file are "valid"/parse correctly, and if not, automatically open an issue via the GH API! Maybe this could later even be extended to testing compliance with the addon template structure... |
|
I would recommend to look on other excellent dependency tools like bundler (ruby) or npm (node.js) which do dependency management exactly the same way we want to do. npm example written for an addon: {
"name": "ofxMyCoolAddon",
"description": "a very cool addon which can do a lot of cool things",
"author": "Cool addon author <[email protected]>",
"dependencies": {
"ofxFenster": ">= 1.2.0", // source can be omitted, because it is the same on ofxaddons
"ofxSpecialAddon": "http://some/special/source/path" // can be local, git or http path
},
"openFrameworks": "0071",
"tags": ["working on linux", "experimental", "needs fix on windows"]
}I think the node management is a little bit smarter, because its pure json and we don't need any further interpreter. /edit/ If we say every addon needs a ofxaddons app can then easily parse this file or file a issue if an addon doesn't have one or send an automated pull request with a template of this file :) /edit2/ another advantage on relying on some standardized format is the fact that we are open to further enhancements in some directions because we can use already existing parsers for the dependency management in every possible language. |
|
Where would this file live? On Tue, Jul 24, 2012 at 4:41 PM, Benjamin Knofe <
|
|
I'd say in the addon root directory. |
|
I don't really like it honestly. I know it's nice to leverage existing the regular expression to parse something that is on one line and contains and the human readable factor + we already have README parsing in ofxAddons that's my 2円 neway On Tue, Jul 24, 2012 at 5:45 PM, Christoph Buchner <
|
|
+1 for root directory. @obviousjim Beside how the file is formatted, I think its no overkill to add more (standardized) information to it, because there is need for more meta-information and personally I don't like the proposal for adding marker like snippets cluttering the names/descriptions/issues on github (like the proposal mentioned on the ML). I think it makes more sense to put all this in a file. Also I think if there will come other features, we could add it to the file. Do you agree on storing everything meta in a file and you just don't like the formatting? |
|
I think what James is driving at is that it should be sufficient to keep it in the README, with a simple However, the simple scheme already gets more complicated if we get more complicated info (in the future), e.g. multiple dependencies as already above, or multiple attributes for one thing (dependency-version, -url(optional), -name, -minVersion,...), and we then might as well stick to an already well-established and proven syntax. Maybe a middle ground is to have a metadata section at the end of the README, with something similar to what ben posted above in it? |
|
*** "I want to investigate some file format that replaces the current
@arturoc isn't that what install.xml is? ;) |
|
how do we mark the seperation of "normal readme" and "meta data in readme". do we agree on a special marker or something like that? From my side I'm fine with adding it to the end of the README but I'm currently unshure how we make this reliable to seperate from normal readme content. |
|
could be: "line starts with one of a list of predefined keywords, then |
|
ok I put everything in a gist to discuss further: https://gist.github.com/3197172 What I not included for now is @arturoc data, because I don't know how to include it. Especially needed packages on linux, because they are named differently on every platform. I think we need more discussion on this. Maybe @arturoc can add the info as he like and we can see if everyone is happy with it. |
|
What I forgot: The syntax is now a valid YAML syntax, which a) renders good in MD and is b) easily parsable. |
|
hm, that |
|
quickly, since this was discussed on the last OF dev chat, I thought I'd mention that I proposed we do the following WRT to the project generator and addons: a) it searches for some sort of file (install.xml or other json/yaml/ etc file in the root directory). I think this is kind of the best of both worlds, allows us to keep the simplicity that exists with alot of addons now, but also helps us solve some of the trickier things the PG is hitting walls with such as link order, additional libraries, etc. I know this doesn't address dependency or capability info but I think this could be part of the file as well. what do you think? |
|
we are working in refactoring the makefiles at the moment, (osx working already!) next step was look into this. my idea is more or less what @ofZach proposes but the file should be something easily parsable by the makefiles, so no json or xml. i guess some properties file with one property per line or something like that + sections for different platforms should do |
|
I like what @ofZach proposes, with a slight alteration to b): if no such file exists, and the pg parses the structure, it should not create the file. why? because if it can successfully extract the info it needs from a (default-like) file structure, it should not put this structure into a file, too. otherwise the addon dev now has to manually handle that file if he changes stuff in the addon, we risk desyncing structure and structure-describing metadata if he does not, and all this is not necessary because the PG can understand the structure anyway. @arturoc: is there a (known) format which is easily parseable by make? the YAML file @benben posted above already looks pretty simple, would that be a workable approach? |
|
btw, some food for thought: we'll probably need some way of dealing with platform-dependent source code, e.g. for ofxOsc, so that those adjustments become unnecessary: https://github.com/bilderbuchi/oscpack/commit/f9a2b5789cc00f87a633a21507a56bf5cfcb479f |
|
i've just added support for this kind of files in the makefile refactoring branch, here's an example for ofxOpenCv: the meta section contains meta tags for the addon, in common you can specify any flags that are common to all platforms and then there's a section per platform where you can add specifics needed only for that platform everything is optional so if the file doesn't exist the makefile parses the file system and if any variable is empty the makefile parses the file system for that value. for example in the case of the openCv file the libraries are specified for some platforms because we need a specific order for those but everything else is empty so all the other settings are parsed from the file system |
|
that looks great! |
|
On second thought, do we really want to mix metadata and build configuration in one file? |
|
i don't mind about the name, don't think it is that confusing since it's in different folders and people won't need to modify yhem most of the time but addon_config.make sounds good too. about the metadata i think it is ok to have it in that file i would add version, probably license and url that way the ofxaddons the PG or any other tool can show that data pretty easily |
|
sounds good. |
|
Hi guys, Sorry for not piping up early. I'm sitting here with James. We've been thinking about this problem for awhile from an ofxaddons perspective. We've been planning on adding an API to ofxaddons and we think it can help a lot with some of these problems. We've come up with a way for addons to declare the other addons on which they depend, which will allow the Project Generator (or other command line tools) to communicate with ofxaddons.com to query for all of the information needed to resolve dependencies and clone them all from github. This is focused on figuring out which addons an app or other addon depends on, not other build requirements. It's meant to replace things like this shell script: https://github.com/YCAMInterlab/ofxTimeline/blob/master/clone_addons.sh We'd like addons and projects to declare their dependencies in a simple text file, something like "addons.txt" which would contain a series of lines like the following: memo/ofxMSAPhysics sha=83eacde45 branch=experimental where the format is: reponame <sha=sha> <branch=branch> (where everything inside brackets is optional.) Our ofxaddons.com code, will look for these files in addons, parse them, and store a resolved dependency graph for each addon. When you query ofxaddons.com for information, it will return everything it knows about a particular addon including the full list of dependencies. That should be enough for the PG (or other cli tools) to successfully clone from github all the needed addons. -- Greg and James |
|
Hey Greg James As i see it ofxaddons should return a list of available addons and the git urls for them but the dependencies and other info should be in the addons file so we can resolve those offline. not sure if what you propose will allow that? take a look at this example no dependencies there yet, my idea was to have just a list of addons the way the other rules are written. also having the possibility of specifying dependencies in the common area or per platform also, i don't mind that much if the meta tags are in a separated file to the compiling rules about the dependencies, i'm not sure that keeping version numbers or hashes is so necessary, i see how it can be useful but it'll also make maintaining this kind of files way harder. perhaps we can have the hash/branch as optional then if someone doesn't include them we default to latest in master and if the addon is downloaded in local whatever version is in the addons folder. |
|
Hey Arturo, Is the addons.make used for other platforms besides linux/android? I'm worried about forcing it on other platforms just for the feature of requiring dependencies. We are thinking of this from the perspective of a user using the PG: We imagined that the project generator would be able to query and see addons from ofxaddons.com. If you select to install an addon to your project it'll also clone the the dependent addons, using ofxaddons to resolve the dependences from the addons.txt file and forcing them into the project projects. This could be done offline if the addon has already been downloaded and the dependencies for it cached. How much do you think the project generator should know about the addons.make? Also the version numbers and hashes are optional. most files will look like this: addon dependencies for ofxMSAPhysicsofxMSAObjCPointer the repo name and sha/branch are not necessary. |
|
addon.make will be used by the PG too and it's optional but can be used in any platform. for example in osx some addons need different search paths than the ones parsed from the file system so they can go in that file, in windows there's some addons with several libraries that need to be linked in a specific order if someone just wants to put the meta tags in an addon they can omit everything else. we could put the meta tags in a separate file or even in the readme but at least the dependencies should go in addon.make so makefiles and the PG can parse them and include them in the project thinking of this a little more, the way software repos work usually is by downloading a file (it could be an xml or json) with all the info about addons in the repo (name, url, author, description...), so in this case the PG could ask for such a file to ofxaddons then with that file it's able to show info for any addon, if it's already downloaded... you can update that file from time to time or it can be updated everytime you start the PG but it's faster than having to make a request to ofxaddons everytime you select an addon |
|
I'm not sure about making our build process reliant on ofxaddons.com. if the relevant info is already in an addon's
I'm curious why you specify both a branch and an sha? those are both refs, one dynamic, one static, which is potentially ambiguous. what am I missing here? |
at last, this discussion seems to have come full circle |
|
@bilderbuchi: You're right, you'd never need both a branch and a sha. Those arguments would be exclusive: if you specify just the branch then we assume you want the HEAD of that branch. If you specify the sha then we'd get you exactly that version. I should have made that clear. I was trying to imply that everything there besides the name of the addon itself would be an optional argument (including the author) you could add if you wanted. |
|
@bilderbuchi what i mean is 1 file that contains the information for all the addons in ofxaddons not 1 per addon, and doesn't need to be included in the addons it's generated by ofxaddons whenever they make changes and downloaded to be used only by the PG to show info about the addons online. to generate the projects or compile through makefiles we would still use addon.make in each addon |
|
@atduskgreg @arturoc: ok, I see. another thing: if you just supply the addon name, how do you deal with duplicates? there are more than a couple addons which have not been properly forked, and exist in several independent forking trees, iirc. also, a version other than the "most recent" may be wanted/needed - you can't resolve that just with the addon name. you'd need to make the addon author a mandatory field, then, and then we might as well just put the git URL into addons.txt, no? would it make sense to encourage usage of version names, and use those for dependency resolution? it would be the sane way to do it, but I don't know if we can entice the addon devs to use versioning. for example, the master branch of an addon could introduce a regression, which would make dependent addons incompatible (so you can't use branch names for dependency resolution anymore). If you use SHA's though, downloaded/checked out addons get into detached HEAD (which will confuse users probably), and any further improvements to dependent addons (i.e. added after that SHA) are not visible until the addon dependency info gets patched by the addon dev - also a pretty fragile solution... |
|
The insufficiency of addons.make we see is that it doesn't let you specify the github location of the the addon, which is the problem we are trying to solve: how to install dependent addons from github. We aren't trying to solve the problem of how to configure the build. Can we add this info to the addons.make? |
|
^ I think this would be a good solution. mainly depends on what arturo feels can be parsed with Make without him tearing his hair out. (damned if I knew why there isn't a json parser for Make available...) I guess the |
|
@obviousjim yes the idea is to have another variable ADDON_URL to have the url for the addon there. |
|
don't you think it would be better to roll them into one field, and parse the name out? less potential for confusion (which addon_url goes with which addon, what if no url is given, etc), and the name parsing should not be difficult (discard everything up to the last |
|
also if it's easier we can have separate files for the meta tags and the compiling rules, the meta tags can be json or whatever format is easier for you and the compiling rules (used both by the PG and the makefiles) the current format in addon.make not sure if it's better to have 2 files, but since addon.make will only be needed by a few addons perhaps it's easier |
|
@arturoc would you see the PG parsing the meta tag file to add dependent addons to the projects that are being generated? |
|
i think the dependencies need to be in addon.make the PG can parse the meta tag file but from the makefiles it's pretty hard to parse anything that it's not plain text |
|
yeah, if we add a project addon's dependent addons to the project's (btw, I still vote for another name for the addons' |
|
Hey Buchi we are on IRC hashing this out a bit faster and will post conclusions here. join us #ofdev on freenode |
|
as a side note, I think the PG should stay fairly minimal -- what I suggest for how it handles dependency is that if you choose an addon via the addon picker, and it notices a dependency, it should try to also highlight that depency in the addon picker. I don't think it should know anything about github, ofxaddons, etc -- it's job is merely to scan the addons folder and in the case of dependencies, to try make sure you have selected the additional addons you might need. It could also potentially highlight missing addons through an alert. more important then dependency, which doesn't seem like a huge issue, is getting all the intricacies of addons that require assets, dlls, apple frameworks etc working with the PG -- ofxOpenNi or ofxKinect for example, complicated, larger addons. I'm not a huge fan of the "addon.make" that arturo posted, because it seems a bit hard to understand and verbose (the multiple "ADDON_LIBS += "), but I can see why it's useful for make files. I wonder if there's another way to format the same thing? I also never really enjoy reading makefiles that much, which might be why I'm not so excited about this approach -- but I can't say that I loved the install.xml either. I also agree with the point raised here that having this named like this can be really confusing since a similar file (which is just a list of addons) is in projects now. adding comments to all of the text files we use now (addons.make, addon.make, libsorder.make etc) can also be pretty helpful -- whatever system we wind up with should be friendly and self explanatory for someone who is poking around. the more automagic we make it, the more we should consider someone trying to grok it quickly and having flat text files without context or explanation makes things harder. I vote for magic + friendly and verbose explanations everywhere. |
|
more comments are good. :-) |
|
also, @ofZach, where do you propose that addon dependency handling/fetching from github/ofxaddons should happen, if not in the PG? or are you opposed to such automated handling in general? maybe this could be hidden in an "advanced mode" of the PG by default? the intricacies you touch upon I believe are all being handled via addon.make now. a better-readable or more structured format is not probable because of parsing difficulties with make, see the above discussion. |
|
I honestly see the PG's job as to deal with what's directly in front of it -- it shouldn't know if your addon is from a repo, something you've made just now, etc. I think it could flag dependencies and try to auto add dependent things it finds locally, but I really don't see it as a tool that would download things or interface with github or ofxaddons. I see that as something a user should and would be doing. |
|
@ofZach: The killer feature @obviousjim and I have imagined of integrating ofxaddons info into the PG is discovery. It would mean the the list of all the amazing things the community have added to OF are right there listed out for people to discover and search through when they're creating projects. If you think having this in the PG will make the PG's job too complicated maybe it would be better in a separate tool that's specifically an ofxaddons client. -- Greg On Jan 21, 2013, at 8:33 AM, ofZach [email protected] wrote:
|
|
i think we'll end up adding something like this at some point in the PG, by now we could make a different tool that does this. probably a command line tool that can be used by a gui tool or perhaps some small library that can be used both by a command line and a gui tool. something like how brew or apt-get works: of-addons update // asks ofxaddons for the latest db of addons (the json file with all the addons) a gui version could call update when it starts, parse the db and show every available addon with description... then call install or upgrade these won't be incompatible with adding addons in the addons folder manually, downloading them in a zip file and uncompressing them there or cloning a github repo of an addon manually |
|
about the format of the config file for the addon it it has to be some kind of plain text format (not xml or json) i think this format is relatively easy, probably the opencv addon is not the best example cause it has to specify every library for lots of platforms. also one of the advantages is that being standard makefile syntax it'll be highlited by any editor. here's how the config for ofxTimeline looks in eclipse, i guess xcode should do something similar: |
|
to add on to the conversation (@atduskgreg @obviousjim) it's important to see the PG as addressing two very major problems with OF for beginners -- specifically, making new projects and adding addons. I think there are other functionalities we should look at down the line, but these two issues have been so major, and such a major stumbling block, that I want to make sure we get them right first before we add more functionality. Which is why I strongly suggest the PG only care about what's in front of it -- I don't mind it knowing about ofxaddons in the future, but it seems like it's a good idea to nail it's two limited goals really well first before we expand its scope. as a small note, additional functionalities for the PG (simple) could involve: parsing an existing project, updating project files, creating project files recursively, addon discovery, addon installation, etc. |
|
Concerning this separate command-line/GUI tool: Sounds good! I can see myself implementing and adopting this, if that is wanted and we end up going that route. I already solved a similar problem with ofStateManager, in Python. I think the next primary goal for the PG should be unifying the two different versions again (PG and PG-simple). |
|
I think this is great - Greg and I should focus on making a separate tool to manage git+ofxaddons+addons directory, and eventually integrate it if it makes sense. |
|
i've changed the name of the file from addon.make to addon_config.mk less confusing with addons.make inside projects and github recognizes the extension and highlights the file: https://github.com/arturoc/ofxTimeline/blob/develop/addon_config.mk |
|
Hello there, The main idea if I get it right would be to do something like :
Then using the PG the same file could be read and a basic update on the project could update the desired Xcode, make or whatever file. The only part I do not get here, is why you decided to avoid json format. Going to basic parsing seems fast and straightforward but not really common except in the world of computer programming C++ software. If the main objective is to facilitate user to understand, and embrace the framework, shouldn't we build this command line on the best practice we can see in other programming language and country ? I just noticed that I already talk about that here : openframeworks/openFrameworks#1063 and made some proposal which did not get much attention. Mixing a readme file and a conf file looks pretty not user friendly as the newbie will read some cryptic stuff he does not care about. More, it could be confusing for github repo as the README is the first thing to display. Finnaly, jQuery just release their new jQuery plugin directory and it looks great and blazing smart. They have written a full doc about how to publish, from naming convention : This separation between machine to machine file conf (package.json) and user file reading (README) seems pretty logic to me. Well, as I said before I do not know where this feature stands right now but I'll be glad to help. Thanks |
|
actually someone has already done such a tool: https://github.com/prusnak/ofx the problem with json is that Makefiles can't parse it |
|
discussion of that new tool is here btw. |
|
Can I just add my $0.02 as a new user of openFrameworks. I've read/scanned through this discussion and it seems that there are two main problems mentioned:
In terms of user frustration, a non-working addon (problem 2) is definitely worse than a lack of information (problem 1). However, in my opinion these are two separate problems with separate solutions. And while the second problem may indeed become quite complicated, the first problem is not. So that is why I hope that we might return for a moment to the original issue and try to solve it. Often it's better to start with a simple solution and incrementally develop when it is needed. Perhaps that might be a suitable approach here as well. Thoughts? :-) |
I think the addon template should also have dependency and compatibility info included in some file (xml?) or other automatedly processable form (special strings in the README?). This was discussed a couple of times already, but the discussion never came to a conclusion. Maybe now is a good time to do this.
What info should be included?
How should it be included?
How do we allow/account for future features like automated addon download by PG (openframeworks/openFrameworks#1233)?
@obviousjim @atduskgreg @ofTheo @ofZach @arturoc: Thoughts?
The text was updated successfully, but these errors were encountered: